The annual EANTC NETCONF/YANG Interop event took place in Berlin during the first two weeks of March. By a wide margin, this is where the deepest and largest number of independent interop tests are taking place for NETCONF/YANG capable equipment. At the interop event, there are also numerous other tests going on besides NETCONF/YANG, such as Segment Routing, EVPN, FlexE, and whatnot. The test case catalog for vendors to sign up for contained just over 60 cases this year, which left little room for boredom.
At the beginning of March, the COVID-19 crisis was not yet in full force. Many participating companies saw what was coming, however, and about one-quarter of the companies already signed up decided to cancel their participation at the last minute. Some companies decided not to come on-site but instead joined remotely. This worked reasonably well for those that were lucky enough to have at least some of their team located in a time zone that was reasonably close and could arrange for at least one person to come on-site in Berlin to connect the physical equipment.
Despite the COVID-19 fallout, the NETCONF/YANG results from this year’s EANTC Interop were actually the strongest so far. For Cisco for sure, but also counting the NETCONF/YANG industry as a whole. In this year’s EANTC event there were 8 test cases defined in the NETCONF/YANG area, out of which 7 were demonstrated by some pair/group of vendors. The test cases ranged from very basic “give me your config, and let’s change one leaf” test cases, to telemetry collection, to single domain network-wide service activation, and all the way to multi-domain L2VPN and EVPN service activation using standard IETF service models.
Three controllers performed at least one successful test combination of managing other systems. The Lumina SDN Controller showed successful test cases with 3 device types. The Huawei NCE Controller with 1 device type. Cisco NSO with 6 device types and 1 controller. Combine this with the test results from the last two years and a pretty nice picture emerges. By unwarranted shares of luck, several of the vendors that were missing out this year were the ones that had the strongest results during the last two years.
So, here is the list of Controllers/Orchestrators/Clients (technically NETCONF/RESTCONF clients) that have at least one successful EANTC test case recorded speaking NETCONF or RESTCONF to a device or another controller during the last three years:
- Cisco NSO
- Huawei NCE
- Keysight IxNetwork
- Lumina SDN C
- Nokia NSP
Similarly, here is the list of Devices or Controllers being managed (technically NETCONF/RESTCONF servers) with at least one successful EANTC test case recorded speaking NETCONF or RESTCONF to a manager during the last three years. Some vendors have multiple products (product families) which are listed here with commas:
- Adva Activator, FSP150
- Arista 7280
- Arrcus QuantaMesh
- BISDN Basebox
- Cisco NSO, ASR9000, NCS540, NCS5500
- Delta AGC7648
- ECI Neptune 1050, 1800
- Ericsson 6371, 6471, 6672
- Huawei ATN910, CX6608, NE40
- Juniper MX204
- Lumina SDN C (interface is RESTCONF-like)
- Metaswitch NOS Toolkit
- Nokia 7750
- UTStarcom UAR500
Out of this list, I would select vendors that have “many” (subjective, I know) successful results:
Cisco, Ericsson, Huawei, Juniper, Metaswitch, and Nokia. The “Big Six” of the NETCONF/YANG world?
NSO has a very high coverage in results with the above listed devices and vendors. That does not necessarily mean everything works perfectly with all of them, far from it, but it does mean we have been able to demonstrate particular test cases specified by EANTC together.
A couple of comments for specific cases: With Juniper, Cisco NSO has long been able to manage JunOS devices over NETCONF. Until very recently, that was with YANG files provided by Cisco. Now, these results are using the YANG files provided by Juniper. The Lumina SDN Controller is listed above as a successful test case. We managed to execute the test specified by EANTC, but the management protocol in use was, strictly speaking, not RESTCONF, but an ODL variant which is based on draft RESTCONF RFCs. With both Nokia and Ericsson, what has been tested are products out of their router portfolios. This has worked nicely. Both companies have many products in other domains and those seem to have different, and not quite as well functioning, NETCONF servers.
One general observation from this year is that there are now quite a few OpenConfig YANG models exposed by devices that actually work well enough to pull off one or more EANTC test cases. That has not been the case in earlier years. The maturity of several implementations is distinctly higher and the OpenConfig team has put in some work to fix incompatibilities and model bugs. The road is still quite rocky, but the trend is exciting. In the end, what matters most is the actual working systems that customers pay for.
When it comes to ConfD, what we see is a mixed bag. Some of the products listed above use ConfD for their NETCONF (and RESTCONF) implementations. Some of those products just shine their way through all the NETCONF/YANG test cases and this is not only with NSO. It’s just solid systems using ConfD in a clean way. Then, there are other ConfD implementations that have issues that go beyond the basics that work perfectly. The issues come on a deeper level including transactionality properties: being able to commit the entire configuration in a single transaction, modifying the configuration without side effects, or implementing OpenConfig in a way that works properly.
Some might say it’s self-evident, but now we have field evidence of this self-evident fact: using ConfD does not in and of itself guarantee a correct and fully interoperable NETCONF/YANG solution. It does certainly make it easier by removing entire classes of problems, and provides a starting point at a higher level. Many of the most interoperable NETCONF/YANG systems out there are ConfD based, and they are doing great. In order to succeed, however, you still need to understand the customer and how the system you are building will be integrated into the greater whole to succeed. ConfD users can benefit from participating in the “NETCONF and YANG Automation Testing” program which provides guidance and self-testing for NETCONF and YANG interop as well as best practices.
I have written a lot about what this means in practice in the past. If you feel you need a refresher, you can do a one-stop shop and have a look at “Network Programmability with YANG”, especially the chapters towards the end.
Get the latest EANTC results, the full reports from the last three years are below: