Discussing Data and Automation in Dublin

The Tail-f team were out in numbers (here’s two of us) this week, visiting Dublin for the annual TMForum Management World 2012. This was a first for us in many ways including the fact that we have never had a booth to demo in before. An exciting couple of days with a full demo station and meeting table pretty much throughout the whole event. We focused on the NCS product with many interesting conversations taking place with service providers and some of the larger software vendors that take an interest in configuration management and network automation.

We focused on two things: programmability and data quality. We wanted to find out if the people managing the service provider networks and those in charge of ensuring the delivery precision of network-related services were happy with the current situation. We couldn’t believe some of the horror stories that we were told over the table in our booth.

The enormous amount of configuration changes that are still being performed manually (CLI-typing, Web UI-clicking) even in the very large and self-proclaimed forward-looking carriers for all types of consumer and business services is truly not for the faint of heart. Work-orders being printed and forgotten in the office printers. Network engineers being forced to use a mix of command line interfaces, web interfaces and scripts across equipment from various vendors. People driving into operation centers on a Sunday to type a couple of commands in a CLI, wait for two hours and then remove the configuration because the customer wanted more bandwidth for a file transfer. I’m going to label this problem “human orchestration” for later use.

The data quality situation is, if possible, even worse. Going into the meetings we knew that maintaining the integrity of the data in an inventory is perceived as hard and that people were struggling to keep a high percentage of correct data in their databases. We had heard that 60-80% correctness was classified as “good” (which struck us as a little low). I was promptly corrected down on those number several times by operations managers from some of the largest service providers in Europe. These are organizations that have spent exorbitant amounts of money on industry-leading inventory systems and are working very hard to exceed 40% correctness. 40%. Imagine trying to make your way through unknown terrain using a map that is 40% correct. This is the direct effect of keeping configuration in partial files, on excel spreadsheets hosted on half-forgotten file servers on corporate LANs, mixed in with home-grown databases containing blobs of vendor specific configuration dumped into them. Visualizing these partial configuration extracts slowly making their way across enterprises I’m dubbing this one “configuration sprawl” for later use.

We return home to Stockholm with the strong impression that the most pressing matters for many service providers are to:

  • Increase the amount of automation for simple services just to get network engineers out of typing and clicking the same repetitive commands over and over for basic services. They are bored of it and humans are inherently more error prone than software
  • Improve that data quality in the network inventories by staying in touch with the source of configuration (it’s the network stupid!) and making sure to reduce out-of-band changes unless for reasons not related to customer services

We demoed a lot, learned a lot, shook our heads a lot and had some excellent beer and food. All in all a great experience and with three full days of un-Irish weather. If you know what I mean.