Using Automation for a DataCenter Network Core Migration

My relationship with network automation and programmability

The thought of network automation and programmability continues to be beside me for any lengthy time, lengthy prior to the term had any meaning for me personally. Previously I have tried personally TCL/Expect, after which Perl along with regular expressions to control network equipment and obtain performance metrics through Telnet and SSH. Although Perl is really a effective scripting language, it needed lots of try to build things on your own, as there wasn't any community with this particular use of the word what. I've been researching Network Programmability for a while now, using various sources:

• I've attended Cisco Live DevNet sessions and labs (Cisco Live Berlin & Barcelona) and 2 different DevNet Express occasions in Athens (2018 & 2019).
• I've adopted Kirk Byers’s email course on Python for Network Engineers and signed up for several web based classes for Python along with other development tools.
• I've adopted both seasons of NetDevOps Live number of webinars in the Cisco DevNet Team.
• I've became a member of several communities using forums for many known Free Network Automation Frameworks and that i follow automation experts, teams and projects on Twitter along with other Social Networking.

The Electricity Core Internet upgrade project - every change is definitely an chance

Project needs and project launch

A couple of in the past, we began focusing on a large network upgrade project with my workmate K.D., with respect to our Department, because the group of network engineers responsible of seeing it through. The work required the majority of our attention, beginning in the research stage along with a practicality study, resulting in writing up specifications documents to have an RFQ, selecting someone, and lastly towards the actual project design and implementation phase, that I had been also assigned because the role of Project Manager.

It contained switching the Electricity Network Core in our Datacenters to Nexus Electricity Switches, separating the consumer Access Network Layer in the Electricity Server Access and Core Layers. For 13 years, you and servers systems were mixed together. Exactly the same pairs of Legacy Cisco 6k Catalyst Switches were operating because the core switches for everything.

For that switch to be effective, we wanted to organize first for a number of large-scale infrastructure upgrades and adjustments to Power, Cabling and Rack space in multiple structures. With individuals complete, we'd still need:

• seamlessly integrate the brand new Core Switches towards the existing network and migrate towards the new design (upgrading all of the old Core Switches was needed first).
• leverage VDC technology to isolate critical systems in a number of separate L2 fault domains
• introduce OTV and MacSec file encryption across Electricity sites
• give a new strong, scalable and resilient Electricity Core
• minimize impact to bank services.

Learning in parallel

By finish of The month of january 2018, i was deep in staring at the relevant technologies and merchandise available. Together with my workmate, we attended Cisco Live Barcelona, gathering information that may be useful towards the project, but simultaneously checking up on current developments, including all we're able to learn about DevNet. The DevNet Express occasions were already going strong around the globe at that time. I was finally in a position to attend one, the very first in Athens, A holiday in greece, come June 2018. Attending this type of live event, could be transforming using the event structure allowing you to return in your tracks, find the journey, and begin exploring. At first, we didn’t think there'd the perfect enough to rehearse and apply our additional skills towards the project, therefore we didn’t arrange for it to occur. However, as time continued, such occasions presented themselves almost naturally.

After DevNet Express Athens, I'd taken time to discover much more about Network Automation Frameworks, taken an e-mail course from Kirk Byers about while using Netmiko Python Library, began using Ansible, signed up for Python courses, and tracked lower any possible resource, using Twitter, Youtube and LinkedIn. My workmate had been learning python and studying lots of relevant material on Cisco DevNet. We frequently exchanged understanding, cooperating, discussing options and discussing ideas, each one of these taking roles, pushing for change and simultaneously ensuring i was keeping a stable feet on the floor, looking for practical ways to use the use of our additional skills.

Use Cases - network automation tools used

By November 2018, i was within the project implementation phase, moving things around, destroying and rebuilding infrastructure to get at our goal. Which involved invoking many Disaster Recovery Processes. Rather of applying individuals standardized DR processes by hands, I began writing small python Netmiko based automation scripts on their behalf. It had been fun and easy also it labored. Besides, what may go wrong, i was already performing Disaster Recovery, right? Next, picking out practical suggestions for Network Automation within our workplace began getting simpler for all of us. I prepared a couple of more scripts for small everyday operational tasks (a few examples here and here)

We first migrated towards the new Electricity Core after which arrived at the purpose, where i was finally prepared to migrate the Electricity servers in the old switches towards the new Electricity access layer, therefore we wanted to discover where exactly every server was connected. That will let us:

• verify steps and coordinate the migration
• make certain you will find none left out once the migration has ended
• perform cleanup later

It had been a 15-year-old atmosphere, with added layers over time, so writing a multiple-use script for this demonstrated a bit more difficult than other cases. It involved studying switch addresses from your stand out file, integrating using TextFsm and NTC-templates to show command output in structured data, gathering arp information in the core switches, mix-referencing it with mac address table information from each access switch, and lastly conveying the outcomes to stand out files. Automating this method permitted us to operate it repetitively, supplying a practical method to check our progress between migration home windows scheduled with groups of system admins and making certain success for that project. You'll find the code and a bit more detailed explanation here.

Following the effective migration of both Electricity Core and Servers Access layer within the new design, we'd to do acceptance tests for the entire atmosphere. I'd notice the Python-based Automated Testing Framework produced by Cisco and released in public places, known as pyATS. When I had almost no time to undergo the documentation and obtain to some extent where I possibly could setup something helpful, I switched to Twitter for help. Cisco DevNet’s Stuart Clark responded by contacting the Cisco pyATS team towards the save. These were kind enough to supply me with information on ways to use the Genie CLI for creating results with minimal effort. I setup my Electricity testbed very rapidly and effectively collected data between your tests, by running Genie Learn and Parse instructions, as well as in the finish compared results by running Genie Diff against result sets.

Using pyATS and Genie, we could draw quite interesting and helpful conclusions effortlessly and speed, for instance we recognized the Mainframe routing process switched the traffic in one Nexus peer to another throughout the tests, resulting in further analysis. We easily verified that services were effectively absorbed by active links and equipment throughout the tests, which all switch processes maintained the required information whatsoever occasions.

Project Conclusion - Current Status

The Electricity Network Core upgrade project continues to be effectively concluded by finish of May 2019. We've many userful stuff here from your automation attempts and we're still learning, finding smaller sized projects, on the situation by situation basis, and investing time carefully in selecting possible new learning pathways, as our time is efficacious. The PyATS & Genie suite remains ready for use in the Electricity atmosphere each time we have to gather status data “before” and “after” big changes.

We're feeling certain that our python scripting skills provide us having the ability to complete gaps within our monitoring atmosphere, manipulating data directly interior and exterior the network equipment or through keeper APIs. We're also thinking about writing code for automated network recovery purposes and network traffic engineering. At this time, we're putting our needs in queue and arranging the little development projects to pay for them, ensuring we're not neglecting our other responsibilities and separate projects.

Future plans - Practical Approach

I've since prepared more scripts, geared to not waste time in order to standardize tasks, tackling additional Network Automation tools, for example Nornir for any use situation described on DevNet Code Exchange. Within the several weeks in the future I'd like to visit much deeper with PyATS, having its library framework within python scripts, to ensure that a network operational status might be switched right into a monitored operational condition variable and convey relevant alerts via a separate alerting tool like Nagios, when things fail.

My work partner, proficient with Network Management tools for example HP NNMi and Cisco Prime Infrastructure, continues to be concentrating on Rest APIs, and removing information from your Management Tools. Cooperating, establishing a database model for monitoring our network equipment or perhaps developing a complete self-service CI monitoring platform, appears possible over time.