Show notes
The Iconic Mr. Nick Russo joins us once again for today's topic of CI / CD ... See Why?! Today we are picking up where Nick and I stopped in ZNDP Episode 30 - Designing for DevOps, if you didn't get a chance to listen to that episode yet you can find it at zigbits.tech/30, If you need some context around DevOps you should definitely listen to Episode 30. For today's topic we are discussing CI/CD which is Continuous Integration / Continuous Delivery/Deployment. And we are live in 3...2...1!!! :)
Continuous Integration in CI/CD
Traditionally, developers would work in their own feature/topic branches for months, then deal with what is called "merge hell" towards the end of the project when entering the "integration testing" phrase. This is typically weeks or months as combining many silos of work is fraught with problems. This goes way past software development; the "divide and conquer" approach for any complex project is best accomplished by continuous lateral communication. CI means merging code together regularly (daily) which kicks off a comprehensive set of tests to ensure code functionality and quality. CI provides immediate feedback as to any problems that arise.
Commonly associated with software development, but can be applied more generically. In this use case, to a complex Ansible library used at a large customer.
Continuing from last time (ZNDP Episode 30), we discussed our specific purpose, along with the processes and tools to support it. How do we ensure quality? How do we know the machine built the right product? We checked the inputs, what about the fabrication steps?
Consider an automotive assembly operation (way overly simplified). First build the frame, then put the wheels on, then the body, then the glass. If the frame is defective, why put the wheels on? Let's try to achieve "Quality at the source" through all the integration steps. This runs whenever code is committed or merged.
The CI Process in CI/CD:
1. Lint all the code. Look for syntax/styling issues, and static code analysis to detect security threats. This takes seconds. Fail fast!
2. Unit (filter) tests. Execute the individual units locally to ensure they function.
3. Role tests. Roles typically rely on filters, so run those against some virtual devices.
4. Playbook tests. Playbooks typically rely on roles, so run those against some virtual devices. For cost/runtime savings, you can stub out the virtual devices and provide mock data to the rest of the playbook, but this is less effective. This takes tens of minutes.
It usually doesn’t make sense to continue to the next step if a previous one fails. For example, if static code analysis reveals a security flaw, executing the code may pose a security risk.
Bonus feature: CI integrates with chat programs (chatops) to notify developers on-the-fly about activities. New comment/issue, code committed, pipeline pass/fail, etc. Common chat tools like Slack, Ryver, and Mattermost are used for this.
Consider even more creative examples, such as Nick's OSPF Cisco Live discussion. Configs and markdown READMEs are up on github. Here is Nick's Github Repository link. There is no real code here, just text documents and a diagram.
Nick's Troubleshooting OSPF - BRKRST-3010 Cisco Live Session:
- Session Presentation
- Video Recording
What's the value of CI in CI/CD?
First, linting the markdown documents ensures they look fresh and high quality. Second, maybe we can do some basic quality checking of the device configurations. There are 19 devices in the lab, and there are two folders (final and initial configs) each with 19 configs, for a total of 38. We can ensure that there are exactly 38 config files. We can also search the files for critical information, such as the author's name and email (for assistance/questions). Last, we can search the files to ensure the hostname of the device (R1) is the same as the file filename (R1.txt) once the file extension is removed. It's frustrating to log into a device with mixed up hostnames.
Is this a killer example of CI? Certainly not. But it's better than nothing and helps prove the value of CI goes far beyond the software development world. A little bit of quality checking sets the basis for expansion later and provides you a degree of confidence.
Continuous Delivery and Continuous Deployment in CI/CD
After code has been integrated, code can be delivered or deployed. These steps always occur after CI, which has integrated/tested the code extensively. Which one does CD stand for, "delivery" or "deployment"? Both actually.
Delivery
Fully tested/integrated code is ready for delivery into production, but is manually deployed. After deployment, testing/monitoring still occurs automatically. Think of a big "easy button", all the previous steps were automated but a human makes the final push.
Deployment
The next logical step after continuous delivery, continuous deployment deploys code into production automatically after CI tests pass.
In Nick's work environment with Gitlab CI, we only do continuous delivery. Remember, baby steps. We do not have an infrastructure-as-code solution, so continuous deployment doesn't make sense. Many in the DevOps community feel that continuous delivery is a must-have, and I agree. Continuous deployment may not be appropriate for every business, but should remain a target/goal for many. It carries high risk for network devices that are typically deployed in lower density than servers (in terms of availability).
Nick's best example of CD is how his website (http://www.njrusmc.net/) is deployed.
AWS today doesn't have S3 as a build target for CodePipeline, so instead, I use the AWS CLI from a CodeBuild step to copy files from CodeCommit to the S3 bucket hosting my static website. Both before and after this copy, I run linters for HTML and Python, then a detailed health check on the website's HTML code, ensuring good-looking HTML and no dead hyperlinks. I am notified via email when pipelines begin, and when they complete with a PASS or FAIL status.
Initially, I was doing continuous delivery for my website. Once my code passed all tests and was staged for production, I manually copied the files into the S3 bucket. Then I started the post-install automated validation tests. I decided to upgrade to continuous deployment both to show a public example of how it works on a simple project, and to simplify my code updates.
In a network-based infrastructure-as-code environment, commiting an updated YAML file containing a list of firewalls ports/protocols kicks off the CI process to validate the changes. The continuous deployment process would idempotently (changing only what needs to be changed) update the firewall configurations based on the changes. The manner of this update could be anything, from SSH commands to REST, NETCONF, or gRPC API calls.
How to get started with CI/CD:
Nick has used 3 different products for 3 purposes:
1. Travis CI: Easy to use and free for open-source projects. Integrates seamlessly with Github, and I use this for all of my production-ready Github projects. It's probably the best place to start! Travis.ci.org
2. AWS CodeBuild/CodePipeline: More complex to use since these tools are wholly integrated with the AWS ecosystem of services. A pipeline contains many stages, one of which could be a build. The build has sub-stages as well. I use this to deploy my personal website and the source CI/CD code is publicly available to help you get started. It isn't free, but is very affordable for small projects. Integration with CodeCommit (private Git repo) is very good, and works well for private code.
3. Gitlab CI: Supported online (gitlab.com) or on-premises with a private installation. I use the private option at work. The initial setup is more complex as you have to create "runners" and register them to Gitlab. Runners can have a variety of executor types, like "shell" (what I use), "docker" (for spinning up docker containers within one runner), and more. Once the runner is registered, it's just as easy as the others. The omnibus installation even comes with Mattermost built-in for simplified ChatOps!
There are endless others, but many have a similar configuration mechanism. Define a YAML file that determines the build/test sequence, usually something like install, pre build, build, post build .... or install, before script, script, after script. Minor language differences for similar concepts.
Nick's examples with CI/CD:
Ansible playbooks (Travis CI)
Cisco Live Sessions
Website (AWS CodeBuild/CodeDeploy)
Pete Lumbis example with CI/CD:
Cumulus Networking (Gitlab)
Call to Action:
What topics would you like us to spotlight on our next Design episode?
Guest Expert: Nicholas (Nick) Russo
Today's guest Expert has become an icon within this industry. He holds Cisco Certified Design Expert (CCDE), and two Cisco Certified Internetwork Expert (CCIE), one in Routing & Switching and the second in Service Provider. He is just coming off an amazing week at Cisco Live US 2018 in Orlando, Florida where he shattered the Cisco Live presentation bar with his Troubleshoot OSPF session!
How to stay engaged with Nick:
Website: http://www.njrusmc.net/
Twitter: https://twitter.com/nickrusso42518
LinkedIn: linkedin.com/in/nicholas-russo-63297541
Nick's Github Repository: https://github.com/nickrusso42518/
Publications:
CCIE Service Provider Version 4 - Written and Lab Exam Comprehensive Guide By Nicholas (Nick) Russo
CCIE and CCDE Written Exam - Evolving Technologies Study Guide By Nicholas (Nick) Russo
BGP Traffic Engineering Server for Leaf-Spine Data Center Fabrics By Nicholas (Nick) Russo
Related Resources:
ZNDP 030 - Designing for DevOps with Nick Russo
Making Work Visible: Exposing Time Theft to Optimize Work & Flow
...
Our TOPPODCAST Picks 
Stay Connected
