In my career, I’ve inherited a lot of different software projects. I’ve built several, sure, but I have also had a lot of experience in managing and maintaining software that has been developed by another team. I have received projects with a properly built and maintained CI/CD pipeline with frustrating infrequency. My first task for these projects is to work with my teams to build an appropriate pipeline that enables us to do our best work.

Building the CI/CD is one of those “invisible” tasks that is unappreciated by a lot of product owners. It’s difficult to sell it to a project manager or other stakeowners since it’s not an obvious user-facing feature. It’s hours of work in order to do something that results in an identical product to what was already being created. But it’s an investment that pays off dividends over the lifetime of a product.

Visions for Good CI/CD Pipelines

The minimum viable product for a CI/CD pipeline is one that builds your product. The mechanics of that depends very much on what you’re building. Can you just run “xcodebuild” or “npm build” in your repo root? Do you need to run “cmake” and generate the build system before you build the app? Does your preferred programming language have a library distribution system or do you need to clone a few other repositories as siblings or children directories in order to gather your dependencies? Whatever it is, the primary purpose of the CI/CD pipeline is to create artifacts that can be distributed to the appropriate deployment environments. That’s just table stakes.

The next step for a good CI/CD system is to run automated test systems. There are many potential tests that could be run. Not all are appropriate for every build. But you should run the appropriate tests at the appropriate times. At a minimum, you should be running the code-level unit tests that are appropriate for your product when building. You might also consider running UI-related tests at an appropriate interval (they may be time consuming and running them on every build may be expensive with a low return on value). You might also consider having integration tests that verify the integrity of the systems that your product communicates. You can delay those tests until the actual deployment phase. Which brings us to…

The D in CI/CD says “deployment,” but I’ve found that it’s often an afterthought. You should be deploying constantly. It’s my experience that teams frequently set up deployments for an internal process. For instance, mobile application development might deploy to AppCenter so that test engineers can quickly grab new builds. However, it’s much less frequent that the pipeline covers the entire deployment chain. It frustrates me to know that developers use Transporter or upload builds using the Google Play Console when that can be done automatically by the build pipeline.

The CI/CD pipeline doesn’t stop at deployment, though. A good pipeline can include some post-deployment verifications. This is especially useful for services and web-based software. You can easily verify deployments with some simple HTTP requests. Importantly, you can check to make sure that your software is working as expected and rollback to a previous deployment if it isn’t. For software that runs on user devices (typically mobile or desktop apps), you might consider merely launching the product. I can’t tell you how many launch-time crashes can be caught with just running the application from a cold start. You can also automate launching the app with some user state already present. UI tests can catch launch time issues, but it’s relatively easy to set up some means of post-deployment testing so you might consider doing it anyway.

The I Has It

Once you’ve got a solid D phase running, you’ll need to focus on the Integration. Too frequently, saying “it builds” is the standard for CI/CD pipelines. You can (and should!) do much better. Modern software development provides a lot of tools to ensure that we maintain consistently good processes.

First, let’s ask a very important and potentially hurtful question: How many warnings does your build emit? If the answer is larger than zero, you need to fix it. I know that there are a lot of products that build very well with many warnings, thank you very much. Yours might be among them. But you can’t guarantee that those warnings are and will always be ignorable. You also can’t guarantee that you aren’t creating more warnings. It’s my experience that once you begin to tolerate some warnings in your product, you create an environment in which you tolerate all warnings. There’s a magic threshold in which you simply stop paying attentions to warnings and new ones will proliferate in your product. When they accumulate, fixing them becomes a significant burden. The warning snowball continues to grow over the lifetime of a product. Stop that avalanch before it starts, set higher standards and treat all warnings as errors.

At the risk of revealing something about myself: I have a pathological hatred of code formatters as an individual contributor. As a team member, I begrudgingly accept them. As a manager, I strongly encourage my teams to evaluate the values that linters bring to their processes. My greatest appreciation is that it cuts out nearly all of the conversation around the “nitpick” and style on code reviews. In any case, formalizing the coding style with a formatter creates a standard for the team that increases cohesion in the project and makes it easier for any individual to drop into any source file. There’s a loss of individual expression and preference, so consider the costs of formatting (an consider using very liberal rules) before imposing it on your team.

Similarly, you might consider using a linter on your projects. Linters are often similar to formatters and you’ll frequently find that the same tool does both, but it’s worth mentioning separately. You may consider other analyzers as well. If you write code that compiles down to native CPU architectures (typically, C, C++, Rust, etc.), you likely have a whole wealth of analyzers that cover all sorts of different needs. You can utilize these analyzers to make sure that your threads execute as expected, that your memory is used efficiently, and that your program is correct in expectation and syntax. Treat analyzer warnings like errors. Break builds and fix them immediately.

You should also consider evaluating your dependencies. If you’re not doing this, then you’re potentially exposing yourself to security threats. There are a several tools available to you to validate and verify your dependencies. Some systems have it built in to the default toolchain (“npm audit” comes to mind). In some ways, this might require you to live at head for some of your dependencies (and you might choose to do so for all of them), but the alternative is that you expose yourself and your customers to potential errors or threats. Audit your dependencies actively. You may not be able to immediately respond to new alerts, but proactive maintenance is much easier than trying to update dozens of dependencies later.

Good Pipelines Empower

The above is just a brief survey of the types of tasks that a good CI/CD system can do. Those tasks open up possibilities for your engineers and your product.

Do your engineers ever complain about the difficulty of maintaining some legacy code? If you’ve invested in automated testing, then you can allow them to gleefully excise and rewrite whatever parts they need to with the knowledge that they can’t break the app. Unit tests ensure that they meet formal expectations for how the code works. UI tests guarantee that the app operates as expected. The ability to immediately create and distribute test builds to test engineers and stakeholders allows real time feedback into whether the rewrites were effective. A good CI/CD pipeline combined with good tests and processes lowers the risk significantly.

Do you have a checklist of things to do to deploy your product? Do you still fret over every release? A consistent CI/CD pipeline encodes that checklist into automated tasks and deployments become simple. You can confidently release software and do so frequently. The time from code merge to release can shorten from days or weeks to mere minutes.

Do you nervously check on the health and safety of your services every time that you update them? If you have a robust CI/CD pipeline, you can integrate it with your monitoring software and let a system take care of it for you. If your team has accidently deployed a server-burning performance bug or shipped a common crash case, you can automatically rollback to a previous version. Imagine sleeping well and never worrying about the nightmare situation of middle-of-the-night pager alerts. Instead, you’ll wake up to the slightly disappointing email that there was an error and the enduring satisfaction of knowing that your automated system was robust enough to continue offering consistent uptime for your users.

Good CI/CD pipelines also require you to maintain high standards. You can’t ignore warnings or security issues. You can’t let minor coding smells slip or else the linter will break the build. You can’t merge code that doesn’t match the team’s predetermined style or the formatter will balk. You have to proactively fix them as you go. The errors will be publicly displayed on your build server and your team will adjust to them. You can use these tools to set a high expectation for excellence and your team may gain the satisfaction of being formally verified at producing a high quality product.

Finally, you get avoid the snowball effect of accumlated warnings and risks. It’s always easier and less risky to clean up the repository as you go rather than delay and fix a lot of things all at once. Imagine the effective time difference between incrementing the minor versions of your dependencies a dozen times compared to incrementing a dozen libraries at one time. In the first case, any source level changes are obvious and immediately appreciable. The testing burden is light as you can likely identify specific items to verify. In the latter, the effecive “blast radius” may encompass large portions of your product. You may need to stop all other development in order to implement and test the changes to avoid potential regressions. It’s much easier and less risky to keep things up to date by amortizing the test and dev burden across the entire lifetime of the development cycle.

Update Your CI/CD Pipeline Today!

This is just a brief summary of the potential and power inherent in fully specified CI/CD pipelines. A proper discussion could fill a book. I hope that this is enough to entice you to reconsider your CI/CD processes and imagine what a new stage in the pipeline could do for you, your product, and especially, your team.