Pipeline Patterns for CI/CD Pipelines

Continuous Integration / Continuous Delivery pipelines are crucial for getting your ideas to production. There can be a lot to unpack in your pipelines as they can mimic your Software Development Life Cycle [SDLC] and include opinions or governance that are prudent to your organization and industry. At Harness, we have the privilege of seeing hundreds of different organizations’ approaches in the software delivery process around pipeline strategy and automation. In our CI/CD Pipeline Patterns eBook, we describe the most common pipeline patterns we see for the CI and CD processes.

What Is a Pipeline?

A pipeline is typically made up of one or more steps where source code is turned into a build that is tested and deployed across one or more environments until it eventually reaches the end user in production. Pipelines can be inclusive of configurations and infrastructure code and are core conduits of the deployment process. 

Why Use Pipelines?

As systems become more complex, distributed, and modular, our expectation is that systems will always perform and be available. Having a safe, fast, repeatable way of getting your changes into production is critical. Unfortunately, scripts would not hold water or would need to be modified and executed dozens to hundreds of times as software development is an iterative exercise and will need to deploy your artifacts to more than one spot.

Difference Between Good and Bad Pipelines

Simply put, good pipelines and pipeline executions are fast and repeatable, and great pipelines are fast, safe, and repeatable.The book Accelerate states that elite performers have a lead time of less than 1 hour and a change failure rate of less than 15% for production deployments. Therefore, a great deployment pipeline will complete in under an hour and catch 95% of anomalies and regressions, before code reaches an end-user. If your code takes longer than an hour to reach production, or if more than 2 out of 10 deployments fail, you might want to reconsider your pipeline design and strategy.

How to Choose a Specific Pipeline Strategy

The strategy and structure of pipelines tends to follow what they are mostly driven by or triggered by. The driving goal of the pipeline has an impact on the structure and pattern the pipeline follows. Pipelines can be driven by environments, tests, services, outcomes, and/or people. For example, for a very distributed application, getting the application into multiple environments might be the key driver, the pipeline is very deployment-centric, and a typical trigger would be a deployable artifact. If your organization has a lot of manual approvals, the person or persons might be the driving factor. In the eBook, we go through these drivers in detail.  

Pipeline Design Patterns

Pipelines can be as unique as organizations; though as industry marches towards standardization in releases, patterns start to appear. Below is a subset of patterns that are outlined in the eBook and are platform-agnostic. Depending on the goal and feedback loop, the end environment of a pipeline might not be a production environment. 

Button Push Pattern

There is a duality when embarking on a pipeline journey: either the pipeline is completely automated, or completely manual. The Button Push Pattern allows for human approval at the finish line rather than entirely removing the human element. This method allows for one more validation either for compliance, regulatory, or confidence-building reasons before deployment / continuous deployment. 

Test Automation Pattern

Test automation can take on a life of its own. Modern test frameworks mimic application development, which have orchestration needs and packages to be deployed along with the application itself. Modifying test coverage to cover application/api changes is an expectation with automated tests. Having a pipeline dedicated to test automation can be a prudent choice.

Full Approval Pattern

Remember when you played Telephone as a kid? You would pass a secret message from person to person, allowing for influence over the phrase to see how different the original message at the beginning was at the end of the line. That could be a pattern in your pipeline. With the Full Approval Pattern, most, if not all promotion steps require manual approval.

In the eBook, we go through the pros and cons of the different patterns, score each pattern, and talk about relevant industries that benefit from each pattern.

Download the Pipeline Patterns eBook

Grab your copy of the Pipeline Patterns eBook to take a deeper dive into the patterns we observed. Depending on if any downtime can be allowed, deployment patterns allude themselves to different levels of safety in the delivery process. If you are interested in learning more, a great spot to visit also is the Harness Community where you can interact with folks solving CI/CD problems and moving the DevOps needle every day. At Harness, we are excited to share what we learned throughout many journeys we are privileged to be part of.