Pipeline Patterns for CI/CD Pipelines

CI/CD Pipelines

Continuous Integration / Continuous Delivery pipelines are crucial for getting your ideas to production. There can be a lot to unpack in your pipelines as they can mimic your Software Development Life Cycle [SDLC] and include opinions or governance that are prudent to your organization and industry. At Harness, we have the privilege of seeing hundreds of different organizations’ approaches in the software delivery process around pipeline strategy and automation. In our CI/CD Pipeline Patterns eBook, we describe the most common pipeline patterns we see for the CI and CD processes.

What Is a Pipeline?

A pipeline is typically made up of one or more steps where source code is turned into a build that is tested and deployed across one or more environments until it eventually reaches the end user in production. Pipelines can be inclusive of configurations and infrastructure code and are core conduits of the deployment process. 

Why Use Pipelines?

As systems become more complex, distributed, and modular, our expectation is that systems will always perform and be available. Having a safe, fast, repeatable way of getting your changes into production is critical. Unfortunately, scripts would not hold water or would need to be modified and executed dozens to hundreds of times as software development is an iterative exercise and will need to deploy your artifacts to more than one spot.

Difference Between Good and Bad Pipelines

Simply put, good pipelines and pipeline executions are fast and repeatable, and great pipelines are fast, safe, and repeatable.The book Accelerate states that elite performers have a lead time of less than 1 hour and a change failure rate of less than 15% for production deployments. Therefore, a great deployment pipeline will complete in under an hour and catch 95% of anomalies and regressions, before code reaches an end-user. If your code takes longer than an hour to reach production, or if more than 2 out of 10 deployments fail, you might want to reconsider your pipeline design and strategy.

How to Choose a Specific Pipeline Strategy

The strategy and structure of pipelines tends to follow what they are mostly driven by or triggered by. The driving goal of the pipeline has an impact on the structure and pattern the pipeline follows. Pipelines can be driven by environments, tests, services, outcomes, and/or people. For example, for a very distributed application, getting the application into multiple environments might be the key driver, the pipeline is very deployment-centric, and a typical trigger would be a deployable artifact. If your organization has a lot of manual approvals, the person or persons might be the driving factor. In the eBook, we go through these drivers in detail.  

Pipeline Design Patterns

Pipelines can be as unique as organizations; though as industry marches towards standardization in releases, patterns start to appear. Below is a subset of patterns that are outlined in the eBook and are platform-agnostic. Depending on the goal and feedback loop, the end environment of a pipeline might not be a production environment. 

Button Push Pattern

There is a duality when embarking on a pipeline journey: either the pipeline is completely automated, or completely manual. The Button Push Pattern allows for human approval at the finish line rather than entirely removing the human element. This method allows for one more validation either for compliance, regulatory, or confidence-building reasons before deployment / continuous deployment. 

Button Push Pipeline Pattern

Test Automation Pattern

Test automation can take on a life of its own. Modern test frameworks mimic application development, which have orchestration needs and packages to be deployed along with the application itself. Modifying test coverage to cover application/api changes is an expectation with automated tests. Having a pipeline dedicated to test automation can be a prudent choice.

Test Automation Pipeline Pattern

Full Approval Pattern

Remember when you played Telephone as a kid? You would pass a secret message from person to person, allowing for influence over the phrase to see how different the original message at the beginning was at the end of the line. That could be a pattern in your pipeline. With the Full Approval Pattern, most, if not all promotion steps require manual approval.

Full Approval Pattern

In the eBook, we go through the pros and cons of the different patterns, score each pattern, and talk about relevant industries that benefit from each pattern. 

Download the Pipeline Patterns eBook

Grab your copy of the Pipeline Patterns eBook to take a deeper dive into the patterns we observed. Depending on if any downtime can be allowed, deployment patterns allude themselves to different levels of safety in the delivery process. If you are interested in learning more, a great spot to visit also is the Harness Community where you can interact with folks solving CI/CD problems and moving the DevOps needle every day. At Harness, we are excited to share what we learned throughout many journeys we are privileged to be part of. 

The Women of DevOps: Ana Margarita Medina

The Women of DevOps: Ana Medina, Senior Chaos Engineer at Gremlin

Cloud Cost Intelligence

Introducing Intelligent Cloud Cost Management

Harness Cloud Cost Management uses intelligence and automation to simplify cloud cost management and save you up to 75% on your cloud bill.

Discover Dollar Logo - Save On Cloud Bill

Discover Dollar Saves 70% On Their Cloud Bill

Discover Dollar made idle cost management easy with Harness. See how you can too.

Webinar Blog

Streamlining CI/CD and Optimizing AWS Cloud Spend

Through Harness, Brad was confident he’d be able to make Drone as ubiquitous as Jenkins. Learn how Drone went from zero to 100 million downloads.