I’d like to start this post declaring upfront that I’m an inpatient chap, with full appreciation for people who get #@&$ done quickly. As a tech evangelist (and former software engineer), delivery of innovation and product features is what I live for.
A few months ago I joined Harness and began an exciting journey with Continuous Delivery. My daily routine for the past few months has been:
- Make coffee
- Watch Jez Humble talk about CD. Man, that guy has some killer shirts.
- Make another coffee
- Talk with 3-5 customers a day about their CD needs and challenges
Continuous Delivery isn’t a new concept
Many of Jez’s talks resonated with me (see below links). I was constantly reminded of the bollocks and frustrations I went through as a software engineer many years ago when apps and projects started to fail towards the end of the software lifecycle. Jez even brought back memories at college when our Windows 98-developed Java apps ran completely different on the college SPARC stations. Truth is, the practice of Continuous Delivery (CD) has existed for many years, so I kept asking myself “Why hasn’t CD been solved yet?”
Most Customers struggle with CD
Since May this year, our team at Harness has conducted close to 300 customer feedback meetings. I’ve been fortunate to sit on about 100 of these but also dived into the notes and feedback from many of the others. Here’s a quick summary of what I learned:
CI/CD is the process of building, testing and deploying code to production frequently and continuously. CI is about taking code to artifact (build) and performing tests (unit/regression/integration) to ensure artifacts are ready to be deployed to customers in production. CD is about automating all these steps so you can reduce the amount of time (and risk) it takes to deliver code into production.
Continuous Integration (taking code to artifact) for the most part is largely solved. Almost a religion of Jenkins, Bamboo, CircleCI and Travis CI implementations.
Continuous Delivery (taking artifacts to production) is largely not solved.
- 1-2% customers have nailed it
- ~50% customers are in the early stages of a CD project
- ~50% customers are in-flight trying to make their CD platforms work
Continuous Verification is about ensuring that deployments meet the right performance and quality standards for your customers
The customers we spoke with also had a real desire to disband/reallocate their release teams and empower developers to deploy their own code directly into production. Power to the developer is a good way of putting it. One customer told us “we simply can’t scale our release teams like our dev teams; it just doesn’t make financial sense. We want our engineers focused on building innovation for the business.”
Build vs. Buy?
As crazy as this sounds, the only viable choice customers have had over the past five years was to build and script their own CD platform from scratch. Take 3-5 engineers full-time (or 15-20 engineers if you are large) for 1-year and build out a basic bespoke CD solution to fit their apps, services, environments, and tools. This is exactly what Netflix did, and where the open-source project ‘Spinnaker’ originated from. Many others customers have built on, and extended, the capabilities of their existing CI solutions like Jenkins and Bamboo with varying degrees of success.
When our CEO Jyoti Bansal was CEO at AppDynamics, one large Fortune 500 bank told him “We have four thousand software engineers writing code and seven hundred DevOps engineers writing automation scripts.” It was at this point Jyoti realized that customers had very little options available to them other than “Let’s cobble together a CD platform.” Our other co-founder Rishi Singh built Apples CD platform, and at the time he couldn’t believe that no solution existed to help companies do CD.
Building a Continuous Delivery Platform is Hard
Think CD is just about provisioning infrastructure and configuration management? Think again. I was astounded at the CD requirements and features customers were sharing with us over the past few months. Everything from managing environment variables to secrets to security groups to serverless. Not everything in CD is obvious; for example, security is a big deal. Companies want to empower their developers, but also control and audit how they can deploy across their apps, services and environments.
Key capabilities & requirements of a CD platform:
- Continuous Integration – how do you build and test your code?
- Triggers – When and how do you execute your CD pipelines?
- Pipeline management – How do you structure and manage your pipelines across different apps, services, and environments?
- Infrastructure management – How do you provision infrastructure or compute for each of your apps, services, and environments?
- Configuration management – How do you version control and deploy your application run-time configuration?
- Deployment workflows – How do you deploy your apps and services to different environments? (e.g. Canary, Blue/Green, Rolling)
- Automated Testing – How do you ensure artifacts are ready to be deployed to production?
- Verification & health checks – How do you verify the impact and health of artifacts in production? Specifically performance, throughput, errors, and quality of service.
- Rollback – How do you manage failure and anomalies in production?
- Security, auditing & secrets management – How do you manage API keys and credentials across your apps, environments, and pipelines?
- Analytics & reporting – How do you baseline, debug and report on deployments?
- Notifications & workflow – How are teams informed of deployments, events, and status?
Over the next few months, I’ll be digging into many of the above to highlight how customers are solving these requirements and challenges.
I’d love to hear stories around CD in your organization. Where did you start, what challenges did you run into, and how did you solve them?