4 Reasons Your Jenkins Pipelines Are Brittle
In this article, we go over 4 reasons your Jenkins Pipeline can be brittle. Hint: it's partly in the plugins.
When we first launched Harness back in October 2017, one of our initial challenges was educating the market on the difference between a Continuous Integration (CI) solution and Continuous Delivery (CD) solution. At that time, I wrote a quick comparison blog on the differences between Harness and Jenkins and the fact that CI != CD.
Since then, I've been struck by the number of customers who went down the Jenkins Pipeline path and, as a result, are in a world of pain. Literally, on Monday this week I had a 30-minute call with one of our customers about what life was like trying to use Jenkins Pipeline for a 20+ microservices application.
The customer said:
"It was painful—actually, it was a lot of pain."
This was how their journey was described. I've heard another customer describe Jenkins as "DevOps duct tape."
Jenkins documentation defines Jenkins Pipeline as:
Jenkins Pipeline (or simply "Pipeline" with a capital "P") is a suite of plugins that support implementing and integrating continuous delivery pipelines into Jenkins.
and this leads nicely to reason number one why Jenkins pipelines are brittle.
1. Jenkins Pipeline Relies on (Lots of) Plugins
Now, before you slap me with a wet fish, allow me to explain—because I love a good SDK just like the next developer. Extensibility is a good thing in software, BUT it can also be a bad thing—particularly when extensibility isn't managed or structured properly.
For example, let's imagine I have a simple Docker application and I want to build a simple deployment pipeline using Jenkins. I go to to the Jenkins plugin site and search for "Docker" plugins and get:
It's 26 different plugins related to Docker, sorted by relevance. In fact, someone was even kind enough to name their plugin "Yet Another Docker," which is about as helpful as a poke in the eye with a stick.
Maybe I'm being overly dramatic here, so let's just click on the first plugin called "Docker" and see what's what:
Unfortunately, at a second glance we hit reason number two as to why Jenkins pipelines are so brittle; I spot 8 plugin dependencies with another 7 optional dependencies. You can see where I'm going with this. In a world where apps/services change more frequently than Tesla's stock price, it seems logical that plugin dependencies could be problematic as technology stacks and plugins naturally evolve. The last thing you want is a broken deployment pipeline because the pipeline itself is broken vs. the actual software artifact or build that's being tested.
Simply put, which Jenkins Pipeline plugin should one use? And what happens when one needs to upgrade plugins and maintain version dependencies? Do I need a plugin for DockerHub, Docker, Kubernetes, and every other technology stack or tool my app interacts with? How many plugin dependencies do you think this list of plugins would have? Over 100 isn't unrealistic.
2. Jenkins Pipeline Relies On Jobs/Scripts
Let's put aside plugins for a second, and focus on the core concepts or entities of a deployment pipeline:
- Build or Artifact (e.g. containers, AMI, Function)
- Variables & Secrets
- Deployment Workflow
- Verification or Health Check
- Release Strategy (Blue/Green, Canary, ...)
- Tests & Tools (test, security, monitoring, ...)
- User Groups & Users (RBAC)
How many of the above actually exist today as first-class citizens in Jenkins Pipeline? More importantly, how many of them require you to write Jenkins Jobs or Shell Scripts to manage or perform the above tasks?
Answer: nearly all of them.
Jenkins Pipeline is another way of saying: "I hardcoded my deployment pipeline with scripts." Hardcoding things like environments, variables, secrets, dependencies and release strategies would have worked back in 2008 when all you had was an apache web server, a few instances of Tomcat/WebLogic/WebSphere, and a chunky Oracle database. However, it doesn't work in 2018 with public cloud, containers, and microservices. You're going to spend more time maintaining your deployment pipelines than actually deploying new versions of your app.
In fact, it's not uncommon for customers to have a team of DevOps engineers solely focused on maintaining deployment pipelines as applications, technology stacks, and tools change every week. Instead of DevOps teams adding innovation to deployment pipelines, they spend their time fixing or dealing with pipelines that break.
This is not good. The more you script the core components of your deployment pipeline, the more you maintain those scripts as applications and technology changes. You want your deployment pipelines to be dynamic in the sense that they can automatically adapt or change based on the meta-data that exists in your cloud provider or DevOps tool ecosystem.
In addition, the majority of Jenkins pipelines I've seen still prompt the user for manual inputs like "Please provide the Major and Minor build version you would like to deploy." Why can't this information be dynamically pulled from the build or artifact repository? Shell scripts by nature will make your deployment pipelines brittle. The more complex your deployment pipeline, the more brittle your pipeline will become.
3. Debugging A Jenkins Pipeline Is A PITA
How about we ignore those pesky plugins and sexy scripts for the time being. Let's imagine we've hard-coded the best deployment pipeline using Jenkins Pipeline for our application that has 20 microservices, with each microservice having a dev, QA, staging, and production environment. Quick question, do we need one Jenkins pipeline or 20 pipelines for our application? And can those pipelines be executed in parallel?
Anyways, let's imagine we push our big red "Build Now" button to kick off our deployment pipeline and get this visual:
The above screenshot doesn't look too bad at first glance. We can see our Jenkins pipeline has 10 steps, and they're all green.
Notice the long list of "Shell Script" executions, and specifically the lack of context those windows have. What exactly are those shell scripts doing and telling the user who is watching the deployment? What if those scripts fail? Are we able to understand in detail what happened? Or are we simply just seeing a console confirming that a shell script executed?
Again, not a massive problem if our application is relatively simple with a few microservices/environments. But this could be a world of pain if we have to observe 20 different pipelines, each with hundreds of shell script outputs.
Understanding the real status and health of a Jenkins pipeline (and your application) isn't an easy task. One customer told me, "Jenkins Pipeline really lacks any sort of global context, dependency model, or high-level view of a deployment pipeline in a microservices world."
4. Deployment Verification and Rollback
Simply put, Continuous Delivery is not just about deployment. Everyone does Continuous Delivery for a reason, normally to increase the velocity of your innovation and business so customers spend more cash.
Not knowing the business impact of deployment in production is a big deal. Not having a rollback strategy is also a big deal. I'm not talking about running a Jenkins Job to run a simple load test or unit test. I'm talking about observing how customers are impacted by a production deployment and managing that risk in real-time. At Harness, we call this Continuous Verification and Smart Rollback.
To achieve this in Jenkins Pipeline you have to write a Job or shell script. Why can't Jenkins Pipeline just integrate with your monitoring ecosystem and automatically tell you whether your deployment was successful? It's possible. We do this every day for our customers at Harness, so it can be done.
Knowing the success of deployment isn't a deployment step or job completion. Success is a deployment having a positive impact on your business. Conversely, a deployment having a negative impact on your business makes your deployment pipelines brittle. It leads you into a false sense of security that everything with your deployment or application is okay— when actually it's not.
CI != CD
In closing, Jenkins is unquestionably the top dog for Continuous Integration (CI) use cases. But Continuous Delivery (CD) is way more than the building and testing of code into artifacts. Continuous Delivery is about taking those artifacts into production where they will delight customers.
Remember, Jenkins Pipeline is just a suite of plugins that allow you to build your own CD platform. Meaning, you're actually the one coding and maintaining your CD process as your applications and services evolve.
Today, many real Continuous Delivery solutions exist that integrate with and complement Jenkins for CI. Harness is one, but if you search you'll find others. Try them! Don't be that customer or team that thinks building your own CD platform is doable with DevOps duct tape.
What's your experience with Jenkins Pipeline and Continuous Delivery? Yay or Nay?