Control and Velocity in Production: Combining CI/CD and Feature Flags

Needing to deliver software faster and more safely is a foregone conclusion. Engineering teams are constantly trying to deliver software faster than before, but without introducing more risk to the system.

As part of this evolution, software delivery pipelines were born, most notably CI/CD. However, as teams push the boundaries of CI/CD, they find that they’re running into new problems at the tail end of their software development life cycle.

In this blog post, we’ll explore this tail end of the problem: deployment. And you can expect to learn about where feature flagging ties into this.

Why Software Delivery Pipelines?

Whether you call it a pipeline, process, lifecycle, or any other term, the idea is the same - there’s a steady stream of changes and new feature releases being developed, and they need to make it into the hands of customers.

Especially as you scale, you need a standard way of delivering software: the pipeline. This ensures that what you ship consistently passes muster, and also minimizes your risk (and of course your potential for rework and fire drills).

Delivering software more quickly and safely is why teams adopt CI/CD. And now, teams are also adopting feature flags to extend that software delivery pipeline even further. The same way in which the end result of CI, the artifact, is consumed by the CD pipeline, the end result of CD, the deployment, is consumed by a feature flag system to take that next step.

But why do teams need to adopt feature flags? Aren’t we good to go after code has been deployed to production? Aha! Therein lies the problem: what happens once we’re in production.

The Limitations of CD

Let’s start with the obvious statement: CD ends when code is deployed to production. What control you have once the deployment is live is restricted to rolling back the entire deployment. You have no control over the contents of the deployment, and you can’t solve for single points of failure in production without pulling the entire deployment out of production. Let that sink in for a minute: you lose control the moment you push to production. You might feel a pit in your stomach just at the thought of doing something that ridiculous.

The reality is, however, that many teams are living this every day. They need the speed and control before things are pushed to production, but they have limited control of what happens once it’s sent over the proverbial wall. This can create some interesting problems before and after deployment.

Before Deployment

• Deployments get larger, compounding risk and causing the deployment process to take longer and add more stress.
• Deployments are bottlenecked by individual changes that are not ready.
• Have to build app configs, DB flags, or environment variables to enable release control to specific user sets.
• Multiple “versions” or long-lived source control branches for distinctly different feature set requirements by customers.

The kicker here is that even after all of this is done, deployments can still be incredibly stressful, and hitting “Go” is as dramatic as hitting the big red button in the movies. The biggest thing here is that if something goes wrong, it’s all hands on deck. Not only is this stressful for DevOps, it also slows down the velocity of new features because development teams have to do rework.

Beyond that, creating multiple experiences based on the customer profile (e.g. free vs. paid user) can be burdensome and doesn’t always work as expected. But the thing is, that control is something that’s necessary after deployment - not just for engineering teams, but also for product, sales, and customer success teams.

After Deployment

• If 50 changes are deployed and only one has an issue, the whole deployment needs to be rolled back.
• Clunky to control which customers get access to specified features, and engineering usually has to do it manually.
• Hard to use data to understand how a specific change or new feature is performing compared to expectations.
• Real-world data often causes some sort of unexpected behavior and requires rework, or in the worst case scenario, a full deployment rollback.

Despite our best intentions, we often find that we end up with a real mess on our hands in production. The clunky customer experience controls often break, long-lived feature branches cause weird merge issues, and if something is just broken… well… say hello to a long list of stressed and irate engineers. Despite knowing what needs to happen in production, it’s incredibly difficult to get that control and achieve the right objectives without a proper system in production that establishes control and removes all of these barriers.

The Role of Feature Flags

There will be no surprises when I say that a feature flag management system is what needs to be in place to solve the limitations of CD. And doesn’t it make sense? Teams are already trying to implement feature flags with the tooling that they have in place.

Not having control after a deployment is scary - and risky. It’s not just developer stress levels that go through the roof, there are also tangible business impacts to not having control in production - especially when something goes wrong! And that’s the first role of feature flags.

Control in Production

If you have a feature flagging system in place, you realize a lot of value immediately. Knowing that you don’t lose control of your new changes and features once they’re deployed is huge for development and DevOps teams. Allow me to list out of the benefits of using feature flags to get control in production:

• You have a first line of defense. If something goes wrong, it can be handled at the feature level, in production. No more rollbacks of monolithic deployments because of one bad feature. Think of it like creating an easy-access kill switch per feature. We go more in-depth on the topic in this blog.
• Simplify feature access control for customers. Instead of a runtime config, you’re able to granularly define which set of users get access to individual features. An immediate use case here is the delineation of beta vs. GA or paid vs. free users.
• Decouple deployment from release. Deploy doesn’t have to mean that all users get access immediately to everything in the deployment. But with only CD, that tends to be the case. Feature flags allow you to deploy everything in an OFF state, and then you can choose when the feature is live to customers - and who gets to access it.
• Empower non-technical teams and take the load off of engineering. In a world without feature flags, engineers unfortunately become the bottleneck to customer needs, having to manually turn on and off features and permissions. Once in production, feature flags let you hand over control to customer-facing teams so they can manage the customer experience, and you can let engineers get back to their main job.
• Test code in production. You can only test so much in pre-prod environments. Once real data starts flowing in, that’s when you know how good or bad the solution really is. Feature flags essentially allow for safe testing in production by rolling out multiple solution implementations (or even just one) to specified user cohorts and see what best solves the customer problem, or if the feature is ready for primetime. We go more in-depth on the topic in this blog.

Velocity in Software Delivery

The other side of the coin is how feature flags let you realize higher velocity in your software delivery process. Recall, first, that CD does increase velocity, but there tend to be diminishing returns as deployments get larger because of 1) more risk, and 2) more features/changes that need to be completed. So how do feature flags benefit this?

• Deploy without release. Push code into production that’s not ready by putting it behind a flag, and test features with small user cohorts before you roll out to all users. Now, deployments can go live as planned and on time as long as things are put behind a flag.
• Deploy in smaller chunks. This at first seems counterintuitive, but if you think about it, by deploying more often and in smaller chunks, don’t you deliver faster and with less risk concentration? Feature flags allow you to ship things as they’re ready, which ultimately lets engineers move at their own pace - not the pace of the slowest or biggest release.
• Relieve the engineering bottleneck. Feature flags free up the engineering bandwidth that was previously spent making manual changes for customers. It’s a double whammy for customers, who get requests get resolved much faster, and get new features sooner.
• Feedback, feedback, feedback. How do you know if you’ve built the best solution for your users? The answer is always feedback - but how do you get more feedback and faster? Can you turn a 3-month process into a 1-month process? Using feature flags, this is what teams are able to start doing better. It may seem like adding more feedback points would slow down the process, but in practice it turns out it’s faster!

It Doesn’t Make Sense to Separate CD and Feature Flags

Feature flags are to CD what CD is to CI. It’s the natural follow-on that makes it not CI/CD but CI/CD/FF. And the same logic applies too - you need CD to do CI properly, and you need FF to do CD properly. In fact, let me ask it another way: why wouldn’t you do CD and FF together?

Like Batman and Robin, it’s the combination of the two that truly does justice to the goal of velocity and control in software delivery. CD is about shipping as fast as possible, but that velocity is rate-limited by the reliability and quality of what you ship. Feature flags remove this ceiling by consistently making shipping safe, freeing up engineers to focus on what matters most - not on the stress of software delivery.

In fact, teams that use feature flags regularly outperform teams using only CD on the DORA metrics:

• 80% increase in deployment frequency
• 66% reduction in time to go from commit to deploy
• 90% of incidents resolved within one day

Teams want real velocity and real control, not to get stuck part way there. They want to minimize risk and stress, and maximize the quality and quantity of new features. While feature flags or CD on their own bring teams closer to that vision, it’s not until Batman and Robin work together that this future is truly realized.

True Continuous Delivery With Harness

The thing is, most feature flag implementations result in a disconnect with CD, which introduces unnecessary risk and leaves management and visibility fragmented - not what you want for mission-critical systems. You miss out on metrics, audit logs, governance, and security.

That’s where Harness comes in. Harness CD and Harness Feature Flags together provide the common enterprise thread required to perfect Continuous Delivery in all its forms. Of course, you don’t need to have both - but then again, having Batman and Robin together only makes perfect sense.

Final Thoughts

At this point, you should feel like you have a fairly good understanding of the high-level implications of keeping CD and feature flags separate, and really why it doesn’t make sense to do that.

To be clear: you’re missing out on the full value of both CD and feature flags if you’re not thinking about using them together. And in fact, you’re probably creating more headache for yourself using just one and not the other.

Harness meets you at each stage of your software delivery life cycle, whether that’s CD, feature flags, or something else. If you’re ready to check out how we do what we do, you can sign up for free forever or reach out to us for a formal demo.