Today, we’re going to talk about something people often look for when they’re looking for a feature flag solution: testing in production!
You may not use those words to describe it, though, so let’s first take a look at what it means. Then, we’ll dig into why you might want to do it, and how.
From Joke To Reality: What Is Testing In Prod?
About a decade ago, “testing in production” was a not-uncommon joke. It was a shorthand way of describing something you definitely would never do… until eventually, some tools started encouraging you to do exactly that.
Testing in production, in short, means using live data and live users to figure something out. It can be performance related, user behavior related, or anything else that you want to see, learn from, and react to based on what’s really happening when your apps serve live users.
What ISN’T Testing in Prod?
Let’s be clear, too, on what testing in production is not. It is not constantly doing deployments and rollbacks. And, it is not being cavalier about what you release. When testing in production, you are not using your live data to make sure things work. That’s still what testing and QA is for.
Testing in production is all about learning, iterating, and helping inform your decisions with the best and most accurate data possible – after code has already gone through the standard deployment process.
Why Test in Production?
Writing tests, good QA, having PMs and other folks review changes prior to deployment – this all goes a long way in getting things right. But, there are some things that are elusive to try and learn with mock data, or at different scale, such as:
- What does your real load do to your latest changes?
- What do users do that’s different than expected?
- Are you seeing the performance change you hoped to see?
- Are users reacting the way you intended them to?
- How does your change react under various stress situations?
You can try to learn some of this on your way to prod, but the most effective way to answer these questions – both technical and user ones – is taking things for a spin with actual users and actual traffic.
Why Use Feature Flags to Test in Production
Feature flags are critical to effectively testing in production. We say this because feature flags allow you to turn things on and off instantly against any criteria you want without the complexity of a rollback or redeployment.
If you’re not familiar with feature flags, we’ve got a great blog on How To Get Started With Feature Flags, as well as another blog on 5 Feature Flag Use Cases You May Not Have Thought Of.
Without feature flags, you are limited to blue-green deployments, canaries, and other very valuable but slower (and more expensive!) forms of production tests. Feature flags bring the cost of production tests down to almost nothing. They make testing in production simple and easy to coordinate.
Below are some examples of tests in production you might want to run. Typically, these tests relate to tangible business outcomes such as customer satisfaction, or revenue gains:
- Of three possible solutions to this customer problem, which best serves customers?
- I have two ways to implement this algorithm. In test, they are about equal, but which performs better when large amounts of data come in via my one million users?
- I want to validate whether this new UX will impact user retention and time-in-app as I expect it to.
- I’m expecting this simplified purchasing process to decrease the number of abandoned carts on my ecommerce page. I want to test this against two groups of users.
- I am rolling out a new set of long-lived operational flags we can flip during scheduled maintenance or outages. Does it behave the way I expect against my whole user base?
What to Know When Using Feature Flags to Test in Production
By definition, testing in production can – and should – cover a lot of territory. You can test load, behavior, resiliency, and more. Therefore, there’s no one-size-fits-all answer. It all comes down to what you want to do.
However, there are a couple things to keep in mind when thinking about feature flags and testing in production for the first time:
- A feature flag gives you options you don’t otherwise have. Instead of thinking “What am I doing that could use a feature flag?” you can think “I’m making a change, let me put it behind a flag just in case.” It’s always better to have the option, and the cost is close to nil.
- Agree up front what you’d like to learn and what your goal in testing will be. Who owns the test and how will you learn from it?
Using Harness Feature Flags to Test in Production
Harness Feature Flags is a complete feature flag management tool that allows users to create and manage flags both in code and through a UI. Harness uses a flexible targeting model that lets you apply your flags any way you want – against users, regions, clusters, accounts on certain billing levels, or anything else you can think of. Take advantage of this to test from a wide variety of angles simultaneously.
In addition to the ability to rig tests, there’s a layer of automation that can also be instrumented. This can especially be helpful if there are standardized tests in production that you run, or you know exactly what you want to happen when certain behavior occurs. For example, you could run three different versions of a new UX, and based on which causes users to stay on the app for the longest time, you want to automatically roll out that feature to the whole user base and remove the other two. That delivery, verification, trigger-based rollout can all be automated using Harness.
Testing in production has gone from a quip to a reality with the spread of modern CD and feature flags.
Feature flags specifically help increase the velocity and responsiveness while lowering the risk and cost. This is critical, because the best tests to learn from are the ones that most match reality. And, reality means production.
While it can initially be difficult to take the leap into production testing with feature flags, there are some questions we can use to get started. It becomes a piece of cake to iterate and expand as your team gets more comfortable.
Have you read our eBook on Feature Flags yet? It’s free and doesn’t require an email address! If you’re looking to learn more, it’s a great resource. Download The Basics: Feature Flags 101 today!