What are Feature Flags and How Do We Use Them?

Building software is a great experiment. Many times we are required to do what has not been done before; this is core to innovation work. However, a lot of the iteration and experimenting tends to happen before software is shipped. That's where the concept of test in production comes into play. Building something from scratch or even jumping into an older project mid-stream requires lots of iteration.  

Once we deploy, for many software engineers, the experimentation on the existing tends to drop off and onwards to the next set of features in the backlog. Product owners/managers typically collect feedback to help prioritize the next set of features. From a software engineering perspective this can feel a little “chicken or egg-ish” - without a feature actually deployed in a production environment to get feedback, how do we if know we are on the mark?

Let’s illustrate this with an example. Customers are asking for an updated UI view for their dashboards; they want something that surfaces the most relevant information faster. Now that the product team has collected those requirements, it’s up to engineering to figure out how to best build it. There might be a few potential solutions:

1. Create a hard-coded view of the dashboards that’s based on customer feedback and create that for each customer.
2. Allow the customer to decide where they want certain visualizations with a drag and drop UI and give them powerful filters to decide exactly what they want to see.
3. Build an ML solution that learns how the customer prefers to see information and then automatically makes changes to the dashboard that shows the most relevant information.

With some potential solutions decided, now it’s up to the engineering team to decide which of the three to implement. And before a customer ever sees it, they have to choose 1 of 3, build out the MVP over the next few sprints, and then ship it. This is the first time they’ll get any feedback, and if it’s not the right solution then it has to be built again!

Now, imagine if this new feature cycle could be improved for product development teams. If you’ve been paying attention to developer tools over the last few years, you might have seen a new category popping up: feature flags. Feature flags as a concept were born to increase development velocity and to solve this very problem through practices like progressive delivery.

While feature flags as a mature category of hosted tooling is new, feature flags - and what they solve - have been around a little longer. Let’s take a look at what feature flags have grown into, and how to use them.

The Story of Feature Flags

Feature flags, at their core, are the ability to wrap different versions of your code in conditional statements that you can turn on and off at will. You may have previously thought of them as app configs, or even part of your company’s rules engine. 

With Harness Feature Flags, we’ve taken these concepts you may be familiar with and taken them even further - adding more sophisticated rules, governance, user management, an intuitive UI, and more. 

Feature Flags lets you separate feature release from code deployment, build more effective strategies for rolling things out and testing with your users, and have more insight than ever into how changes impact your application performance.

Ultimately, feature flags give engineering teams the ability to deliver more features, with less risk. For leaders, it’s a great way to improve engineering velocity and developer experience, and to reduce the risk associated with feature development. And for developers, it’s a great way to work more efficiently, with less stress - think fewer late nights before a deployment, and fewer firefights post-deploy. 

What Are Feature Flags (or Feature Toggles)?

Feature flags are a way that developers can conditionally turn certain sections of their code on or off. You can think of feature flags as extending Continuous Delivery into Continuous Deployment - a way to put changes into production behind a flag and turn them on in a controlled way later (or hide and remove them in the same way).

But you can also think of feature flags as a new way to think about building and releasing applications - by clearly defining features and components that can be changed and toggled on and off individually, you allow for experimentation, controlled rollouts to different users, as well letting more people (like PMs, sales, and support) turn things on and off for customers.

Together, this results in giving more power and control to developers to create more and better features faster, while at the same time ensuring that engineering is not the bottleneck for the business in having to make changes for individual customers or ensuring a perfect production deployment for every new feature or feature update.

If we want to simplify even further, feature flags essentially create private swim lanes for developers where they can ship a feature directly to customers and then have control over who sees it, get feedback, and turn it on and off as needed. It’s like a light switch (a pretty complex one)!

Who Uses Feature Flags?

People oftentimes consider feature flags either an engineering tool, or a tool for product managers. The reality is, it’s both. Flags can help software development and DevOps teams lower their overhead and increase their velocity, and they can help product managers better control releases, coordinating launch timings and creating a feedback loop more effectively.

Research indicates that 95% of engineering leaders want to implement a feature flag solution for themselves. This comes on the back of 97% of leaders saying they’re under pressure to deliver faster, and 65% of whom say they find it difficult to achieve that velocity increase in a safe manner. It’s abundantly clear that engineering teams are likely the biggest beneficiaries of implementing and using a feature flag solution, and indeed, they tend to be the primary users. And when they do implement a feature flag solution, they end up delivering 66% more features per application per year!

Feature flags are great for users across the organization beyond product and engineering. For sales and support teams, feature flags provide a way for them to directly manage betas and new features for their customers, and to track down who’s using what in case of any issues. And for management, flags can provide new forms of visibility into what’s happening in development and how new features are being tested with users.

On a ‘fun fact’ kind of note and since we’re in the ‘who uses feature flags’ section, it’s cool to note that big companies like Netflix, Flickr, Google, Reddit, and more use feature flags! They’re definitely worth looking into and absolutely are an important part of the software delivery lifecycle.

Why You Should Consider Feature Flags

Whether or not the use of feature flags is important to your software delivery lifecycle comes down to what it is that you need to accomplish. While feature flags are a great tool to have for any team, they are particularly useful if you have these problems:

Problems in the Business

• The risk associated with new feature releases ends up slowing down developers since they have to thoroughly validate design and implementation before releasing.
• There are often multiple possible versions of how a new feature will look and function, and it’s up to the developer to choose one (not always the right one) and implement it.
• The decision of when to release new software features is controlled by developers instead of the business.
• Sunsetting old features introduces risk to newer features that might have dependencies on the old code.
• It’s all or nothing - a feature is rolled out to everyone at the same time instead of doing a feature rollout incrementally for verification purposes.
• When teams are beholden to engineering release cycles (e.g. once a month), multiple features are released simultaneously, introducing complexity and risk that can result in deployment war rooms, large rollbacks, and dissatisfied customers.

Problems in Engineering

• Releasing a feature takes too much coordination and time across stakeholders.
• Merging new features, or managing separate feature branches for changes, is cumbersome and complex.
• You want to test changes to see how they impact your application and end-users before rolling them out more widely.
• You want to provide access to new features, or to test changes with certain portions of the user base sooner - by location, by customer plans, by customers opting into beta testing, or by any other criteria or logic you can think up.
• You want to reduce the risk of rolling out new code by having instant kill switches that don’t require rollbacks or new deployments to fix a disrupted environment.

To simplify, we can consider that feature flags are a great solution if you’re trying to do the following things:

• Increase engineering velocity and ship more features.
• Decrease feature development risk by A/B testing multiple solutions with customers.
• Reduce the risk and toil associated with production failures and rollbacks.
• Decouple engineering deployments from feature unveilings to customers.

Different Types of Feature Flags/Toggles

It’s natural to want to put all feature flag use cases in a single bucket, but it’s more helpful to instead view flags themselves as a means to an end and as having categories of use cases.

Here are some examples of the different kinds of flags that could be used in a system:

• Release features - Using flags to let the right people decide who to turn a feature on for, and when, without complex engineering deployment coordination required.
• Experiment - Using flags to learn something about how a change impacts things. This can be a technical experiment, such as how a new UI impacts server load, or it can be a user experiment such as how a new button impacts conversion.
• Ops flags - Letting you have permanent control around key parts of your app so you can turn them on and off as necessary without a full new deployment cycle. Add in RBAC, and this is a robust way to control who has access to modify your production application. 
• Control access - Using flags as a way to manage betas, early adopter list, trial access, and more.

Implementing Feature Flags

Implementing feature flags can seem really simple at first, but it turns out that like with any system, the devil is in the details. Building an awesome feature flag solution in-house certainly has its perks, but it can be deceptively difficult. In particular, we’ve seen three distinct sets of issues arise with feature flag solutions as an organization’s implementation matures.

When a feature flag solution is nonexistent or immature and an organization is standing it up for the first time, they tend to run into issues with finding value in the tool quickly. Especially with the need for such a solution and the pressure to make it happen quickly, it’s critical that value is demonstrated quickly, otherwise it may end up leaving a sour taste. Within that, there’s a dichotomy of architecting for eventual scale versus getting an MVP out the door to prove value. This often results in complex design or architecture that isn’t scalable and only works for small, targeted use cases.

As feature flag solutions enter a middle stage of maturity, or as they start to scale, the problem is quite obvious: scaling is hard. Especially if built in-house, ensuring that the tool can be used by multiple users and multiple teams for a variety of use cases, all while maintaining a common set of best practices and building on a single infrastructure can be incredibly daunting. And as this is being figured out and teams have code in production, it can quickly result in stale code or issues keeping production tidy when code changes occur. Add onto that the need to integrate neatly into everyone’s workflow and you’ve got a behemoth of a problem to solve.

Building on that is the scenario we often see with highly mature feature flag implementations, where they primarily end up as separate systems from existing software delivery or CI/CD pipelines that teams now have to learn how to use, and to figure out how to port their existing processes or pipelines into the feature flag solution. At this stage, there’s the obvious cost of maintenance and future development to keep up with new changes or requirements. Technical debt is bound to build up by this point, and it can become unmanageable without a dedicated team - and perhaps even its own codebase.

Sure, 95% of engineering leaders may want a feature flag implementation, but as it turns out, it’s not so easy to build something that works well over time.

Build or Buy: Feature Flag Management

You may be thinking, “Feature flags seem pretty simple, can’t we just build this ourselves?” 

The real answer? It’s complicated. 

We’ve seen in most cases that an internally-built tool will end up causing a lot of overhead and putting a very low ceiling on the benefits you can receive from feature flags. It may be good for your first couple of flags, but it can be hard to make an internal system scalable, performant, and robust for all the things flags can do as your organization gets more and more comfortable using them - and more reliant on them.

For a deeper look at the build vs buy conversation, you can check out an article we wrote earlier this month, Feature Flags: Should I Build or Buy?

Why Use Harness for Feature Flag Management

Harness provides a feature flag platform made for getting you up and running quickly while being able to scale to any level. We make it easy to get started and quickly see the benefits of feature flags. With Harness Feature Flags, that means you can get your first flag out the door in minutes by following three steps:

• Create a free trial account.
• Create a new flag in just 3 clicks.
• Add one of our SDKs to your code in a few minutes.

And, you’ve got a flag working!

If that’s not quite clear, then it’s worth knowing that the fundamental way flags work in an environment like Harness is that SDKs, added to your application as a library, will receive updates on flag statuses to evaluate with your users (you should choose between real-time streaming or pulling updates on intervals). These SDKs are secure, collecting no data you don’t want to share, and they are designed for performance and stability. 

Harness Feature Flags is fast, secure, and focused on helping teams get the most out of feature flags - from your first initial use all the way to global governance, sophisticated reusable templates, and developer-first experiences.

And if you’re wondering how we stack up against LaunchDarkly, Optimizely, Cloudbees Feature Management, Split, and other FF tools - those pages are being worked on right now and we’ll update this post with the links when they’re live! 

If you’d like a little more information on our architecture and on how to get started, you can check out the feature flag documentation or request a demo.