Using Feature Flags for Effective Incident Management

Whether you call it an incident, outage, surprise, or unplanned work, your application isn't working as expected, and you need to deal with the problem. That’s where dependable incident management comes in. You never want to be responding to an incident without a plan or prior systems set in place. You may not always be able to stop an incident, but you can always dampen and remediate its effects with good planning.

How Feature Flags Can Help for Incident Management

Feature flags can help you improve your incident management process in a number of ways:

• Kill switches and circuit breakers: You can use feature flags to disable features or services quickly and easily without having to redeploy your code. This can be helpful if you need to quickly isolate the source of an incident or prevent a problem from spreading. This can prevent a feature from causing problems for your users.
• Throttling: You can use feature flags to throttle traffic to certain features or services. This can be helpful if you're experiencing performance issues or if you need to prevent a feature from being overloaded. This can help to improve the performance of the service for more critical or demanding users.
• Debugging: Feature flags can help you debug incidents by providing you with insights into which features or services were recently enabled or changed. This can help you quickly identify the root cause of an incident. If it’s not obvious at first, you can also enable and disable different features and services to see how they impact performance and study the affected systems.
• Automation: You can use feature flags to automate tasks related to incident management, such as adjusting logging levels, rate-limiting services, or disabling features. This can help you reduce the time it takes to resolve incidents.

Tips for Effective Incident Management

Here are some tips for using feature flags for incident management:

• Create a flag management plan. This plan should define how you will create, manage, and use flags for incident management. It should also include a process for reviewing and updating your flagging strategy on a regular basis. We’ve seen this be very effective for teams using feature flags at Harness. Remember: a little extra time spent on this before can save a lot of time (and money) after.
• Use flags to implement operational resilience practice. Operational resilience practices are designed to help you minimize the impact of incidents on your users. For example, you can use flags to implement circuit breakers or to throttle traffic to certain features during an incident.
• Monitor your flags. It is important to monitor your flags to ensure that they are being used as intended. You should also monitor the impact of your flags on your application and users.
• Testing your incident response plan. Use feature flags to simulate different types of incidents during your incident response testing. This will help you to identify any areas where you need to improve your plan. If, however, an actual incident occurs, treat it as another (harsher) test of your incident response plan and make sure to apply all lessons learned during it. Even if you had an effective incident response, it’s important to acknowledge what went right.

By following these tips, you can use feature flags to improve your incident management process and reduce the time it takes to resolve incidents.

Benefits of Using Harness for Incident Management

In addition to the general benefits listed above, Harness Feature Flags also offer a number of differentiators that can be particularly helpful for incident management. Here are some specific examples of how you can use Harness Feature Flags with OPA, Pipelines, and Git Sync for incident management:

• Use OPA to enforce policies on feature flags: You can use OPA to enforce policies on feature flags, such as requiring that certain approvals be obtained before a feature flag can be enabled. This can help to prevent accidental or unauthorized changes to feature flags during an incident.
• Use Pipelines to automate the process of enabling and disabling feature flags: You can use Harness Pipelines to automate the process of enabling and disabling feature flags. This can help to streamline your incident management process and reduce the risk of human error. For example, you could create a pipeline that enforces that a feature be tested in non-production before the feature is enabled in production.
• Use Git Sync to keep your feature flags in sync with your Git repository: You can use Harness Git Sync to keep your feature flags in sync with your Git repository. This can help to ensure that your feature flags are always up-to-date and that you have a complete audit trail of all changes. For example, you could use Git Sync to create a branch for each incident fix and then use that branch to track all changes to feature flags related to that incident.

Beyond incident management, here are some other key benefits of using Harness Feature Flags for your project:

• Automated lifecycle management : Harness Feature Flags helps you manage your flag tech-debt by detecting potentially stale or deprecated flags and automatically removing them if needed‍
• Integrated with CI/CD: Since Harness covers a range of software delivery products, you can easily integrate your feature flags into CI/CD as a unified pipeline that helps maintain the flags as part of the software development lifecycle.

Conclusion

By using Harness Feature Flags for incident management, you can improve the resilience of your application and reduce the impact of incidents on your users. Incident management starts before you write your first line of code and continues after your code is in production. By following the tips above, you can effectively contain the blast radius of any unexpected incidents and restore the user experience.

If you want to use Harness Feature Flags for incident management, get started for free now!