Migrating From Homegrown Feature Flags

At Harness, we recently did something pretty difficult. Actually, we did two difficult things, with the second challenge compounding the first.

We launched a new product - Harness Feature Flags - which is always tough, and exciting. On top of that, Harness - like many companies out there - already had feature flags we used day to day internally, through a home-built system that had evolved over the years.

In launching our own commercial feature flag solution, we also worked on migrating the entire company off of the existing internal system over to our new Feature Flags product. Replacing infrastructure (which is what feature flags really are at a certain point of maturity) is never easy, but it’s especially tough when you have a wide variety of stakeholders - from design and support to PM - trained to use the existing solution. And yet, we managed to do it.

So, let’s talk a bit about why we did it and what we learned.

Migrating Feature Flags: Why We Had To Do It

First, why did we migrate? Half of the answer is obvious: to drink our own champagne. Internal usage is an accelerant to product learning, and while you always need to be careful not to assume that you represent your customer identically, having Harness use Harness (like we’ve always done for CD, and now CI) means we have a constant, easy to access stream of real world usage and feedback. 

Every company aims to be great at this, but in reality, lots of companies never quite get the appetite to do the hard work to replace the system that’s working with their own new solution. I know of a lot of DevOps companies out there that are still using their competitor’s tools because they simply could not interrupt work long enough to make their aspirations of dogfooding real. Not at Harness - for better or worse!

That’s only the dogfooding side, though. There is another very real reason to migrate and it’s actually on the value and engineering sides. Internal solutions (as I discussed in my “Should I Build or Buy” piece) are expensive and put a very low ceiling on the value you get with feature flags. Specifically, some of the limitations we had with our system here were:

• Only boolean flags. We never built out the functionality for more complex flag configurations
• Limited custom rules engine
• Limited tech start support, stopped working as we added new services and products
• Difficult to secure properly (do we really want to build granular RBAC for our internal tool?)
• Poor UX, people needed very specific instructions in how to use it

The result of all this is that while we were using feature flags, which is better than not, and even in a pretty mature way since we had people all across the org using them, we also lost a lot of time because of the difficulty and limitations of the system. We were probably a fairly advanced use case and we still struggled to make it efficient from any angle.

By introducing a proper product to do this (albeit our own), we were able to solve essentially all these challenges instantly while also wiping out the engineering burden necessary to maintain the system globally. 

Solution vs Internal Comparison

Here’s a quick look at the internal system we had before Harness Feature Flags replaced it, compared to what we have now with our commercial solution.

Challenges We Faced While Migrating Feature Flags

The good news is that migrating feature flags from our internal system was perfectly easy. It took about an hour and there were no challenges. 

Wouldn’t it be great if that was true? Feature flags are embedded into some of our core processes - they’re part of how support works, part of how PMs work, and more. You can’t make a change like that without some hiccups. 

We can talk in another post about the technical parts of how we migrated, but here I’ll focus a little more on the process and people side. What kind of challenges did we encounter?

• When we imported all of our existing flags into Harness Feature Flags, they were… a bit difficult to understand. Previously, we didn’t have naming and descriptions, so what we ended up with was a dashboard of flags that unless you knew what you were looking for, you’d never figure it out.
  
• We did not take enough time to communicate or train folks ahead of this change. Generally, a good rule of thumb is probably that however much time you’ve taken to communicate and train people, add more time because it’s not enough. We saw our support team and some engineers from other teams have important things they needed to get done that suddenly they didn’t know how to do. This is preventable, and we can learn from it for future changes.
  
• We actually probably made this change too late, but there are some small UX improvements that will make a big difference for our teams using the product that we could have caught even earlier if we had been more aggressive with our timeline. Similar to the rule above about enabling folks, when you’re considering your timeline for when to start testing your own tools, accelerate that timeline in almost all cases.
  
• We could have made support and escalation paths clearer. If there was a problem with the existing solution, go to the Ops team. But if you had a question or issue about the new, commercial solution, what do you do? Do you file a ticket, like a customer would? Reach the PM? Ping the lead engineer? We didn’t make this clear enough, and it caused some unnecessary back and forth.

So, we certainly had some challenges - which we’re not afraid to share here - but we were able to learn from them and are in the process of making both product and process improvements internally as a result. This is what drinking your own champagne is all about! 

Results Are Early

Because we just made this change, we’ll avoid talking about the amazing impacts of migrating feature flags just yet - we want to give it some time so that we can circle back and share more knowledge once we have seen the impact after a few months. But, so far the response is promising and we have freed up our Ops teams to spend time on better problems.

Conclusion

Harness did two things that are tough in any circumstance: launching a new product, and replacing an existing tool. We did them back to back while bringing a massive customer on (ourselves) to learn from. 

I probably don’t need to encourage companies to want to do this. In my experience, the difficulty in dogfooding is never in the desire but in the ability to commit to getting it done. But, it’s absolutely worth it - you may need your leadership to put a stake in the ground, but on the other side of a change like this is invaluable learning of a powered-up organization able to move faster and build together more effectively.

Interested in learning more about feature flags? Book your demo today.