Ancestry Cuts Downtime by 50% Using Harness Governance

Authors:

Table of Contents

About

Ancestry®, the global leader in family history and consumer genomics, empowers journeys of personal discovery to enrich lives. With Ancestry’s unparalleled collection of more than 27+ billion records and 18 million people in our growing DNA network, customers can discover their family story and gain a new level of understanding about their lives. For over 30 years, they've built trusted relationships with millions of people who have chosen them as the platform for discovering, preserving and sharing their story.

The Deployment Pipeline Wild West

Ancestry’s software deployments were like the wild west, and it was up to Chief Architect Russ Barnett to be the sheriff. Each Ancestry application team built their own custom Jenkins pipeline to deploy software. Some teams built sophisticated pipelines that allowed them to deploy multiple times a day. Some teams could hardly reach production.

We were acting like we had an advanced CI/CD culture with the capability to deploy multiple times a day. The truth is we didn’t have the infrastructure to support that for all our teams.

Russ Barnett | Chief Architect | Ancestry

The lack of consistency across teams started to affect Ancestry’s uptime. Faulty pipelines, insecure deployment strategies, and unreliable rollbacks caused downtime for some of Ancestry’s key applications. In one instance, a key application that allowed Ancestry.com to function went down for four hours.

Mock up computer pc on table with books and lamp.

Russ knew he needed to standardize Ancestry’s deployment process. He attempted to onboard all the application teams onto a centralized testing tool. This would help reduce the number of issues reaching production. After a year and a half, the tool only had 20% adoption.

Any pipeline change we needed to make was a complete nightmare.

Russ Barnett | Chief Architect | Ancestry

Russ then tried to make canary deployments standard across the Jenkins pipelines. Canary deployments would make it easier to roll back and identify production issues. It took a developer 2 weeks to onboard 2 systems. Ancestry has 10 systems per team across 70 teams. That equated to roughly 13 years of developer effort to implement canary deployments.

Russ needed a way to reign in their uncontrollable Jenkins pipelines. So he turned to Harness.

Standardization Leads to Stability

Using Harness, Ancestry was able to onboard 350 systems in a year, with full adoption projected at the end of year two. This was an 85% reduction in effort when compared to Jenkins.

These new standard pipelines were easy to change and govern. Canary deployments became the de facto Ancestry deployment strategy, and the centralized testing tool’s adoption rate rose to 97%.

Example Canary Pipeline in Harness

Once teams are using a Harness pipeline, it’s easy for us to add additional mandatory steps.

Russ Barnett | Chief Architect | Ancestry

The increase in pipeline governance resulted in a 50% decrease in production outages. The remaining outages resulted from infrastructure problems, and none of the outages were related to deployment issues.

Harness also helped increase developer productivity with a 3x increase in deployment velocity.

Our teams were initially skeptical of getting rid of their Jenkins pipelines for Harness, but now they’re big fans of the Harness UI and the ease it brings to the deployment process.