How DORA Metrics Improve DevOps Performance
| Harness Blog
Explore DevOps Research and Assessment (DORA) survey findings and share what you need to know about achieving Continuous Delivery and the DevOps philosophy on speed and stability.
DORA metrics measure DevOps performance through four key indicators: Lead Time for Changes, Deployment Frequency, Mean Time to Restore (MTTR), and Change Failure Rate.
These metrics shift focus from individual outputs (like lines of code) to cross-functional team outcomes that are statistically correlated with improving organizational performance and business outcomes.
Elite-performing teams deploy on-demand (multiple times per day) with lead times under a day, compared to low performers who deploy monthly with lead times of one to six months [Source: DORA, "State of DevOps Report", 2024, dora.dev/research/2024/dora-report/].
Peter Drucker once said, “If you can’t measure it, you can’t improve it.” The same is true for DevOps. To efficiently and effectively deliver better software, teams need the visibility, data, and decisions to drive DevOps capabilities.
The DevOps Research and Assessment (DORA) team is Google’s research group best known for their work in a six-year program to measure and understand DevOps practices and capabilities across the IT industry. DORA’s research was presented in the annual State of DevOps Report from 2014 - 2019. The group also produced an ROI whitepaper, providing insights into DevOps transformations.
DORA’s research identified four key metrics that indicate software development and delivery performance from the six years of study data. DORA team’s lead, Nicole Forsgren, co-authored a book called Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations. This book shared the DORA team’s findings explaining the science and research behind the capabilities that delivery and development teams should invest in to drive higher software delivery performance.
What Are the Four Key Metrics?
Measurements for developer productivity and performance like lines of code, velocity, and utilization focus on individual or siloed team outputs. In the spirit of cross-functional delivery teams, tracking cross-functional team outcomes versus individual outputs allows organizations to achieve their organizational goals with more focus and speed.
Through their book Accelerate, Nicole Forsgren, Jez Humble, and Gene Kim identified key characteristics for building high performance technology organizations. These top-performing organizations focused on engineering outcomes over outputs and teams over individuals by tracking four measures: Lead Time, Deployment Frequency, Mean Time to Restore (MTTR), and Change Failure Rate.
Let’s define each of these terms and discuss practical methods for measuring these metrics.
Lead Time
Delivery Lead Time is the total time between the initiation of a feature request to the delivery of that feature to a customer. In lean manufacturing and value stream mapping, it’s common to capture the Lead Time for a process like deploying a service. Capturing the total time it takes from source code commit to production release helps indicate the tempo of software delivery.
Deployment Frequency
Deployment Frequency is how often an organization deploys code for a service or application. Frequency indicates the tempo of software delivery. The theory behind Deployment Frequency also borrows from lean manufacturing concepts and how it translates to controlling the batch size of inventory to be delivered. Highly-performing organizations do smaller and more frequent deployments.
Mean Time to Restore (MTTR)
Mean Time to Restore or MTTR refers to incident resolution. Where there is a failure, the Mean Time to Restore is the average time it takes to restore the service when an incident occurs. MTTR is a measure of time similar to Lead Time.
Change Failure Rate
The Change Failure Rate is the percentage of changes made to a service where the change results in remedies, incidents, rollbacks, or failed deployments. Change Fail Percentage is a measure of quality. Based on the DORA team’s research, high-performing teams were somewhere between the 0-15% range.
How Do You Use Accelerate Metrics?
When tracking these metrics, it is important to consider time, context, and resources. Data analysis requires consistent measurement over time. Different levels of leadership can then understand these results based on context. Was there a lack of tooling or automation to aid in deployments, triaging incidents, and testing our services? Were there changes in architecture, planning, or goals during this time? Similarly, tracking these metrics per service and across various teams can provide additional insights into what’s going well and what is not.
These metrics are meant to encourage improvement, discussion, and delivery across anyone with a stake in the software service or application. Accelerate metrics can be used to compare the performance of different teams. By comparing the performance of different teams, organizations can identify teams that are performing well and teams that could use improvement. This information can be used to identify best practices that can be shared across teams.
This could be especially true if the deployment frequency was daily or weekly. If the deployment frequency was infrequent, but the Change Failure Rate was high, this could indicate that the deployments were not well planned and could have contained major or large feature changes.
Context, timing, and resources matter in these conversations. But these four key metrics influence one another and often help unravel stories and insights that would otherwise be harder to understand. Looking at the duality of speed and stability is one method for analyzing your DevOps performance.
Beginning the Journey of Continuous Insights
Now that we understand the four key metrics shared by the DORA team, we can begin leveraging these metrics to gain deployment insights. Harness’ Continuous Insights allows for teams to quickly and easily build custom dashboards that encourage continuous improvement and shared responsibility for the delivery and quality of your software.
Continuous Insights provides real-time delivery analytics,automatically giving DevOps and team leads insight across all applications, environments, versions, and deployments within the Harness platform. There are many ways to track these metrics. For example, the open-source project Four Keys gathers and displays your DevOps performance data from your GitHub or GitLab repos.
Improve Your DevOps Toolkit With the Harness Platform
The blog post shares how to measure and track DORA metrics. Like many other forms of manufacturing and production, software delivery can be tracked, understood, and improved. If you’d like to leverage these four key metrics for continuous insights for free, try Harness today.
There are many articles and topics currently available for delivering business value better and faster. Subsequently, many more organizations and teams are adopting common software practices - even outside of the usual use case of delivering software (take, for example, SpaceX teams working agile, or leadership teams that use scrum practices and have stand up calls). This progression into faster, better, cheaper, easier leads into the topic of this blog post, where we’ll provide a beginner’s guide to DevOps and what it takes to achieve and incorporate DevOps into business value delivery.
Starting With Then vs. Now
A great perspective to consider for understanding DevOps is how technology has changed how we build, connect, share, and educate. Up until the 2000s, the de facto turnaround times for photographing and sharing pictures would be 1-3 weeks, if not longer. This process would involve film cameras and darkrooms to process photographs. Today, mobile devices allow us to snap hundreds of images and share them immediately online. These changes in the world don’t only apply to photography but many other industries, including Financial Services, Banking, and IT Technology.
The DevOps Lifecycle
Software delivery, in the past, consisted of traditional waterfall methodologies that offered little flexibility to the nature of change and demand for technology services today. Software delivery now is about using DevOps to leverage new insights through people, processes, and technology, giving organizations the ability to deliver business value continuously.
For our technology organizations, this process for delivering business value is modeled through the DevOps Lifecycle.
Getting To A DevOps Culture
Very often, our challenges with DevOps do not lie in the technology or processes, but in how we work together to deliver with our cross-functional teams. In order to understand what building a DevOps culture means, we need to comprehend what culture encompasses and how our people, process, and technology work together to enable a successful DevOps lifecycle.
I like to consider DevOps to be a layered process made up of:
· Mindset
· Values
· Principles
· Practices
· Tooling
Getting to a DevOps culture involves articulating change both from the bottom and top of our organizations and is, therefore, no easy feat. Before we can introduce or discuss tooling, we need to understand how our practices and principles help us achieve our outcomes. Before we can set practices, we need to understand the underlying principles of use. Each inner layer requires the outer layer before we can truly get to defining and strengthening a DevOps culture.
Here are my tips for getting to a DevOps culture:
· Develop from mistakes
· Focus on the important
· Capture collaboration
· Perfect your tooling
The DevOps Principles
To summarize up to this point, DevOps is the collection and collaboration of people, processes, and technology to deliver software value continuously. Let’s discuss some core DevOps principles to eventually share some tools and practices you can use for your DevOps journey.
Create With the End in Mind
High-performing teams work best when leaders and individuals understand their role in the team. Therefore, creating with the end in mind involves reassessing and often asking big picture questions, like “Who is this delivery for?” and “Why are we doing things this way?” Realizing the deliverable is about keeping the vision in mind and making adjustments along the way.
Famous photographs in history involved many adjustments and attempts. Creating with the end in mind means knowing we may not have everything right the first time around. Similarly, life isn’t without its own distractions, conflicts, and changes. Keeping the target outcomes in mind helps us align to initiatives and actions and question when they don’t align.
Cross-Functional Teams
Many productions and deliveries involve more than one person. For example, our best MCU movies were made thanks to the help of multiple production teams and companies coming together to build what we know today and some of the best superhero movies. We succeed with software delivery in the same way by having fully responsible cross-functional delivery teams.
The best engineering teams I’ve been on contain independent and autonomous individuals that know how they can contribute to the work in progress. They’re also willing and unafraid to collaborate with others in order to get this work done.
Commitment to the Journey
“I am not here to be right. I am here to get it right” - Brené Brown, from the book Dare to Lead.
Even with our cross-functional delivery teams and creating with the end in mind, our teams still may not have everything right the first time around. Committing to DevOps is about accepting this and any failures that come from the journey. Committing to the journey doesn’t mean you can’t define what it involves. Experimentation is also a part of how we will adapt and adopt novel approaches from delivering faster, cheaper, and better software.
The DevOps Principles as Guides
Use these principles as the foundations for how to work together, use tools, and define and improve processes. In photography, there are three components that serve as building blocks to the final images we capture: aperture, shutter speed, and ISO. In DevOps, it’s our people, processes, and technologies. Achieving high-performance with DevOps involves all of these components working together and adjusting to our environment, situation, or team.
The key to getting to the results you desire: Not getting blindsided by any single component of getting to the result.
The DevOps Practices and Tools
Now that we understand the mindsets, values, and principles of DevOps, we can begin sharing DevOps practices and tooling. I like to consider this section like the one that calls our teams and individuals to gear up! No reasonable photographer would leave for a photoshoot without their camera and camera gear. Similarly, DevOps practitioners and teams need the right tools to get the results needed.
Source Code Management
Source code management is about allowing teams to collaborate, version, correct, and revert software code. This is fundamental for many teams to increase not only visibility and collaboration, but also start automation workflows.
If you are an individual just starting out, I recommend learning the basics of Git, such as how to add, commit, push, merge, branch, and clone code. If you’re a team, I recommend discussing practices and standards you want to use for collaborating. Always fit your Git structure and workflow to your codebase and team dynamics.
Continuous Integration
Continuous Integration (CI) takes code away from local machines to package code into a deployable artifact in a CI server. CI is about automating the integration of newly committed code changes. This process is often triggered by code changes pushed into your Git repository. Using checks and other tools, teams can version and release artifacts that have been vetted for quality and security.
Continuous Delivery
Continuous Delivery takes artifact code to production environments. CD aims to deliver code changes of all kinds to users in a fast, repeatable, and safe manner. CI/CD consists of both CI and CD pipelines working together to deliver code to users.
Infrastructure as Code
Infrastructure as code is a practice that allows users to define hardware through code. The goal of delivering in a DevOps matter is to improve how we can provision, configure, and maintain hardware systems. Infrastructure provisioners like HashiCorp’s Terraform and AWS Cloud Formation, configuration automation tools like Ansible, puppet, chef, and Package Managers like Helm allow teams to accomplish much more when software scales and demands increase.
Secrets and Identity Management
Security is everyone’s responsibility. Secrets management and identity management tools make accomplishing DevSecOps easier by securing and managing sensitive information. Teams should consider how they are handling secrets and users within their architecture. HashiCorp Vault, KeyCloak, KMS, and other similar tools can help you achieve your goals around security.
Much More to Be Discovered
There are many other practices and tools DevOps teams and organizations can use to achieve their desired outcomes. What works for one team may not work for another - hence the need to innovate and experiment in a controlled and well-led manner. Other practices that can increase your team’s ability to deliver value include value stream mapping, progressive delivery, GitOps, and chaos engineering.
Some Key Takeaways
People, processes, and technology belong together in a DevOps culture. While practice makes perfect, it’s more about getting the right answers along the way than coming in with the mindset that everything has to be right the first time around. Just remember that no leader, individual, or team in DevOps is alone. We started this blog post defining what it requires to build a DevOps culture, change mindsets, and set principles and practices, so I leave with one takeaway to ask you: What does DevOps mean to you?
To understand this space, it's helpful to review the origins of continuous delivery, current trends, and a variety of tooling approaches.
Continuous Delivery Origins
The term CI/CD can be traced back to the 2000s, when Java, Javascript, and PHP emerged. Several developers came together to write the agile manifesto, the first principal of which is "Our highest priority is to satisfy the customer through early and continuous delivery of valuable software". The agile manifesto paved the way for the release of the first CI tools in 2001, but technological limitations of the time kept CD a pipe dream.
Still, once builds were automated with local unit testing, it was natural to look to deploy to a test environment for functional testing and from there to look to replicate those deployments in production. This led to early continuous delivery tools like AnthillPro(https://en.wikipedia.org/wiki/AnthillPro). However, the norm was to to build custom scripts on top of continuous integration tools like Jenkins to create full deployment pipelines.
Other CI tools picked up on this, and soon, every CI tool touted its ability to create Continuous Delivery pipelines. For five years, CI was synonymous with CD, and it was generally accepted that you could manipulate your CI tool to act as a CI/CD solution.
However, companies realized they were spending too much time scripting CD. As all of you know, wherever there are custom scripts, there’s an opportunity for a new solution. This is where CI decoupled from CD and Continuous Delivery tools entered the market.
Continuous Delivery Fundamentals
Continuous Delivery is the process of deploying build artifacts into an environment. An example of an artifact is a docker container, and the environment could be for testing or for production.
Continuous Delivery tools help create pipelines to standardize releases. Fundamentally, pipelines can be broken down into five stages:
Testing and QA - These tests are normally performed in addition to tests that are run during the build process. A CD tool should allow you to incorporate your existing test automation (regression testing, etc.) into a pipeline.
Change Management, Governance, and Approvals - These steps identify a version as appropriate for production release. Typically the tests and other risks are reviewed.
Deployment Strategies - This is the approach taken to deploy applications to production. Examples include canary deployments, blue/green deployments, and rolling deployments. Deployment strategies are defined in the CD pipeline.
Verification - This is the process of verifying the success of a production deployment. Monitoring tools should plug into deployment pipelines to help verify.
Rollback - If something in the verification step reveals an issue, then a pipeline should be able to roll that deployment back.
CD flow" id="" width="auto" height="auto" loading="auto">
Some CD solutions emphasize a particular area, but all will have these basic concepts. I suggest doing more research on what CD is and why you should care.
CD Tool Categories
For the purposes of this exercise, we’ve broken down CD solutions into four categories.
Open-Source - As the name suggests, this category will feature open-source tools that are “free” to set up and are managed by a community.
CD Platforms - These solutions built products around the entire software delivery life cycle. CD makes up a substantial portion of its value proposition.
Cloud Provider CD Tools - These solutions are created and maintained by the major cloud providers and are optimized to work on that specific cloud.
CI Tools - These are tools optimized for Continuous Integration, but can be extended to accomplish CD with enough custom scripting.
Things we looked for in good CD tools: easy installation, intuitive user interface, enterprise-grade features, low maintenance cost, and GitOps functionality. This list is not exhaustive but should give you an overview of the most popular players in the market.
Open-Source CD
Open-source tooling is attractive for many reasons. Popular projects have hundreds of dedicated engineers bringing diverse experiences to a world-class product. Not to mention there’s no sticker price.
The downside of open-source is the level of expertise needed to build and maintain the solution. The maintenance effort can often outbalance the “free” nature of the products. Open-source CD tools aren’t optimized for large enterprises with granular security control needs. That being said, smaller shops might find open-source to be an attractive solution.
Best - Argo CD
Argo CD is the cream of the open-source crop. This feature-rich solution automates Kubernetes deployments using GitOps technology and supports a wide variety of config management tools, SSO integrations, and webhook integrations. It reduces administration toil by making application definitions, configurations, and environments declarative and version-controlled. Argo CD’s stated goal is to make application deployments automated, auditable, and easy to understand. It boasts 5k stars on GitHub.
Like all open-source tools, you’ll want to consider the maintenance effort and lack of governance, but overall this is a solid option to deploy code. Argo CD struggles as you look to move beyond just deployment to full continuous delivery. Migrating through a series of environments based on the results of automated testing is largely outside of Argo CD's scope today. Perhaps it would be more accurate if it were named "Argo Deploy" rather than "Argo CD".
Spinnaker is an open-source solution brought to you by Netflix. There are currently 300 open Spinnaker issues on GitHub.
Spinnaker’s major issue is setup and maintenance effort. On average, it takes Spinnaker users 6-8 months to stand up a pipeline, and by that point, enough engineering resources have been spent to justify the purchase of a different tool.
Check out our comparison of Spinnaker vs Harness to see how we stack up.
Proceed with Caution - GoCD
GoCD is an open-source “middle child.” The solution was sponsored by ThoughtWorks, and is now maintained by a handful of former Thoughtworkers. It has plenty of features like pipelines as code, native Kubernetes integrations, elastic agents, JSON and YAML support, and a modern interface. There’s a solid amount of plugins available on the market, which helps it fit in with your current tool stack.
Unfortunately, it’s a noticeably newer tool that lacks advanced features found in more tenured solutions. There’s quite a bit of potential for improvement here, but at the moment your effort would be better spent somewhere else, especially with its future in doubt.
CD Platforms
Software delivery encompasses Continuous Integration, Continuous Delivery, cloud cost management, feature management, etc. Software delivery platforms decided to provide an end-to-end platform that makes software delivery a one-stop-shop.
Each company has approached this in a different way, and therefore, each company has had varying results.
Best - Harness
Harness is a modern CI/CD platform that offers Continuous Integration, Continuous Deployment, Cloud Cost Management, Security Testing and Feature Experimentation. Harness was founded on Continuous Delivery roots and that still remains one of the company’s cutting-edge products. It leads the pack when it comes to governance with fine-grained RBAC, full audit trails, and proprietary integrated secrets. It’s also worthy to note the incredibly easy installation, which reduces time to value considerably.
The great thing about Harness is its à la carte model. Teams are free to pick Harness modules without purchasing the entire platform. Every Harness product is developed to be standalone in quality and functionality.
GitLab offers a thorough software delivery platform that includes a solid Continuous Delivery tool. All the CD fundamentals exist here, but the tool lacks advanced governance features such as automatic verification, and automatic rollbacks.
The hardest part about using GitLab is its all-or-nothing model. If you want to use any of the GitLab tools, you need to use all the GitLab tools. So if you work in an opinionated tooling organization, you’ll have to convince them to leave behind some of the tools they already use. Our advice for GitLab? Stick to Git.
Check out our comparison of GitLab vs Harness to see how we stack up.
Proceed with Caution - Digital.ai
Digital.ai attempts to be a DevOps swiss army knife. Instead, it resembles a butter knife. The company acquired XebiaLabs in 2020 in an apparent attempt to provide Continuous Delivery, but it struggles to keep up with a modern Kubernetes world.
A quick glance at the product section of their website will show you the breadth of their offerings. We recommend going with a platform that emphasizes software delivery specifically.
Is it any surprise that the three biggest tech companies in the world have solutions for Continuous Delivery? Amazon, Google, and Microsoft each created their own flavor of CI/CD. Something to note is these tools are optimized to work with their corresponding cloud environment. Therefore, these tools are a viable option for organizations running a single cloud environment.
Best - Azure DevOps
Azure DevOps includes CI/CD pipelines (Azure Pipelines) as well as several other DevOps modules. As expected from Microsoft, Azure DevOps offers fine-grained RBAC and audit trails, making it a smart solution for companies in highly-regulated industries. Microsoft's focus seems to be more on GitHub Actions, but ADO remains a decent offering.
Azure DevOps is not strictly limited to Azure environments. This is a tool that you can use across your application stack.
Good - Google Cloud Build
Cloud Build, Google Cloud’s CI/CD platform, is CI focused, but has some deployment capabilities. This CI/CD platform provides standard pipeline features and works well with a modern tool stack.
Honestly, this tool is solid. But it lacks the intuitive design of Azure DevOps. This is still a solid deployment option if you’re on GCP and need something to get started with.
Proceed with Caution - AWS CodeBuild/CodeCommit
CodeBuild is a fully-managed CI/CD tool. It primarily focuses on Continuous Integration, but does offer some basic deployment strategies. CodeBuild exclusively works on AWS environments, making it a no-go for other cloud users.
Amazon’s deployment tool is confusing to set up and manage. Save yourself some time and choose something else.
Tools that aren’t actually CD tools…
CD is a new buzzword, and everyone wants in on the action. Unfortunately, this is making it harder to distinguish between CI tools and CD tools. The reality is most CI tools can be extended to CD, but that involves custom scripting, script maintenance, and a less than ideal end solution. We think these CI tools deserve respect for their integration capability. But we also think CI and CD are separate disciplines that have different value propositions.
Best - CircleCI
CircleCI was built for speed and scalability. Build configurations are stored in your application repo as YAML, making it easier to track changes and contribute to your testing process. The Orb system allows you to extend pipelines using third-party functionality with minimal effort. These features create a standardizable CI solution that can easily be adopted. CircleCI definitely rates as one of the better CI tools currently available.
Check out our comparison of CircleCI vs Drone to see how we stack up.
Good - Travis CI
Travis CI was the first CI-as-a-service solution. It was designed for open-source projects: testing them is, and always will be, free - the creators of Travis CI offered this as a way to give back to the community. Additionally, the tool offers a streamlined experience that is easy to set up and use. Configurations are created using YAML. Travis CI lacks many of the enterprise-grade features of other CI tools (think: security and governance), but that, in turn, means it requires little effort to maintain.
Jenkins is old. It was created in a different technological decade and built to solve Continuous Integration. I want to give credit where credit is due. Jenkins out-competed Bamboo (https://www.atlassian.com/software/bamboo), TeamCity (by Jetbrains), and other CI tools of the early 2010s to become the industry standard DevOps solution. But industry standards have evolved. Any solution that requires dedicated hosting servers and 3-5 full-time engineers to manage it will have a hard time standing up to modern tools.
Each category of Continuous Delivery tools could fit a specific need. Open-source tools work well for smaller engineering teams that don’t need robust governance or a new tool bill. Platform solutions help transform the entire software delivery process and are great tools for enterprises concerned with security. Cloud deployment tools are a great option for companies hosted in one cloud. And the CI tools... well, the CI tools are great to build the artifacts deployed by your CD solution.
Complimentary Tools
Continuous Delivery tools work best when existing tools are plugged into new Continuous Delivery pipelines.
For instance:
By connecting your code repository (GitHub, Bitbucket, etc.) to your CD pipeline, you can automatically trigger your pipeline every time a new artifact is committed.
By integrating terraform or ansible into your pipeline, you can create an automated infrastructure-as-code process. This basically replaces the functionality of tools like Chef and Puppet.
By integrating APM and Log tools into CD pipelines, platforms like Harness can use machine learning to verify the success of your deployments.
The more of your existing toolset you can integrate into a CD solution, the more sophisticated your pipelines will be.
Define Your Needs
At the end of the day, your organization will have specific Continuous Delivery needs, and you’ll need to decide which Continuous Delivery tool best solves those problems. Some key questions you should ask before choosing a solution:
How much time do I have to get a delivery solution up and running?
How much effort can we put into maintaining a solution?
Is there a migration planned in the future?
Do we have the expertise to script things ourselves?
There are definitely tools I recommend avoiding for your own personal sanity, but as an employee of Harness, I have an implicit bias towards the Harness software delivery platform. I recommend taking a look at Harness even if it’s just to create a baseline you can compare to other tools. Try it free today.
Get Started
Get Started with Harness AI
Try the full platform free. No module restrictions, no credit card.
