DevOps DORA Metrics – Everything You Need to Know

How quickly can your team deliver a new project?

Why can’t you take on new work?

How often are we deploying?

How quickly are we able to recover from production issues?

These are all questions that engineering leaders must discuss with management, as well as internally. Engineering organizations are increasingly expected to deliver software more frequently and with better quality. To meet this demand, software teams are embracing Agile development methodologies and DevOps to varying degrees. As they try to get more efficient in their agile practices, software delivery teams need software metrics to measure their development speed and stability. 

The complexity of distributed teams, outsourced projects and remote work makes it even more critical to have the right metrics defined to measure the overall DevOps performance.

Dora Metrics: The 4 Key Indicators of Elite DevOps Performance 

There are 4 key metrics that many organizations use to measure their delivery performance. These metrics were suggested by the DevOps Research & Assessment (DORA) research program, and hence are known as DORA metrics.  The 4 DORA Metrics are:

Software Delivery Lead Time

Software Delivery Lead Time (also known as Lead Time for Changes) is the amount of time it takes a commit to get into production - from the time the software development team starts working on a task to the time the customer gets the feature. This includes the time spent in build, test, integration, implementation and deployment, and does not include the time spent in design, prioritization or backlog

Deployment Frequency

Deployment Frequency is the number of times code or software is deployed to production or “shipped”. This metric helps organizations determine and set their delivery cadence.

Mean Time To Restore (MTTR) 

MTTR is the time it takes to restore a failure in production. A failure can be an unplanned outage or a service failure.

Change Failure Percentage

This is the percentage of deployment causing a failure in production. It is the measure of the number of times “a hotfix, a rollback, a fix-forward, or a patch” is required after a software deployment or a service change.

‍

How to Measure DevOps Success: Why DORA Metrics Are Important

The four DORA metrics provide a great baseline to measure the tempo, rhythm and responsiveness of an engineering organization.

A.  Development Velocity

The first two metrics are a measure of software delivery performance “tempo”, also known as Development Velocity. It is important for organizations to understand their development velocity. 

Lead Time

Lead Time helps organizations understand how quickly they can deliver software. It gives you a sense of the efficiency of the development teams. With shorter lead times, you can deploy to production in smaller deployments and more often. This enables faster feedback on what is getting built and allows for quicker course correction. Conversely, longer lead times signify bottlenecks in the development process.

Lead Time is a great metric to track, especially if looking at the trend over time. It shows whether there are any issues, and if things are getting better or worse. And at the end of the day, how quickly you are able to respond to the business. However, what is more important is to get further breakdown of the different stages. 

• Are there any bottlenecks? What stage are they in? 
• Are PRs taking long to Review? If so, why? 
• What is the Merge Time? 
  

These are all good questions to ask. With a combination of data from the Project Management, SCM and the CI-CD and Deployment systems, this breakdown can be achieved.

Deployment Frequency

Deployment frequency helps organizations set their release schedules. Many SaaS organizations chose to deploy builds frequently - some even on a daily basis. Smaller, more frequent builds help reduce risk. However, not every organization will want to or need to deploy very quickly or frequently. SaaS organizations may need to deploy very frequently. On the other hand, for certain business applications, deployment frequency of once or twice a year might be sufficient - their customers may not be happy with frequent changes. 

‍

‍

Software teams should evaluate the needs of their business and ensure that the velocity of their development process (the Lead Time and Deployment Frequency) matches their business need. The principles of Lean and Agile can still be applied by delivering software in small batches rather than delivering as large monoliths.

B.  Software Quality

Software development organizations often put a lot of emphasis on improving the development velocity. However, perhaps even more important is the quality of their output. 

Change Failure Percentage

Change Failure rate can give organizations a sense of how frequently they are shipping out code that causes issues. Ideally, the Change Failure Percentage should be as low as possible indicating good quality code.

Mean Time To Resolution (MTTR)

MTTR is the time it takes to restore a failure in production. A failure can be an unplanned outage or a service failure.

‍

The above 2 metrics measure the reliability and stability of the software that is delivered. Together, these are a good indicator of the quality of the development output.

As teams grow, it is critical to find a balance between how much and how often to deploy vs how stable the product is. A higher development velocity is important but should not come at the expense of quality or the stability of the delivered software.

What is a DORA Report?

Engineering leaders need to be able to evaluate the performance of their organizations on an ongoing basis. The DORA report is a great way to start getting some initial insight into the development velocity and software quality. With these metrics, you can start to see if there are any bottlenecks in the development process, and the quality of their output.

It is however important to not view the numbers as absolute. Being able to track DORA metrics on a regular basis helps you see the trends - this could be a better indicator of issues.

The DORA report measures five key metrics:

• Deployment frequency
• Lead time for changes
• Mean time to recovery
• Change failure rate

The DORA report finds that high-performing organizations are able to deploy software more frequently, with shorter lead times, and with lower change failure rates. High-performing organizations also have better operational performance, meaning that they are able to resolve incidents more quickly and with less impact on their customers.

‍

Getting Started with DORA Metrics

Accurately measuring DORA metrics can be a challenge for most organizations. Much of the data that is needed to calculate these DevOps metrics lies in various systems across the DevOps toolchain - project management, SCM, CI/CD, service desk, issue tracking, and other systems. The data must be parsed, broken into spreadsheets, and then correlated to get the right DORA metrics.

As an example, consider an organization that uses Jira for their planning, GitHub for SCM, Harness for CI/CD, ServiceNow for service desk, PagerDuty for production monitoring, and various other tools for testing, security, etc. This is a very common scenario in many organizations where they select different tools that meet their needs for different purposes.

To get a metric such as Lead Time, you need to correlate the data from Jira with data from GitHub and Harness, so you can accurately understand how much time was required from the start of the task till it made it to Production.

Similarly, for accurate MTTR, you need to correlate data from PagerDuty back to ServiceNow and then to Jira. This can be quite a challenge.

Beyond DORA: Other Agile DevOps Metrics 

In addition to the 4 DORA metrics, there are several other metrics that can help engineering teams determine their efficiency, productivity and their bottlenecks. Some of these are:

PM Hygiene

• Requirements Clarity
• Sprint Distribution
• Details of User Stories 
• Prioritization
• Acceptance Criteria & Test Cases
  

Sprint Hygiene

• Story Point Calculation
• Estimation Accuracy
• Story Point Mix
• Dependencies & Interactions
• Distribution of workload between teams
  

PR Lifecycle

• Time to First Commit,
• Time to Review,
• Time to merge,
• Number of reviewers
• Items stuck in various stages
  

CI-CD Effectiveness

• Number of failed builds
• Number of failed deployments
• Number of rollbacks
• Number of hotfixes
  

Incident Management

• Number of Hops
• Bounce backs
• Time spent on resolution
  

Code Hotspots, Technical Debt, Security Considerations and a lot more.

Summary

Metrics are the foundation to understanding the efficiency and effectiveness of your engineering organization. With the right software metrics, you can make data-driven decisions and demonstrate alignment with the business toward customer-centric outcomes.

DORA metrics provide a good foundation to start measuring development velocity and software quality. Tracking DORA metrics regularly helps you see trends and point out problem areas. However, DORA metrics can be hard to obtain since data resides in different tools deployed across the DevOps toolchain. You need to correlate data from various sources such as GitHub, Jira, PagerDuty, etc, which can be difficult, time-consuming and frustrating.

Accelerating DevOps with DORA Metrics and Harness 

The Harness Software Engineering Insights module can help organizations solve this challenge. Software Engineering Insights enables engineering leaders to proactively manage and demonstrate the performance of the engineering organization to business stakeholders with established success metrics, such as DORA and the SPACE Framework. With over 40 out-of-box integrated data sources (including GitHub, GitLab, BitBucket, Jira, PagerDuty, and Azure), you can find and remove software delivery bottlenecks while gaining a better understanding of engineering operations. 

Interested in learning more about how Harness Software Engineering Insights can help improve your engineering outputs? Request a demo. 

‍