Top 3 Challenges in Measuring DORA Metrics in Azure DevOps

Azure DevOps platform is a leading DevOps platform that provides integrated experiences for developers from planning and coding to CI/CD. Microsoft has invested in deep integrations not only between these products but also with Azure Cloud, with many features such as one-click deployment from DevOps Server onto the Azure Cloud.

However, DevOps analytics is an aspect that is quite challenging for enterprises to implement with Azure DevOps products. Extracting even the basic North Star metrics such as DORA metrics can require considerable time, effort, and cost. 

DevOps Research and Assessment (DORA) Metrics provide a way to quantify DevOps effectiveness. The 4 key metrics are Deployment Frequency, Lead Time for Changes, Mean Time to Recover and Change Failure Rate. Both Azure DevOps Server 2019 and Azure DevOps Server 2020 have the capability to provide some of the raw data needed to calculate DORA metrics.

However, converting raw data to an actual metric is challenging due to several reasons. Some of these are the reliance on Power BI as a platform for data collection and analysis, setting up complex queries to slice the data, limitations on the types of data that can be easily imported from various Azure DevOps products into Power BI, and the time and effort needed to maintain the analytic framework. Frankly, to do this right it is almost like building and maintaining a new product. This blog covers the limitations and pitfalls of computing DORA metrics with Azure DevOps and in general creating a robust DevOps Analytics capability within the Azure DevOps suite.

Baseline solution components and capabilities needed to build an analytics solution with Azure DevOps are: 

• Azure DevOps Server - the data source for analysis
• Analytics Service - For historical data that you can query
• Analytics Views - A simplified way to filter data from Azure DevOps into Power BI. Helps avoid complex querying but limits data types and data detail.
• Power BI - ability to slice and dice the data and present in usable reports and dashboards
• PowerShell scripts to extract data through the Azure DevOps REST APIs for data not yet available through the Analytics Service
• Knowledge of Azure DevOps APIs to extract and load into Power BI the data that is not obtained via Analytics Views
• Deep knowledge of SQL to build the right OData queries.

Extracting DORA metrics from Azure DevOps involves more than just exercising few APIs, executing SQL queries and charts and graphs in Power BI.

DORA Metrics are most effective when measured against the process used by the team. When teams follow slightly different process gates and stages, the complexity in obtaining DORA metrics increases by an order of magnitude. Moreover the unique structure of Azure DevOps with concepts such as Teams, Areas and the like, imposes many restrictions in how data can be viewed, aggregated and shared. These and several other reasons make it almost impractical to build your own solution to extract metrics that can drive meaningful change in the organization.

Challenge 1: Functionality Limitations, Integration and Maintenance Costs

Analytics Views provide a simplified way to specify the filter criteria to pull data from Azure DevOps Server to a Power BI report. Thus it simplifies the task of creating views in PowerBI. However, there are four severe limitations to this functionality. 

First, Analytics Views only support Azure Boards and its objects. As a result, the views are limited to Backlog Items, Bugs, Tasks and WorkItems. Second, Analytics Views only supports views by month, half-yearly, 30 days, and today (Reference). Third, each view supports a flat list of items - i.e. it does not support a hierarchy of work items. And fourth, Analytics Views does not support reporting on plain text or HTML fields. As a result, it is not possible to extract and analyze descriptions, comments, and other such data.

Thus the basic Analytics Views are very limited even for the supported data types in Azure Boards. For any other slices of Azure Boards data, one has to create custom views.

These limitations imply that one cannot easily extract data from Azure Boards by sprint, view rolled-up metrics by team, and other such critical perspectives needed by the leadership to effectively manage their teams. 

A primary limitation to easily extracting DORA metrics such as Lead Time to Change and Deployment frequency is that Analytics Views does not support data types such as Azure Pipelines and Tests. Hence, one must rely solely on the fidelity/hygiene of data within Azure Boards. This can be error-prone and lead to highly inaccurate or incomplete metrics. Microsoft explicitly states that it has no plans to support these data types that help calculate accurate DORA metrics. 

Absent an approach with Analytics Views enterprises are left with using Analytics with Custom Queries to extract raw data directly into Power BI and creating dashboards and reports in it. To report and slice data for these data types, one has to create Custom Queries in Power BI. These custom queries are quite complicated and error-prone. Here’s an example of a custom query to extract data about pipeline runs.

 Source

Each of the query strings has to be replaced/parametrized with the appropriate entity name. For example, the pipelineName must be substituted with the Name of the Pipeline, and so on. This is highly error-prone and does not scale for organizations running many different pipelines across teams and projects. Maintaining and managing these queries is a monumental task, especially given the constant change in enterprises.

The key challenges with this approach are:

• Not all data types are supported across Azure DevOps Server versions. Azure DevOps 2019 and earlier versions do not support extracting Azure Pipelines Data via Analytics.
• Scalability of queries - Analytics Service is optimized for reading large data volumes. However it is not optimized for writing server side queries to filter data. This has deep performance implications.
• Learning curve on executing these OData queries and maintaining those queries over time.

The only choice these limitations leave to enterprises is to build custom rest api queries and do a lot of the slicing and dicing on their own data lake. This is essentially a parallel development effort that could take up as much resources as a core product module. The question that enterprises must ask themselves - is this a smart choice or should they look for a commercial out-of-the-box solution to extract DORA metrics from Azure DevOps Server?

Challenge 2: Can't Easily Slice Data by Teams to Generate Reports

It is common practice to view DORA Metrics by team, organization, project, product or other such slices. These help leaders identify which areas need attention and which best practices to adopt from one team to another. Azure DevOps recommends the use of Area Paths to group work items by team, product, and feature area. Whereas this approach works for more static definitions like Product or Project, which do not change often. The notion of teams is no longer defined in hierarchical or organizational terms alone and is seldom static. In an Agile environment it is common to find people contributing to projects that do not follow the traditional definition of team. Several users have expressed frustration with this architectural choice by Microsoft. 

The ability to have a fluid definition of teams using multiple criteria such as organization, project, issue tag, and such is critical because ultimately metrics have to be measured at the level where change can happen to improve it. That is at the team level. 

It is possible to do this with Azure DevOps and Power BI. However, to overcome this limitation and view the data across different teams, users need to incorporate a Team Slicer, which is another complex set of filters applied to the queries. The figure below shows the series of 12 to 15 steps one must execute to apply a team-based filter to baseline Azure Boards Data.

Security and RBAC are important considerations. To ensure a fully functional reporting system that adheres to the enterprise standards one essentially needs to build another RBAC type layer on Power BI to control visibility and access as per the company policy. Here are two key limitations why this functionality must essentially be built on Power BI.

First, a prerequisite to obtain view access to data is that the user must be a member of the project. As a result, for management and other matrix functions to view reports in Power BI, they must be at least a basic member of the project. This limitation results in a proliferation in Project Membership just to ensure that reports can be consumed by these personas. This presents significant security or access control issues for the enterprises.

Second, anyone with access to the project can see the analytics views. This presents a security challenge since enterprises may need to restrict access to certain reports even though the user is a member of the project. 

Maintaining History of Data - Bringing in data from Azure DevOps isn’t enough for maintain and report on historical data or view data via timeline views. To create timeline views users must build custom views. Creating Custom Views requires additional steps such as query formulation, filter criteria and the like.

Scalability - Assuming enterprises are able to built these custom views, scaling is a challenge. Here’s a note from Microsoft’s site on reports and data set sizes - “Scope your data to generate reports that quickly refresh. For large datasets, refreshing non-filtered views can fail. To narrow your results, add the right set of filters. The more filters you add, the faster the view loads and refreshes in Power BI.”

The ability to account for these factors, design, build and maintain the integration between Azure DevOps and Power BI is a lot to manage and execute for busy development organizations wanting just to extract DORA metrics. 

It is essentially a parallel development project involving multiple FTEs.

Challenge 3: Time and effort to pull data from heterogeneous tools

Heterogeneity is a reality in most enterprises, including those that have fully adopted Azure DevOps as their platform. Several factors result in heterogeneity in the development toolchain.

Acquisition of a new team or company brings in new toolsets. This is a very common cause of heterogeneity, which makes it difficult for the acquirer to get a better understanding of the development metrics. Likewise, outsourced teams often use different tooling. Bringing in this data to understand the output of outsourced teams creates new sources of discontinuity in the process of measuring DORA metrics and other engineering metrics. And, third party tools such as Jenkins for CI-CD also introduce heterogeneity in the toolchain. Many organizations that adopt Azure DevOps use Jenkins for just their CI-CD needs. Likewise, though not directly related to DORA metrics, third-party Security Scanning tools, Feature Flag products like Launch Darkly and Artifact Repositories like JFrog Artifactory are also commonly seen in organizations that use Azure DevOps for a majority of their DevOps needs.

Heterogeneity adds another layer of complexity as data from other tools now have to be fed into Power BI and analyzed. This means more API calls, data modeling, query building, and other steps for each such tool. 

In conclusion, whereas Azure DevOps as a platform is an excellent choice for development teams, extracting DORA Metrics and other in-depth development metrics from Azure DevOps is a major undertaking. If you want to go beyond some vanity metrics such as commits and issues closed to truly measuring DORA Metrics, be prepared to invest significant time and resources to build and maintain a DevOps Analytics solution in-house with the Azure DevOps toolset.

Interested in learning more about how Harness Software Engineering Insights can help improve your engineering outputs? Request a demo.