The Engineering Knowledge Graph: The Missing Link to Unlocking Engineering Productivity
Increasing engineering productivity is a high-priority, challenging goal for most engineering organizations. Technology leadership teams seek various ways of removing bottlenecks to improve efficiency. Harness has been leading the charge here, helping organizations of all shapes and sizes better understand their bottlenecks across their entire SDLC DevOps lifecycle. This experience gives the Harness team a unique vantage point into which patterns are essential to unleash engineering productivity.
This blog will share a fundamental pattern - the Engineering Knowledge Graph (EKG).
First, let's talk about why you need EKG. As the organization and product grow, understanding the tools and assets owned by different teams becomes very challenging. Furthermore, it isn't easy to know how the teams are organized from a development/project perspective and not purely an HR perspective.
For example, it is difficult to understand which projects, labels, and components in JIRA, which repositories in Git, which jobs in Jenkins, which alerts in PagerDuty, and which environments and services in AWS are owned by the frontend team. Sometimes people assigned to projects and groups are also difficult to ascertain and forever changing.
Without having a documented EKG, it is challenging for organizations to
- Make data-oriented decisions
- Automate processes
This often results in sub-par engineering execution, where leaders make gut-based decisions. However, as the team, product, and organization grow, gut-based decisions and manual processes often reach a breaking point.
Challenges Caused by Lack of a Mature EKG
Challenging or nonexistent metric collection and reporting - Chances are, you've been a part of a team that struggles with data collection and reporting. Companies that find roll-ups and comparison challenging are most likely struggling with comprehensive metrics collection. Without the correct data, most decisions are gut-based. It is challenging to understand high-level trends for engineering leaders.
Identifying the right owners - When a team or team member cannot locate the right resource for an issue, this is an immediate sign that the organization doesn't have a mature EKG.
- As an example, your security team identifies a failure and doesn't know whom to contact for that service
- A CI job fails, and it's hard to find the right owner
Challenges with actionability - Related to ownership, a non-mature EKG will result in organizations often struggling with end-to-end automation of DevOps processes. Take, for example, a failing CI job. If this failure is not routed to the appropriate party for remediation, this is a failure in action. Another example of challenges with actionability would be a security alert that does not immediately route to the appropriate developer. For any issue, an automated job must notify the relevant party.
Building a mature and up-to-date EKG can help an organization maximize engineering productivity.
Why is Building and Maintaining an EKG System Challenging?
With the explosion of DevOps tools and the centralization of a few, the organization's intelligence map is sprawled across multiple SDLC/DevOps tools. The segmentation model differs from application to application. For example, tickets in JIRA might use labels and components to differentiate from each other. In GitHub, it could be repo names or it could be Tags. All this information is not something that one can query an Active Directory system or HRIS system. Correlating this segmentation map of people and assets from different DevOps tools and maintaining it becomes a fundamental part of building a mature EKG system.
How Can Harness Software Engineering Insights Help Create an EKG?
Harness Software Engineering Insights helps an organization create and maintain its EKG in one place. Once Harness is integrated with an organization's dev tools like JIRA, Git, ADO and others, Harness will automatically start co-relating information to build an EKG and maintain it.
How Can an EKG Help?
Building the right metrics and insights across projects and teams - A mature EKG allows one to understand how each project and team is trending, understand bottlenecks, and compare and contrast different projects and pivot points. It also gives an excellent understanding of inter-team dependencies and communication bottlenecks to the engineering leader.
Automating end-to-end workflows within engineering - Because of the EKG, one can customize the automation workflows for each entity or team and realize end-user actionability by automating the last mile.
Automating workflows between engineering and other stakeholder terms - The EKG can help external teams identify the right owners and different segment rules for various subsystems and automate workflows. For example, the SLO definition for team A could differ from the SLO definition for team B. However, since the EKG would capture what constitutes team A or B, it would be easy to identify SLO failures and route them to the right team member.
How Harness Can Help
Building and maintaining an EKG is a prerequisite to unleashing productivity in today's landscape. Although most organizations start documenting EKG across various Wiki pages, this becomes limiting as the data becomes stale quickly and cannot be leveraged for insights or automation.
If you want to learn more about how to automatically create and maintain an EKG system for your organization, with an eye toward increasing engineering productivity, request a personalized demo.