6 Myths of Engineering Intelligence

Over the past few years, I've engaged in numerous conversations with Engineering Teams and Engineering leaders, leading me to identify several myths surrounding engineering intelligence. Let's debunk these myths one by one:

Myth 1: Engineering intelligence is solely about metrics.

While this misconception is prevalent, it doesn't encompass the entirety of engineering intelligence. Metrics do play a crucial role, but to drive engineering teams towards excellence, one must consider both metrics (measurement) and insights (improvement). To better understand this, consider the analogy of weight loss. Your weight measurement is the outcome metric, but simply measuring it won't help you shed pounds. You need insights: when you feel most hungry, if you're getting enough sleep or water, etc. Additionally, you need to measure input metrics like sleep quality, daily water intake, and calorie consumption per meal. By focusing on input metrics and observing trends, you can improve your outcome metric. Similarly, engineering intelligence requires both metrics and insights.

Myth 2: Outcome metrics are sufficient.

This myth raises a crucial point: there are outcome metrics and input metrics. To change the outcome metrics, you must focus on the input metrics. However, understanding which input metrics to prioritize requires insights. For example, you might have a slow build process or code reviews that act as bottlenecks. Your requirements may frequently change, or your codebase could be a complex monolith causing significant regressions. It could even be an issue related to engineer burnout or disengagement. To optimize and address these problems, you need insights such as trends in scope creep and subjective feedback from developers. Therefore, engineering excellence is achieved through a combination of measuring and gaining insights.

Myth 3: Measuring developers is the same as measuring developer productivity.

These two concepts are distinct and often confused. Measuring developers focuses on individual performance, while measuring developer productivity examines the work produced relative to invested resources. Efficiency measurement assesses how well resources are utilized to achieve output. To use an analogy, it's about having the right people on the bus, ensuring the bus functions correctly, everyone has the same destination, and determining how fast the bus can reach that location. An effective engineering insights tool should measure both individual developer performance and overall productivity and efficiency metrics.

Myth 4: Metrics can be easily manipulated or gamed.

To achieve engineering excellence, it's crucial to consider both insights and metrics. While it's true that metrics can be manipulated, most frameworks include counter metrics to mitigate this issue. For instance, if someone focuses solely on gaming velocity metrics through fake pull requests, it will inevitably impact quality metrics. Thus, a comprehensive approach involving multiple metrics and counter metrics helps maintain accuracy and discourage manipulation.

Myth 5: The data is not 100% accurate or hygienic

When it comes to data-driven KPIs and insights, the accuracy of the numbers is often seen as a challenge. However, it's important to recognize that even if the data is not 100% accurate or perfectly clean, the insights and KPIs still provide valuable directional guidance regarding which bottlenecks to focus on. Instead of striving for absolute accuracy, it is more beneficial to treat these data points as smoke signals and leverage the improvements in these trends.

While data accuracy and hygiene are important considerations, it's essential to understand that data-driven insights are not reliant on perfect data. Imperfect or slightly inaccurate data can still provide meaningful insights that help identify areas for improvement. By focusing on the trends and patterns in the data, organizations can make informed decisions and take necessary actions to address bottlenecks and enhance performance.

Rather than aiming for 100% accuracy, it's more practical to view these KPIs and insights as tools that provide a directional sense of the underlying issues. They act as indicators pointing towards potential problem areas or opportunities for optimization. By leveraging these trends and improvements, organizations can make meaningful progress towards their goals, even if the data isn't entirely flawless.

Myth 6: Surveys provide better insights than metrics.

While surveys are valuable for capturing subjective signals related to developer happiness and experience, they are not the sole solution for improvement. To drive engineering excellence, both insights and metrics are necessary. Surveys can have recency biases and may not capture signals that individual developers care about, such as optimal resource allocation or investment trends. To achieve a well-rounded understanding, insights should be derived from both surveys and data sources.

In conclusion, driving engineering excellence requires investment in an engineering intelligence tool that provides both insights and metrics. Such a tool should leverage surveys and data sources to surface valuable insights. Remember, measuring developers is distinct from measuring developer productivity and efficiency metrics—it's essential to focus on both. If you're interested in advancing your engineering excellence goals, I invite you to book a demo for Harness Software Engineering Insights today!