Measuring Product Management Success: An Engineering Leader's View
Are we on track with the release? If I had a dollar every time an engineering leader is asked this question….. Predictability is one of the most important facets of engineering leadership. However it's also the most difficult one to master.
Being predictable is deeply rooted in how well the sprints are planned. For most engineering teams, achieving predictability is a tough balance between doing the right things versus doing things right. Why? Because most engineering teams ultimately answer to the business and business can sometimes be unpredictable.
Here are two common examples for unplanned scope creep.
- A lighthouse customer requires a bug to be fixed
- A feature must be released within a week to win a multi-dollar deal.
There are many such examples. To address this challenge it helps to categorize and I like to think of scope creep in two categories:
- Bug creep
- Feature creep.
Generally bug creep is a lagging indicator of the quality of previous releases. However today we will focus on feature creep , which is a big contributor to the overall scope creep.
Consistent feature creep is a lagging indicator generally of poor product management or customer/sales management practices.We have seen tons of burnout in teams where the product managers kept changing priorities based on their last customer conversation or the stories weren’t groomed correctly or the requirements were not very clear or where the customer was promised a new feature to be delivered in a couple of days. Extending the concept of wartime and peacetime CEOs into product management, these feature creeps are probably important if you are in a wartime product phase. However the keyword here is “consistent.”
Unlike sales, measuring success in product management (PM) can take time. In spite of engagement and adoption metrics, It can take quarters and sometimes years to truly judge some of the decisions and judgement of product managers. However one area one could measure product managers, is the quality of their product management output - the stream of work requirements. Whereas measuring PM Work Quality may seem difficult to measure, in my experience there are 5 simple proxies that when put together provide a composite picture of PM Quality.
1. Do requirements change too often?
Some change in requirements is par for the course because being receptive to feedback and responding is important. However, a consistent volume of requirement changes is a sign that it was not well thought out. Here too the metrics have to be measured in a nuanced manner. Measuring total change is a general sign but does not help fix the problem. The requirements changes have to be categorized by product area and feature to be able to identify the root causes and address. Furthermore, teams must also measure requirement changes that result in rework since these are the most expensive. Such changes must be minimized or eliminated.
2. Are the requirements clean and have acceptance criteria?
Acceptance criteria is an often overlooked aspect of PMs output. Often a user story provides a basis for feature requirements but the user story sometimes leaves a lot of room for interpretation. The purpose of Acceptance Criteria removes the ambiguity in expected outcome and provides Developers and QA a clear guideline on when to consider a story to be done. Acceptance Criteria creates a backstop for teams against scope creep. Acceptance criteria must be clear, concise, testable and must be written from a user perspective.
3. Are PMs prioritizing technical debt?
PMs like defining and prioritizing new features because that’s an exciting part of their job and it also gets the most attention from sales, customers, peers and other teams. This often causes technical debt to be neglected. Technical debt typically comes in two flavors - an immediate customer facing issue and strategic technical debt. The latter is very hard for engineering leaders to negotiate into the sprints and releases because it is hidden and doesn’t surface until it becomes a major issue. Most of the time PMs choose to have blinders on to the strategic technical debt and kick it down the road in lieu of more fancy features, until it blows up one day. Measuring the technical debt that’s worked into releases is another sign of healthy Product Management hygiene.
4. Are platform features or security fixes being prioritized?
Security is a complex topic even for PMs within the Security industry. Hence most PMs tend to ignore security in their plans. Likewise, platform features are hidden and highly technical in nature. These too do not get enough attention from PMs. For example, implementing SSO is pushed out because it wasn’t perceived as helping sell the story better. However, security is important and PMs must think of security first. This security first thinking should be driven by PMs and not necessarily by the Security team because PMs have the capacity to schedule work into a sprint or release. Measuring the investment in Product:Security:Platform is essential to keep a healthy balance. Typical ratio is 90% product and 10% everything else. A healthy ratio would be 60% Product, 20% Security and 20% Platform investment. The exact ratios may vary from one company to another but the essential point here is to measure and maintain a healthy balance. Here Engineering leaders too have an obligation to articulate for the right investments and to partner with PMs since they understand the technical nuances and impacts much better than the PMs.
5. Balancing Relative Investments in Product Features, Security and Platform Capability
It is essential to keep a healthy balance. Typical ratio is 90% product and 10% everything else. A healthy ratio would be 60% Product, 20% Security and 20% Platform investment. The exact ratios may vary from one company to another but the essential point here is to measure and maintain a healthy balance. Here Engineering leaders too have an obligation to articulate for the right investments and to partner with PMs since they understand the technical nuances and impacts much better than the PMs.
These 5 measures of Product Manager’s work help you quantify the quality of a Product Manager’s output instead of relying on intuition and gut to judge PMs output.