The Measurement Trap

The Hidden Costs of Over Reliance on Metrics

The accompanying podcast was created with NotebookLM

The adage "If you can’t measure it, you can’t manage it" is often misattributed to Peter Drucker, yet even so, this oversimplified view of management does not capture the depth of Drucker’s philosophy. In fact, according to the Drucker Institute, Drucker never said this; instead, the phrase is more accurately associated with W. Edwards Deming, a pivotal figure in the quality movement. However, even Deming's full statement is often overlooked: “It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.” Both Drucker and Deming recognized that while measurement is important, an overemphasis on what can be quantified can lead to significant blind spots in management.

Drucker’s actual view on measurement was more cautionary: “What gets measured gets managed—even when it's pointless to measure and manage it, and even if it harms the purpose of the organization to do so.” This perspective highlights the potential dangers of focusing solely on measurable outcomes while neglecting the broader, often intangible aspects that are crucial to an organization’s success.

The recent struggles at Nike provide a compelling case study of the pitfalls of such an approach. In the article "Nike: An Epic Saga of Value Destruction," Massimo Giunco outlines how Nike, under the leadership of CEO John Donahoe and President of Consumer, Product, and Brand Heidi O’Neill, made a series of strategic decisions that prioritized measurable outcomes over the more nuanced, intangible elements that had long defined the company’s success.

Starting in January 2020, Donahoe’s vision for Nike included eliminating product categories, shifting the company to a Direct-to-Consumer (DTC) model, and making the marketing strategy heavily data-driven and digitally focused. Initially, these changes seemed promising, especially during the pandemic when Nike’s DTC business thrived. However, as the dust settled and normalcy returned, the cracks in this strategy became apparent.

The elimination of categories, a decision intended to streamline operations and reduce costs, ultimately led to the loss of valuable expertise and innovation in product creation. The shift away from wholesale to a DTC-led model not only alienated long-standing retail partners but also failed to account for the complexities of consumer behavior, leading to a breakdown in inventory management and a loss of market share to competitors. Furthermore, the overemphasis on digital marketing and performance metrics, while easily measurable, eroded brand equity and led to diminishing returns.

Giunco’s analysis underscores how Nike’s leadership, in their pursuit of measurable success, overlooked the critical importance of brand loyalty, product innovation, and market dynamics—elements that are not easily quantified but are essential to long-term success. The company’s recent financial struggles, including a $25 billion loss in market cap in a single day, are a stark reminder of the dangers of an overly data-driven approach.

While this cautionary tale of Nike is unraveling in real time before our eyes, this overemphasis on the measurable has been happening for decades. In 1956, V. F. Ridgway published an influential article titled "Dysfunctional Consequences of Performance Measurements" in the journal Administrative Science Quarterly. Ridgway argued:

Quantitative measures of performance are tools, and are undoubtedly useful. But research indicates that indiscriminate use and undue confidence and reliance in them results from insufficient knowledge of the full effects and consequences. Judicious use of a tool requires awareness of possible side effects and reactions.

Ridgway’s cautionary words are particularly relevant in the context of Nike’s current challenges. While data and measurement are invaluable tools in management, they must be balanced with a deep understanding of the intangible elements that drive true value in an organization.

Peter Drucker, who placed human relationships and community at the core of effective management, would likely have advised Nike’s leadership to look beyond the numbers. As he once told a client, “It is the relationship with people, the development of mutual confidence, the identification of people, the creation of a community. This is something only you can do… It cannot be measured or easily defined.” Deming also echoed this sentiment, observing that “the most important figures that one needs for management are unknown or unknowable.”

Nike’s story serves as a powerful example of how an overreliance on measurable outcomes can lead to the erosion of the very elements that once made a company great. While the road to recovery for Nike may be long and costly, the lesson for leaders is clear: not everything that matters can be measured, and not everything that can be measured matters.

While I wouldn’t advocate for eliminating numeric goals in management altogether, as they can certainly be useful, it's crucial to remain mindful of both the purpose measurements serve and, more importantly, what they fail to reveal. For instance, consider KLOC (thousands of lines of code), a metric that was once used to gauge productivity in software development. It didn’t take long for people to realize that KLOC doesn’t capture the full complexity of the work and can be easily manipulated to give a misleading sense of productivity.

I’m often asked, particularly by those outside the tech world, how we measure productivity, quality, and efficiency. My short answer is that we rely on our Engineering managers to assess these factors, taking into account a broad range of elements. The longer answer involves our belief in empowering teams to focus on business outcomes that drive customer success. Although these goals are often quantified, they don’t tell the whole story. When teams struggle to move key performance indicators (KPIs), it’s important to also consider what they’re learning and how quickly they’re doing so. Understanding what doesn’t work can be just as valuable as understanding what does.

There’s also a tendency to emphasize short-term metrics, such as conversion rates, while overlooking more difficult-to-measure but likely more significant long-term metrics, like customer lifetime value. One metric I’m particularly focused on is experiment velocity. This metric provides insight into how many experiments we’re running, their outcomes, and the size of the wins. However, it doesn’t tell us how much we’re learning. In fact, if we can learn without the cost of running an experiment, that’s even better, but the metric itself doesn’t capture that.

In the end, while measurements are undeniably an essential aspect of the tech world, they are only one piece of the puzzle. As effective managers and leaders, we must recognize that the numbers we track can significantly influence the behavior of our teams, often in ways that aren’t immediately apparent. For instance, an overemphasis on hitting certain metrics can lead to unintended consequences, such as encouraging short-term thinking or driving teams to focus on achieving the metric itself rather than the underlying business objective. This can result in a narrow focus that overlooks the broader context or more strategic, long-term goals that are harder to quantify.

Furthermore, metrics can often fail to capture the intangible elements that are crucial to success, such as team morale, creativity, and the quality of decision-making. These are aspects that don’t easily translate into numbers but have a profound impact on an organization’s ability to innovate and adapt. When leaders rely too heavily on what can be measured, they risk neglecting these critical factors, potentially stifling the very qualities that drive sustainable success.

It’s also important to consider that metrics can create a false sense of security. Just because something is being measured doesn’t mean it’s being managed effectively. A focus on the numbers can sometimes mask underlying issues that the data doesn’t reveal, such as cultural problems, communication breakdowns, or a lack of alignment with the company’s core mission and values. These are the areas where true leadership is required, where the ability to see beyond the numbers and understand the deeper dynamics at play becomes essential.

Ultimately, while metrics are a valuable tool, they should not be the sole driver of decision-making. Great managers and leaders need to balance the quantitative with the qualitative, understanding that the most important insights often lie in what the numbers don’t show. By being aware of the limitations of metrics and considering their broader impact, leaders can foster a more holistic approach to management, one that values learning, innovation, and the human elements that are critical to long-term success.