You Need More Metrics.

or, how to create a shared reality.

👋🏽 Abhi here, back again for another dispatch from the front.

Last time, on Roundup…

In our last chat together, we took a step back to frame up three ideas:

1. The value of data is mostly in answering and organizing around four fundamental questions:

   1. What is happening in our business?

   2. Why is it happening?

   3. What’s going to happen next?

   4. What should we do about it?

2. By extension:

   The mandate [of a data team] is to…focus their work on [building the] company’s operating system…make these questions easy enough to answer that companies ask them more often; and the answers clear enough that companies act in response.

3. The best way for “data teams” to do this is to focus their energy on making sure two of the most important components of a company’s OS (The Big Two) land and land well.

   Business Reviews are how a company monitors, introspects and recalibrates. Planning Cycles are how a company decides how to invest its resources in and for the future. Taken together, these two processes shape most of how the company works and what it produces. As such, they are incredibly high-leverage.

   

Easier said than done!

Business Reviews

Over the past year, I’ve been working in the community with many of you to help stand up these mechanisms and get them thriving. And after helping nearly 100 teams go through this process now, I’ve seen some common patterns of both failure and success.

I want to use our space today to share some quick points of emphasis around how to get one of these mechanisms right: the Business Review.

Much has already been written about starting, structuring, and running Business Reviews (hereafter, “MBR”), so I won’t reinvent the wheel here. Instead, I’ll point you to Cedric Chin’s (a must-follow for data folks) excellent deep-dive on how Amazon runs their MBRs. Amazon, one of the most data-driven companies in the world, is one of the great success stories for how being data-driven impacts enterprise value through operational excellence.

Amazon didn’t realize the promise of a data-driven culture through modern data tooling (their internal data tooling is as inspiring as their externalized data tooling) — the tooling that powers MBRs are often low-tech, and the final presentation layer is famously often analog PDFs! It did so by focusing on process: by running great MBRs, and integrating these deeply into the operating system of how the company runs.

For companies that aren’t quite ready or willing to go Full Amazon, there are also plenty of gentler on-ramps to a Business Review culture.

For whatever flavor of MBRs you choose to implement or support, I offer one point of emphasis: A company must have enough of the right metrics for a Business Review to be successful.

This is the most common reason I’ve seen MBR programs fail to reach their potential.

MBRs at Amazon are known to contain 100s of metrics.

When I share this with people, most wince. To them, this feels like an anti-pattern. “We have too many dashboards already,” complain analysts, “stakeholders can’t even use the data they have”. Meanwhile, operators worry that too many metrics will force them to spread their focus too thin; that metric-ocracy will flatten innovation and risk-taking.

And yet, this is a feature, not a bug, to MBRs at Amazon, and an essential part of why they work there.

One of the primary outcomes we want from MBRs is to create a shared reality around the physics of the business. We want leaders to see reality the same way so they can reason collaboratively about the past, present, and future of the business.

It turns out this is very difficult to do with a sparse set of metrics.

Reality is a set of facts; for businesses, the most important facts are metrics.

Metrics are how we express how a company turns its inputs into outputs — how the company creates, captures, transforms, spends, and distributes value. Context on how that happens starts out tacit and diffuse, and when we design metrics we’re taking that tacit, diffuse context and encoding it into legible, explicit knowledge.

Companies with few metrics to work with are poor on facts. Their view of reality is murky and incomplete and, because context remains tacit and diffuse, inconsistent.

These companies have more uncertainty than understanding; more questions than answers. They tend to have slower feedback loops as questions have to be answered asynchronously and “offline”. They tend to harbor more finger-pointing (“Marketing brings terrible leads!”; “Sales doesn’t know how to close!”). Opinions abhor a vacuum, and HiPPOs rush to fill the void of metric facts (“If we have data, let's look at the data. If all we have are opinions, let's go with mine.”).

Meanwhile, companies with richer sets of metrics have a picture of reality that is in far higher resolution. That makes for a better foundation for building alignment, understanding, and continuous learning. There is little private knowledge and operators can speak the same language and generate both questions and first-pass answers live and in-context, together.

At these companies, more metrics doesn’t have to mean less focus — if done right, it can mean more. Operators can still stick to a bounded, manageable set of goals and targets and [what Amazon calls] “Output" Metrics. But a wider range of “Controllable Input” Metrics can help narrow the aperture on how and where to innovate and experiment. And over time, operators develop a “fingertip feel” for the full constellation of metrics, their patterns, and the relationships between them — an intuition that can be as fast and reliable as more robust analyses.

At these companies, more metrics doesn’t have to mean more dashboard sprawl — if done right, it can mean less. Dashboard sprawl is a result of metrics done badly, not a necessary consequence of having more of them. Dashboard sprawl happens when leaders don’t understand the mechanics of their business and when they haven’t established data feedback loops that govern metrics through their lifecycle. More, well-designed metrics help address the former; well-run MBRs help address the latter. There’s relatively little dashboard sprawl at Amazon — the primary interface is the MBR.

So, must you have 100s of metrics before starting an MBR program? Well, no…

Most companies aren’t sufficiently complex (or sufficiently blessed with data volume) that they need 100s of metrics to express the reality of their business. Most companies don’t have analysts with the skill, experience, and context to design 100s of metrics that generate more light than heat. And most company operators and leaders don’t have the analytical maturity to make sense of that many metrics. (Though, importantly, with respect to the latter conditions, I’d argue that generating and consuming lots of metrics is the best way to build the requisite maturity!)

But it’s crucial not to start too small either. I hear weekly from companies and data teams that want to build their Business Review program incrementally. These companies reason that they can start with ~10 metrics and gradually build muscle around this bone. I wouldn’t recommend it.

Overwhelming the inertia of an organization enough to create effective data-driven feedback loops requires considerable activation energy. Anemic MBRs fall short of the energy required. By being too minimal, they fail to be viable; they fail to generate the aha moments around understanding and alignment needed to keep these programs going.

Instead, the data teams that I’ve seen be most successful with these programs start by taking a step back. They take pains to help business leaders codify the tenets, principles, and purposes for their Business Review program. They help leaders model their understanding and assumptions around the business and its drivers and translate these into metric definitions that are clear and consistent. They help these leaders understand that metrics are the lifeblood of a data-driven company. They understand this themselves. They treat the production of high-quality, high-signal, reliable metrics (not dashboards, not data models, not analysis write-ups) as their most important deliverable and work backwards from metrics to structure everything they do.

Easier said than done! (but more on that next time)

From around the web

When Action Beats Prediction

While Cedric and I are both obviously big fans of tightly-regulated, metric-driven control systems, Cedric provides this counterpoint to the perspective above — when/why should we stop thinking and start acting? The truth, as always, is somewhere in between.

Unifying Large Language Models and Knowledge Graphs: A Roadmap

A title meant to bait me if there ever were one. I don’t know if knowledge graphs will become more prominent in the coming years, but I know that they should. I have no doubt that KGs can enhance “AI” reasoning. But I also think KGs are the most powerful way to manage/model an enterprise’s data (and that the other ways we model and represent data can mostly be treated as projections from a KG). I fantasize about a future where more “analytics engineers” and “data engineers” become ontology engineers.

Metis: Building Airbnb’s Next Generation Data Management Platform

Self-recommending, as with all posts out of Airbnb’s Data Platform group. Metis is their approach to unified metadata ingestion, management, and routing. I’m looking forward to a resurgence of posts like this like we haven’t seen in 5+ years. The tech world swings across cycles of “build” vs “buy”, and in 2023, the pendulum is starting to swing back to “build” (which will inevitably fuel the next generation of startups to “buy” in 3-5 years).

Data Falsificada (Part 1): "Clusterfake"

A fun deep-dive into the forensics around doctored data, and a new reason new reason not to use Excel.

EU lawmakers pass landmark artificial intelligence regulation

🍿

A Personal Retrospective on Prophet

I’m glad to see Sean finally taking accountability for his deleterious impact on global GDP.