Discover more from The Analytics Engineering Roundup
The perfect north star metric
And why it should be driven by value to the customer and not value to the business.
Benn stirs the data Twitter pot this week with Startups shouldn’t care about revenue. His argument: revenue is a poor mechanism for rallying a company around a singular objective because it doesn't offer timely feedback and is too far removed from the day to day of most people's inputs.
Here's why I agree with him:
Revenue is a lagging indicator of success. This isn't just a reflection of my values orientation (see profits are exhaust), it's an important causal gotcha I've learned from Nick E. our Director of Revenue. Healthy revenue growth reflects investments made months or quarters ago in other areas of the business. You can impact revenue on a shorter time scale — raise prices, lower operational costs, hire more sales folks — but if you don't also make investments towards customer value you'll find the impact of these to be short lived. In other words, if you focus your organization solely on revenue, you’re not only incentivizing the wrong things, you’re doing so on the wrong timeline.
What should this look like instead?
Benn references an incredibly thorough article from last year on the variety of classic metrics startups use for their north stars and what business types they are appropriate for.
Heuristics only get you so far though! One of my favorite examples where this traditional wisdom broke down is Twitter and MAU. MAU is a classic social network north star metric: the more monthly active users, the more eyeballs and the more ads one can sell. The problem is that this metric is entirely disconnected from what is valuable to the customer about the platform. It's hard to know what to optimize for, and how successful your efforts will be. Twitter has paid the price for this several times with wall street by poorly estimating their user growth and as of 2019 they no longer report this metric at all.
I'd like to take a different approach. What if we designed north star metrics that captured value to the customer rather than value to our organizations?
This orientation would force us to do things differently, and actually understand what value the customer derives from what you're building/providing first. This means doing thorough user research and listening with an open mind about what your customer is telling you — what do they rave about? What do they miss most when your service goes down? What is their biggest friction point? Are you measuring all of these things today? Then and only then can you align the company around a vision of the problem being solved for the customer.
Taking Twitter again as an example, consider what happened when the company raised the character limit to 280. This decision was almost heretic given how foundational this limit was to the platform at the time — as foundational as no edit button! But when you look at the data on how often folks ran into the character limit when writing in different languages, customer pain is suddenly painfully apparent.
Imagine doing oodles of growth hacking to raise your MAU and discovering a major friction point like this years later. Turns out the character limit actively hinders customer value in the platform — expressing themselves quickly to talk about what's happening right now — and thereby creating content that the entire network depends on.
Assuming you're sold on this idea of designing metrics around customer value, let's talk about how to design your perfect north star metric:
Clearly define your customer jobs to be done and make sure you measure all of the ways this shows up in your product
Measure more than one kind of activity if there's more than one job to be done. Challenge yourself and your team to ask more questions about who is the customer and what problems do they solve (or try to solve!) with the product if you only see one job to be done. Odds are you missed a valuable opportunity to expand!
Decide how often you expect to see these activities from your customer when your product is fully utilized — is it something they will do every day? Every week? If it's only once a month, is it actually valuable to them? The goal here is to flip the script — you aren't measuring velocity of activity based on how fast you want your business to move and iterate, you're focusing on how often your customer is getting value. It's harder and requires you to be very honest with yourself about what you expect your customer to be doing.
Test your assumptions by asking yourself if your resulting metric is explainable and predictable. In other words, is it easy to communicate who you're building for and what problems you're solving, and is it easy to trace work being done in the company to positive customer outcomes? If the answer to either of those is “no” then you need to do more work to understand your customer and not more work to educate your employees.
Elsewhere on the internet…
By Emily Thompson
Emily's writing is a text book for every aspiring data leader who wants to move their business more towards self service. This week she covers a framework you can use to prioritize work that breaks you out of the spiral of reactive analytics work. This one is a must read!
By Iliana Iankoulova
This is a very thorough read on designing your data systems to be a true self service platform. I especially loved principle #4 — making sure everyone has a stake. Iliana writes about how their team set up rotations in Slack to support data questions across all data roles and I can't say how fundamentally important this type of thinking is for a growing data organization. Everyone needs to stay connected to customer value!
Also, don't miss this thread from Katie Bauer on why calling data science and analysis “magic” erases the hard work teams put into the process and focuses too much on a culture of random strokes of genius instead of persistent hard work:
by Stephen Bailey
I really enjoy writing that is not only exceptionally thoughtful and challenging but also presented with a careful and tight metaphor. Stephen's writing this week on wandering is exactly this — sometimes you have to take the slow and often meandering path to learn something really valuable. Wandering is a great metaphor for dogfooding your product — you often hear that employees don't have time to learn the tool or platform, that it's more efficient if they use what they're already familiar with but that's not the point — Stephen's point is that the time spent wandering through your product is well worth it because you are putting yourself in your customers shoes. You are exploring what it feels like to use the product not just what your dashboards tell you. If you have to learn how to use the product first, you're experiencing the same journey as many of your first time users — what friction do you experience? What doesn't make sense to you? Your fresh eyes are a feature, not a bug and you'll probably learn a ton in the process!
Thank you Stephen for a great thought experiment!