Data as an assembly line (w/ Cedric Chin)
Cedric Chin runs Commoncog and Xmrit, a free tool to create and share XmR charts.
Cedric Chin runs Commoncog—a publication about accelerating business expertise. He joins Tristan to talk about the analytics development lifecycle, how organizations value (or misvalue) data, and why “data teams are not some IT helpdesk to be ignored.”
This is Season 6 of The Analytics Engineering Podcast. Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.
Listen & subscribe from:
Key takeaways from this episode
So the thing that made me want to reach out to you and have this conversation is that I published a precursor to the full analytics development lifecycle (ADLC) post that I put out. And in it, I talk about how the role of data analysts is to do insight generation. And I made the statement that analytics isn't fundamentally an assembly line
And it was around, really how we should perceive the value that's created by data analysts and by data practitioners more generally, and how we should think about the value that data generates. I was representing what's probably almost conventional wisdom.
But your pushback on this topic was, what if we do conceptualize data more as an assembly line? Why don't you try to frame what you see as the value that data can provide to organizations when it's living up to its fullest potential.
Cedric Chen: Maybe I should start with the conventional view of data in the data world. Data professionals I talk to say they answer questions, generate insights, the business person asks a question and then the data professional give them some answer after sometimes very hard work of figuring out correlations, relationships, whatever.
And then the business person, and this is a common experience that I've heard from every data professional, the business person goes, okay, cool. And it does nothing with it, right? Not always, sometimes it really does lead to impactful stuff. But a lot of times it turns out that the business person sort of carelessly asks you as a data person or the data team a question that actually doesn't matter to the business. They're just curious. And then they don't know that you have to, you as the data professional have to go through hell and high water to get the answer to that question.
And a data professional told me after reading a whole bunch of stuff that I covered, after I'd gone through not just what Amazon did, but also other companies like Amazon who use these ideas and how they actually use data, that it's almost as if the problem with most businesses and most business people is that there's too many questions they can be asking. And there's no way to narrow down the set of questions that really matter in the business. And because you can't differentiate between what's a good question, what's a bad question, the data team is like this service desk that is overwhelmed with questions.
It turns out that in really good data-driven companies of the type that I dug into, the term that I'm using is process control or statistical process control, which is the set of ideas that was used to ramp up production for World War II and then was used to transform industrial Japan post-World War II and led to the creation of the Toyota production system. This is a particular style of using data called statistical process control. And the basic idea is just, how does your business work? Your business is a system. It's a process. It has some inputs. It has some outputs. And you can figure out what those inputs and outputs are.
Amazon has a framework that is much simpler to understand, which is controllable input metrics and output metrics. Obviously, the ultimate output metrics you care about in the firm, in the company, are things like financials, profit, revenue, free cash flow. But there's a complex set of inputs and intermediate inputs as well that lead out the other end. And there's some lag from the inputs to the outputs. Your executives running the company, running this complex machine, need to have an idea of how the inputs flow true to the outputs.
Now, given that frame, how do you actually piece together the causal model of your business? And the core thing you have to grapple with is that it's not that easy to say, “OK, we've driven these inputs, and then we get some outputs out the other end.”
The problem that we have to deal with as business people is variation. Most people can't deal with variation. Variation just means that something wiggles. And we know that when we step on a weighing scale that our weight wiggles, right? It doesn't just stay the same. But it's even worse in business. Metrics can wiggle, sales metrics can wiggle upwards of 40, 50%. And that's perfectly normal. Nothing else has happened.
And so if you have a way of differentiating between a real wiggle that you have to worry about and you should investigate and routine variation, which is a normal wiggle you can ignore, you have a way of separating signal from noise. And you have a way of finding out things that actually move the needle on the metrics that you care about or the outcomes that you care about.
Now what this does, if you're able to differentiate between routine variation and exceptional variation is that you unlock the most common of human learning loops, which is trial and error. And that data usage context results in a way of using data and asking questions of a data team that is very different from a company that doesn't have this weekly process of figuring out how the business actually works.
I follow along with the fundamental insight here, which is to say that, what we should be doing is building a causal model for our organization. And that requires defining key metrics and understanding the relationships between the metrics.
And then having a reasonable statistical view of what's noise and what's actual variation. And then making decisions based on that. But obviously the devil is in the details. Does this model of the world always work or did Amazon happen to have a perfect business model where they could just observe the inputs and outputs and draw causal relationships really effectively?
Yes, it works. But there are certain things that are harder to measure using this methodology than others. So marketing attribution is not solved by this method. It's really, really hard. And if you look at early Amazon, they defaulted to things that they could actually tell there's an impact, even if the impact is vague.
They didn't do TV ads. They did it for a while and they realized that, we can't really find controllable inputs and controllable outputs. We can do controllable inputs on the cost of the amount of money we spend on TV ads, but we can't tell the output, right? So let's stop. So what did they do? They defaulted to affiliate marketing.
This was affiliate marketing in the 1990s. You can imagine how bad that was. And they could sort of model that behavior. The other thing that they could do was they could model word of mouth growth. They could say that within a certain number of months, a certain percentage of people who turn out to be new users in this new purchases in this first month, right? They didn't even have the term “cohort”. They called it “vintages”.
So by the fourth vintage, would be like 70 % that will become a steady state number of customers, right? It is possible to figure out.
One of the nice things about running Commoncog is that it's a dinky little business, and it doesn't really matter if I tell you about how my business works, and I can tell you my metrics.
We wanted to figure out how the newsletter grows. The first step is just measure. And then every week, take a look at your WBR. We do a full Amazon WBR. You go take a look and figure out what routine variation looks like. So you develop a fingertip feel of what normal looks like.
And what's some of the controllable inputs that you can think about? Well, LinkedIn posts or Twitter posts. And it turns out that there's a linear relationship between visits to the site when I post on Twitter and if I ramp up my Twitter posting, there's a relationship and increase in visits from Twitter. But there's no change in new starter signups. Similarly for LinkedIn. Maybe some of them do sign up, but it's not exceptional variation. It's not special variation. One day, October last year, we saw exceptional variation in in-depth readers, which is people who read more than one page, and unique new starter signups. were like, holy shit, what happened?
And it turned out that a 20,000 substack, an investing substack, had linked to Commoncog. Now, that's interesting. And it turned out, by the way, that the author of the substack had tweeted just one week before and no exceptional variation on any of our metrics. And this person had 70,000 followers on Twitter, but zero budget on metrics. So what's the obvious thing? The obvious thing is, OK, we need to go run an experiment.
What if in terms of the bang for buck for for our effort, it makes more sense instead of posting on social media or hiring someone to post on social media, to go and hunt down subsets and try to get them to link to us. It could be by buying ads and we have to experiment with that, or we could try to be friends with them so that they link to us naturally. There are a range of experiments that we could do because we now know that this is a lever that will result in a change in the output metric that we care about.
Nothing you're saying should be hard. And it's certainly not technically hard if you're good enough. For some reason BI tools don't generally draw XMR charts, right? You have a tool that does that, right?
All the data tools are good enough. Yeah, we have an open source tool that we made because BI tools don't generate XML charts and we want people to steal it. We want BI tool vendors to steal the code.
You want to drive change in the industry so that people can do this instead of BI tools.
The funny thing is that the most people who are using the tool are business people who want to get results. And they don't have data teams, so they can't bother to deal with the data team and wait for them. So just give me the CSV, and I'll paste it into this tool so that I can run the experiments, which is sad. Commoncog is for people who want to get good at business, because I want to get good business. So I bring them along for the journey.
The open source thing is more like I feel for data people, I have an affinity for and I empathize with them. Plus my wife is a data analyst, so I feel her pain. And it's just I'll give you an answer to the question you did in articulating. Why is this not more widespread?
What I was poking at originally was that maybe it's not relevant in all cases and maybe it is to a greater or lesser degree relevant in different cases, but I think that you would probably take the belief that it's generally a useful tool in most contexts. And my guess is that it somehow has to do with organizational dynamics. Is that right?
Yes, it has to do with power. It has to do discipline. It has to do with will not skill. You have to bear in mind that Amazon did all of this in 1997 with Excel 1997, which sucks. And they could forecast within a 3% error rate their growth of their business at the time.
Amazon got to a billion dollars on Excel. So if they got into a billion dollars running the WBR on Excel, what excuse do other organizations have? You have the modern data stack. You have everything that we didn't have back in the day. And literally, the way it worked was that every Sunday night in a shared folder, all the departments would drop Excel files into a shared folder.
So if the tools are not the problem, what's the problem? The problem is the social technical dynamics, right? The WBR is not just a metrics review meeting. It's also a political tool.
If somebody who is an executive asks one of your metrics owners under you in your department a question that you cannot answer or you don't know the answer, you'll be shamed in front of the entire organization. So what happens in that kind of political context? You really care about data and you don't ask stupid questions of your data analysts because what matters in this dynamic is you have a set of output metrics you have to hit by the end of the quarter.
You have to figure out what the input metrics are. And the rule is we don't talk about the output metrics. We only talk about the controllable input metrics. So you need to go figure out what those controllable input metrics are. So now a fire is lit under your ass to work with your data team. And the data team is embedded inside your organization to figure out what those controllable input metrics are. And the way you figure it out is that you do trial and error quickly. You drive this and see if it pushes the output metric after some lag. No, doesn't work. Let's try another one.
And you have to instrument, and it's OK. If you say that you’re still instrumenting, they say, “OK, it's fine; you can't push the controllable input metric. We'll give you some time to instrument.” Everybody understands that in the org, right? But once you do, you better figure out what the controllable input metrics are. And then you start setting targets. And then it becomes a very tight process where there's a fire under us because every Wednesday morning you have to present to executives.
It's a system that once it's working, I can see it being incredibly effective and self-perpetuating. It is also a system that I can really imagine being hard to create in the first place. Because most executives at a company do not actually want to subject themselves to that type of scrutiny in front of all of their peers.
One beautiful thing when you have a business review that measures the company end to end is that when there's a problem in one part of the company, you can say, hey, you in the different department, can you help this guy out? We're part of a team. Let's work together. Because things that change in one department, especially if you figure out controllable input metrics, output metrics. One person's output metric is sometimes another person's input metric. And so everybody has a bird's eye view of a very complex business.
Amazon's WBR is designed so that you can do 500 metrics in exactly 60 minutes. You don't go over time except during the holiday season. But the point is just the practice that matters.
There was a trend, over the last 12 months or so in the data community, at least the data people that I'm connected to on LinkedIn, talking about metrics trees. That's just as good. The point is that you need some kind of practice like this. And unfortunately, it requires shoving down by the CEO. At least all the successful examples I've seen has been somebody in a position of power on the executive team enforcing this.
And when it works, it is a wonderful environment to work in as a data person, because everybody is motivated on the same business problems. Everybody has the same causal model of the business. And the data team is not some IT help desk to be ignored. They are critical to achieving your goals.
One of the reasons data people don't talk about this more on LinkedIn is that it's not something that's controllable for them. It requires the type of executive sponsorship that then gets books written about it.
Probably many data people have never worked in a context like this. I think it probably was more common in an era of where they were more physical manufacturing type processes that we were trying to measure. So probably the experience set of people operating this environment is just actually not that high. Would you agree with that?
Somebody in my community, observed that this kind of operational excellence only emerges when you have a very low return on invested capital. Not very low, single digit. Because if you think about it, if your margins are super, super high, your return invested capital is super high, because like Google, you can be sloppy. It doesn't really matter. Who cares? You have a network effect. You're just generating gushes of cash.
If your return on investor capital is negative, then it doesn't matter. Your operational excellence is just staving off the inevitable. You're just going to die. You're in a textile mill in America, and you're fighting against it. But if you have a 3% return on investor capital, an operational excellence can double that for free, for effectively free.
These kinds of methods tend to spread in organizations or industries where you have a single digit return on investor capital, because if you don't have it, then you die. It just so happens that Amazon is a low margin business that just happened to hire somebody from manufacturing to deal with the scale of volume in their fulfillment centers.
And then the ideas spread out and they use it to crush their competitors because the competitors just did not understand what was going on in their businesses. I often tell people that maybe you can get away with this in high margin businesses, right? But if you are in a low margin business and you up against Amazon, Amazon has process control and you have something naive like the North Star metric framework, you're going to get crushed because you are just working towards one metric.
Amazon is working towards like 16 and they can spin off things to like, let's just undercut you in this particular way and we'll process control that and process control the cost that we're spending to undercut you. And we can do pincer movements. So it's an organizational capability that is super powerful.
This newsletter is sponsored by dbt Labs. Discover why more than 50,000 companies use dbt to accelerate their data development.
Let's have Tristan do an bonus 3 hour episode with Cedric :)