The space between data dogmatism and data nihilism
Or how one A/B test forever changed my view on analytics
There’s a fascinating article making the rounds on Linkedin discussing Nike’s recent financial stumbles.
The article claims that Nike, whose stock is down 32% in 2024, made a number of “data driven” changes to their organization over the past 4 years. These decisions were based off of a series of rational seeming insights and precepts, building off the idea that having a set of clear, measurable targets around their product development and GTM strategy would allow them to build data flywheels and eliminate inefficiencies.
Before we go any further - I want to say that I’m not necessarily endorsing this particular read of Nike’s recent financial performance - but it does point at a fundamental paradox that the most discerning data leaders I know grapple with regularly:
It is very important for all organizations, but particularly organizations that reach a certain scale, to be able to build a data model that maps the inputs and outputs of the business in order to understand what is happening in the organization and plan for the future
Often many of the most important features about how successful organizations are run is difficult if not impossible to fully quantify and there can be an inexorable pull towards the things that are easiest to measure
A good data leader can create well defined constructs that map your business and allow you to understand the mechanics of your organization.
But a great data leader needs to know how to do this while also balancing the unquantifiable, the unmodelable, the easy to miss, subtle but important ways that metrics can mislead.
And most importantly they need to be able to do this while recognizing its still important to measure your business.
Reality is sticky
I distinctly remember my first collision with this paradox. I was a junior analyst running my first A/B test - we were trying to increase our trial signups for the SaaS company I worked at.
In my infinite 22 year old wisdom, I realized that by removing the nav bar on our trial signup page, I got a 16% bump in trial signups.
16% increase in trial signups!! This is HUGE! I’ve altered the course of the business forever.
After all, I could go plug 16% extra trial signups into our financial plan, see a 16% boost in Sales Qualified Leads, a 16% boost in closed deals, a 16% boost in revenue. I had just earned the company many times my salary - all with a single A/B test.
Except…
I was about to learn one of the core insights of the data practitioner. Reality is sticky. It’s quite complex and we need to be incredibly cognizant that the map matches the territory.
Because what actually happened was that those 16% trials who only signed up because I mucked around with the signup page, well, they weren’t great trials. They didn’t take the steps to get onboarded into the product, answer our sales calls or close.
What I’d thought was a 16% increase down the line looked more like 16%, 12%, 2%.
To be data driven or not to be data driven, that is the question
It’s really easy to look at situations where organizations have narrowly optimized towards a metric and fall into data nihilism - I know because I’ve found myself there in the past.
When we don’t feel like we can create mappings that fully capture the complexity or nuance of the world. Like the whole endeavor is doomed from the start.
But the fact of the matter is that we only know that narrow data driven optimization doesn’t work … because of our data systems.
Yes my early A/B test failed to have the desired impact because I was narrowly optimizing for the wrong things. And I figured that out - by looking at the data. Eventually, we figured out the right things to be optimizing for. The tests that would actually change our outcomes.
We talked to customers. We did a lot of experiments. We thought extremely hard about how to solve the actual problem facing our organization. We found tools that allowed us to wildly exceed our previous ability to learn from our data.1
At the end of the day, the way to resolve the paradox is … pretty simple. We recognize that it’s incredibly important for a functioning organization to have a strong sense of its business fundamentals. We invest the time and energy into mapping those to give us a sense of what we can measure and we work to find the areas where we might have gaps in our quantification.
We think really deeply about whether a “quick fix” that seems like its going to be the silver bullet really is that, or if there is deeper work we need to do to drive the change we want to see.2
We don’t fall prey to data dogmatism, where we believe the map always matches the territory, nor do we become data nihilists and think that none of it matters. Instead, we focusing on tuning our sense of reality, what can be measured, what can’t and everything in between.
This newsletter is sponsored by dbt Labs. Discover why more than 30,000 companies use dbt to accelerate their data development.
You might have heard of the most impactful one, they happen to sponsor this very newsletter.
Not to say that silver bullets never exist, that is also too easy - there really are simple tweaks you can make that will generate shocking / outsized returns. But they are often the 118th simple tweak you try.