Making friends with the truth

Near, far, wherever you are

There’s a concept that’s been echoing around my head. I believe it has a huge amount of explanatory power as a framework, both inside the domain I encountered it (meditation) and in just about every area of life.

For any state, concept or ideal we might want to pursue - there is a far enemy which is the opposite of that desirable quality, but that there is also a near enemy which looks superficially like the goal we are seeking but is actually in opposition to it.

Today, lets talk about the near and far enemies of the truth.¹

Many of us spend our lives searching for something approximating truth.²

One way we can conceptualize this search:

• Far enemies of the truth - the opposite of the truth. As far away as what you’re seeking as you can get

• Near enemies of the truth - something that looks like the truth. That very smart and competent people can legitimately and earnestly believe is the truth. But not the truth itself

• The truth

Or to put in a different format:

This concept has become load bearing to me for understanding the world. Not just in terms of identifying individual far/near enemies and how they crop up, but in developing a higher order theory of what it feels like to move along the pathway of far enemy / near enemy / the truth. And while finding a singular unifying truth is out of scope of today’s post (sorry), what we can do is talk about getting closer to the truth across specific domains.

Specifically:

• What the process of riding this wave from far to near to truth feels like

• Apply this thinking to data work - both at an individual level and an industry level.

We begin with a historical interrogation unto the nature of gravity

Let’s walk through an example together - gravity. Think quick! Why, when i drop my iPhone, does it fall until it hits the ground.

Do you know why? Really know?

I’m going to perform a dramatic oversimplification³ of our understanding of gravity as follows:

• Far enemy (pre-Newtonian physics) - things fall because it is in their nature to do so

• Near enemy (Newtonian physics) - things fall because of the force of gravity and Newton’s laws

• Truth (general relatively) - things fall due to the curvature of spacetime⁴

Going from pre-newtonian to newtonian physics was really helpful for humanity! They had wide ranging impacts across science and engineering. This is a crucial thing to remember about near enemies - they can be extremely useful.

Some things that are true about near enemies:

• They are closer to the truth than far enemies. That means that they can be useful, interesting and provide value under some circumstances. Often, going from a far enemy to a near enemy is a huge sign of progress

• By the very nature of their proximity to the truth, it can be more challenging to relinquish them. These can be ideas that have served us well, that have helped us in important ways. But they are not the truth itself.

Going from a far enemy of the truth to a near enemy feels exhilarating. A previously untheorized, unknown aspect of reality sits before you, at last explained and known. The implications can be clear, precise, blinding.

Moving on from a near enemy to the truth feels different. In my experience, it feels less like a blazing moment of breakthrough than a subtle realization that your current model isn’t matching reality. It involves a searching, often subconscious, to resolve this tension.

And often when the tension is resolved, it can feel terrible.

Remember - our near enemies are dear to us. They’ve cleared up what was once opaque and brought sense to an aspect of reality that we previously didn’t have. But they are not the truth itself.

Bringing this to the modern data analytics stack

Ok so astute readers among you may have noticed that we haven’t been talking about the data industry, the ostensible topic of this newsletter.

But also, we haven’t not been talking about the data industry or doing data work.

We’ve spilled much ink on the Roundup over the years talking about what it feels like to do data work, to develop an organizational ontology and make decisions using data. We’ve talked about it at the individual level, at the organizational level and the industry level.

The last two years have involved a lot of hard learning for all of us. And I think I know why.

Tristan gave a good example of this last roundup - talking about moving on from the term “The Modern Data Stack” that for a time was extremely useful, conveyed real meaning, had explanatory power, helped us reason about the industry.

Let’s take the same concept and zoom in a few levels - to an individual organization that’s trying to use data to make better decisions. We’ve learned a lot over the past decade about what that means and now to do it.

So how can we become data driven?

Far Enemy - Yolo mode, Hippos, Political Game of Thrones

The far enemy of data driven is simply to not know what you are optimizing for and not having quantitive systems in place to track that to reality.

This is organizational decisionmaking by law of the jungle, by way of the powerpoint deck.

There’s a couple ways this tends to play out in practice.

It might be rule by executive fiat - where top down decisionmaking sets the tone and direction for the organization.

It might be board room theatrics where theories are presented, but where the optics and grandeur of the presentation carry greater weight than the content within.

It might be a game of patronage where factional groups in the organization band together to prioritize a particular strategy,

There have been and will continue to be organizations that are successful while looking like the examples above. There are a lot of reasons organizations succeed and a lot that they can fail.

But for building organizations that are maximally dynamic requires the ability to integrate, understand and react to new information, and the ways of working described above make this notably more difficult.

Near enemies of the truth in data - Spreadsheet brain, SLOTHs, Moneyball for Everything

The near enemy of data driven is becoming so hyperfixated on tracking metrics that you forget to have a business strategy or make any decisions.

That might sound like:

• We need to be data driven and that means every move we make needs to be tied to metrics.

• We need a dashboard for every initiative we launch. We need north star metrics. We don’t want to be making decisions off of vibes, we’ve got to have the cold hard truth.

• Can we prioritize this experimental new feature? Well it sounds interesting but how are we going to know it’s working?

Katie Bauer has written the definitive piece on this, coining the term SLOTH - Statistical, Logical and Over-Thinking Hesitaters.

The dangerous thing about SLOTHs, however, is that their fundamental behavior is not bad. They want to be objective, to use metrics to calibrate and broaden their understanding, to find opportunities that are not obvious through casual observation or pure instinct. They see data as a useful tool, and they help wielding it to its fullest extent. This is a flattering and refreshing situation to be in as a data professional—it’s always nice to feel wanted—but it can get out of hand quickly.

Beware the SLOTH

The whole piece is brilliant and you should read it. What it does particularly well is capture one of the most dangerous aspects of near enemies of the truth - they feel right at first and require subtle and nuanced perception to identify.

If you are working at an organization with no understanding of data strategy, no attempt to connect to ground truth, at first you’ll find working with a SLOTH is a breath of fresh air.

It’s only much later, when you realize that despite all the checkins, dashboards and planning spreadsheets that you recognize that you aren’t making decisions all that much faster.

Approaching the truth - you make the best decisions you can with the information you have

One of the real problems of writing a piece like this is that now that I’ve walked through far enemies and near enemies, a reasonable and astute reader will be expecting to be walked through the One True Method for becoming optimally data driven.

While we aren’t fully there, we do have some promising evolutions in how we think about and use data, helpfully made possible by the combination of technological leaps in the hardware and software powering data and many thousands of smart people from the Community putting their heads together to solve problems (that’s you!).

We’re starting to recognize that you might need less dashboards but more metrics and a more systematized way to engage with them.

We’re starting to realize the value in having strong heuristics around when to wait for more data and when to just go ahead and make a dang call.

We’re starting to realize what we need to do when the chart wiggles.

It’s not just the metrics, its not just the data, its the sociotechnical system that enables you to capture data, reflect upon it and integrate into your decisionmaking.

Snap back to reality

So that’s it? We’re on the cusp of figuring out how to make organizations perfectly data-driven, cracking the code and entering an era of organizational enlightenment?

Well, no.

You’ll remember I promised I’d come back to the gravity thing. In the example above, I listed General Relativity as The Truth as far as we know about gravity. But it’s not, not really. Because despite being, to this day, our best explanation for gravity on a macro scale, it is wildly incompatible with our best explanation of gravity on a micro scale, provided by quantum mechanics.

The lesson here is that both of these things can be true.

• At a fundamental level, we have no fucking clue what gravity is or how it works

• That being said, we can still use General Relativity to make GPS work

How do we square those? Me, I like to think of it like this.

Source

All this to say that the trends we’ve identified in data work might not be the universally optimal solution to creating dynamic organizations with a ground truth connection to reality. But at the same time, they have brought us to a new level of understanding, a peak a little higher than we’ve gotten to before. And so, as the story has unfolded time and time before, we look to the horizon again and see where our friend the truth lies, and we set off on the journey.

This newsletter is sponsored by dbt Labs. Discover why more than 30,000 companies use dbt to accelerate their data development.

1. While not the first place I encountered the framing, this book serves as a helpful introduction to the topic

2. There is absolutely no way I can properly caveat how impossible it is to define the word “truth”, so lets pretend, you and I, that I have done so to your satisfaction

3. Do read this for a more thorough explanation though https://en.wikipedia.org/wiki/History_of_gravitational_theory

4. if you’re thinking WAIT THERE’S MORE - hold on, we’ll get there