One dbt
My reflections from the biggest theme of Coalesce 2024.
A few weeks ago I wrote about the feature announcements from Coalesce. This issue I want to zoom in on our One dbt theme I announced from the main stage. It’s a big deal for us, and will be important for the long-term health of the community.
One dbt.
IMO, the biggest underlying stressor in the dbt Community over the last several years has been the underlying tension caused by the steward of the OSS roadmap (dbt Labs) going through the growing pains of becoming a real software business.
Many, many humans use dbt Core as a central part of their jobs. It has become an important professional dependency for many of us, and so there has been some stress in the community associated with topics like:
Is the license going to change?
Will Core continue to get new good features or will dbt Labs abandon it?
We’ve done our best to answer questions like this over the past couple of years. We’re not changing the license—it’s staying permissive open source, Apache 2.0. We continue to ship new and important features to Core—check out the new incremental functionality and unit tests, both launched this year, both things I’ve wanted for a long time.
But despite this, many folks in the community correctly identified a fundamental point of friction between dbt Labs and the dbt Community, one that had created real underlying tension.
I think that we (dbt Labs) have—unintentionally!—created a context that caused a bifurcation in the dbt Community into two somewhat distinct sub-communities: the Core community and the Cloud community.
The Core community was much larger, full of early adopters, and loved the permissionless nature of dbt Core. They didn’t mind that running Core came with some friction, because it was such an important tool in their professional tool kit that learning to operationalize it became a point of pride, not a problem to be solved. Often, but certainly not always, folks in this community were skewed toward the Data Engineer persona.
The Cloud community was smaller. It was just as passionate about dbt, its fundamental principles, and its impact. Its members often, but not always, tended to be a part of larger organizations and talked about best practices more frequently inside of their own, internally-facing analytics communities. Often, but not always, folks in this community skewed toward the data analyst persona.
The product itself did not absolutely force users to make a choice—core OR cloud—but it didn’t make it easy to use the two versions of dbt together.
All of this drove a gap between the two communities.
And we exacerbated this gap in the way we set up dbt’s documentation, dbt’s onboarding experience, our product launches, and countless other aspects of the interactions that members of both of these communities had. We continually—although unintentionally—drove a wedge between these two groups, one tiny hammer blow at a time.
Over time, identities started to form. “I’m a Core user.” “I’m a Cloud user.”
Divisions in a community are bad. Divisions in a community that ossify into identities are worse. Instead of having a pragmatic conversation about feature set and pricing, you’re having a conversation about who someone is.
We certainly haven’t fixed all of the underlying problems yet. But at Coalesce this year, we called it out publicly, and we committed to bringing these communities, and the products, closer together. This was an important part of our “One dbt” theme and we talked about it everywhere. Here’s how I framed it in my keynote:
In the data industry, we are obsessed with these zero sum debates. Snowflake versus Databricks, Data Engineer versus Analytics Engineer, data team versus the business, dbt Core versus dbt Cloud. It is incredibly easy to frame things this way, as fights, as battles.
But none of these should be OR conversations. They should be AND conversations.
We need to focus on how to bring people together, not pit them against each other. Snowflake AND Databricks, Data Engineers AND Analytics Engineers, dbt Core AND dbt Cloud. I have always wanted dbt to be a unifier, not a divider. dbt works across Clouds, across Data Platforms, across Teams.
Reactions at Coalesce
The feedback we got from attendees throughout the conference was, quite honestly, incredible. Cloud users expressed an excitement about building bridges to Core users, often who worked on peer teams at the same organization. Core users expressed excitement about being able to use Core alongside Cloud-native experiences.
You know how sometimes in a personal relationship there will be this underlying unsaid thing. Maybe that thing has gone unsaid for years, and maybe it’s been there for so long that the two people in the relationship have forgotten that it’s even there. But it continually saps energy and joy from the relationship.
Eventually someone works up the courage to say “you know what I’m really pissed about [this thing].” And then all the sudden the floodgates open.
At first the conversation is tense. It’s hard to talk about these kinds of things, after all, which is why that topic went unsaid for so long. But eventually, as both sides go deeper, the conversation develops some safety. Trust is built. Both sides display empathy for the perspective of the other that they slowly come to understand. Eventually—and this “eventually” can take hours or years—the relationship is stronger than ever.
That’s what it felt like started—started!—to happen at Coalesce 2024.
If I had to summarize the response from the scores of attendees I spoke to about this divide and about our One dbt message, it was: “You’re goddamn right. Now let’s fix it.” And behind this response was this incredibly complex cocktail of emotions: relief, frustration, gratitude. It was a lean in, not a lean out. A “let’s do this.”
Making it Real
But now we have to manifest One dbt in the product. Let me give you two examples of things we’re thinking about. Standard caveats apply: this is not an official roadmap, but these are real ideas we’re thinking through.
Example 1. We need to create an easy way to ingest Core metadata into dbt Cloud so that Cloud users can do things like:
create cross-project references to upstream Core projects
view the entire DAG in dbt Explorer
Right now, doing this is possible, but hard. In a One dbt world, the idea that some users (and some workflows) will be on Core and some will be on Cloud is a default assumption.
Example 2. We currently have two CLIs: there is Core (obviously), but there is also the dbt Cloud CLI. Many folks don’t even know about the Cloud CLI, but it has many advantages over Core and is my personal favorite dbt developer experience. The problem is, moving back and forth between those two CLIs is friction-ful and users tend not to do it.
In a One dbt world, we wouldn’t force users to choose “which CLI to install.” In a One dbt world, the idea that you would have to choose wouldn’t make any sense—there should only be one.
I could go on; there are a lot of examples of these types of problems. dbt Cloud has a ton of product surface area at this point, and most of these product experiences have not been designed with a One dbt mindset. There’s real work to do.
Reflecting back
There are not many people, or many companies, that have the very good fortune to steward large open source products or lead large communities. I am consistently humbled by the trust the dbt Community has placed in me, and in us, over the years.
As I am not shy about saying, I don’t know all the answers, and I am not perfect. As I reflect on the journey that got us here, I can’t help but be self-critical.
But I’m so very grateful for the opportunity to name this problem and for the opportunity to correct it. There is a tremendous amount of goodwill in the global community of dbt users, and a genuine desire to build a bright future together. Creating space for commercial innovation while keeping the community healthy and aligned is, I believe, a very achievable goal, and is absolutely central to how we need to ensure a sustainable future for dbt over the very long term.
Thanks for your patience along the journey, thanks for your support, and I’m looking forward to pushing forward together.
There is only one dbt. LFG.
- Tristan