Ep 1: Robert Chang on Scaling Metrics @ Airbnb
Robert's team at Airbnb built Minerva to track over 12,000(!) metrics and 4,000(!) dimensions with consistency. How'd they do it?
Robert Chang is a product manager for the data platform at Airbnb, where he helped build and roll out Minerva, Airbnb's internal metrics store.
In this conversation with Tristan and Julia, Robert dives into why they built it, what it took to get it done—and crucially, what you should do if your company doesn't have the resources to build your own internal metrics store.
Listen & Subscribe
Listen to the full episode from the player below, or find your player of choice from the links beneath it.
Listen & subscribe from:
Show Notes
Key points from Robert on Minerva and the metrics layer:
Why build a metrics store?
I think a key insight that we realized at Airbnb is it is definitely very important to have table-level standardization, but it's not enough because when people consume data, they don't really consume tables. If you have your leadership team or your stakeholders, they think in terms of business questions, which are roughly translated to metrics and dimensions, not really like the underlying table that's really abstract.
Pushing standardization to the next level from tables to metrics is something that we really believe in, and that's something that we started to see about a few years ago. So we've been working on that project since 2017 - for almost four years. And more recently, we've seen a lot of attention being put on the similar concept in the industry.
I would venture to guess that this is where the industry is gonna go, and the concept of metrics store will play a pretty critical role in the modern data stack.
What is a metrics store?
So, the infrastructure that I helped to build at Airbnb is called Minerva, and essentially it's what we now call a metric store. And a metrics store, if I want to put succinctly, it's basically a single source of truth metric repository, where users can define metrics and dimensions once, and then they can use it everywhere, be it analytics, experimentation, or reporting.
For our team, our vision statement is "define once use everywhere" and it's all centered around metrics and the idea of a metrics store is what make it possible.
What kind of timeline and resources did it take to build Minerva @ Airbnb?
So, it's a four-year project in the making and we're still pushing the frontier.
The project started out in mid-2017. I was still at data science then, and I wasn't the PM of the team, but I actually worked with the Minerva team very closely. There's actually only one engineer, not even spending his full time working on this project, but the idea is that we should be able to build a data engineering framework that bridges the gap between tables and metrics.
Minerva is not really a concept that was born out of the blue. We had an experimentation platform, and that platform allows users to define metrics in which they can then keep track of his experiment results. It's called metrics repo, and the metrics repo has been very successful, but the success is only limited to the context of experimentation.
A lot of people started asking questions like, well, if we define all of these metrics for the experimentation use cases, why couldn't we leverage it to apply it to reporting or analytics and beyond? And that was actually a very valid question that was sort of the genesis, the starting point of Minerva, and Minerva actually was internally called Global Metrics.
The idea is that defined once use everywhere. And we wanna use it, use these metric definitions beyond just experimentation, but also in other applications as well.
So it started out in 2017, and then it took the engineer about a year to kind of put everything in place. And starting in 2018, the turning point for us is that we started to identify a few very important company-level reporting needs that allowed us to partner with them and try to help them understand the value of Minerva.
Then we started working with them very, very closely, and they're basically our alpha and beta users that give us feedback on what are the kinks that need to be fixed so that eventually it can be ready for prime time.
We have been very lucky because that was also the time where Airbnb was- well, we have always been trying to prepare ourselves to go public, so we were doing a lot of data readiness type of work, and one of which is the rebuilding of a lot of Airbnb's data models.
So, around that time, basically, a group of very experienced data practitioners at the company get together and try to rethink: ok, what is the right architecture going forward that will serve us in the years to come? And so there's a lot of innovation in the data engineering space, but then there's clearly the question of: ok, once you build all these data models, how do you translate these models and tables into analytical datasets that are analysis-friendly?
That was where Minerva came in, because it's the perfect paradigm that bridged the gap between the data engineers and the data scientists. So, by working with them, we onboarded a few very important use cases at Airbnb.
Then, in 2019, we just kept hardening our metric infrastructure, we tried to drive adoption and we build the Minerva API, which allows more users to connect to our metric store and to build more innovative applications on top of it.
And 2020 was a year where we kind of crossed the chasm. By 2020, most of the people at Airbnb recognize Minerva as the single source of truth for metrics.
What makes a metrics store successful?
I would say that like a metrics store is interesting that it also has network effects that the more metrics and dimensions that you added to the metrics repository, the more incentive for people to use it, right? Because the activation energy for them to adopt the technology and the chance of them finding something useful for their particular use case just lowers significantly.
So, Minerva has been very successful for various reasons. One is because of the general kind of company direction, where we were trying to go public, we need to harden our data quality. That definitely helped. The fact that we were able to partner with important stakeholders that own important use cases that also helped.
And then finally, the network effect also is another a big reason why Minerva has been adopted very widely across the company. So, we're very humble and very happy that this concept really can materialize at Airbnb.
What should smaller companies do at the metrics layer, who may not have the resources to build an internal tool?
I think when you have some sort of data warehouse, data lake that's built on top of Snowflake let's say. I think the first question is probably not how you standardize metrics. I think probably the first step is to do table standardization. So it just goes back to Monica's data hierarchy of needs. I think you kind of tend to build things layer upon layers.
So, I would say to the extent that you can, try to solve and try to tackle table standardization first. And that probably means that you should have talent who are very strong at data modeling skills, people who can translate the business questions into metrics and dimensions and the associated entities, and then do the corresponding data modeling.
And then once you've built out a set of highly-vetted, high-quality, core data tables, hopefully using tools like dbt, that's one step forward to building standardization in your warehouse.
Once you have those tables, then naturally you would probably start to see some of the problems that we experienced at Airbnb. But, I think that the advantage of having table standardization is enormous, and it'll buy you years to come.
At Airbnb, we had Core Data table that, pre Minerva, that paradigm of everything is centered around Core Data lasted for five or six years.
So, just having tables standardization actually takes you a long way. And I think as the company and as the organization becomes more and more mature, that you will start to see the need of having metrics standardization. But, of course, if there are technologies that enable everyone to skip the middle step, the table standardization and go straight to the metric standardization, that would be even better.
But, I actually haven't seen that out there, and I just think that this is still a pretty novel and pretty new concept. People welcome the idea, but it's not very clear how exactly the same success can be replicated into different organizations yet.
And just because we have been somewhat successful at Airbnb, I would not claim that this can be easily done in other organizations, even though I do believe in this idea.
Links from the episode
Robert’s instant classic post from 2015 on Doing Data Science at Twitter
How Airbnb Achieved Metrics Consistency at Scale, a deep dive from Robert’s team on the Minerva project
Monica Rogati’s the Data Science Hierarchy of Needs - everyone starts from the bottom, with “can you count?”
More from Robert Chang
If you’ve never read Robert’s writing on Medium before, highly recommend checking it out. You can also find him on Twitter @_rchang.
Keep an eye out for more posts from him on the subject of metrics!
The Analytics Engineering Podcast features conversations with practitioners inventing the future of analytics engineering.
New episodes are published every 2 weeks, along with the companion Analytics Engineering Roundup newsletter.
To get each edition of the Podcast + Roundup to your inbox, subscribe below: