A Recipe for Data Intuition
If running a data team is like running a restaurant, is the craft of analytics like cooking? And if it is, where do you get your recipes?
This week marks the start of the second October since the beginning of the Pandemic. It has been a solid 18 months or so since the great 2020 TP shortage, 17 months since murder hornets and the month of cancelled plans, and about 16 months since everyone privileged enough to be cooped up and working from home started getting really good at cooking and baking. Raise your hand if you still have pounds of flour and yeast from 1.5 years ago sitting around, reminding you of your unrealized Great British Bake-off dreams ✋.
The sudden inability to travel beyond the confines of my ~800 sqft apartment instantly heightened my cravings for childhood food recipes, and for what felt like the longest few months of my life, I got really really good at cooking comfort food thanks to a network of YouTube and blog recipes from around the world.
Prompted in part by writing on the craft of analytics from the past week, and in part by the first meaty discussions in the dbt Community Slack on the topic, I started thinking: What is the analytics equivalent to my COVID cooking binge research? Where does one go to learn how to work and think like a data professional? Where are the influencer blogs and YouTube channels for Analytics 'recipes'? 1
Why do we call it “data intuition”?
We, those fortunate to already hold various data related jobs, often talk about how "data is hard". In our leveling guides we speak of developing strong "data intuition". In our characteristic industry navel-gazing, we grow concerned that if we let non-data folks self-serve answers, they won't be able to "ask the right questions". Why is this so hard? What does it mean to ask the "right" questions and how do you know that you are? How does one develop this "data intuition", exactly?
What a curious word to use to represent the mastery of translating data into insight — “intuition”:
Intuition is a form of knowledge that appears in consciousness without obvious deliberation.
By reducing the work of working with data to a sub-conscious process, we are writing it off as a kind of innate ability at worst, or at best, something intangible that requires raw talent to be cultivated through years of meticulous observation and practice into a kind of art form.
At it turns out, this was not quite unlike what learning to cook was like prior to the French Industrial Exposition of 1834.
A cooking revolution
While cookbooks have existed since Roman times, and were popularly copied even before the development of the printing press, the concept of learning to cook was still a highly non-scalable 1:1 affair. Learning to cook was either aimed at producing basic sustenance, and happened through the observation of someone in one's immediate vicinity (aka your grandma and her pot pie recipe); or if you wanted to make a living out of "High Class Cookery", it required a formal apprenticeship with someone already well known for their ability to perform the artform (aka Jiro Dreams of Sushi — you know you binged it all again in April 2020).
It was only after the French Revolution — or more specifically, about 12 years after Eponiné walks the streets of Paris "on her own" on the eve of the 1832 uprising — that Europe first began to adapt to the concept of "cookery for the masses". The French Industrial Exposition of 1834 was the first time the category of food preparation appeared at an expo, and it was also the first exposition targeted to the emerging middle industrial class. The success of the French Expo inspired the Great Exhibition of 1851 in London, whose success in turn led to the establishment of the National Training School of Cookery in London — the first place in Europe to produce teachers of 'Plain Cookery' who would emerge to support the growing number of post-industrialization working class women who were no longer learning to cook from their mothers. The Boston Cooking School, the first formal culinary institute in America, followed shortly thereafter in 1879 with a similar aim as its London counterpart.
It was only after the establishment of the Culinary Institute of America in 1946 that learning to cook came with theory alongside the customary hands-on learning.
Today, hands on learning for the masses has been replaced by YouTube cooking channels and recipe blogs and aggregators. And the theory of "cookery" has been reduced, thanks to Amazon and Netflix, to four very basic components: salt, acid, fat and heat. Watching YouTube and learning about the leidenfrost effect while combining these four basic elements does not automatically make one a Chef. But it does pave the path to complex recipes that were once locked away in someone's grandma's cookbook, or behind the ivory doors of culinary apprenticeships.
A recipe for data “intuition”
The Analytics Profession is still stuck in the days of the French Monarchy. Within organizations we have the equivalent of your grandma's cookbook — a set or recipes hand scribbled and passed down through generations of analysts. For those who have yet to land their first data gig, their options are 1) “Plain Cookery” via toy datasets and made-up problems or 2) an intensive apprenticeship in a related field (e.g. Finance) to break into the ranks of "High Class Cookery" like Data Science.
Of course building data “intuition” is hard! We side step the theory that goes into teaching someone to develop reproducible data products, and lock away real world analyses that drive successful business outcomes behind layers of SSO. Without access to the recipes of others and a formalized theory of cooking up analytics insights, many aspiring data professionals are left standing in front of their fridges stocked full of big data, learning how to cook by throwing data ingredients together at a notebook until they generate a spark.
What were the recipes that first taught you how to produce actionable insight? What parts of those do you think we should distill into the theory behind the Analytics Profession?
- Anna
Elsewhere on the internet...
It’s been a busy week in modern data stack investment circles!
Congratulations to Materialize for Raising their Series C and Acceldata on their Series B!
Amplify Partners welcome Emilie Schario as their first data-strategist in residence 👏 If you aren’t familiar yet with Emilie, you should absolutely start with her and Taylor Murphy’s 2020 Coalesce talk on "Running your data team like a product team", and then promptly sign up for Coalesce 2021 to hear her tell you why she wants to remove the “Data Scientist” job title from your vernacular. Congratulations Amplify Partners!
phData take one more step towards end-to-end data services with the acquisition of Tessellation, an Analytics and BI Consulting company.
Snowflake announced a partnership with and investment in Anaconda (!) — but it seems like we need to wait 1.5 more months to find out what this means for the MDS 😉
Also this week…
The modern data stack ecosystem visual to end all visuals (until the next one) from Matt Turk:
Or the full resolution version in PDF if you’re not interested in testing your vision today Airbyte is moving away from the MIT license used for their OSS core towards adopting the Elastic license. This will primarily restrict the ability of others to provide Airbyte as a managed service and pave the way for monetization of Airbyte Cloud.
Bobby Pinero expands on the ideas in his “Every Analyst is a Finance Analyst” piece with “The First Operator”. Bobby argues that analytics and finance should sit together on an early stage start-ups org chart because of the amount of overlap in the team’s objectives at this stage of the company’s growth.
‘Til next time! 👋