Analytics and the craft of software engineering

We've walked 500 miles, and have 500 more to go

Hi, I’m Kshitij, and I’m a software engineer on the Core team here at dbt Labs.
Since Tristan is out on vacation, I’ll be your substitute writer this week.
Today, I’d like to share with you one of the most cherished parts of my job – the freedom to think deeply about technology past the next quarter or OKR.

As all data columns are legally required to, let’s start with ChatGPT.
If you’re up tossing and turning over whether ChatGPT will take our data jobs: don’t.
This tweet from one of my heroes, John Carmack, helped me sleep soundly, and it might help you:

Software is a tool to help accomplish something for people - many programmers never understood that. Keep your eyes on the delivered value, and don't over focus on the specifics of the tools.

This focus on delivering value dovetails with a piece Benn Stancil wrote nearly a year ago around how analysts should be better compensated than analytics engineers since their skills are harder to acquire and hone.
On a personal note, I’m thrilled for my analyst friends who have landed better opportunities and compensation as analytics engineers, but I’m disappointed that they needed to be perceived as “being technical” to get there.
Seeing them thrive within analytics engineering, and not feel the pressure to transition into software engineering for better opportunities, makes me excited to come into work to build a tool that serves them.

However, if you are interested in becoming a software engineer – or how to think like one – I’d like to direct you towards John Carmack’s interview on the Lex Fridman podcast.
The entire 5.5 hour(!) episode is an incredible masterclass in the craft of programming video games (like Doom and Quake).
But it is this clip highlighting the importance of getting the right amount of feedback from our tools that has stayed with me.

What if dbt showed you red squiggly lines underneath query bottlenecks, invited you to step through each one frame by frame, and offered suggestions for potential solutions?
As software engineers, we sometimes forget that these tools we take for granted don’t exist for other professions.
It's therefore heartening to see projects in the dbt community that enhance the data development workflow, including:

1. Alexander Butler’s dbt-osmosis automates the headache of YAML config management.

2. Nicholas Yager’s Whetstone analyzes dbt project artifacts to recommend improvements to data modeling, testing, and documentation.

3. New to me, fzf-dbt from Infused Insight enables autocomplete for dbt models with a fuzzy finder.

4. flow state tool from my colleague Sung Won Chung dares to imagine what hot reloading could feel like for dbt models.

I’m grateful that this collaborative spirit is thriving within the dbt community because it assuredly is not in Silicon Valley.
Ben Thompson astutely points out in “The End of Silicon Valley (Bank)” that the myth of Silicon Valley is shattering because when push came to shove, many prioritized saving themselves over saving the community.
I’m sad to see the camaraderie and magic vanishing in Silicon Valley, but at least I can chuckle at Pedro Pascal memes in dbt Slack.