The dbt Labs data team is moving to an all Iceberg lake with a mix of compute engines to power transformation, analytics, and agentic experiences. The team has been able to move quickly towards this architecture because the entire ecosystem has been laying the groundwork for years. All of it’s coming together to make this new open world a reality fast.

On this episode, Tristan discusses the reality on the ground for data practitioners. Where’s the Iceberg ecosystem today? What can practitioners realistically expect when attempting to run on top of Iceberg in production?

Tristan is joined by Anders Swanson, a developer experience advocate at dbt Labs. Anders has spent a lot of time over the years navigating open-source data ecosystems and tracking their progress.

They unpack the open standards shift, define the core building blocks (query engines, object stores, catalogs), and dig into why external catalogs have become a fourth namespace tier across platforms. Anders outlines a pragmatic, phased adoption model for Iceberg integrations, explains why metadata performance and resiliency are hard requirements, and clarifies why vended credentials exist and what they solve.

Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.

The call for papers is open for dbt Summit 2026. We invite data practitioners, platform leaders, and executives to share real stories of how data gets done at the world’s largest gathering of dbt community members. If you ship fast, reduce costs, improve trust, or bring governed AI to life, the dbt community wants to hear from you.

Submit a talk

Coalesce is now dbt Summit. Join the world’s largest gathering of dbt users, where data leaders and practitioners come together to shape the future of data analytics and AI.

Listen & subscribe from:

• Spotify

• Apple Podcasts

• Amazon Music

• TuneIn

• Youtube

• RSS feed

Key takeaways

Tristan Handy: I wanted to have you on because of work you’ve been doing internally to summarize the state of the Iceberg ecosystem. We’ve talked about Iceberg a bunch lately with folks deep in specific parts. Your work is more of an overview: where we’re at with platform integrations, what’s easier now than a year ago, and what’s still hard. Before we dive in, I want to define a few terms. When you say “query engine,” what do you mean?

Anders Swanson: It’s the thing that does your work. When you issue a CREATE TABLE or a SELECT statement, it’s what returns data or stores it somewhere for later.

Object store.

It’s the cloud service where you can store an object. An object is anything: a blob.

Catalog.

In this context, a catalog knows what tables and views exist and where they are, and how you can fetch or write to them.

Let’s talk internal versus external catalogs.

An internal catalog is what you get by default in a system like Snowflake or SQL Server. An external catalog is more like another directory, often managed by a different system. As you connect more disparate platforms, you can’t assume one system controls everything.

The complexity comes from duplication. How do you make namespaces unique? Can you plug in many external catalogs?

Abstraction matters. A common pattern emerging is one‑to‑one mapping of an external catalog into a database. That pushes a move to a four‑part namespace: catalog, database, schema, identifier. Spark moved toward this; Databricks Unity Catalog and Snowflake‑style catalog link approaches are in this family.

So the downside?

The devil is in the details, especially metadata performance and resiliency. For example, information schema listing. Users expect listing tables to be fast and reliable. In a federated world, if listing tables takes five seconds, users blame the vendor they’re using—even if the external system is slow. DuckDB draws a line by not mixing external catalog tables into information schema listing today. Snowflake’s catalog link databases appear to cache or mirror metadata so it feels as performant as native tables.

With catalog link databases, Snowflake is doing mirroring.

Yes. Mirroring exists in different flavors across platforms. Delta is sometimes seen as “simpler” because metadata can live in object store, but as soon as you want multiple engines writing, you still need a real catalog.

Sharing across multiple platforms adds another layer. What’s the state of platforms reading and writing to the same Iceberg catalog?

There are phases of integration.

Phase one is the naive approach: you have Parquet and JSON in object storage, and an engine reads it. Reading is easier than writing. You can get a toy example working.

Then you run into versioning and “what’s latest.” The next phase is connecting to an Iceberg REST catalog so engines can ask for the latest table version without users thinking about paths.

Phase three is schema‑scale: it’s never just one table. You need discovery of new tables, keeping schemas up to date, and eventually things like multi‑table transactions.

This maps to dbt Mesh and cross‑platform mesh. Producer vs consumer.

A consumer‑led model requires the downstream team to create pointers (DDL) to external tables. It’s operationally messy. Producer‑led is cleaner: the producer writes to the catalog and it’s just there, immediately queryable downstream.

Are platforms there yet?

Some support writing directly to external catalogs. When it works, it’s great, but there are still kinks. We’re retrofitting race cars designed for isolation to be interoperable without losing performance.

Identity is one of the hairiest issues. Vended credentials.

Vended credentials solve the “two keys” problem. You authenticate to the catalog, the catalog tells you where data lives, but then you need separate object store credentials to read files. Vended credentials means the catalog vends short‑lived credentials so you can access the object store location without managing separate keys.

That doesn’t solve user identity and grants.

Correct. Vended credentials isn’t global authorization. Identity and access across platforms is still hard. Ideally you grant access once and it works everywhere, but enterprises have different identity providers and platforms have different permission models. Today, admins often have to configure grants separately in each platform.

Is this mission creep?

The goal is to reduce how many people have to think about storage details. Big tech had whole data platform teams solving reliability problems in Hive‑era lakes. Iceberg reduces that toil dramatically, but the long tail is still auth, mirroring, and cross‑platform governance.

How does this reshape data teams?

Analytics engineering abstracted a lot of work. Data engineering has also been simplified by replication/orchestration vendors. What remains is the open ecosystem complexity: identity, object store policies, and cross‑platform connections. Many enterprises already have teams with these skills (infra as code, Terraform, Snowflake management), but others will need to grow into them.

Are vendors embracing Iceberg in good faith?

The goodwill and collaboration in the past 18 months feels unprecedented. We’re getting “more problems” because we solved prior ones. The industry aligning on standards feels like F1 teams standardizing components so they can innovate elsewhere.

In your internal writeup about Iceberg, you quoted Wolf Hall: “The making of a treaty is the treaty. It doesn’t matter what the terms are, just that there are terms, it’s the goodwill that matters. When that runs out, the treaty is broken, whatever the terms say.” Explain the relevance here.

When I joined dbt, it was taboo to mention one partner to another. Now vendors openly acknowledge mutual customers and invest in interoperability. On the Iceberg repo you see competitors collaborating on proposals. The goodwill is the standard.

Wrap us up with three things you’re excited for next year.

Push‑based catalog updates so platforms can subscribe to changes rather than repeatedly listing and polling. Progress on the small files problem so Iceberg works better for smaller data too. And more platforms supporting writing directly to external catalogs, unlocking producer‑led sharing and cross‑platform mesh.

Chapters

00:00:00 — Intro: why open standards are accelerating

00:01:20 — What practitioners can expect from Iceberg in production

00:05:00 — Lightning round: query engine, object store, catalog

00:06:20 — Internal vs external catalogs

00:09:30 — The “four-part namespace” and catalog-link style abstractions

00:11:30 — The downside: metadata performance, resiliency, and caching

00:17:10 — Sharing across multiple platforms: reality and tradeoffs

00:19:10 — Iceberg integration phases (1: naive table, 2: REST catalog, 3: schema-scale)

00:24:10 — Producer vs consumer model and cross-platform mesh

00:29:10 — Identity and “vended credentials”: what it is and what it isn’t

00:33:30 — The hard unsolved part: grants and global identity across platforms

00:37:00 — Is this mission creep? What Iceberg is optimizing for

00:39:50 — How roles on data teams evolve in an open ecosystem

00:43:40 — Are vendors genuinely aligned? Why Anders is optimistic

00:46:50 — “The making of a treaty is the treaty”: goodwill as the standard

00:51:50 — Three things Anders is excited for next year

This newsletter is sponsored by dbt Labs. Discover why more than 80,000 data teams use dbt to accelerate their data development.

Demo on-demand

A lot has changed in how data teams work over the past year. We’re collecting input for the 2026 State of Analytics Engineering Report to better understand what’s working, what’s hard, and what’s changing. If you’re in the middle of this work, your perspective would be valuable.

Take the survey

Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.

Listen & subscribe from:

• Spotify

• Apple Podcasts

• Amazon Music

• TuneIn

• Youtube

• RSS feed

Key takeaways

Tristan Handy: You spend a lot of your time thinking about Iceberg and Polaris. Give the audience background on how you found yourself in this niche of high‑volume analytic data file formats.

Russell Spitzer: It’s a bit random. I started at DataStax on Apache Cassandra as a test engineer and quickly got drawn into analytics. I saw big compute clusters and wanted to be involved. A coworker, Piotr, noticed Spark 0.9 and began a Spark–Cassandra connector. That got me into Spark. Over six to seven years I focused on moving data between Cassandra and Spark and into other systems. The interoperability problem across distributed compute frameworks was compelling.

This was pre‑Apache Arrow and pre‑table formats. We were just putting Parquet files everywhere and no one quite knew what they were doing. Pre‑Spark, people explored DSLs like Apache Pig. Eventually the industry converged on SQL for end‑user interfaces.

I later applied to Apple for the Spark team.

Helping build Apple’s Spark infra, or working directly on Spark?

Apple has an open-source Spark team and a Spark‑as‑infra team. I was trying to join the open source team, pushing Apple’s priorities into the project and supporting Spark as a service. During interviews, Anton—another Iceberg PMC—convinced the hiring manager I should join the data tables team, essentially Apple’s Apache Iceberg team.

They ambitiously planned to replace lots of internal systems with Iceberg. Iceberg existed but was early (Netflix started it around 2018/2019; I joined Apple in 2020). At Apple it was Iceberg all the time; convincing teams to move off older stacks, adopting open‑source‑as‑a‑service to save money, and getting onto ACID‑capable foundations. We were successful.

Migrations are hard. How did you make it accessible?

We replaced complicated bespoke reliability fixes with Iceberg. In Hive/HDFS, small‑file problems lead teams to write custom compaction and locking. Removing that toil is a big win. For big orgs, migration is a long‑term investment with ongoing engineering cost. For smaller companies, the key is offloading runtime responsibilities—ideally to SaaS—so engineers aren’t in the loop. Open source limits lock‑in so you can move between systems. Most companies are paid to deliver business value, not to build data infra. dbt is a great example of avoiding hand‑rolled pipeline code. Same logic applies to table/file formats.

Let’s talk Apache governance. What’s a PMC? How do projects run?

Apache projects aren’t owned by one company. Influence is earned by contributing to the community. The PMC governs merges, releases, membership. People move companies; the project stays with them. The goal is to make the project broadly useful. There’s no CEO dictating roadmap and no company can change the license.

Most big projects—Spark, Kafka, Iceberg, Flink—are maintained by employees of companies with vested interests, but governance is consensus‑driven. Vetoes are for technical issues (security, future‑limiting design), not ideology.

Is Iceberg for the top 20 tech companies or for everyone?

Not everyone needs Iceberg. OLTP belongs elsewhere. But for analytics, we should move past raw Parquet partition trees with folder‑name partitioning. In the Hadoop era, lakes were dumping grounds; schema evolution was painful. Many are still moving from CSV to Parquet. Over time, better encodings and table formats become default.

Decoupling compute and storage changes everything versus co‑located HDFS. Defaults tuned for HDFS (like 128MB Parquet files) don’t always hold for S3. We want elastic storage and compute; no one wants to pay for compute because storage grew.

Walk us through Iceberg versions.

v1: transactional analytics—ACID commits instead of fragile Hive/HDFS patterns. v2: row‑level operations—logical deletes via delete files so you don’t rewrite 10M‑row data files to remove one row; later compaction physically purges (key for GDPR). v3: expanded types—geospatial and variant for semi‑structured data; Variant was standardized across vendors and Parquet so everyone can write/read consistently.

v4: two thrusts—streaming and AI. Reduce commit latency, make retries faster under contention. Historically writes took 10–20 minutes, so commit latency didn’t matter. For streaming (writes every minute/five), it does. We’re evolving commit and REST catalog protocols so clients can specify intent (add these files, ensure these exist, then delete those) and let the catalog resolve conflicts server‑side.

On AI: Iceberg doesn’t yet serve some vector/image‑heavy patterns well. We’re exploring changes in Iceberg, Parquet, or both, without breaking existing tables.

Talk about Polaris and the catalog layer.

Polaris is an Apache incubator project (PPMC). Incubation proves we operate like an Apache project (community‑driven, trademarks donated). Iceberg defines the REST catalog spec/client; Polaris implements a catalog that speaks that spec. Many of us work across projects (Parquet, Iceberg, Polaris), which helps align boundaries.

Horizon, Polaris, external catalogs—what’s the story?

We’re simplifying: Snowflake can act as an Iceberg REST catalog, or you can use an external REST catalog. External can be Polaris (managed by Snowflake or self‑hosted) or another REST implementation. Interoperability means everything talks the same REST.

What is Polaris trying to be best at?

A broad, interoperable lakehouse catalog. It can act as a generic Spark catalog (HMS replacement) and aims to support multiple table/file formats. Architectural choices differ (KV vs. relational storage, where transactions live, policy enforcement vs. recording, identity integration). Polaris aims for base implementations that are pluggable—e.g., AWS/GCP/Microsoft identity.

Identity and scope—where does the catalog stop?

There’s a “business catalog” for discovery/listing versus a “system catalog” that must know table layout to govern access. Polaris can vend short‑lived credentials for the exact directory of a table’s files for a load operation; that requires understanding layout. Purely relational metadata often needs to delegate that decision.

Will identity/grants slow broad adoption?

Possibly. But many once‑complex things become default—compressed files, columnar formats, soon encryption. With collaboration (like Variant), we’ll land broadly accepted patterns.

Chapters

00:01:30 — Guest welcome and interview start

00:02:00 — Russell’s path: DataStax Cassandra, Spark connector, interoperability

00:05:20 — Joining Apple’s Iceberg team and early Iceberg momentum

00:06:20 — Why migrations resonated: replacing bespoke Hive/HDFS compaction/locking

00:09:10 — Apache governance 101: PMCs, consensus, and corporate influence

00:15:40 — How decisions land without votes; when vetoes apply

00:18:30 — Who needs Iceberg and where it fits

00:22:20 — Lake → lakehouse and warehouse → lakehouse in the cloud era

00:25:20 — Iceberg versions: v1 transactions, v2 row‑level ops (GDPR), v3 types

00:28:10 — Standardizing Variant across vendors and Parquet

00:31:10 — Iceberg v4 goals: streaming commit/retry improvements and AI use cases

00:33:40 — Commit latency and server‑side conflict resolution

00:37:20 — Polaris as an Apache incubating project (PPMC)

00:39:30 — Iceberg REST catalog spec and Polaris implementation

00:42:30 — Clarifying Snowflake Horizon, Polaris, and external REST catalogs

00:45:10 — What Polaris aims to be best at; pluggable identity providers

00:48:00 — Identity scope: business vs. system catalogs and credential vending

00:51:00 — Will identity/grants slow mass adoption?

00:52:50 — Wrap‑up

This newsletter is sponsored by dbt Labs. Discover why more than 60,000 companies use dbt to accelerate their data development.

Demo on-demand

In this episode of The Analytics Engineering Podcast, Tristan talks with Lauren Anderson, who leads the enterprise data platform at identity company Okta. Lauren discusses how identity sits at the center of two seismic shifts in data—AI agents and the open data lake—and why central governance and a shared semantic layer are critical. She lays out how analytics engineers and data engineers should divide responsibilities as agents begin to write a growing share of analytical queries.

A lot has changed in how data teams work over the past year. We’re collecting input for the 2026 State of Analytics Engineering Report to better understand what’s working, what’s hard, and what’s changing. If you’re in the middle of this work, your perspective would be valuable.

Take the survey

Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.

Listen & subscribe from:

• Spotify

• Apple Podcasts

• Amazon Music

• TuneIn

• Youtube

• RSS feed

Key takeaways

Tristan Handy: Before we dive into the current day, can you share a little bit about your background and how you came to the role that you’re in today.

Lauren Anderson: I’ve had a 20‑something year career at this point. I have basically spent my entire career in analytics some way, but my first data job was at a big bank. I won’t name it. There’s only a few big banks you could probably guess. I worked for the finance org and I did compensation planning and administration, with a side of sales tracking and analytics. I was part database analyst, part customer support for people that made a lot more money than I did.

I was there for seven, seven and a half, eight years. Towards the end of it, I became the owner and creator and almost business architect for our brand‑new sales tracking data warehouse. At a very young age, I got to think about how relational databases should come together for the outcome of both analytics and reporting—dashboards and whatnot—but also operations, which was paying compensation every month. It got me super excited about this world of data and being able to architect pipelines and the end‑to‑end flow for real‑world outcomes.

What do you think allowed you to be successful in that era? I often think the things that enabled success then aren’t the same as what make data folks successful today.

When I took it over, we ran compensation out of an Access database. I was new, the person who designed it left, and there wasn’t much documentation. It worked the first month, then broke the second—right before a payroll deadline. I rebuilt it as a long series of SQL queries with inline comments and step‑by‑step checks that produced a clean file. That willingness to throw away the brittle thing and rebuild with clarity and documentation gave me early success. The meta‑skills:ability to learn, take chances, and figure out the best path—still apply, but the technology is completely different now.

You’ve split time at Okta into two stints. How would you characterize the work?

Okta was my first truly B2B company. I realized quickly B2B data is my sweet spot. I love thinking about customers as businesses and how business users interact with our products and features. Okta data is complex—many products, features, and highly configurable use cases—especially with large customers. That variety is exciting. In simpler retail flows you see a lot of the same patterns; in B2B, the variety is the appeal.

What’s your current role?

I lead our enterprise data platform, engineering, and architecture function. For enterprise data used to make business decisions, we own ingestion into the warehouse, transformations, and delivery—dashboards, reverse ETL to third‑party applications, other data stores, and internal apps.

How big is the central function and how do you engage with the business?

We’re about 50 people across data engineering and analytics/data science in a company south of 7,000 employees. We support every business unit. Engagement spans a maturity curve. One end is platform self‑service: teams land data via approved connectors, build transformations in dbt on our implementation, and build dashboards in Tableau we administer. Governance and roles are defined centrally, and teams assign people to those roles. The other end is a white‑glove model where we partner through the full lifecycle—question, discover existing assets, requirements, data work, build, interpretation, validation, and end‑of‑life of the data product. Our sweet spot is the middle: we own enterprise “gold” pipelines for company‑level metrics—monitored and governed—while domains build and later graduate via a path‑to‑production under stronger governance.

Okta is known for identity and security. How does security‑first actually work in practice?

Reinventing controls every time slows you down. We invest in repeatable frameworks. Any new source goes through third‑party risk review, classification, and decisions on masking or exclusions. We help teams through that; after a couple times, they can engage directly with risk while we stay in the loop and monitor. As our classifications and expectations got clearer, review cycles shrank from weeks to days. It’s not all roses—it takes time—but we all operate as security practitioners. That shared mindset builds trust and reduces corner‑cutting.

How much do users need to know?

We don’t expect everyone to know everything. We provide dbt frameworks and minimum testing standards, plus SMEs to guide teams. The culture is to ask when unsure.

Will agents write more analytical queries than humans in the next 12–24 months?

Macro, yes. For us, more like 24–36 months because we’re careful. The key is safe, ethical AI consistent with being a security company.

How are you thinking about agent access?

Central governance. Ideally, agents query centralized, agent‑ready stores. Run governance once: policies, roles for users and for data, tracking and logging on a central plane. The semantic layer is essential. Creating semantic views must get easier and more automated, and semantics should inform policy application.

Why are agents different from humans in access patterns?

Row‑level security to the extreme. Conversational intelligence data should be limited to what the requesting user can access. Aggregations could be broadly accessible with anonymization, but detailed content should remain constrained. You might also limit allowed functions on large unstructured objects. Identity for agents matters—Okta Secures AI looks at distinct identity patterns to secure agents across applications.

Where are you with MCP and agent building?

Early, building support and insight use cases. Progress is fast, but nothing broad in production yet.

How should analytics engineers and data engineers participate?

Analytics engineers should own semantics—tooling, vendor choices, onboarding use cases, and the shared business language. Data engineers should optimize for consistency and scale, notice overlap across agents, and provide a platform others can build on with confidence in governance and security.

Will you standardize an agent development platform?

Yes, in partnership with engineering and shared services. Our current pull skews to the business, so we’re leaning toward accessible, governed platforms that serve both business and engineering with central governance.

Any assumptions you’re rethinking?

Treating everything like a relational model. Many initial agent questions are intentionally simple, where speed and reasonable accuracy trump perfect sophistication. The important thing is to start, observe, and mature.

Chapters

00:02:28 — From bank analytics to owning a sales DW

00:05:00 — Rebuilding brittle Access → SQL with documented checks

00:08:30 — Ops accountability then vs. optimization today

00:11:00 — TripIt, marketing analytics, and moving into tech

00:13:14 — Why B2B data became Lauren’s sweet spot

00:16:00 — Current role: ingestion → transform → delivery at Okta

00:18:10 — Operating models across business units and the path to production

00:22:20 — Security-first in practice: repeatable frameworks over friction

00:24:23 — Third‑party risk, classification, and shrinking review cycles

00:28:00 — Policies, masking, and the need for a central governance plane

00:30:20 — Frameworks for dbt, testing, and SME guidance

00:32:11 — Will agents outwrite humans? Macro yes; Okta timeline nuance

00:33:48 — Central governance and agent access patterns

00:37:19 — Semantic layer as bridge and policy carrier

00:41:00 — Function limits on unstructured data and Okta Secures AI

00:42:35 — Early MCP experimentation and support use cases

00:43:03 — Roles: analytics engineers (semantics) and data engineers (scale)

00:46:10 — Enabling an org-wide agent platform with shared governance

00:47:43 — Solve governance once, serve business and engineering

00:49:30 — Simpler questions first; rethinking relational assumptions

This newsletter is sponsored by dbt Labs. Discover why more than 60,000 companies use dbt to accelerate their data development.

Demo on-demand

In this episode, Snowflake VP of Product Management Chris Child joins Tristan to unpack Snowflake’s AI roadmap and what it means for data teams. They discuss the evolution from Snowpark to Cortex and Snowflake Intelligence, how to govern agents with row- and column-level controls, and why Snowflake is investing in Apache Iceberg and the Open Semantic Interchange initiative. dbt Labs recently open sourced MetricsFlow, the technology that powers the dbt Semantic Layer, to align with the goals of OSI.

Chris also shares a vision for the next five years of data engineering: fewer bespoke pipelines, more standardization and semantics, and a bigger focus on business context and data products.

Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.

Check out the dbt VS Code extension

Listen & subscribe from:

• Spotify

• Apple Podcasts

• Amazon Music

• TuneIn

• Youtube

• RSS feed

Key takeaways

Tristan Handy: Where have you spent your time professionally?

Chris Child: I didn’t end up in data on purpose. I found myself here through a series of hops. I was working at Redpoint Ventures and got excited by a company we invested in, RelateIQ. I left to join RelateIQ, building an intelligent CRM. We captured emails and meetings and built profiles of everyone you interacted with. We were acquired by Salesforce. Looking at what sales teams needed, I realized they also needed product usage data, marketing data, and campaign data, with a platform to pull it all together. That led me to Segment. I joined when it was about 50 people. Segment was mostly analytics.js then, loading different JavaScript on your webpage for tracking. We had just built the first warehouse connector to Redshift and got huge usage sending click and user data to Redshift.

The original Redshift connector was a nightmare to work with.

Like many startup things, one engineer built it in a week. Suddenly a ton of people used it, and enterprise customers depended on it. We had to rebuild it several times. You could see the future there. Folks I worked with went on to start companies like Census and Hightouch, thinking the CDP should be built on top of the warehouse, which Segment evolved toward. We also built a Snowflake connector because customers demanded it in addition to Redshift.

It’s funny to think back a decade to how small Snowflake was.

A couple customers demanded it; we built it, and we were sending a ton of data. That led to the realization that a customer data platform is one instance of a data warehouse, and there are others you need. Seeing how fast Snowflake was growing, I wanted to build the next layer of infrastructure.

I joined Snowflake seven and a half years ago. I’ve had three key roles. First, I built areas of the product: the UI, billing, product-led growth engines and free trial infrastructure, and application capabilities for connecting into and building on Snowflake. After Sridhar became CEO, he asked me to reconnect product and sales by leading solutions engineering, reporting to the CRO. Leading a global technical seller org was very different for a product person, but it helped align teams at scale.

About eight months ago, I returned to lead data engineering: how people bring data into Snowflake, how they transform it—spending a lot of time with dbt—and work around Iceberg and interoperability for worlds where not all data sits in Snowflake.

I didn’t realize the path started in investing. Are you a finance person way back?

My undergrad is in computer science. I started programming in fifth grade on an Apple IIe, learned C before high school, and followed that thread. In college I noticed business folks often made the decisions. I wanted to learn that side. After college I joined a consulting firm, then private equity, then an MBA. I realized I didn’t want to be a finance person. I moved to venture as a bridge to building products, but I wanted to build, so I jumped into operating roles.

Tell the story of Snowflake and AI. In the 2010s there was huge demand for easier, scalable, cloud-oriented data solutions. Then 2022 happened, ChatGPT launched, and the world changed. How did Snowflake respond, and where are you today?

Even pre‑2022 we saw customers putting their most important business data into Snowflake, then pulling data out for things they couldn’t do inside: training ML models and other analyses that SQL wasn’t a great fit for. Customers told us they didn’t like losing governance and lineage when data left. We invested in ways to bring more of that work to Snowflake.

Snowpark was the first big step: a runtime for non‑SQL code (Python, Java, Scala) with APIs inspired by Spark, plus capabilities like forecasting. It’s great for some workloads, but most customers don’t train most ML models inside Snowflake yet. We also acquired Applica for document extraction using early LLM techniques, and Neeva for web search based on LLM approaches.

When ChatGPT arrived, we saw two major influences. First, people wanted to chat with data they’d brought into Snowflake and transformed with dbt. That’s hard because LLMs are great with unstructured data and less great at turning business questions into correct SQL. Second, LLMs are very good at writing code, including Python and even dbt code. They’re not perfect for data engineering code yet, but they help.

Our goal is to help customers activate important enterprise data safely in AI models, deploy agents at scale under existing governance, and keep up with exploding data volumes without 10x headcount.

What are the key product pieces—Cortex, Snowflake Intelligence, etc.—in the Snowflake AI stack?

First, you need a great data foundation. That isn’t new: get the data in one place, apply good governance and permissions, know your data, tag PII, and raise the standard of care.

AI raises the bar because agents can expose sensitive data faster than dashboards. OSI (Open Semantic Interchange) work is part of this; LLMs need explicit semantics and cataloging they can consume, not tacit knowledge hidden in downstream tools.

Companies with strong hygiene move faster with AI. Roles matter; if a product manager role has access to certain rows and columns, an agent acting within that role can safely answer questions. Agents can run inside or outside Snowflake, but should assume appropriate roles when querying.

On the AI stack, after the data foundation, Cortex provides higher‑level APIs for unstructured processing, RAG, and structured processing. You can choose models (OpenAI, Anthropic, Mistral, Gemini, Llama, etc.), but most folks don’t want to manage prompts and GPUs. Cortex AI SQL lets you express intent like sentiment filters or fuzzy joins. It’s powerful for exploration but non‑deterministic, so you need care in production. Costs map to tokens at higher abstractions, with budgets and guardrails similar to variable compute in the cloud.

At the top, Snowflake Intelligence is a UI and agent framework. You define agents with access to specific datasets and semantic models, plus gold queries and usage guidance. It looks like a chat interface over your governed data. Inside Snowflake, we’ve deployed a GTM assistant that blends product usage, Salesforce, notes, docs, and content—structured and unstructured—respecting row‑level security for every seller while giving leaders broader access.

Let’s talk open formats and Iceberg. Why lean in when it opens up the data?

Our aim isn’t to lock up data, it’s to help customers get value. Snowflake began as a reaction to Hadoop—betting on SQL at cloud scale with our own formats and catalog because they didn’t exist then. Those proprietary pieces let us evolve quickly. Iceberg is now almost as good, and we’re contributing to make it better.

Openness is a win for customers and expands the universe of data Snowflake can query, run Cortex on, and power Intelligence with. The tradeoff is standards move slower. Variant type support is a good example—we contributed our approach and shepherded it into the v3 spec.

Next up, the community is wrestling with fine‑grained access control beyond table‑level policies. It’s hard and will take time, but the outcome should be better for everyone.

Give us your view on the future of data engineering.

Data volume is exploding, including unstructured data that’s now usable. You can’t hand‑build every pipeline. Demand is also exploding as agents query more things in more ways. Teams must operate at a higher level: automate, standardize, and reduce bespoke pipelines.

Expect more shared semantic models across consumers and packaged semantics coming from systems like SAP. You’ll also build data‑engineering agents to do work and monitor pipelines. The role looks more like architect and manager, allocating budgets, deduplicating work, and—most importantly—deeply understanding the business. The best data engineers shift from code output to data products, with clear semantics and context.

Talk more about context.

The day‑to‑day activity shifts, but the output is still data products. Great data products come with instructions, definitions, lineage, quality expectations, and how to get correct answers to common questions.

We need that context captured where work happens—models, visualization, quality systems—and made available everywhere: catalogs, agents, and UIs. As you build, you should also document, and those semantics should flow consistently into tools like Snowflake Intelligence so agents can reason correctly.

A big part of the challenge is selecting just‑enough context per question.

Chapters

• 00:01:50 — Chris’s path: RelateIQ, Segment, Snowflake

• 00:05:40 — Roles at Snowflake: product, solutions engineering, data engineering

• 00:09:00 — Snowflake and AI: foundations before ChatGPT

• 00:11:40 — Why keep ML and non-SQL work closer to governed data

• 00:13:40 — Applica and Neeva acquisitions, enterprise search context

• 00:14:50 — Two big AI influences: chat with data and code generation

• 00:16:50 — Scaling agents while preserving governance and cost controls

• 00:18:40 — Why governance must live at the data layer (roles, rows, columns)

• 00:22:00 — Inside vs. outside Snowflake: how agents assume roles

• 00:23:02 — Cortex: higher-level APIs over many LLMs

• 00:24:06 — AI SQL: joins/where by intent and the non-determinism tradeoff

• 00:27:40 — Cost models, tokens, and guardrails

• 00:29:10 — Snowflake Intelligence: agents over a governed foundation

• 00:32:10 — Open formats and Iceberg: Why Snowflake leaned in

• 00:36:00 — Standards tradeoffs: variant type and community progress

• 00:38:40 — Fine-grained access control for Iceberg: thorny but necessary

• 00:40:40 — The future of data engineering: scale, unstructured data, agents

• 00:43:20 — No more bespoke pipelines; standardized models, and semantics

• 00:44:50 — Data engineers as architects and business partners

• 00:50:00 — Code vs. context: data products and shared semantics

• 00:53:10 — Capturing context where work happens (models, viz, quality)

• 00:55:00 — Selecting just enough context for agent reasoning

• 00:56:30 — Closing

This newsletter is sponsored by dbt Labs. Discover why more than 60,000 companies use dbt to accelerate their data development.

Book a demo

In this episode, Tristan Handy sits down with Chang She—a co-creator of Pandas and now CEO of LanceDB—to explore the convergence of analytics and AI engineering.

The team at LanceDB is rebuilding the data lake from the ground up with AI as a first principle, starting with a new AI-native file format called Lance and building upward from there.

Tristan traces Chang’s journey as one of the original contributors to the pandas library to building a new infrastructure layer for AI-native data. Learn why vector databases alone aren’t enough, why agents require new architecture, and how LanceDB is building a AI lakehouse for the future.

Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.

Check out the dbt VS Code extension

Listen & subscribe from:

• Spotify

• Apple Podcasts

• Amazon Music

• TuneIn

• Youtube

• RSS feed

Key takeaways

Tristan Handy: You’re the founder and creator of the Lance file format and LanceDB. Before diving into vector search and vector databases, tell us about your background.

Chang She: I love talking to analytics engineers because that’s my background. I started about 20 years ago in quantitative finance. As a junior analyst, you do a lot of data engineering and analytics, which got me into open-source Python. I became one of the co-authors of the pandas library—initially to solve my own problem of not wanting to do analytics engineering in Java or VBScript.

You worked for a hedge fund?

Yes, AQR.

Did they know you were contributing to pandas? Hedge funds aren’t known for open source.

My roommate and colleague at the time was Wes McKinney. He showed me a proprietary Python library he was working on. It was life-changing. I started using and contributing. He spent about six months convincing the fund to open-source it. This was around 2010, and they were ahead of the industry in that respect.

I didn’t know pandas started at AQR. That’s fascinating. So much of your circa-2010 analytics work was done in early pandas?

Exactly. We went through several iterations, even debated the name. Because it was a hedge fund, there was a lot of econometrics and “panel data,” so Wes named it “pandas” for panel data analysis.

That origin story isn’t widely known. You then founded two companies, sold one to Cloudera, and were there during an interesting time.

Wes and I created DataPad—cloud BI before cloud BI really took off—and sold it to Cloudera. I spent about four and a half years in the Hadoop “big data” world, where I met my co-founder. He worked on HDFS at Cloudera, and several ex-Cloudera folks are at LanceDB today. After that I moved into machine learning at Tubi TV, working on recommender systems, ML serving, and experimentation/AB testing. That exposed me to embeddings. We dealt with videos, poster art images, and synopses—data that doesn’t fit neatly into pandas or even Spark data frames. That inspired me to build better infrastructure for these data types—what we now call “classical” machine learning—which led to LanceDB.

So that’s our bridge to vectors. You experienced these problems at Tubi, then founded the company. And Tubi used dbt?

Heavily. Thank you for creating it—it was critical to our stack.

Give us a non-technical intro: what are vectors used for?

Many people focus on the latest models and techniques. My perspective: everyone has access to similar models—your differentiation comes from your data and how effectively you connect data to AI. Vectors are a way to represent any kind of data in a form models understand: high-dimensional arrays of floating-point numbers—1,500, 3,000 dimensions, etc. Early statistical models might have a few interpretable dimensions; now you can have thousands where individual dimensions aren’t necessarily interpretable, but the space captures semantics.

Beyond RAG, vectors power internal model representations, recommender systems, and personalization—the original mainstream use case.

Search is also a good use case. How is vector search different from full-text search or Command-F?

Full-text search (e.g., Elasticsearch) returns documents containing the exact terms you searched. If you search for “customer,” it finds “customer/customers,” but might miss “user,” “adopter,” “organization,” etc. Vector search uses dense representations where semantically similar words and documents live near each other in high-dimensional space. Search for “customer,” and you get results that include semantically related terms.

Would you combine vector and full-text search?

Yes—hybrid search. Early RAG demos often used pure vector search for speed. Now enterprises need production-grade relevance. Many combine keyword and vector search with a re-ranking step to reach higher precision/recall.

Early RAG pipelines often chunk text, embed, and call it done. But more thoughtful pipelines do something closer to feature engineering, right?

Absolutely. Thought goes into what you feed the embedding model. For example: add a document- or section-level summary alongside each chunk before embedding; include multimodal features—artistic descriptions, literal captions, tags; create multiple embedding columns (e.g., different prompts/modalities) and search across them with re-ranking. High-quality retrieval requires feature-engineering-like decisions before embedding.

Let’s talk vector file formats (Lance) and vector databases (LanceDB). My crude belief: a vector database is a standard database with additional indexes. True?

Not wrong, but my hot take: with Lance and LanceDB, we’re building a lakehouse for multimodal data that includes vectors. Many “vector databases” are optimized only for vectors and struggle with other data types and workloads. The category needs to evolve—either toward new-generation search engines or new-generation lakehouses. We set out from day one to build the broader lakehouse, not just a vector index.

Outline your AI-enabled data lake vision. I’m familiar with Snowflake and Databricks’ lakehouse. How do you see the world differently?

We assumed everyone would use Parquet and tried for months to support AI workloads—search, training, preprocessing—on it. We couldn’t make it work well. Talking to computer-vision and ML practitioners, no one had something effective. That gave us confidence to build a new format.

In AI you manage vectors, long documents, images, and videos. The first problem is storage. With Parquet, mixing wide blob columns with narrow metadata columns leads to out-of-memory issues due to row-group design. If you shrink row groups to fit blobs, read performance tanks.

Even once data is in Parquet, AI needs random access and secondary indexes. Parquet doesn’t support efficient random row access: retrieving scattered rows forces reading entire row groups. With media, that’s prohibitively expensive—both for search and for training (e.g., global shuffle). Data evolution is also hard: with table formats like Iceberg, backfills often mean copying entire datasets. Copying petabytes of media is a non-starter. These issues motivated Lance.

I have a good mental model of Parquet with structured data. With images or video, do you put them in blob columns?

Yes. We use Apache Arrow types. Images/audio/video are large binary columns. Vectors are fixed-width list columns (e.g., 1,536-dimensional). But Parquet’s row-group mechanics and lack of random access make these workloads painful.

So Lance was the first thing you built. It has solid traction on GitHub. Who uses a file format—users or vendors?

Both. Frontier labs use Lance to store training data—e.g., for image/video generation—replacing stacks like TFRecords, WebDataset, Parquet, and BigQuery. Large tech companies and vendors also build on Lance: Databricks, Tencent, Alibaba, Netflix, NVIDIA, Uber, among others.

Databricks uses Lance?

For parts of their AI-specific offerings.

You’ve raised several rounds—the format is Apache-2 licensed. How do you commercialize?

Our commercial offering is a data platform for large-scale AI production: vector search, data preprocessing, training/serving cache, and an analytics engine for curation and exploration. It supports ML training workflows and AI application development, solving the hard distributed-systems problems along the path. We partner closely with big vendors; we’re generally not competitive because goals and customer bases differ. Cloud providers seek platform consumption; we focus on an AI-optimized data platform for specific workloads and users.

The commercial product is called LanceDB, but you prefer to position it not just as a database.

Right—we’re an AI-native data platform/lakehouse for multimodal data, with Lance as the common format.

How does this space play out over the next two to three years?

Two big predictions. First, multimodal will be 100× bigger—more usage and more data. Audio is exploding; video generation is resurging; robotics is next. Second, our data infrastructure isn’t ready for agents driving search and retrieval.

Let’s unpack both. On multimodal: unlike structured analytics, where every company needs it, multimodal workloads seem concentrated. Do all enterprises really need this?

I think every enterprise becomes multimodal. Take insurance: tons of documents to digitize, extract, search, and analyze; drones capturing images/video to assess risk and improvements over time. Existing businesses become more efficient; AI-native entrants gain structural advantages. Multimodal data underpins both.

It’s a heavy lift. Will every Fortune 500 insurer build these capabilities in-house, or will vendors package them?

Likely both—just like analytics engineering emerged as a role, with adjacent talent re-skilling. We see the same with AI engineering.

What titles are hands-on with your product?

AI researchers and AI engineers. Many app developers building AI features now carry the “AI engineer” title.

On agents: how do their access patterns change platform requirements?

RAG was one-shot: ask, retrieve, answer. Agents iterate: they decompose problems into sub-questions, refine queries and results, and run many steps in parallel. Load skyrockets—humans type slowly; agents can issue hundreds of queries simultaneously. Queries are more varied and selective, and agents are creative in combining modalities and sources: schemas, SQL over structured data, prior analyses and charts, document stores, image/video metadata, etc.

Traditional vector databases aren’t designed for this breadth and scale. If you bolt together multiple specialized systems, your “agent stack” balloons into a maintenance nightmare. Our approach: put all data in one place with a single system that supports vector search, keyword search, filters, key-value lookups, re-ranking, analytics, and efficient random access—on top of an AI-native file format (Lance).

For listeners whose curiosity is piqued, any resources you recommend?

Chang She: Yes—our blog series by Weston Pace, the tech lead for Lance format. It dives into encodings, I/O, and has great reads for analytics engineers: lancedb.com/blog .

Chapters

• 00:00 – Intro: Analytics meets AI

• 03:20 – Chang’s background and how Pandas began

• 06:40 – Lessons from Cloudera and metadata

• 08:30 – Multimodal data and LanceDB’s origin story

• 10:00 – Why vector search matters (beyond RAG)

• 12:00 – What are vectors and why do we use them?

• 15:00 – Full-text vs vector search

• 18:00 – Feature engineering in AI use cases

• 21:15 – Lance format

• 28:00 – Storage, scale, and the problem with Parquet

• 35:30 – Building a business on open source

• 41:00 – Two big bets: multimodal data and agents

• 46:00 – Every company will become multimodal

• 50:00 – Agent access patterns will redefine data

• 54:00 – Why dbt-style workflows matter now more than ever

This newsletter is sponsored by dbt Labs. Discover why more than 60,000 companies use dbt to accelerate their data development.

Book a demo

What does agentic coding look like in analytics engineering? Mikkel Dengsøe, co-founder at SYNQ, recently wrote a series of posts on his experiences as an analytics engineer with agentic coding tools. In this episode of The Analytics Engineering Podcast, he walks through a hands-on project using Cursor, the dbt Fusion engine, the dbt MCP server, Omni’s AI assistant, and Snowflake.

Tristan and Mikkel cover where agents shine (staging, unit tests, lineage-aware checks), where they’re risky (BI chat for non-experts), and how observability is shifting from dashboards to root-cause explanations delivered to the right person at the right time. Along the way: practical prompts, why “one model at a time” keeps you in control, and a testing philosophy that avoids alert fatigue while catching what matters.

To see real-world use cases of agentic coding and to learn directly from data and AI leaders, join us at Coalesce 2025 in Las Vegas, Oct. 13-16.

Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.

Listen & subscribe from:

• Spotify

• Apple Podcasts

• Amazon Music

• TuneIn

• RSS feed

Key takeaways

Can you talk a little bit about your background?

Mikkel Dengsøe: Yeah, so I can start from the beginning. I've been in data for, I think it's coming up to 15 years now, and started my career in data at a Danish shipping company, which was very much zero to one. When I came in, there was no data warehouse, and the only way we could know how many containers were shipped was by an IT guy pulling that out of the system every six months. I then spent two years there building up their data warehouse on SQL Server, which was super fun. After that, I spent five years at Google, which was a very different gear.

That's a natural transition. Just global shipping company straight to Google.

Exactly. And that was very much a hundred-to-end where, in my case, I worked with the ads data and you get a perfectly curated data table that you can work with and everything kind of works. Then after that I joined a company called Monzo. For those who are not familiar, it's a scaling fintech out of the UK and that was very much the one to a hundred. When I joined we were 30 data people, but scaled to a hundred over two years. We had 10,000 dbt models and we built every internal tool under the sun for dbt. Super interesting. And then three and a half years ago I went on to found SYNQ alongside Peter and Steve, which is a data observability platform.

Tell us a little bit more about SYNQ.

We are a data platform that primarily works with companies using tools like dbt already, but have issues going from important data to business-critical data. That might be customer-facing dashboards, machine learning models, or something else. They want better monitoring—we often deploy anomaly monitors—and they also want workflows such as incident management for when things go wrong. We were founded in 2022, so now we're in early stages of working with scale-ups and startups, and now also onboarding enterprises and larger companies. It's been a fun journey.

In your series of blog posts, you went through the modern data stack and said, “What's the most current version of this tool and how effectively can I AI-ify that?” Whether that's using Cursor to build dbt models or using the agent experience inside of Omni—what made you decide to get into this and write about it?

The first part of it is just: it's super fun to tinker with these tools and try them out. It's magic. And we were also building an MCP server at SYNQ, so I had a lot of interest in seeing how it works with others and what we can learn. It was also driven by being able to have conversations with our customers. When they ask about it, being able to speak from the point of view of having actually tried this and seen what works and what doesn't.

The early days of using Redshift were such a visceral experience relative to what came before. If I hadn't interacted with it directly, I wouldn't have understood how big a state change cloud data was. This feels like another one of those moments: if you don't have hands-on experience, you're not going to really get it. Fair?

Spot on. And I think pretty much every data team should be doing this unless they have a very good reason not to. The risk and the stakes can be pretty low if you use it for internal workflows like data modeling and writing tests. You're still in control. I recommend everybody do it.

What tasks did you try to accomplish?

It's three different blog posts: the data modeling part, the testing part, and then exposing it in Omni's AI agent where people can ask questions about the data. There's a fourth post: once the data is live, how can you use the SYNQ MCP to do things like root-cause analysis and planning changes. I started with data modeling. I had raw data from different JSON sources, some XMLs, some profiles—extracted and put into Snowflake—and then did the data model.

So the data was already loaded into Snowflake?

Yeah, exactly. For the data modeling, I started from the sources and then worked through staging, marts, and finally metrics using the semantic layer. Each step looks a little different when you use AI tools because the behavior differs. In terms of tooling, I used Cursor with the dbt-MCP plugged in. If you're not familiar, dbt-MCP lets you, via prompt, interact with dbt tools—execute dbt build, get models, or get everything upstream of a given model—so you can chain work without explicitly doing it.

Cursor + dbt-MCP. What model did you use?

I just used the default in Cursor, which I believe is Claude. There's an important distinction: Cursor is really good at writing code, but it can't execute queries on your behalf. If you want to extract raw data and query Snowflake to get rows out, you have to do that in Claude Desktop. That became key. Early on, as I built models, the first thing I did was get a snapshot of sample data from Snowflake—10,000 rows of a source. I fed that into Cursor and said, “These are examples of what this data looks like.” Using that data, Cursor could model in a clever way. For example, a column called quarter like “2025 Q1”—Cursor understood to translate it into a datetime and do the transformations.

I've used the dbt MCP server a decent amount—less in Cursor, more in Claude Desktop. Your stack was Cursor + Claude models + Claude Desktop. And Cursor cannot directly execute queries in Snowflake, but Claude Desktop can. Is that because there’s tool use Claude has that Cursor doesn't?

I believe so. In Claude Desktop, if you write queries against dbt-MCP, Claude can visualize a graph, show outputs of a SQL statement, etc. Cursor, as far as I know, couldn't. My middle ground was to take sample data out of Snowflake, put it into a CSV, and feed that back into Cursor so it could look at raw data.

As part of its own context window?

Exactly. That was key for my workflow. Then when I wanted to write unit tests, I could use real data examples from the sample. Or when automatically documenting the data, I asked Cursor to specify examples in the docs based on the most common occurrences within a column. Letting Cursor peek at raw data was a core pillar.

It's a little hacky, right? Cursor should really be able to interact directly with Snowflake or Databricks to investigate the shape of the data. Agents should be empowered to do that.

I would say so. There might be a way I didn’t know about, but I patched the gaps by uploading into the context window.

So that's the state of the art today.

Seems so. To be clear, I think the limitation is IDE differences—Cursor vs. Claude Desktop—rather than dbt-MCP itself.

Once you had sample data in context, did you have to suggest conversions, or did it naturally do them?

It got the defaults pretty right, but I guided it on what I wanted from the source data. I wanted control over everything, so I asked it to do one model at a time rather than auto-generate a whole stack. That way I could review each step and stay in control.

Your prompt workflow was “Build me a model with this name that stages the data from this table,” basically?

Yeah. When it proposed code I didn't like, upstream it was usually simple (regex to parse dates, etc.). Downstream, in marts and metrics, I started describing my ideal data product: user jobs-to-be-done and the final output. That’s when Cursor got creative and invented metrics I hadn’t anticipated—like “apartment price relative to time on market.” I pruned ones I didn’t want, but some were good surprises.

Which layer did it help most?

Testing. Modeling was good—especially staging—but testing accelerated significantly. SQL is a bit like English; for simple datasets you can express intent easily. Testing can be much harder and more verbose.

Roughly how much more effective did you feel?

Modeling: multiples faster. It nailed the tedious parts—regex, casting, pass-throughs—so staging/intermediate layers flew. In marts/semantic metrics, the benefit was brainstorming. It helped me think of metrics I wouldn't have.

Did the dbt Fusion engine help?

Yes. Fusion shows lineage and whether a column is pass-through. For example, if a column is pass-through with no transforms, don't add another not_null or unique if there's one upstream. I bounced between the IDE to check this and codified it as a testing strategy. That's already top-10% testing hygiene.

Any MCP feature requests surface?

The more context and tools the agent has, the more it can do. In the fourth post, for root cause analysis, we used the SYNQ MCP. We collect all your Git commits and have history, so the agent could correlate recent code changes with incidents. Requests depend on the job at hand.

Let's move to testing—why was it the most additive?

Testing is hard; many teams don't know how to do it and alert fatigue is common. A huge share of tests we see are not_null/unique, which doesn't reflect real data risks. First thing I did in Cursor for testing was provide our internal testing philosophy as guidelines: test heavily at the source, don't retest pass-through columns, focus on business and metric anomalies in marts. That worked really well. For sources and staging, it generated relevant tests. Then for marts, I asked for unit tests and gave it a thousand sample rows from Snowflake. It wrote very relevant unit tests I’d otherwise spend a lot of time on.

Examples?

Simple ones like: when you pass a string value in the date column, does it transform correctly to datetime and match the expected format? These just worked. Then at the metric level, it looked at raw data and proposed assumptions—like square-meter price should be between X and Y—sometimes segmenting by postcode. Very thoughtful, though I'd replace static thresholds with anomaly monitors so they don't go stale as prices move.

So at least 5× on testing?

At least. Apart from swapping static thresholds for anomaly detection, it nailed testing and did so in a lineage-aware, layer-appropriate way.

Tell me about the BI layer.

Many teams start at the BI layer with a chat interface. I think that's risky because it's used by business users and you only get so many chances before trust drops. I moved into Omni. You create a “topic” (a data model you can join with others) and then specify an AI context: instructions for how the LLM should behave. For example: if a user asks about price, always return square-meter price; never make up fields not present in the mart; if asked about provenance, mention the source. Writing AI context is a new skill for our industry.

Were you using Omni’s AI assistant to create assets faster, or to let users self-serve?

The latter—so users could ask questions instead of going to a dashboard. It could have been any BI tool with similar functionality; we just use Omni internally.

And how was the experience as a consumer?

Amazing when it works, but I'd hesitate to give my VP of Marketing access. It gets things wrong maybe one in five times, and it's not obvious why if you're not a data person. For analysts doing exploratory work, it's great—they can inspect and dig in. I wouldn't replace company-wide dashboards with a chat bot yet. Omni does log freeform queries and feedback, so there's a path to iterate the AI context over time.

The last thing you did was use AI plus SYNQ to monitor production infrastructure. What does observability look like in the future? Historically it's looked like dashboards—Datadog for data pipelines. Is it just more effective monitors, or fundamentally different?

Fundamentally different. We’re heading to a place where observability tools can tell you what's wrong at the right time, with just the right context, delivered to the right person—inside or outside the data team. Done well, there may be few dashboards; instead you get an LLM-summarized root cause delivered from a monitor that might be auto-created. Less “active tool you poke at,” more “proactive explanation.”

Still technical observability (pipelines/data issues), or business observability?

More the former. Teams at the edges—Sales Ops managing Salesforce, engineering teams creating web events—often need to be notified about data issues. Business KPI movements require a different experience for marketers, etc.

Automated remediation?

Gradual. You can imagine an issue occurs without a dedicated test; the system proposes a new test. But 80% of issues come from root systems elsewhere (someone typing in Salesforce), and closing that loop is still hard. In the article’s fourth part, we had a data issue and I asked the SYNQ MCP through Claude Desktop to do root cause analysis. It walked the same steps a data person would: inspect the model, check errors, examine lineage and upstreams, review recent commits, and documented each step to the root cause. That works now.

At the beginning you said there’s no good reason not to use these tools today. What reasons do you hear for not trying?

People are busy. But if you look at a risk curve, lowest risk is modeling and testing—you're in the driver's seat and it boosts productivity. Higher risk is replacing your BI tool with a chat bot; higher still is customer-facing experiences. The first two are hard to argue against.

Enterprise IT approvals might be one blocker—approved models, data access, etc.

True. For example, our MCP can query raw data to detect if an issue happens in a segment, and enterprises might hesitate there. Also, “MCP” as a term can be confusing. But it's actually simple and explainable, not a black box. Setting up dbt-MCP can still feel hacky in enterprises; if it lived natively in cloud environments, it’d be easier to adopt.

You can set it up locally—no permissions/procurement—and just play. We also shipped the MCP server as a remote MCP in cloud, though that introduces auth/permissions considerations.

If I had to pick a persona, it's the analyst. Analysts have had a tough decade: more tools, harder workflows, less time to tinker. MCPs and AI workflows are a turning point. At Monzo, we had a philosophy that you should be able to have an idea on your commute and have it implemented by midday. As we grew to 10,000 dbt models and long CI checks, that faded. I can see a world where this returns. MCPs can help. I'm excited.

I love that. Analytics engineers think “infrastructure, correctness.” Analysts think “idea to validation fast.” Excel was always the analyst’s best friend because it's fast and flexible. MCPs make it easy to plug tools together and get answers quickly again.

One company we work with—Voi, a scooter company out of Sweden—has a strong data leader, Magnus, who is very bought into metrics. Their data team doesn't produce dashboards; they produce metrics. In an AI world with MCPs, flows, and curves, that's a clear decision.

I believe there's no such thing as the wrong BI tool—different tools have different trade-offs. Probably true for models/IDEs too: Claude Desktop vs. Claude Code vs. Cursor—no single “right answer” as long as the underlying context and metric definitions are shared.

Agreed. What really matters across workflows: consistent metric definitions, documentation for columns and fields, and high-quality data. Those foundations matter even more when an LLM is in the loop; you may not have a human sanity-checking every result.

Chapters

• 00:00 — Tristan’s intro

• 01:10 — Mikkel’s background: shipping → Google → Monzo → SYNQ

• 03:08 — What SYNQ does (data observability for business-critical data)

• 04:15 — Running the experiment

• 06:23 — Scope: modeling, testing, BI agent, observability

• 07:17 — Tooling: Cursor + dbt MCP server + Snowflake + Omni

• 09:38 — Sampling real data into the agent’s context

• 13:14 — Modeling workflow: one model at a time

• 15:14 — Where agents help most: testing > modeling

• 18:10 — dbt Fusion engine: lineage-aware checks, fewer redundant tests

• 19:50 — Feature requests and root-cause via commit history

• 20:57 — Testing philosophy: source-heavy, pass-through aware, metric-level

• 22:49 — Unit tests from samples; thresholds vs anomaly monitors

• 25:10 — BI agents: great for analysts, risky for broad rollout

• 31:54 — The future of observability: explain first, dashboards second

• 36:10 — Adoption curve: safe places to start

• 40:49 — Analyst superpowers return

• 42:04 — Metrics over dashboards

This newsletter is sponsored by dbt Labs. Discover why more than 60,000 companies use dbt to accelerate their data development.

Book a demo

If you're a data practitioner, you likely understand Iceberg as a user, why it's important, and how it's changing the way that we build data systems. But you may not know a lot about what going on beneath the surface.

There are multiple ways to interface with Iceberg catalogs, multiple versions of the Iceberg REST spec. There's several leading catalogs that implement that spec. All this in an ecosystem that includes companies of all sizes, in proprietary and open-source code, and in academic and commercial contexts.

In a few years, all this ambiguity will be behind us, but right now it's very much evolving in real-time. To get an update on the status of the Iceberg ecosystem and to walk through all the developments, Tristan talks with Christian Thiel. Christian is one of the lead architects of Lakekeeper, of one of the most widely used Iceberg catalogs.

To learn more from some of the leaders in the Iceberg ecosystem, join us at Coalesce 2025 in Las Vegas, Oct. 13-16.

Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.

Listen & subscribe from:

• Spotify

• Apple Podcasts

• Amazon Music

• TuneIn

• RSS feed

Key takeaways

Walk us through your background

Christian Thiel: I started in natural language processing, then moved into machine learning applications in manufacturing. Like many people, I realized that the biggest barrier wasn’t the algorithms but the data—its availability, quality, and accessibility. That led me deeper into data architecture and engineering, eventually to building Lakekeeper.

What is Lakekeeper, and what are you building now?

Lakekeeper is an Iceberg catalog implementation—a technical requirement for building distributed, composable analytic systems based on Apache Iceberg. But our vision goes beyond that. We see the future in data collaboration and reliable sharing of data, supported by clear contracts.

For listeners new to Iceberg, what makes it so important?

Iceberg allows organizations to store data once, in an open format, and then use the compute engine best suited for each workload. It’s a foundation for building modern, composable data platforms while avoiding vendor lock-in. If there’s one thing that should be open, it’s the data at the center of your platform.

Some folks might say this sounds like Hadoop all over again—lots of open standards that are hard to integrate. Why is this time different?

The ecosystem has matured. Even big vendors like Snowflake and Databricks are embracing Iceberg, which shows there’s a strong shift toward openness. Plus, the tooling and infrastructure are much easier to deploy today. A modern Iceberg setup is far less complex than a Hadoop environment used to be.

Let’s talk about what’s happening under the hood. How does Iceberg work?

Iceberg organizes data using a metadata hierarchy. At the top, there’s a JSON file that stores high-level table information: snapshots, schema, and locations. Below that are manifests and other layers that keep track of files. This hierarchy is what makes things like time travel, atomic transactions, and schema evolution possible.

What about ongoing maintenance?

There are two key tasks. First, expiring old snapshots so you don’t accumulate unnecessary files. Second, compaction—combining many small files into larger ones

Catalogs are another critical piece. What role do they play?

Catalogs manage the top layer of metadata and coordinate transactions. They make atomic updates possible, allow multiple writers, and handle governance—things like access control and multi-table transactions.

How enterprise-ready is Iceberg today?

Very ready. A year ago, there were still gaps, but today, performance and feature parity with native tables on platforms like Snowflake and BigQuery are strong. Governance and authorization models are still evolving, and different catalogs implement them differently, but the core functionality is there.

Speaking of catalogs, how should someone pick between options like Lakekeeper, Polaris, Unity, AWS Glue, or Gravitino?

Christian Thiel: It depends on priorities. Lakekeeper focuses on performance, extensibility, and ease of use. Polaris is developer-focused but less user-friendly. Unity is tightly integrated into Databricks. Glue now supports the Iceberg REST spec, which makes it more interoperable than before. Gravitino is another option aimed at enterprise-scale environments.

Recently, DuckDB announced DuckLake. What’s your take on that?

It’s interesting, but there are two concerns. First, it uses a database schema directly for the catalog, which creates interoperability issues—similar to the early JDBC catalog in Iceberg that the community eventually moved away from. Second, it was built without community involvement, and openness without adoption isn’t really openness.

That said, for heavy DuckDB users, it could offer optimizations that make queries extremely fast, and if the broader ecosystem adopts it, it could become a viable open format.

What’s next for Lakekeeper?

We’re continuing to invest in table optimization, enterprise features, and data collaboration tools. Our vision is what we call the “unbreakable lakehouse,” where contracts and collaboration guardrails make shared data more reliable. Long-term, we see Lakekeeper as enabling truly collaborative, open data ecosystems.

Chapters

• 00:00 – Introduction

  Tristan Handy introduces the episode and the focus on Apache Iceberg.

• 01:40 – Christian Thiel’s background

  From natural language processing to data engineering.

• 04:30 – Introduction to Lakekeeper

  What Lakekeeper is and its role in the Iceberg ecosystem.

• 06:00 – Why Iceberg matters

  How open table formats enable flexibility and reduce vendor lock-in.

• 11:40 – How Iceberg works under the hood

  Metadata hierarchy, catalogs, and how state is managed.

• 21:30 – Maintenance and optimization

  Snapshot expiration, compaction, and keeping tables performant.

• 24:20 – Catalogs and governance

  Access control, multi-table transactions, and security.

• 31:40 – Enterprise readiness

  How Iceberg is evolving for production use in large organizations.

• 42:10 – Choosing the right catalog

  Overview of Lakekeeper, Polaris, Unity, Glue, and Gravitt.

• 47:20 – DuckLake discussion

  Pros, cons, and ecosystem adoption challenges.

• 52:00 – The future of Lakekeeper

  Data contracts, collaboration, and building the “unbreakable lakehouse.”

This newsletter is sponsored by dbt Labs. Discover why more than 60,000 companies use dbt to accelerate their data development.

Book a demo

In this episode of The Analytics Engineering Podcast, Tristan Handy talks to Sean Falconer, senior director of AI strategy at Confluent, about AI agents. They discuss what truly makes software "agentic," where agents are successfully being deployed, and how to conceptualize and build agents within enterprise infrastructure.

Sean shares practical ideas about the changing trends in AI, the role of basic models, and why agents may be better for businesses than for consumers. This episode will give you a clear, practical idea of how AI agents can change businesses, instead of being a vague marketing buzzword.

Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.

Listen & subscribe from:

• Spotify

• Apple Podcasts

• Amazon Music

• TuneIn

• RSS feed

Key takeaways

Sean, can you give us the TLDR on your career and what you're working on today?

Sean Falconer: I've always worked at the intersection of data, engineering, and AI. From academia studying computer science, into industry as a founder, then to Google, I worked on conversational systems and privacy/security in AI. Currently, at Confluent, I'm leading our AI product strategy, balancing both technical and go-to-market roles.

You moved from being deeply technical into marketing and sales. What drove that transition?

I was forced into it as a founder. Initially uncomfortable, but it taught me huge respect for marketing and sales. I had to learn by making many mistakes, eventually building out entire marketing and sales functions. I realized how challenging and critical these roles are.

You were at Google before ChatGPT launched. Did you foresee the transformative nature of these technologies?

Honestly, no. Having seen earlier disappointments in conversational AI (like Microsoft's Alice), I was skeptical initially, even as ChatGPT emerged. It wasn’t obvious we'd soon experience this revolution.

You’ve written about three waves of AI. Can you describe these?

Yes. Wave one was predictive AI, traditional ML models trained for specific tasks like fraud or spam detection—effective but rigid. Wave two introduced generative AI, or foundation models, trained on vast general datasets, flexible but lacking specific business context. The third wave, agentic AI, involves AI systems that can reason, dynamically choose tasks, gather information, and perform actions as a more complete software system.

Do foundation models replace traditional ML methods?

Sometimes they can, but it doesn’t always make sense. An LLM might do sentiment analysis well enough, but a traditional model may be more efficient and cheaper. Think of using an LLM as cutting steak with a chainsaw—possible, but unnecessary.

Let's clarify "agents." What makes software truly agentic?

It’s software that can dynamically decide its own control flow: choosing tasks, workflows, and gathering context as needed. Realistically, current enterprise agents have limited agency to ensure reliability. They're mostly workflow automations rather than fully autonomous systems.

You mentioned a spectrum of agency. Is this similar to autonomy in self-driving cars?

Exactly. Highly autonomous agents are appealing but not practical yet. Most enterprise success stories involve structured workflows with clearly defined boundaries.

Why have agents taken off more in enterprises than consumer apps?

Enterprises have many well-defined, high-value tasks perfect for automation. Consumer scenarios demanding high agency—like planning complex trips—are still too unreliable. Enterprises can benefit significantly even from limited agentic capability.

Is an agent just a microservice?

In many ways, yes. An agent functions like a microservice with extra capabilities (using LLMs for decisions). Deployment considerations like state management and long-running tasks differ slightly, but fundamentally it’s similar.

What tools and frameworks help build effective agents?

Start with frontier models like GPT-4 or Claude. Frameworks include LangChain, Microsoft Autogen, and CrewAI. But for real-world deployment, treat it as rigorous software engineering with observability, scalability, and robustness in mind.

Are organizational barriers bigger than technical challenges?

Yes. AI efforts are often mistakenly tasked to data science teams rather than cross-functional software teams. Successful companies create dedicated teams blending software engineering skills and data expertise to build reliable agentic systems.

What pitfalls should teams avoid?

Avoid building monolithic agents. Break systems into smaller, well-defined units in a multi-agent architecture. Use event-driven frameworks to avoid rigid, hard-to-maintain dependencies.

Chapters

• [00:00] Introduction: What's all the hype about agents?

• [01:10] Meet Sean Falconer: A journey from engineer to AI strategist

• [04:10] Learning marketing as an engineer-founder

• [05:50] Inside Google's AI efforts before ChatGPT

• [09:00] What does it mean to run AI strategy?

• [10:45] Three waves of AI: Predictive, Generative, and Agentic

• [16:30] Will foundation models replace traditional ML?

• [18:30] Defining agents clearly: Beyond the buzzword

• [22:00] The spectrum of agency: From controlled workflows to open-ended tasks

• [25:30] Why agents fit better in enterprises than consumer apps

• [28:00] Agents as microservices: A practical view

• [35:00] What tech stack is needed to build effective agents?

• [37:50] Organizational challenges in adopting agents

• [39:30] Models that are favorites for developers

• [43:30] Why software engineers are best placed to build agents

• [46:00] The technical stumbling blocks in building agents

• [48:00] Concluding thoughts: Beyond POCs to production agents

This newsletter is sponsored by dbt Labs. Discover why more than 60,000 companies use dbt to accelerate their data development.

Book a demo

In this episode, Tristan talks with Andy Warfield, VP and senior principal engineer at AWS, where he focuses primarily on storage. They go deep on S3, how it works, and what it unlocks. They close out talking about Iceberg, S3 table buckets, and what this all suggests about the outlines of the S3 product roadmap moving forward.

Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.

Listen & subscribe from:

• Spotify

• Apple Podcasts

• Amazon Music

• TuneIn

• RSS feed

Key takeaways

Operating systems, garage sales, and Xen

Tristan Handy: You’ve done a lot over the last 20 years. Before we get into specifics, can you just share a little about your journey as a software engineer?

Andy Warfield: I just like playing with computers. I studied computer science in Ontario for undergrad, then moved to Vancouver for grad school, then to the UK for a PhD. I worked on operating systems, low-level stuff. I got to work on a hypervisor called Xen, which ended up being used by a lot of cloud providers, including Amazon.

After that, I did a couple of startups, one around Xen. Then I became a professor at UBC, teaching operating systems, networking, and security. Later, I did another startup in storage, and eventually I joined Amazon.

Now I have this highfalutin role—VP and engineer—working across S3, other storage services, and now a bunch of analytics services too. I get to cause trouble in lots of different parts of the cloud.

VP slash distinguished engineer—does that mean you just get to march around telling people how to improve their stuff?

People love that! I’d say about half the time I’m causing trouble—starting things and encouraging new ideas—and the other half I’m helping teams dig out from those ideas. Sometimes I take over a team if we’re doing something especially interesting or innovative, just so I can be closer to the action.

That sounds like a pretty good gig if you can get it.

It’s amazing. I’ve been here nearly eight years, and I still love this job.

The rise of virtualization and the origin of Xen

I want to talk about Xen. You said you were always interested in operating systems, which is kind of a niche fascination. What drew you in?

When I was a kid, we didn’t have much money, so I built computers from garage sale parts in Ottawa. In high school, I found this federal government warehouse that sold off old equipment. I started a little business buying pallets of hardware for cheap, fixing them up, and reselling.

It was chaotic—but I learned a lot. I dealt with machines like IBM DisplayWriters with 8-inch floppy disks and massive dot-matrix printers. Getting them working meant diving into their software and systems.

Eventually I played with Linux, hacked on the kernel, and that all led me into OS research and development.

Tristan: So what is a hypervisor, and why did virtualization become so important in the 2000s?

Andy: There were two big drivers: server utilization and isolation.

Companies had racks full of 1U servers, most of which sat idle most of the time. But they couldn’t share workloads because apps weren’t isolated well—config conflicts, shared resources, etc.

Virtualization allowed multiple operating systems to run on the same hardware, with isolation. It also let you consolidate servers, which had big cost and efficiency benefits.

There was also a technical challenge: x86 processors weren’t designed to be virtualized. That made it a really interesting research problem. We wanted to see if it could even be done—and done efficiently.

Tristan: And Intel eventually started building virtualization support into the hardware?

Andy: Exactly. Our work on Xen and similar projects showed it was possible. That pushed Intel and AMD to add features like VT-x, which made it easier and more performant to run hypervisors.

Tristan: How did AWS end up using Xen?

Andy: I wasn’t part of those internal conversations, but the story goes that a small startup in Cape Town, South Africa, was building a control plane for Xen. That team got picked up by AWS and became the basis for EC2.

Understanding Amazon S3

Tristan: Let’s switch to S3. I think a common mental model is that S3 is just a big pool of SSDs. But that’s clearly not the whole story. How do you explain what S3 actually is?

Andy: That’s one of my favorite questions.

Early on, S3 was like a storage locker. You’d rent space to stash things you didn’t need right away—backups, static files, CDN origins. Latency wasn’t great, but durability and availability were.

Things really changed when the Hadoop community built S3A—an adapter to let Hadoop use S3 instead of HDFS. Suddenly, we had people doing real analytics on S3. The system had enough drives to support massive parallel reads.

Today, workloads are way more demanding. Performance, consistency, and latency matter. We’ve been evolving the system constantly to meet those needs.

Tristan: Are we talking about billions of hard drives?

Andy: I can’t share exact numbers, but yes—it's a lot of hard drives. Some of our largest customers have data spread across millions of drives. And most drives are shared across multiple customers.

Tristan: And these aren’t SSDs?

Andy: Mostly spinning disks, actually. Hard drives are terrible at latency, but they’re cheap and good for bursty workloads. Spreading your data across many disks lets you take advantage of parallelism.

S3’s durability, performance, and scale

Tristan: Let’s talk about S3’s durability promise: 11 nines. How do you achieve that?

Andy: We use erasure coding—a form of RAID-like redundancy that lets you split data into parts and parity blocks. Then we store those shards across different availability zones.

We constantly monitor for failures. Disks die all the time, so we have fleets of processes repairing and maintaining durability. It’s not static. It’s a living system.

Tristan: You must have incredibly precise failure models.

Andy: We do. We track failure rates, temperature sensitivity, vendor behavior—everything. That allows us to be proactive and surgical in how we manage risk.

From Parquet to Iceberg to S3 table buckets

Tristan: I want to talk about table formats. Parquet is everywhere now. And then we got Hive Metastore, then Iceberg. Why did S3 launch table buckets?

Parquet is great, but it’s just files. Customers kept asking for more structured semantics: schema evolution, upserts, ACID transactions.

We saw Iceberg adoption grow rapidly—especially among our largest analytics customers. But they were struggling with operational complexity: too many small files, custom compactors, brittle catalogs.

So we launched S3 table buckets to bring native Iceberg support to S3. That includes:

• Automatic compaction

• A REST catalog

• High-performance access

We wanted to make it easier to treat Iceberg as a storage primitive, not just an analytics backend.

So this is a shift in philosophy—S3 isn’t just object storage, it’s now table-aware?

Exactly. Historically, S3 was just where you stored objects. Now, we’re thinking more about what those objects mean.

We also launched S3 object metadata tables—a way to semantically describe and query your object store, especially useful for AI workloads using retrieval-augmented generation (RAG).

The future of open data and S3

What does the future of S3 look like? Where’s this going?

We’re headed toward more structure, more semantics, and more performance.

Inference workloads are scaling fast. AI models are hitting S3 hundreds of thousands of times per second to do vector lookups. That’s changing how we think about indexing, metadata, and latency.

We want to make S3 the best place to do open, flexible, high-scale data work—from tables to training data to retrieval.

Chapters

[01:42] Meet Andy Warfield

Andy shares his background, including startups, professorship, and his current role as VP & Senior Principal Engineer at AWS.

[05:10] From garage sales to hypervisors

Andy describes his early passion for hardware, OS development, and the origin story behind the Xen hypervisor.

[08:50] Why virtualization took off in the 2000s

Exploring why isolation, utilization, and technical curiosity fueled the rise of hypervisors.

[14:30] Xen vs. VMware and the road to AWS

How Xen became the default for EC2 and the technical differences between virtualization approaches.

[17:35] The origin of EC2 and S3

How a team from Cape Town helped launch AWS compute—and the early days of cloud services.

[20:00] What is S3, really?

Andy breaks down the mental model behind S3: not just object storage, but a scalable data platform.

[22:49] How many drives? More than you think

Why S3 storage spans millions of drives—and how AWS uses scale to deliver performance.

[28:10] The 11 nines durability model

Inside S3’s approach to reliability, failure tolerance, and background repairs using erasure coding.

[32:00] Tail latency and engineering for bursty workloads

Why slow requests matter, and how S3 teams optimize for streaming, AI, and analytics use cases.

[35:20] Iceberg, metadata, and table buckets

The emergence of Apache Iceberg as a table format—and AWS’s new structured storage approach.

[38:00] Why S3 added a REST catalog and compaction

How AWS is simplifying the operational burden of working with Iceberg at scale.

[40:00] A new mental model for object storage

S3 is no longer just about storing files—it’s about managing semantics, lineage, and trust.

[44:00] Looking ahead: S3, RAG, and semantic metadata

How S3 is preparing for the next wave of AI, inference, and context-aware applications.

[47:20] Is Iceberg ready for enterprise?

Andy shares thoughts on enterprise readiness, performance tradeoffs, and real-world adoption of table formats.

[49:05] Wrap-up and reflections

Tristan and Andy reflect on the conversation and where data infrastructure is headed next.

This newsletter is sponsored by dbt Labs. Discover why more than 50,000 companies use dbt to accelerate their data development.

Book a demo

Specifically, I gave myself a challenge - could I go from a never-before-seen dataset to a production-ready Semantic Layer using a combination of tools:

• An agentic coding CLI (I used Claude Code for this experiment)

• The dbt MCP server

• A terminal interface (in this case Warp)

Before we go any further, if this is at all interesting to you, I suggest that instead of reading my findings here that you sit down and try this yourself. I'm quite confident you'll find it both illuminating and worth your time.

We'll get to my findings in a bit. Long story short - it was successful enough that it shifted my thinking about the near-term trajectory of data work.

But first, let's talk about why experiments like this matter so much right now.

Sensemaking in the age of AI

You've probably been hearing some variant of these takes multiple times a day:

"An agent is just an LLM run in a loop"

"AI agents are coming to replace white collar work"

"I don't even know what an AI agent is, this is just marketing hype"

And about a billion more. All of these represent our collective attempts at sensemaking in this unique technological moment. But honestly, the noise can be so overwhelming that it's tempting to just tune it all out and wait for the dust to settle.

I don't think that's an option for data practitioners. Instead, we need to develop our own internal compass for sensemaking - and that means getting our hands dirty.

To do great data work is to be a great sensemaker. My theory of sensemaking requires holding two paradoxical skills in tension:

• Build strong mental models about the world and use them to take decisive action

• Constantly scan for misalignments between your models and reality, then adjust accordingly

Organizations and institutions need time to metabolize change and adjust their mental models. There's a physics to it. And that physics takes time.

But when the underlying reality is changing rapidly, the best thing you can do is go make direct contact with that reality. Don't wait for the consensus to form - go see for yourself.

Because things are not the same as they were even 6 months ago:

• We've gotten the first wave of models optimized for agentic work (OpenAI’s O3, Claude 4 and Gemini 2.5)

• We've started building real infrastructure to connect these models to our systems (MCP and other emerging protocols)

• LLM-based coding has shifted from autocomplete to actual agents (something longtime Roundup readers saw coming)

That’s a bunch of big changes! It can sometimes feel like keeping up with everything here is a full time job. And with my last couple months being pretty tied up with other things I felt like I owed it to myself to set aside some time and go deep here.

The experiment: Two hours from zero to Semantic Layer

I chose the weather source dataset on the Snowflake marketplace precisely because it was both interesting and completely unfamiliar to me. I booted up Warp (dbt MCP server already configured - that might add additional time here) and got started.

In two hours, I went from raw data to a working dbt project1 with:

• Documented source definitions

• Tested data models

• A functional Semantic Layer with queryable metrics

It felt incredible. A bit unbelievable. Of course this was just a simple project and nothing in here would be particularly difficult for an experienced analytics engineer - but it would have taken a whole lot of time and effort.

Some observations from the process:

The experience was exhilarating. Watching an abstract goal decompose into concrete tasks, then seeing those tasks execute in real-time feels like witnessing something total new. It was also addicting - in this interface has a “just one more level” feeling of playing a great video game.

The cognitive load is different. It was cognitively demanding but not in the same way that coding is cognitively demanding - I have a sense that I’d be able to sustain longer blocks of “pairing” with Claude code before getting mentally depleted than normal coding.

The tools aren't optimized for data work yet.

• It first attempted to build out a bunch of models that depended on each other, but didn’t check if the first model actually ran. Then there was an error part of the way through it’s dependency and we had to do a bunch of unthreading.

• It’s competent at writing SQL (and dbt-style SQL). I don’t expect this to be the bottleneck for AI augmented development.

• It is not very good at understanding what columns or models it has access to at a given time - I expect this to be an area where the models will be most useful when assisted by deterministic tooling.

What this proved (and didn't prove)

This experiment convinced me that agentic workflows have moved beyond “pure speculation” and into “definitely worth exploring and net useful for many teams today”. It feels pretty similar to the earlish days of coding assistants like copilot. Not yet for every team but definitely for some and on a steep acceleration curve.

This was just a simple experiment and I walked away thinking just as much about what I don’t know as what I learned.

• I still don't know if my models are logically sound (validation would take as long as building)

• Enterprise-scale datasets might break this approach entirely

• The actual utility of what I built remains untested

• And even with all of this, there are just as many organizational bottlenecks that the data team face as technical. What implications does this have there (if any).

But here's the thing: in two hours, I accomplished what would have taken me at least a full day manually - not just the modeling, but documentation, testing more. That is worth paying attention to.

Your move

When facing a question as vast as "How will AI reshape data work?", it's easy to get paralyzed. But the answer isn't in think pieces or Twitter debates - it's in running experiments.

My mental model shifted because I made contact with reality. Right now, data teams not using agentic workflows are doing just fine. But things are moving fast. It’s worth it, at the very least to get a sense of what the state of the world is here and think about how you might adapt to it.

So here's my challenge: Block two hours next week. Pick a dataset you don't know. Try to build something real with these tools. Report back and let me know.

The future of data work is being written right now, in thousands of small experiments by practitioners who refuse to wait for the dust to settle. If you're reading this, you have the expertise to contribute to our collective sensemaking.

What will you discover when you stop reading about AI and start building with it?

In this episode, Tristan talks to Solomon Hykes, the founder and creator of Docker. They trace Docker’s rise from startup obscurity to becoming foundational infrastructure in modern software development. Solomon explains the technical underpinnings of containerization, the pivotal shift from platform-as-a-service to open-source engine, and why Docker’s developer experience was so revolutionary.

The conversation also dives into his next venture Dagger, and how it aims to solve the messy, overlooked workflows of software delivery. Bonus: Solomon shares how AI agents are reshaping how CI/CD gets done and why the next revolution in DevOps might already be here.

Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.

Listen & subscribe from:

• Spotify

• Apple Podcasts

• Amazon Music

• TuneIn

• RSS feed

Key takeaways

Tristan Handy: I want to get you to give a little background on yourself, where you've been, what you've been up to for the last couple decades. I think many people will know you as the person who kicked off an avalanche that changed how we interact with compute environments by inventing Docker?

Solomon Hykes: Docker is the thing I'm known for. Pre-Docker, I grew up in France. I studied programming in a French school called EpiTech. It was a brand-new, unconventional school where you learned through nonstop programming, which I loved.

Eventually, I got exposed to startups, despite being a complete outsider. I met someone who told me about them, and it stuck in my mind. Still in France at the time, I moved into my mom's house in the suburbs of Paris and worked out of the basement.

By complete luck, I got into an early version of Y Combinator in 2010. That got us on the path to what would become Docker three years later. In 2013, we pivoted to Docker from our previous company, dotCloud.

Tristan Handy: The original thing was called dotCloud, right?

Solomon Hykes: Yep. It was about container technology and its potential, but we didn't quite know how to take it to market. DotCloud was about deploying and hosting people's apps—platform as a service—competing with Heroku and many clones.

Tristan Handy: When did Heroku become a thing?

Solomon Hykes: I became aware of it in 2009. Just as I was struggling in France with container tech. When we joined YC in 2010, we packaged that tech into dotCloud, our hosting platform. Our differentiator was using containers under the hood when others didn’t. That let us support many language stacks and even run databases in containers—which was unheard of at the time.

Platform as a service was a tough business. Most startups went out of business or got acquired early. Eventually, we pivoted from selling the car to building an ecosystem around the engine—that became Docker.

Tristan Handy: Did you pivot because selling the car wasn't working? Or because people kept pointing at the engine saying, “Give me that”?

Solomon Hykes: Both. It was hard to market platforms. Developers expected free hosting, and hosting costs money. Margins were tight because of AWS. It always felt like pushing a boulder uphill. Meanwhile, people wanted to run things locally. There was no good ecosystem for that. Docker provided transparency, flexibility, and portability.

Tristan Handy: Can you define Docker and containerization, and how it differs from virtualization?

Solomon Hykes: Sure. Virtualization splits a physical machine into virtual ones using VMs—each with its own memory, compute, and storage. It gives flexibility, but with overhead.

Containerization does something similar but at the operating system level. Instead of virtualizing the machine, you split the OS itself. It’s mostly done with Linux, which can subdivide itself into isolated units. Containers are more lightweight, letting you run hundreds or thousands, unlike VMs where you might manage a handful before hitting limits.

Docker didn’t invent this, but we solved new problems with it.

Tristan Handy: I remember creating my first Docker container around 2015. I expected a slow boot-up like a VM, but it was instantaneous. Where is the OS in that setup?

Solomon Hykes: Great question. Docker relies on Linux. When you're on a Mac, it runs Linux behind the scenes—today via virtualization. Back then, we used lots of early, rough tools and kernel patches to make Linux containers work. Docker put all the pieces together in a coherent way.

Tristan Handy: So containerization wasn’t new, but Docker made it accessible?

Solomon Hykes:
Exactly. The Linux kernel had features like namespaces and cgroups—building blocks for containers. But they weren’t user-friendly. We made a developer-centric abstraction on top of those tools.

And Linux provided a massive compatibility layer. Unlike Java, which required writing your app in Java, Docker containers could wrap apps written in any language, as long as they ran on Linux.

Tristan Handy: So Docker is like infrastructure as code—a primitive that enables the whole concept?

Solomon Hykes: Yes! And because we wanted ubiquity, we avoided pushing too many opinions. We let developers build on top of it in many different ways. That’s what helped Docker become a de facto standard.

Tristan Handy: How fragmented is the Linux world under the hood? Did you have to do much abstraction work?

Solomon Hykes: We were lucky. The Linux kernel is extremely stable and consistent. But everything above it—distros, package managers, tooling—was chaotic. That chaos created the opportunity for Docker to provide a consistent experience.

Tristan Handy: Were there any drawbacks? Like “Docker sprawl” the way VMware saw VM sprawl?

Solomon Hykes: Definitely. With power comes chaos. Teams would run dozens of Docker containers, each configured differently. Docker doesn’t enforce opinions—by design.

Tristan Handy: And what happened when you left Docker in 2018?

Solomon Hykes: I took time off, became a full-time dad. But I also realized how many unsolved problems remained. Especially around CI/CD pipelines and software delivery—what we now call the software factory.

That led me to start Dagger.

Tristan Handy: So Dagger is like “containers for pipelines”?

Solomon Hykes: Yes. Just as Docker standardized app deployment, Dagger aims to standardize and containerize software delivery. CI/CD pipelines today are often duct-taped together with YAML and bash scripts. We’re bringing consistency and modularity to that space.

Tristan Handy: Will there be a “Daggerfile” like there’s a Dockerfile?

Solomon Hykes: Sort of. But this time, we’re opinionated. Dagger is narrowly focused on CI/CD. That lets us provide APIs, SDKs, and a deeper abstraction stack. We give platform engineers a DAG-based system to define repeatable, containerized steps.

Tristan Handy: And what’s the role of AI and agents in all this?

Solomon Hykes: Great question. We didn’t plan for it, but our community showed us the way. People started building AI agents that run in Dagger pipelines—automating things like writing tests, submitting PRs, and optimizing builds.

That blew our minds. Agents blur the line between development and delivery. They need programmable environments. Dagger is becoming an ideal platform for that.

Chapters

01:30 – Early Days: From France to dotCloud

Solomon shares how his early programming experience and startup journey led to the creation of dotCloud.

04:00 – The PaaS Struggle and Birth of Docker

The team pivots from platform-as-a-service to focusing on the container engine itself—what would become Docker.

07:00 – What Is a Container, Really?

Solomon explains containerization vs. virtualization in plain terms and why it changed the game for developers.

11:00 – The Developer Experience That Won the World

The magic of fast, lightweight Docker containers—and how that first “wow” moment felt.

14:00 – Building a Ubiquitous Standard

Why Docker stayed narrow by design, resisting feature bloat to maximize compatibility.

18:00 – DevOps Before DevOps

How Docker avoided language tribalism and achieved mass developer adoption by choosing Go and CLI-first tooling.

21:00 – Complexity and Container Sprawl

Docker made infrastructure easy—but created new operational challenges at scale.

24:30 – Why CI/CD Pipelines Are Still Broken

Solomon outlines the gap Docker never got to fix: modern software delivery remains brittle and ad hoc.

27:00 – Enter Dagger: DevOps for the Modern Age

How Solomon’s new company is treating pipelines as composable software, not brittle scripts.

30:00 – Building an OS for the Software Factory

Dagger helps platform teams manage the complexity of software delivery with reusable, testable components.

33:00 – Agent-Native Workflows: A Surprise Use Case

AI agents begin using Dagger to reason about pipelines, generate code, and submit pull requests autonomously.

37:00 – Reimagining the Dev Loop with AI

Why the boundary between development and CI/CD is collapsing—and how Dagger fits the agent-powered future.

41:00 – Scaling Trust in Delivery

Tristan and Solomon reflect on how developer tooling evolves and what a stable, fast delivery layer enables.

45:00 – Final Thoughts: What’s Next for DevOps

The conversation closes with predictions on intelligent automation, composability, and the future of platform engineering.

This newsletter is sponsored by dbt Labs. Discover why more than 50,000 companies use dbt to accelerate their data development.

Book a demo

Lonne reflects on the evolution of ETL, the unexpected staying power of legacy tech, and why AI may finally erode the switching costs that have long protected incumbents. The future of the AI and standards era is bright.

Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.

Listen & subscribe from:

• Spotify

• Apple Podcasts

• Amazon Music

• TuneIn

• RSS feed

Episode chapters

00:46 – Meet Lonne Jaffe: background & career jurney

Lonne shares his career highlights from Insight Partners, Syncsort/Precisely, and IBM, including major acquisitions and tech focus areas.

04:20 – The origins of Syncsort & sorting in mainframes

Discussion on why sorting was a critical early problem in hierarchical databases and how early systems like IMS worked.

07:00 – M&A as innovation strategy

How Syncsort used inorganic growth to modernize its platform, including an early example of migrating data from IMS to DB2 without rewriting apps.

09:35 – Technical vs. strategic experience

Tristan probes Lonne’s technical depth despite his business titles; Lonne shares his background in programming and a fun fact about juggling.

11:55 – Why this history matters

Tristan sets up the key question: what lessons from 1970s-2000s ETL tooling still shape the modern data stack?

13:00 – Proto-ETL: The real OGs

Lonne traces the origins of ETL to 1970s CDC, JCL, and early IBM tools. Prism Solutions in 1988 gets credit as the first real ETL startup.

15:40 – Rise of the ETL market (1990s)

From Prism to Informatica and DataStage—early 90s vendors brought visual development to what was once COBOL-heavy backend work.

18:00 – Why people offloaded Teradata to Hadoop

Exploring how cost, contention, and capacity drove ETL out of the warehouse and into Hadoop in the 2000s.

20:00 – Performance vs. price: Jevons Paradox in ETL

Why lower compute and storage costs led to more ETL, not less—and how parallelization changed the game.

22:30 – Evolution of data management suites

How ETL expanded into app-to-app integration, catalogs, metadata management, and why these bundles got bloated.

25:00 – Rise of data prep & self-service analytics

Tools like Kettle, Pentaho, and Tableau mirrored ETL for business users—spawning a whole “data prep” category.

27:30 – Clickstream, logs & big data chaos

How clickstream and log data changed the ETL landscape, and the hope (and letdown) of zero-copy analytics.

29:10 – Why is old software so sticky?

Tristan and Lonne explore the economics of switching costs, the illusion of freedom, and whether GenAI could break the lock-in.

33:30 – Are old tools actually… good?

Defending mainframes and 30-year-old databases like Cache. Sometimes the mature option is better—just not sexy.

36:00 – The new vs. the durable

Modern tools must prove themselves against decades of reliability and robustness in finance, healthcare, and compliance.

38:20 – GenAI in data: The early movers

Lonne highlights why companies like Atlan and dbt Labs are in the best position to win—distribution, trust, and product maturity.

41:00 – TAM and the Jevons Paradox, again

Revisiting how price drops expand TAM. Some categories vanish, others explode—depending on elasticity of demand.

43:15 – Unlocking new personas with LLMs

Structured data access for non-technical users is finally viable, but “it has to be right”—trust and quality remain the barrier.

46:00 – Real-world examples: dbt’s MCP server win

Tristan shares how dbt’s Metadata API became a catalog replacement for a traditional financial institution—an unplanned AI GTM success.

48:30 – Agents, not interfaces

New pattern: LLMs as agents interacting directly with infrastructure via APIs. Tool use is becoming table stakes for AI integration.

50:30 – Are LLMs birthright tools yet?

Discussion around adoption of ChatGPT Enterprise, Claude, etc. Lonne suggests adoption is accelerating fast—and the usage model matters.

52:00 – Looking ahead

The conversation ends with a reflection on GenAI’s near future in data workflows, TAM expansion, and what the next episode might tackle.

Key takeaways from this episode

Tristan Handy: You've had a long career in tech. Maybe start by giving us the 30,000-foot view of what you've been up to over the last couple decades?

Lonne Jaffe: I’ve been at Insight Partners for about eight years now, working mostly on deep tech investments—AI infrastructure companies like Run AI and deci.ai, both acquired by Nvidia. I’ve also done work with data infrastructure companies like SingleStore. Before Insight, I was CEO of a portfolio company called Syncsort, now Precisely. It was founded in 1968.

Prior to that, I was at IBM for 13 years, working in middleware and mainframe technologies. Products like WebSphere, CICS, and TPF—foundational systems for enterprise computing.

Tristan Handy: And Syncsort's origin was in sorting, right? Literally sorting files?

Lonne Jaffe: Exactly. In the early days of computing, sorting was a huge part of what you did. Much of the data was hierarchical—stored in IMS—and had to be flattened into files to process. The algorithms were optimized to run in extremely resource-constrained environments.

Tristan Handy: Fascinating. And I assume as compute and storage improved, the data integration landscape evolved?

Lonne Jaffe: Yes. We saw a move from hierarchical to relational databases, then toward ETL tools in the 80s and 90s. The first real ETL startup was probably Prism Solutions in 1988. Informatica and DataStage showed up in the early 90s, followed by Talend and others.

Tristan Handy: It seems like we got a whole bundle of tools over time—ETL, CDC, app integration, metadata, and so on.

Lonne Jaffe: Yes, often bundled together, even though data prep and app integration were treated separately. That persisted for longer than you'd expect. At Syncsort, we acquired a company with a "transparency" solution that allowed IMS applications to use data stored in DB2 without rewriting code—a clever way to manage switching costs.

Tristan Handy: Speaking of switching costs—why are these legacy tools so sticky?

Lonne Jaffe: Great question. In many cases, no customer loves the product. They’d switch in a heartbeat—if it were easy. But rewriting jobs and ensuring reliability is a heavy lift. The best outcome is a new system that replicates old functionality. And for many organizations, that’s not worth the risk.

Tristan Handy: But if generative AI could reduce those switching costs?

Lonne Jaffe: That’s the potential. Code generation, agents that explore and iterate—those could erode the moat that’s protected these incumbents for decades. Not tomorrow, but it’s a real possibility.

Tristan Handy: It also seems like some of these systems are more robust than people give them credit for.

Lonne Jaffe: Absolutely. Mainframes are IO supercomputers. Products like InterSystems Cache, used by Epic, are incredibly performant. But new systems must match or exceed those capabilities in reliability and scale, which is a high bar.

Tristan Handy: As you look at the evolution of the modern data stack, how do you think about its impact on the market?

Lonne Jaffe: In the 2010s, we saw disaggregation—tools like Fivetran, dbt, and Snowflake each tackled a slice of the old enterprise bundle. But the TAM isn’t infinite. Some categories may compress or vanish entirely if price drops aren’t offset by new demand.

Tristan Handy: Do you think AI expands or compresses the data stack?

Lonne Jaffe: It depends. High elasticity of demand—like with dashboards or analytics—can drive massive TAM expansion. But some categories, like logo redesign or simple data movement, might get commoditized. For more complex workflows, AI agents accessing platforms like dbt or Atlan could dramatically increase value by automating common tasks and enabling new personas.

Tristan Handy: We’ve seen an example already—a customer replaced their data catalog with our dbt Cloud metadata server and AI interface.

Lonne Jaffe: That’s telling. If AI interfaces can connect to tools like dbt and generate value—self-service, documentation, lineage—it changes the game. Especially for organizations already standardized on those platforms.

Tristan Handy: What’s your view on how these AI interfaces get distributed?

Lonne Jaffe: ChatGPT Enterprise, Claude, and others are spreading fast. Eventually, you’ll want those tools to search files, access internal metadata, and interact with your data stack—not just answer questions from the open web.

Tristan Handy: It makes a lot of sense. If AI is going to serve enterprise users, it needs access to the real data. Otherwise, it’s just a toy.

Lonne Jaffe: Exactly. A model that can’t query or verify against your actual environment won’t be reliable. And data quality and observability—something dbt Cloud is already good at—become foundational.

This newsletter is sponsored by dbt Labs. Discover why more than 50,000 companies use dbt to accelerate their data development.

Book a demo

Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.

Join Tristan May 28 at the 2025 dbt Launch Showcase for the latest features landing in dbt to empower the next era of analytics. We'll see you there.

Listen & subscribe from:

• Spotify

• Apple Podcasts

• Amazon Music

• TuneIn

• RSS feed

Chapters

• 01:00 – Introducing Warp and Zach Lloyd

  • Zach Lloyd explains Warp's origin, mission, and initial vision.

• 02:40 – Why redesign the terminal?

  • Zach describes why traditional terminal UX was ripe for reinvention.

• 04:43 – Enter LLMs: A new direction for Warp

  • Warp evolves into a natural language interface for developer workflows.

• 06:34 – What is a shell?

  • Zach defines shells, how they process text, and their role in the CLI ecosystem.

• 07:58 – Shells vs programs vs built-ins

  • Distinguishing between shell commands and standalone programs.

• 10:00 – Why do developers debate shells?

  • Features, syntax, and licensing behind the Bash vs Z Shell discussion.

• 12:17 – Why terminals still matter

  • The enduring power of text-based computing and scripting.

• 16:40 – What is a terminal, really?

  • Clarifying the difference between terminal hardware, emulators, and modern terminal apps.

• 20:13 – The Warp interface

  • Zach breaks down Warp’s UI: input editor, output blocks, and mouse support.

• 22:48 – Will Warp replace your IDE?

  • The vision of AI-driven development and the convergence of terminal, editor, and chat.

• 27:20 – Rethinking development interfaces

  • Finding the ideal hub for AI-native software development.

• 35:00 – Why the terminal has an edge

  • Advantages of the terminal for cross-project, full-lifecycle developer tasks.

• 37:10 – Bottom-up adoption strategy

  • How Warp approaches growth: focus on individual developers, not top-down mandates.

• 39:50 – Is Warp redefining the terminal?

  • The challenges of innovating in a legacy-dominated space and creating a new category.

• 42:45 – Developer control & context in Warp

  • Customization, context-awareness, and MCP integration in Warp’s AI tooling.

• 46:32 – Closing reflections

  • Zach and Tristan wrap up their thoughts on the future of terminals, AI, and developer tools.

Key takeaways from this episode

Tristan Handy: Can you tell us about Warp, where the idea came from, and where you’re at today?

Zach Lloyd: Warp reimagines the command line to make it more approachable, powerful, and useful for developers. I've been a software engineer for over 20 years and always used the terminal, but never understood why it worked the way it did. I used to learn the minimum I needed and rely on team members when I ran into issues.

After my last startup, I looked at tools I used frequently that could have a big impact if improved. The terminal stood out. I realized better UX—like being able to use a mouse to position the cursor or select output for copy-paste—could unlock a lot of productivity. That was the initial idea about five years ago.

We spent the first couple of years redesigning the interface. Today, Warp is more than a terminal—it's a natural language interface to the command line, powered by large language models (LLMs). You can use it to set up projects, write code, debug production, and more.

Tristan: I want to dig into fundamentals. Can you define what a shell is?

Zach: A shell is a program that parses text input, runs commands, and returns text output. You can run it interactively or through scripts. Terminals, by contrast, are the graphical layer that displays text and captures keyboard input. Shells like Bash, Z Shell, and Fish offer different features, syntaxes, and configurations. Some programs like cp are shell built-ins, which don’t require forking new processes.

Tristan: Why do terminals persist in a GUI-dominated world?

Zach: A few reasons. First, it’s easier to write command-line apps than GUI apps. Second, the interface is infinitely flexible—you can pass endless flags and parameters. Third, command-line programs interoperate cleanly via text streams. And lastly, they’re scriptable. Developers can automate repetitive workflows easily, which is powerful.

Tristan: So a terminal just runs a shell. But I never think of terminals as having features. What makes a terminal more than a simple interface?

Zach: Terminals emulate old hardware—keyboards and text displays. Today’s terminal apps are GUI shells that simulate this behavior. Most are "dumb terminals," just rendering characters. But they can support features like theming, control characters for advanced UI (e.g., in Vim), and even bitmap rendering.

Tristan: Warp looks very different. Can you describe it?

Zach: Warp looks more like a chat or notebook interface. Each command's output is grouped in a logical block instead of being dumped in a scroll. The input area behaves more like a code editor, with syntax highlighting and first-class mouse support. We're aiming for modern UX.

Tristan: So you're blending terminal, editor, and chat. Will people eventually write all their code in Warp?

Zach: My vision is that developers will increasingly describe what they want in natural language, and agents will do the work. Developers supervise the results. That interface needs to support managing many tasks at once. That’s what we’re building towards. It won’t even be called a terminal—it’s a new category of software.

Tristan: The boundaries between these tools are blurring. And maybe the best interface for AI-assisted development isn't an IDE or chat app—it could be the terminal.

Zach: The terminal spans all phases of development—from setup to deployment and debugging. It also supports cross-project work, which IDEs don’t. That’s a huge strength.

Tristan: But terminals are a personal choice. How do you think about adoption and your business model?

Zach: Like editors, terminals are developer-choice tools. We don’t go top-down. Our motion is bottoms-up: get individuals to love Warp, then expand into teams and enterprises for security, privacy, and data controls.

Tristan: Are you trying to reset the baseline for what a terminal is?

Zach: We're not open source, though we’ve considered it. It’s risky. But our focus isn’t on redefining "the terminal." It’s on building the best tool for developers to ship software. That might require a new category name.

Tristan: What’s the dev experience in Warp like? Is it customizable?

Zach: We support theming and shortcuts. But the most important part is AI context. Warp can use any CLI tool to gather context—GitHub CLI, GCloud, etc. We’re also implementing the Model Context Protocol (MCP) and plan to better support custom/internal tools as well.

This newsletter is sponsored by dbt Labs. Discover why more than 50,000 companies use dbt to accelerate their data development.

Book a demo

The conversation covers the evolution of compiler technology, what software engineering has gotten right over the past several decades, and why the data ecosystem is poised for similar transformation. Lucas and Tristan explore why SQL has lagged behind other programming ecosystems, and how new compiler infrastructure could lead to package management, interoperability, and greater innovation across data platforms. It’s a fascinating (and timely) episode: Get ready for the new dbt engine.

Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.

Join Tristan May 28 at the 2025 dbt Launch Showcase for the latest features landing in dbt to empower the next era of analytics. We'll see you there.

Listen & subscribe from:

• Spotify

• Apple Podcasts

• Amazon Music

• TuneIn

• RSS feed

Chapters

• 02:40 The vision behind SDF Labs

• 04:00 What is a compiler?

• 05:00 Components of a compiler: frontend, IR, backend

• 08:00 Syntax vs. semantics and the role of parsing

• 10:00 Logical vs. physical plans in SQL compilers

• 13:00 Historical context: mainframes to LLVM

• 16:00 Cross-architecture portability in Rust & other compilers

• 18:00 What is LLVM and why it matters

• 20:00 Bootstrapping and the self-recursive nature of compilers

• 21:00 Compilers in Java, TypeScript, and dbt

• 23:00 Why compilers are foundational to software ecosystems

• 26:00 The SQL dialect problem in data warehouses

• 29:00 Can SQL get its own LLVM?

• 31:00 How Substrate and DataFusion aim to standardize SQL

• 35:00 Package management and the path toward SQL abstractions

• 38:00 The future of the data ecosystem with a common SQL compiler

Key takeaways from this episode

What is a compiler?

Tristan Handy: What is a compiler?

Lukas Schulte: It's something that takes higher-level human-readable code and translates, compiles, rewrites it into lower-level machine code that is much harder for humans to understand and much easier for machines to understand.

Compilers typically have phases. They have a frontend that deals with the language you're working with, a middle component—usually called an IR or intermediate representation—and a backend that takes that IR and compiles it into machine code.

Compiler phases: frontend, IR, backend

Tristan Handy: How does it all come together?

Lukas Schulte: There’s a preprocessor that handles macros, removes comments, and prepares the text. Then a lexer converts it into tokens. These tokens get assembled into a tree that the compiler can understand. That’s where syntax validation and semantic analysis happen.

From there, we build a logical representation of the operations we want to perform. That transitions to a physical plan, which starts considering the hardware: how many cores, how much memory, which files we’re accessing. After that, optimizations are applied and it compiles to actual machine code using a toolchain like LLVM.

Syntax vs. semantics

Lukas Schulte: Let’s break down syntax vs. semantics.

Imagine the code x = x + 1. That has valid syntax. Its meaning—its semantics—is that we’re incrementing x by 1.

Now, you could also write x += 1. Different syntax, same semantics. So syntax defines structure, and semantics define meaning. That distinction is important when you’re analyzing or transforming code.

LLVM and portability

Tristan Handy: Have we been building abstraction layers like this for decades?

Lukas Schulte: Absolutely. That’s what LLVM does. It provides a consistent intermediate representation that compilers can use to target multiple backends—Intel, ARM, different OSes. Apple invested early in LLVM to support custom chips.

With Rust, for example, LLVM is what lets us build binaries that behave the same on macOS, Windows, and Linux with relatively little effort.

Bootstrapping compilers

Tristan Handy: So there’s this recursive loop—compilers being built with other compilers?

Lukas Schulte: Exactly. Rust wasn’t always written in Rust—it started in C++. Eventually, the compiler was rewritten in Rust itself. Now, Rust compiles Rust. It’s fully self-hosted. That’s common with mature languages—it shows the compiler ecosystem is stable and powerful enough to sustain itself.

Why compilers matter

Tristan Handy: You said once that compilers are the foundation of every software ecosystem. What did you mean?

Lukas Schulte: There are two big drivers in software: abstractions and standards. You want one way to interface with a USB device—not ten. Same for software. You want one standard way to express a Python program, a JavaScript app, etc.

Compilers enforce those standards and make sure the same code works across platforms. That consistency powers things like package managers, shared libraries, and open ecosystems.

SQL dialects and fragmentation

Tristan Handy: Are there ecosystems that are doing worse than others?

Lukas Schulte: SQL does a particularly bad job. Anyone who's used more than one data warehouse knows you can't take the same SQL statement and expect it to work the same way. Casting, case sensitivity, functions—every engine handles these things differently.

Toward a universal SQL compiler

Tristan Handy: Can you convince me this problem is solvable?

Lukas Schulte: Yes. That's what we're working on with SDF—creating a shared intermediate representation for SQL. If we can express SQL logic in a unified form, we can compile it to any dialect—BigQuery, Snowflake, Redshift, and so on.

That allows developers to build reusable libraries, just like in other languages. It also makes governance, validation, and testing easier.

Future of data ecosystems

Tristan Handy: What would that future look like for practitioners?

Lukas Schulte: One major change would be the emergence of robust SQL libraries. Today, there’s no import system for SQL. Everyone writes similar logic over and over.

A shared compiler abstraction would let us reuse components, collaborate across companies, and build an ecosystem of packages for transformations, metrics, and validations—similar to how we use NPM or PyPI.

This newsletter is sponsored by dbt Labs. Discover why more than 50,000 companies use dbt to accelerate their data development.

Book a demo

Welcome to our new season of The Analytics Engineering Podcast. This season, we’re focusing on developer experience. We’ll explore the developer experience by tracing the lineage of foundational software tools, platforms, and frameworks. From compilers to modern cloud infrastructure and data systems, we’ll unpack how each layer of the stack shapes the way developers build, collaborate, and innovate today. It’s a theme that lends itself to a lot of great conversations on where we’ve come from and where we’re headed.

In our first episode of the season, Tristan talks with Wolfram Schulte. Wolfram is a distinguished engineer at dbt Labs. He joined the company via the acquisition of SDF Labs Labs, where he was co-founder and CTO. He spent close to two decades in Microsoft Research and several years at Meta building their data platform.

One of the amazing things about Wolfram is his love of teaching others the things that he's passionate about. In this episode, he discusses the internal workings of data systems. He and Tristan talk about SQL parsers, compilers, execution engines, composability, and the world of heterogeneous compute that we're all headed towards. While some of this might seem a little sci-fi, it’s likely right around the corner. And Wolfram is inventing some of the tech that's going to get us there.

Join Tristan May 28 at the 2025 dbt Launch Showcase for the latest features landing in dbt to empower the next era of analytics. We'll see you there.

Register now

Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.

Listen & subscribe from:

• Spotify

• Apple Podcasts

• Amazon Music

• TuneIn

• RSS feed

Chapters

• 01:35 Introduction to dbt Labs and SDF Labs collaboration

• 04:42 Wolfram's journey from monastery to tech innovator

• 07:55 The role of compilers in database technology

• 11:05 Building efficient engineering systems at Microsoft

• 14:13 Navigating data complexity at Facebook

• 18:51 Understanding database components and their importance

• 24:44 The shift from row-based to column-based Storage

• 27:40 Emergence of modular databases

• 28:44 The rise of multimodal databases

• 30:45 The role of standards in data management

• 35:04 Balancing optimization and interoperability

• 36:38 Conceptual buckets for database engines

• 38:46 DataFusion compared to DuckDB

• 40:44 ClickHouse

• 44:20 Bridging the gap between SQL and new technologies

• 50:55 The future of developer experience

Key takeaways from this episode

From monastery to Microsoft: Wolfram’s journey

Tristan Handy: Can you walk us through the Wolfram Schulte origin story?

Wolfram Schulte: I was born in rural Germany—Sauerland—and ended up in a monastery boarding school after my father passed away. Their goal was to train monks and priests, but that didn’t stick for me.

Later I went to Berlin—back then you had to cross East Germany to get there—and began studying physics. But I realized everyone else understood physics better than I did! One day I walked past a lecture on data structures and algorithms, and I was hooked. I hadn’t written a line of code at that point, but I switched to computer science immediately.

After my PhD in compiler construction, I joined a startup, then landed at Microsoft Research in 1999 thanks to a chance encounter with the logician Yuri Gurevich.

Inside Microsoft Research and Cloud Build

At Microsoft Research, we were like Switzerland—neutral across teams like Office, Windows, and Bing. We’d invent tools and ideas, but often the business units didn’t trust them. That changed when I was asked to build an engineering org.

We created Cloud Build, a distributed build system like Google’s Bazel. It reduced build times from hours to minutes and had a huge impact on iteration speed, productivity, and even morale. People stayed in flow. Builds were faster, cheaper, and smarter—running mostly on spare capacity.

Janitorial work at Meta: cleaning up big data

You later joined Facebook (Meta). What was that like?

A different world. No titles for engineers. Egalitarian, fast-moving. I joined to clean up the data warehouse—what they called “janitorial work.” At Meta, each type of workload had its own engine: time-series, batch, streaming, etc. This made understanding lineage and dependencies across systems extremely hard.

We responded by building UPM, a SQL pre-processor that stitched metadata across engines. It became part of Meta’s privacy infrastructure and compliance tooling, especially after the fallout from Cambridge Analytica.

Databases as compilers

Let’s shift gears. Can you walk us through how analytical databases actually work—like a professor at a whiteboard?

Sure. Think of a database like a compiler:

1. Parsing & analysis: Is the SQL valid? Are the types correct?

2. Optimization: SQL is declarative, so you can reorder joins, push down filters—based on algebraic laws like associativity.

3. Execution: Often done in parallel, especially in modern warehouses.

4. Storage: Columnar vs. row-based; optimized formats like Parquet or ClickHouse’s custom format.

Historically, storage and compute were bundled. Now they’re decoupled. But when the engine understands the format deeply, performance is much better.

The rise of modular and composable data platforms

How did we get from monolithic systems to the composable database architectures we have today?

It started with the rise of big data—Hadoop, HDFS, MapReduce. That decoupled compute from storage. Columnar formats like Parquet enabled analytical workloads. Then came Iceberg, Delta Lake, and similar standards that enabled multiple engines to share data.

Modern databases are modular. For example, Postgres is transactional, but you can bolt on an OLAP engine for analytical queries. You can mix and match based on your workload. The result is a data ecosystem that’s far more flexible—but also more complex.

Engine families: Snowflake, DuckDB, ClickHouse

Can you help us bucket the different kinds of engines out there?

Totally. Here are three buckets:

• Cloud-native engines: Snowflake, BigQuery. They’re optimized for massive scale, often with their own proprietary storage.

• Embedded/single-node engines: DuckDB, DataFusion. Great for local dev or embedded analytics. DuckDB is for users; DataFusion is for database builders.

• Real-time/high-throughput engines: ClickHouse, Druid. Tuned for streaming and extremely fast aggregations.

Each has its trade-offs. Increasingly, projects are combining these. For example, you can plug DuckDB or DataFusion into Spark to speed up leaf-node execution. The whole engine space is getting more composable—and more interchangeable.

The role of SDF in dbt’s future

If you think about the future where SDF is fully integrated into dbt Cloud, what does that enable?

Initially, it might feel the same—but faster, smarter. Longer-term, we can give developers superpowers.

Imagine your dev environment proactively surfaces:

• “This data looks different than yesterday—want to investigate?”

• “You’re missing a metric that’s often used alongside this one.”

• “This join will behave differently on engine X—here’s what to change.”

That’s the kind of intelligent, predictive developer experience we’re building. We’re catching SQL up to what IDEs have done for code. And if we can make logical plans portable across engines, dbt becomes the consistent interface across heterogeneous compute.

This newsletter is sponsored by dbt Labs. Discover why more than 50,000 companies use dbt to accelerate their data development.

Book a demo

I try really hard and go into every conference with an open mind about what I’m going to learn. Tabula rasa. Blank Slate. Beginners Mind. This is actually a really important part of being able to continually grow and develop your analysis of the industry rather than getting stuck in familiar mental grooves.

But for this year’s Data Council, I have to admit I went in with a preconceived take on the newsletter I wanted to be sending out today.

“I’ve been to a whole lot of data conferences that talk about the intersection of data and generative AI”, I’d write triumphantly, “but this was the first one I’ve been to where data and AI felt truly integrated, where the worlds finally converged”.

And you know what? It was true. You couldn’t throw a stone in the convention hall without hitting a booth for AI-assisted data development or using your data in agent systems.

GenAI applications, after all, aren’t just running on models trained on massive datasets built and maintained with many of the tools and open source libraries created by the people and organizations at Data Council. Their usage and utility also depends on strong infrastructure, as Martin has told us.

We saw a lot of very cool data + AI infrastructure at Data Council!

• Bauplan, fresh off their recent fundraise, walked us through the minimum viable data platform

• The Snowflake booth showed how Cortex Agents can sit in your database and perform useful work

• Lloyd Tabb gave a great walkthrough of Malloy and repeatedly emphasized the benefits of writing LLM-based analytics queries with a Semantic Layer as opposed to going straight to SQL

• Jacob ran a session on vibe-coding your data engineering workflows

• MCP was the talk of the town, with notable MCP servers being discussed by ClickHouse, Motherduck and yours truly.

And then of course we had Elias discussing SDF + dbt and walking through a new bit of data infrastructure that I believe is going to play a significant role in the story of how data + Gen AI fit together, the development of the new dbt engine - Rust-based, type-aware and ready to validate your SQL queries are dialect accurate and governed, whether they are written by a human or a machine.

So in a certain sense, I am walking away from this Data Council feeling like the worlds of generative AI and traditional data infra are closer together than ever.

But in another, deeper sense, I’m not.

A familiar kind of weird and a new kind of weird

Three years ago, in his reflections on Data Council, Drew had one request: “Keep Data Council Weird”. At the time, we were wondering if the ecosystem was becoming too vendor+VC driven and hoping that we’d still maintain our spunky outsider energy.

Well, I have to be honest with you, this Data Council felt pretty darn weird.

Partly, it felt weird in a familiar way. I asked Drew if this year felt weird and here’s what he told me:

The venue - a masonic temple - was gorgeous and unlike any conference venue I’ve been to before. My legs hurt from walking up and down 4 flights of carpeted stairs. I watched Elias’s talk from a parapet (is that even the word?)1 in a column adorned theater. I think I saw a crucifix. The bathrooms had couches in them. Scott B and I talked about our skincare routines. I saw a lot of old friends and former coworkers. I befriended [redacted]. My beef with [redacted] grew even deeper. I had a top 3 all-time cheeseburger and a bottom 3 all-time dessert (Mango Piggy). Pete and the whole Data Council team put on one hell of an event this year!

If you’ve been around the block enough times, this is a familiar kind of weirdness. Comforting.

It also felt weird in a different way though:

Because fundamentally, even though data infra + AI are moving ever closer together, there are big differences in how each side moves and progresses.

The reason boils down to this:

Data Infra is heavily engineered, based on building well-understood systems and standards.

It moves at the speed of ecosystems and standards. Three years ago at Data Council I’m sure there were people talking about Apache Iceberg and wondering whether it would become adopted across the industry. We’re big believers in Iceberg at dbt Labs and I expect to see strong and meaningful adoption of Iceberg over the next three years. I think an 80th percentile good outcome for Iceberg adoption looks like a world where organizations are not meaningfully constrained by their choice of data platform and are able to use Iceberg to avoid vendor lock-in and have true cross-platform control of how they operate on their data.

Generative AI is built differently, and it moves at a different speed.

The folks at Anthropic like to say that LLMs are grown, not built. Three years ago when Drew said that we should keep Data Council Weird, we were about 9 months out from the release of ChatGPT, and a year away from GPT-4.

Since then, the price of a query to GPT-4 has fallen by somewhere around 100x. OpenAI is projecting $125 billion in revenue by 2029. The latest paradigm shift, reasoning models, are around six months old.

I don’t know what an 80th percentile “good” (meaning fast) outcome looks like here, but there are people a lot closer to this than me that are saying we’re going to be deploying bio-engineered algae nanobots to fuel the data centers doing recursively self-improving AI by the time we hit three year’s from now’s Data Council.

That, to me, is pretty weird.

The weirdness of two worlds, closer than ever before but apparently moving at blindingly different speeds.

The weirdness of sitting in a talk and getting legitimately excited by the idea that we as an ecosystem can robustly adopt the nearly-decade-old Apache Arrow and then going into the hall to talk to someone who had just walked out of a talk on Bryan’s Foundation Models track and was wondering to what extent 2 year old LLM based coding workflows are going to change whether any of these questions are still relevant.

So what do we do with this?

Look, maybe one day soon, we’ll pinch ourselves, bolt awake and think “man that whole AI thing was crazy”. I’ll look back on this newsletter, cringe a bit about my prognostication and sheepishly admit that maybe I got carried away by drawing out lines on a curve. God knows it’s happened before.

But … maybe not. And in that world, what relevance does the data infra have?

I think it means that all of this matters a lot - even more so in this world. It means that pretty soon, the data systems and data infrastructure we build are going to be powering a whole lot of systems that interface more directly with the world than we are used to.

Because my prewritten take about data systems and AI workflows becoming increasingly intertwined and dependent on each other was right. And now we need to figure out how to make engineered data infrastructure that move at human scale support LLMs that look like they are moving much faster and are still fundamentally mysterious to us.

The real world and the data we represent it with has a lot of complexity. And if we’re about to have AI systems that are 100x cheaper and 100x more powerful than what we have today operating on the tools, systems and standards we build, then they’d better be really good.

I don’t have an exact answer to how we should approach this. I don’t think anyone does.

I do know that I’m looking forward to next year’s Data Council and the one after that and the one after that too. I’m hoping that alongside the new weirdness, we keep the familiar weirdness and that we all continue to share our knowledge, our expertise and perhaps most importantly our mango piggies.

Appendix

As I was writing this, the ever thoughtful Benn Stancil released a post touching heavily on MCP and the dbt MCP.

As with basically everything Benn writes - it’s worth your time. The post probably deserves a full response, so I’ll save commentary for another day, but I recommend you check it out.

The analytics engineering roundup is sponsored by dbt Labs.

If you want to see what the big kerfuffle about dbt + SDF is all about, plus a whole lot more, join Elias and the dbt team for our Cloud Launch Showcase on 5/28 (parapet not included).

Sign up for the Cloud Launch Showcase

Business intelligence is on a collision course with AI.

The collision itself hasn’t happened yet, but it’s clearly coming. The inevitability of this has been clear roughly since the launch of ChatGPT, but no one knew exactly what shape that would take.

Today I want to propose how that collision is going to happen and what will happen in its aftermath.

I think it will be a very good thing for data practitioners of all stripes—those who officially have the word ‘data’ in their title but also everyone else who simply uses data in the service of their larger job. So: I’m all for it.

Before getting into AI part of the story, I need to introduce two specific mental models.

Let’s go.

BI is a Portfolio of Stuff

We all use the term “BI” but have become inured to what an Orwellian term it is. “Business intelligence” isn’t descriptive, it is industry-speak for a bunch of stuff glued together in order to achieve a desired user outcome: know facts about a business using tabular data.

For a long time, BI included a bunch of stuff that it no longer does. Like: data processing. Pre-cloud, BI tools processed data locally and often had proprietary processing engines. They competed on being fast.

With the cloud, that evaporated. Local data processing was anathema. BI tools got easier to build but gave up a part of their value proposition.

In today’s post-cloud world, I would suggest that BI tools have three jobs:

1. Modeling Define the semantic concepts behind your structured data: metrics, dimensions, joins, etc. Think: LookML.

2. Exploratory data analysis (EDA) The iterative process of exploring data in search of useful insights. Highly iterative, flow-state, and unpredictable. Think: Looker explore window.

3. Presentation The aggregation of multiple data artifacts together to present a single cohesive narrative that can be shared out to potentially many others within an organization, all governed by a permission model. Think: Looker dashboard.

Some tools skip modeling and just allow users to do EDA without a model. EDA and presentation are the most core jobs of any BI tool and every BI tool I’m familiar with does both. And it is the fact that BI tools facilitate the EDA process that enables them to govern and share the presentation of that analysis.

Scaling the Criticality of an Analysis

All credit to my collaborator Dave Connors for this mental model 🦞

Generally speaking, artifacts pass through a few lifecycle stages as they mature into data products supporting production use cases. Think about these stages as the ‘production line’ of BI.

Phase 0: Exploratory Analysis

The first thing data practitioners do when faced with a business question is to start developing low fidelity sketches to try to answer it.

The vast majority of the work generated here will be thrown away, so there are low expectations code quality and governance. The primary goals of the best EDA experiences are iteration speed, flow state, and flexibility.

Phase 1: Personal Reporting

At a certain point, some exploratory analysis will cross over into a true insight; your question is answered, your curiosity sated. The question is important enough that you want to make sure you can return to it later. But it is not yet “ready for prime time”—you’re not ready to share it with others and have it be a part of someone’s operating cadence.

Some BI tools have a separate section for your “personal space”—think about your personal folder in Looker.

Phase 2: Shared Reporting

The moment that a report gets shared with another person, the required governance characteristics of a data artifact increase significantly. When you create a report you understand its context; when someone else starts using it they just expect it to be correct.

In phases 0 and 1, there may not be any governance applied—all governance may be applied at the compute layer with grants. But once you share an artifact, it is the governance at the BI layer that determines who gets to see what. This is simply because most data consumers don’t have accounts within the data platform and so the BI tool takes over as the arbiter.

In phases 0 and 1, there is also no auditability requirement. Auditability, change tracking, and general data ops best practices are introduced when artifacts are shared with others in Phase 2.

Phase 3: Production Artifact

When shared reporting reaches a very high level of criticality (frequent access by a large number of end users, agreed upon SLAs, supports a critical business process, dynamic features), it’s officially “in production” and needs to be owned and operated like any other production data asset.

===

If you think about these stages as the ‘production line’ of BI, the most important job of a BI tool is to be the conveyor belt through all of these stages. Start with raw materials, end with a production data product. At each phase of maturity, it’s easy to extend the product to support the next set of capabilities: governance, dynamic filters, SSO, etc. You never think about those things during Phase 0, but as your work progresses, the BI tool makes it straightforward to progressively add those capabilities.

But for this all to work, you gotta start the process inside the BI tool all the way back from Phase 0. You can’t do your EDA in Jupyter & Pandas and expect to ship it to users in Tableau…that’s not how that works.

So: you gotta do your EDA in a BI tool to take advantage of the “production line”. But…are BI tools typically the best way to do EDA? We’ll return to that later.

MCP and AI-as-Aggregator

The final thing we need to understand is the impacts of a context protocol. I wrote about this a few weeks ago:

The easiest thing to do for any technology vendor at the very onset of the AI era was to take all of the domain-specific context that you had and surface it to users in a chat interface. And we did the same thing. It was (and is) quite good—it does a great job of allowing users to ask business questions and answering them with semantic-layer-governed responses.

The problem with this approach is that users don’t actually want to interact with dozens of chat interfaces. They don’t want to remember to go to a given tool to get one type of answer and another tool for another type of answer. There will not be 30 chat experiences all with different context. There will be one…or maybe just a few. But likely a single dominant one.

This is how aggregators work. You likely don’t use a bunch of different search engines—you probably just use one, and it is probably Google. This is how chat will go as well.

The problem is, Google could scrape the web and respond to all queries based on that knowledge. But ChatGPT cannot know all of the information you want to ask it questions about (at least, yet). That lack of business context is the problem.

That’s where a context protocol comes in. A context protocol—a somewhat new topic in the public AI conversation—is a standardized way for services to provide additional context to models via an open protocol. The most promising one today is called MCP, but whether or not MCP wins, the awareness/excitement/support for this idea has developed a ton of momentum and I am fairly convicted that something like this will become real and widely-supported.

There will be a large number of context providers (every source of valuable enterprise context) and a large number of context consumers (different products with AI capabilities). There is no way to create point-to-point integrations to facilitate this. A protocol will be needed if we are going to see the right type of advancements, and I think it will happen.

Imagine that your license to ChatGPT enterprise or Claude Desktop or whatever already came with a connection to all of the metadata about every piece of structured data you had access to. What was there, how trustworthy it was, how suitable it was for the analysis you were describing, etc.

Well, in the intervening weeks since I wrote this, a couple of things have happened. First, this:

…then this:

Clearly this thing is going somewhere.

Just as momentously, I have gotten access to an internal-only dbt/MCP-powered experience in Claude Desktop. In it, I can ask every type of metadata question I might want (powered by our Metadata API) and I can also ask questions about all of our business metrics (powered by our Semantic Layer API).

It is incredible. I don’t want to share too much right now, but … having your data and metadata available in the context of a modern reasoning model is incredible.

BI in an AI-First World

Ok, we now understand: the jobs of a BI tool, BI conveyor belt, and how to get structured data context into your AI-of-choice. We’re finally in position to tackle the coming collision.

Here it is; plain and simple:

1. AI is going to be meaningfully better at exploratory data analysis than any BI tool.

2. If you take away EDA from BI, the ‘conveyor belt’ model breaks down. And the conveyor belt model is the primary reason you use your current BI tool.

3. It is not yet clear how the BI ecosystem will adapt to this new reality.

That’s it. That’s my entire argument. Let’s see if it holds up.

Artificial Intelligence will Far Outstrip Business Intelligence for Exploratory Data Analysis

There are a lot of data tasks that AI is good at. I’ve talked about a lot of these in the context of data engineering here. But the area of data analysis that is going to be most benefited from AI is EDA.

I am confident about that for two reasons. First, I have empirically validated this first-hand. The dbt + MCP + Claude 3.7 combo that I outlined earlier is just dramatically better at EDA than anything I’ve experienced in my life, and it’s getting better fast. But I am not ready to show you that (it’s single-digit weeks away from a public demo!), so you may not believe me. Fair.

The second reason I’m confident about this is the fact that most time spent in EDA is writing code (whether done by hand or via a GUI). And we now know how good leading-edge models are at writing code when supplied with the right context. Whether you want to reference individual developer testimonials or the head of YC or Andrej Karpathy or Google, it all lines up. And it just so happens that the two software engineers whose opinions I trust most in the world—my cofounders Drew and Connor—have gone all in on Cursor over the last 3 months and are not-quite-but-almost religious about the experience.

If you find yourself skeptical of this, here are a few things to keep in mind.

1. You don’t need the LLM to answer ‘why’ questions, or generate hypotheses, to have it be far superior than your current workflow. Rather—it just makes you a lot faster because it can write EDA code a whole lot faster than you can (whether you’re writing Excel formulas or dataframe operations).

2. Accuracy is a non-issue as long as you ask a question that can be governed by a semantic layer. The code written tends to be: get data from the SL, manipulate it in Python, generate a chart using some dynamic javascript library. If you can’t get a dataset governed by an SL query, text-to-SQL does continue to improve with sufficient context.

Just imagine: an interface that allows you to just have your questions answered far faster. You remain the objective function and the creative drive behind the process, AI is simply better and faster than you are at writing analytical code.

IMO that shouldn’t feel threatening, that should feel empowering. I seriously lost it the first time I interacted with our internal data in this type of experience. The primary value prop of a data analyst shouldn’t be writing code, it should be analytical problem solving and generating action.

The Conveyor Belt Model Breaks Down

You typically don’t tend to use your BI tool because it is the fastest or most delightful EDA experience. You use it because, when you have something to publish to your coworkers, you know exactly how to do that.

But what if another tool were so much better at EDA that you would be handicapping yourself if you didn’t use it? What would you do?

There are likely three answers.

First, you could go back to publishing one-off assets. If you ask any AI experience to “give me that in an Excel file” most of them have no problem doing that. So maybe you just go back to shipping attachments. But that doesn’t feel like progress.

Second, having iterated and found the insight you were looking for, you now have to reconstitute that analysis inside of your BI tool of choice. In practice this will likely only happen rarely; it is not a stable equilibrium because every human hates double work.

Third, and hopefully preferable, is that we find some way to pull back in the results of an exploration into the governed framework of the BI tool. Imagine asking “make a PowerBI worksheet out of this analysis.” We will need to get deeper into the MCP era to see exactly how this will play out, but I’m optimistic that it will be possible.

The third option still sees the BI tool as an important governance and presentation layer but pulls out the most strategic responsibility (EDA) from its portfolio.

A Very Different BI Tool

BI tools used to ship with compute engines. Today they do not.

What if BI tools were no longer the primary way EDA was done?

What if their primary job was to render data artifacts a governed, interactive environment?

That is still an incredibly valuable thing, and needed as long as humans are going to continue to interact with structured data (IMO: a long time). But it’s not what BI tools look like today.

Most BI tool vendors want to pull this new EDA experience inside their Chrome—exposing AI-powered interfaces inside their products. I don’t believe this will be how most users do EDA, for three reasons:

1. User behavior
   Aggregation theory will dominate, every knowledge worker inside of a company needs access to this functionality and they’re not all going to think to go to a specific tool first, they’re going to prefer to simply ask data questions in the same place they ask all of their other questions. Claude, ChatGPT Enterprise, whatever.

2. Tool combinations
   MCP is not only powerful because it lets you use a single tool, it is powerful because it is a pluggable framework to pull in all kinds of tools for the model to use. You’ll be able to ask a BI question (”Show me our most important renewals for the coming quarter”) and then immediately act on it in another tool (”Email the main point of contact on the account to set up a check-in meeting”). Having all of these tools interact together inside of a single interface is combinatorially powerful. There is already a large ecosystem of tooling available and community-driven innovation is happening fast.

3. Tech
   Except for MSFT, current BI vendors are not AI research labs. They are just not going to create better models or be the primary destination for all AI interactions within a company.

My Predictions

I think that the BI workflow that has dominated for the past ~15 years is going to change significantly over the next 2. EDA will significantly migrate over to AI interfaces, enabled by MCP.

I think this will be incredibly positive for all knowledge workers throughout a company. It will enable more users to create sophisticated analytics and will enable existing data practitioners to move significantly faster.

I think this will be a headwind to many current BI vendors. BI is extremely sticky and this change isn’t going to happen overnight, but it will be a headwind.

I think there is likely space for new players to innovate: to be the best place to aggregate and govern all of the artifacts built in this new workflow.

I’ll return to this post after six months and see how my predictions are faring!

Today, I’m going to sketch out for you:

1. what exactly is this abstraction of abstractions at the heart of this sea change

2. the lay of the land today: how far things have come, what’s still holding us back, and open questions

3. [EXTRA CREDIT]1 technical weeds: table format convergence, S3 tables, vended credentials, and more

To not bury the lede any further, I’ll be talking about Apache Iceberg™️ , and a further abstraction: the Iceberg REST Catalog Specification (IRC).

The current state of Iceberg isn't easy to navigate. Despite all the buzz, the technology is still young. The ecosystem changes quickly—each day brings something new, from proposals to private previews to updates in pyiceberg.

So what's really going on here? What matters most? Why should you care? And if even Iceberg's creator says we shouldn't have to think about it (more below), why is everyone talking about it?

Over the past year, I’ve been working with many data teams to learn and implement Iceberg in production. I'm convinced of the Iceberg’s potential to impact many more analytics engineering teams. True Iceberg adoption will happen once robustly integrated with all major data platforms but even where it has been integrated there’s last-mile user experience missing that’s dampening the adoption curve. But it’s improving everyday!

So let’s get into it!

Iceberg: A tough nut to crack

Understanding Apache Iceberg is a “tough nut to crack” because it’s easy to get lost in the technical weeds and miss the big picture2. Ironically, Iceberg exists so that most people don’t need to think about it at all! In the first five minutes of his talk at Data Council last year, Ryan Blue, a creator of Apache Iceberg says almost exactly that:

Iceberg should be invisible [in that it should aim to]:

• avoid unpleasant surprises

• don’t steal attention and reduce context switching

Ryan Blue, "Why You Shouldn't Care about Iceberg"

This sounds a lot to me like a powerful abstraction that lets you focus on the task at hand without getting bogged down in details.

“Bogged down in details” is an apt description for data engineering until recently. MapReduce, Hadoop, Hive, and Spark were all powerful tools that got the job done, but no one will claim that these were easy to use. You could never just write SQL — a portion of your brain was always reserved for reasoning about where and how the data was written and avoiding unpleasant surprises and edge cases. Your resulting pipeline could process petabytes of data, and you had the sweat to show for it.

“bigger data → more work” is a reasonable heuristic, but the impetus for Iceberg was an attempt to minimize the cognitive burden with a new abstraction of a table that just works like a database’s table (e.g. Postgres or SQL Server)3. Iceberg isn’t a silver bullet that solves all problems with large analytic data, but it’s a stronger, empowering abstraction.

The IRC (no not that IRC)

The Summer of Love Iceberg Catalogs

Ten months ago now, in June 2024, during what we colloquially refer to as “Summit Season”, two hallmark announcements were made within 24 hours of each other.

“Iceberg steals the Summits spotlight” & “Iceberg wins the table format war!” comprise the the gist of many folks’ reactions. I largely agree, with a small tweak: the real winner was the Iceberg REST Catalog.

What does an IRC do for me?

If you want to know what an IRC is and does, I’ve put that section at the bottom of this post in hopes of avoiding the technical weeds and staying high-level

The IRC reminds me when I first started using Dropbox in college. Countless times, I’d often stay up all night before a deadline writing in Microsoft Word on my Macbook Pro. In the morning I’d run across campus to a desktop PC at the computer lab. In a minute I could get my paper off the internet opened in Word again so I could print it.

It’s easy now to take for granted the power of the abstraction that Dropbox represented even at a time when the Internet already existed. There was complexity behinds the scenes, but the core UX was magic in it’s simplicity: It was a folder of files that was where you wanted it to be and it just worked like a folder should. This is what I feel the IRC represents for us in data.

Coming back to data warehousing, we can replace a “folder of a dozen files” with a “schema of a dozen tables”.

So imagine that you have this schema of tables and access to many query engines like: Databricks, Snowflake, Redshift, DuckDB, Trino, and Spark.

How powerful would it be if you connect any of the engines to the schema above to read those tables, modify their data, and make new tables. Also, that others using other query engines could see those changes.

On top of that, this system does not include expensive copying of data, or working with FTP servers, Google Drive, or directly interacting with Azure Blob storage. You just connect your SQL engine to an Iceberg catalog to read and write your data. This is the promise of the IRC in conjunction with your data platform.

How might your data team operate differently in this world? This is why we’re launching cross-platform Mesh to support these exact multi-stack engine scenarios that more than 50% of our Cloud customers already find themselves in today.

A Year of Progress

So where are we at today with Iceberg and where are we going as we enter summit season 2025 and beyond? The threads I want to pull on are about end-user adoption, data platform vendor integrations, and open source catalogs. I don’t have a crystal ball, but I’ll prognosticate a smidge.

curiosity? high! adoption? lukewarm (but growing!)

At my Iceberg breakout session at Coalesce in Las Vegas last October, I asked the analytics engineers in attendance to raise their hand if they’d heard of Iceberg — all of the hands went up. When I asked those with their hands up to keep them up if they felt they could explain Iceberg to the person sitting next to them, nearly all of the hands went down.

Tellingly, this included the folks who said their teams were already using Iceberg in production. This isn’t a problem: it’s Ryan Blue’s vision in action! More-so, this is the opportunity of Iceberg via IRCs: understanding the technology isn’t necessarily a prerequisite for adoption. Maybe one person on the team sets it up. For everyone else it’s business (analytics) as usual.

Data Platforms are showing up in a big way for the IRC

So what have the data platforms and other independent software vendors (ISVs) been up to in the past year?

HOLY COW — SO MUCH!

It's remarkable to see the entire ecosystem embrace an open-source Apache project as the foundation for their products. The vendors that have integrated deserve a huge round of applause. Yes, they’re just responding to customer demand, and yes, a real reason to invest in Iceberg is you can reallocate engineers away from maintaining proprietary table formats and work that drives more revenue.

Still, the industry's investment deserves praise, especially since taking a more self-interested and cynical approach would have been easier, at least in the short-term.

Six months ago, we predicted internally that most vendors would support the IRC spec within 6–12 months.

Today, after evaluating more private previews than I could possible count, what progress can we observe?

If we can interpret “Iceberg support” as being compliant with the spec as of six months ago, then our prediction is looking good. The only major outstanding work is something known as “external writes”.

However, as I’ve mentioned above, Iceberg itself is still evolving, so our prediction was poorly framed in the first place.

Maybe the right question to ask is

When will IRCs be a stable abstraction such that:

• end users have a stable, fully-featured interface

• the Iceberg spec can continue to evolve under-the-hood without heavily burdening data teams using Iceberg?

Perhaps this moment comes when data platform catalogs support external writes, and this will be true in six months. Time will tell!

OSS catalogs: important but not for end users

Databricks and Snowflake also deserve credit for also open sourcing their catalogs: Unity Catalog and Polaris, respectively. Lakekeeper is another worth calling out for being written in Rust and improving quickly.

When data teams ask if I recommend self-hosting a catalog, my answer is largely “No!”. The exception here are teams that have either or both of

• enterprise security requirements (think: on-prem, self-managed data centers)

• a dedicated data platform team with the know-how to deploy critical data infrastructure.

The challenge here is that of uptime and availability. If the IRC is unresponsive, you can’t query the tables any more. A minority of teams will sign themselves up for this challenge. For most, I think your time is better invested elsewhere.

Beyond this small minority of data teams, the real value of these projects are for:

• data SaaS vendors: who need some catalog functionality

• prospective data platform customers. who need help committing to use a proprietary catalog (”worst case we migrate away and run the OSS catalog ourselves!”)

I don’t say this to cast doubt on the technology, in fact quite the opposite. All of these projects are being used today in production and are “battle-tested”. This usage serves to further refine the IRC as a standard. Everyone benefits from this, even users of proprietary catalogs.

what might data platforms do differently?

IRCs are the clearest option for making Iceberg truly an implementation detail, but adoption is hindered when data platforms don’t truly integrate the concept into their products. Some examples of this include requiring users to:

• create a second catalog within the data platform to make data available elsewhere

• choose a unique object store path for the data when creating an Iceberg table

• mount tables individually and manage their refresh

Some data platforms are taking a cautious approach to Iceberg and REST catalogs, worrying that these might create a disjointed experience alongside their native, proprietary table formats. These platforms are instead focused on streamlining their lake house experience within their own product suite. While this concern is understandable, this becomes a game of chicken. Customers want interoperability so they risk losing customers by having a walled garden.. Iceberg has fundamentally changed how data teams evaluate tools—any platform without a clear Iceberg strategy now receives a "lock-in" red flag during vendor evaluations, even if said team has yet to start using Iceberg.

what questions are on my mind for this Summer’s Summits and beyond?

Iceberg Summit is happening next week both IRL in SF as well as virtual. You should check it out!

As far as what Iceberg announcements I’m hoping for and expecting come June, here’s a list of things that, if announced, would be leading indicators for accelerated Iceberg adoption:

• support query engines to write directly to external Iceberg REST catalogs

• support mounting of a schema’s worth of Iceberg tables

• full support for catalog vended credentials

• any differentiated features that go beyond the scope of the actual Iceberg spec and are focused on UX and developer productivity

If we get all of this and more, I still have some open questions

• what’s the multi-region and/or multi-cloud story of Iceberg catalogs? Right now everything presumes the same cloud and same region or suffer painful egress and latency costs

• how to federate RBAC across query engines? we still heavily rely upon data bases to GRANT access to data. If the data its RBAC is managed in the IRC catalog, how is the query engine configured?

• what are best practices for working with multiple catalogs? more on that in a future post 😉

Thanks so much for reading — as always the comments and my DMs are open.

Should you be left wanting more, there’s four more sections that shy less away from the technical weeds.

Technical Weeds

What about Delta Lake?

Some of you will be frustrated that I didn’t bring up Delta Lake.

At the time of the Tabular acquisition I remember some people speculating things like this

Databricks acquired Tabular to squash Iceberg in favor of their open table format Delta Lake.

It was refreshing to see that cynical take be put to rest so soon when this interview was posted between Michael Armbrurst and Ryan Blue (creators of Delta and Iceberg, respectively). I love this quote so much:

It was never our intention to start a "format war" and have people spend so much time thinking about storage. It should just work and very few people should have to think about it. You should be able to focus on doing analytics.

To achieve this north star of "you don't have to think about it," they aim to standardize the two projects as much as possible. This isn't just lip service! One example touched on was their plan to standardize the VARIANT type implementation by pushing it upstream into parquet itself.

Another great example came through Deletion Vectors (DVs)—a feature that Delta tables had but Iceberg lacked. While Iceberg had a comparable feature called "equality deletes," it wasn't nearly as performant.

Now this work has been merged into the spec, slated for release with the Iceberg V3 table spec. This work represents a true data industry team effort with contributions from engineers at Databricks, Snowflake, Netflix, Google and more. If you're feeling brave, curious, and reading “roaring bitmap” doesn’t send you running for the hills, check out the PR and click around!

There's been much discussion about technology that converts between table formats, like Databricks' UniForm and Apache XTable. While these tools are essential in the short term, they'll ultimately become redundant. I'm seeing strong signals that the Delta and Iceberg teams agree not only on what the most important problems are, but also on how they should be solved. But maybe I’m being overly optimistic!

What about S3 tables?

I’ve long been bullish on the IRC, but the announcement of S3 Tables Buckets and Nikhil Benesch’s analysis made me question the assumption.

I had been thinking of IRC as an abstraction over object storage, i.e. the REST Catalog would deal with creating, naming, finding iceberg files without you having to think about it.

With Table Buckets it’s the converse. you think about S3, but don’t have to create/manage reason about an IRC. This isn’t necessarily a bad thing for both query engine developers nor end users.

For a query engine developer, you could argue that it’s easier for query engines to integrate with S3 than it is to integrate with a still evolving OpenAPI spec. They’re all already familiar with object storage!

For end users analytics engineers like us, IRCs can be a hurdle to initial Iceberg adoption because you have to set one up before you can create a single table. S3 Table buckets radically simplifies this, in that they have their own catalog behind the scenes. Not only is this catalog wildly performant like many products out of AWS, it also automatically handles maintenance tasks like file compaction. This approach has already borne fruit imho given there’s a plethora of Iceberg quickstart tutorials out there now (Snowflake, DuckDB)

I’m still very wary about asking analytics engineers to think about S3 paths when writing SQL to the extent that I still think it is an anti-pattern. This is why dbt by default will manage the path for you when materializing a Snowflake-managed Iceberg table. With S3 Table buckets there’s not a clear notion of a namespace to hierarchically organize tables (think: datababe.schema.table ).

However, just a few months later AWS S3 announced that Table buckets are available via the IRC API, so S3 Table Buckets has proven to not be opinionated on API. Perhaps their approach is the correct one in providing both UXes.

However, there’s an opportunity to simplify. While it is only natural that the S3 team would collaborate with the AWS Data Catalog team, the result is a rather disjoint end user experience.

It should not surprise us that the AWS S3 team wants to bring their expertise to making data lakes management easier and cheaper. I’d count on the team to continue evolving this product over time, so you should keep your eyes peeled as well.

IRCs: what specifically do they do?

At the risk of oversimplifying, what the IRC does is closes some remaining gaps that kept SQL on data lakes from feeling like the SQL you’d expect.

Attention Naming is all you need

One powerful abstraction of traditional SQL databases: all you need to query a table is its name, and you never have to think about where the table’s data is stored. You’ve likely never even thought about how much easier this makes your life until you don’t have it anymore. But, in data lakes, often you need to know the table’s path in the object store (e.g. S3) for it’s data.

refers to example normal SQL three-part name my_db.my_schema.my_table data lake object store path S3://my-data-lake/some/folders/my-table/metadata.json

I believe that asking analytics engineers to think about S3 paths when writing SQL is an anti-pattern. This is why dbt by default will manage the path for you when materializing a Snowflake-managed Iceberg table

sir, were you aware that was a red light you just drove through?

The other problem that the IRC solves is more behind-the-scenes. When I run a query in Postgres, I never think:

• I hope this file lands on disk successfully

• I hope no one else is trying to write to this table right now

• What if someone else deletes the files I'm writing?

We SQL users take this all for granted, but this isn’t possible with a data lake unless you have a catalog! Postgres and many other DBs play “traffic cop” for you so you don’t have to. The IRC fills this role for you on the lake

one API to rule them all

The last problem relates to simplifying how data platforms and query engines integrate with Iceberg. Spark has never had a problem integrating with Iceberg because Iceberg is implemented in Java. But how do you

• integrate the Iceberg Java library if your database is written in Python?

• read from an Iceberg catalog written in Go with a query engine written in Rust?

The IRC solves this problem by proposing a language agnostic API and a spec for a backend service that does some work that a query engine developer previously would have had to build. This is great because it lowers the barrier to adoption by reducing the required engineering effort to integrate.

What about IRC’s vended credentials?

Once you already have an IRC set up and configured (non-trivial work in it’s own right), the next step is to give a query engine access to it. To do so, by default the query engine must authenticate to two things in order to be able to read and write to the IRC:

• the IRC itself (typically with a personal access token)

• the object store that has the files associated with the Iceberg table

Not only is this a high-friction set-up, the experience isn’t very intuitive. For example, in this setup, when you ask the IRC for a particular table that you’d like to read, it will return to you an object store path for a file that has more info. If you don’t have access to this file in (e.g. S3), you’re SoL. That’s why this pattern also requires that the query engine also has direct access to the object store.

However, it doesn’t have to be this hard! With Vended Credentials, you only need to authenticate to the IRC, and the IRC will provision you access to the files in object store. This is a much simpler workflow than what I experienced my first time using IRCs over a year ago.

Vended credentials have been in the Iceberg spec since last June, but only recently has it been supported in platforms like Snowflake, Databricks, and SageMaker Lakehouse after a number of preview periods.

one query engine writing directly to an external IRC is also vastly simplified by vended creds. You just connect to their IRC and write the table directly without ever knowing where the data is stored.

How great to live in a world where when another team needs data from you, you never have to connect to their FTP server, Google Drive, Azure blog storage account to put the data, you just write to their IRC.

A consequence of vended credentials is that the IRC becomes critical path for accessing data, it means that you’ll have to refactor your connection later should you decide to stop using an IRC or select a different one. However the abstraction is more simple because you only need to tell your query engine about the IRC and not about object storage anymore.

The bear case here for vended credentials here is that it introduces a third access model distinct from the native RBAC of storage (i.e. IAM Policy) and the query engine (think database roles and privileges). However, you can’t have a catalog without RBAC, and the closer that RBAC lives to the data the better. It doesn’t make sense that a query engine should have roles for accessing the data, especially in a world where multiple query engines will access it!

In 2021 through early 2023, the data space and specifically “The Modern Data Stack” was arguably the highest energy, most dynamic area of the tech sector, and certainly dominated the discourse.

In late 2022, the ChatGPT moment happened and all of the oxygen, immediately, became sucked up into AI.

The mystery on my mind has been - what is the connection here? Is it just an accident of history that right before we got AI systems that actually work at scale, we were focusing on centralizing and modeling our data?

It’s completely possible that the answer is yes. It’s felt that way at times, but over the past year and particularly over the past few months, as AI systems move outside of narrow chat windows and become more integrated into our workflows, two things are becoming clear:

1. Complex AI workflows are going to draw many of the learnings from data engineering

2. LLMs will need access to the data produced by your analytics workflows in order to be truly useful for many use cases

Both of these facts feel relatively obvious now and immediately valuable in the near term, compared to other ideas that felt more … grasp-at-strawsy like “data teams will be the keepers of your organizational data to custom fine tune models for your business”.

So what changed? Why do these feel practical and useful now compared to even a year ago?

Because we’re actually starting to roll LLM systems out in the real world - and quickly. Even in their nascent state it’s clear that this is not hype, that there’s real value here, today. But it’s also becoming clear that the problems and lessons that brought us to the modern data stack haven’t gone away in this brave new world - although the ways those problems are solved and the systems they are being solved within may change dramatically.

At this point, you are probably familiar with the frustrating experience of going to a new website and being presented with some sort of chatbot interface and not being entirely certain what questions you can ask it.

The chatbot probably works extremely well for the set of context it has access to. But you probably don’t know what exactly it has access to, and that underlying guessing game means that what should be (and often are!) incredibly useful interfaces end up feeling slapped on and piecemeal.

The problem is largely not the models which have gotten extremely good for most questions you might ask of them. The problem is that they often literally don’t have the right information to give you the correct answer.

It does not matter how smart a model is, or how good it is at in context learning, if the only answer or a path to the answer can’t be added into its context because its locked somewhere in a single reddit post from last week, a proprietary document or in your data warehouse. The good news is that we’re quickly moving towards a world where that context isn’t locked up anymore and there are protocols and standards for accessing it.

But what context should be provided? How do you know it’s right? Is it accurate? Who has access to it?

These are all questions we’re going to have to learn to answer in our AI system. And it’s gonna be a doozy.

It is extremely non-trivial to feed the right context to LLMs at the right time

Probably the most interesting thing about adding new knowledge sources to LLM workflows is the way that LLMs magnify both the best and worst aspects of working with a particular system, almost to a fault. The time to answers is quicker, you can pull threads together - you’re always feeling movement. But any cracks in the system you’re using to feed the model context become immediately obvious.

We talk about “context” like it’s a monolith, but the underlying context we’re feeding is ultimately going to look something like “all of the mechanisms that humans have created for storing and conveying information”.

Let’s look at how this is going in practice - the good and the challenges:

LLMs + Internet search:

• What it is: Integrate public web data into LLM queries

• Why it’s great: This massively broadens the ability of LLMs to pull in information. This is really useful for when you require specific, granular information pulled from the real world (I like to use this for finding restaurants).

• The challenges: SEO-bait works extremely well on LLMs. Under the hood, it’s still performing some sort of traditional web search - the same tools and tactics that people use to climb to the top of Google searches work on an LLM. Try an experiment right now - go do a Google search for “Best Pillow 2025”. Do you have any reasonable way of breaking down the answers and getting to ground truth?1 If it’s hard for you, it’s going to be hard for the LLM.

LLMs + Internal company documentation:

• What is it: Search over internal, unstructured data like NotionAI and Slack

• Why it’s great: I want to be incredibly clear - I love NotionAI. It is the single closest I’ve ever felt to being able to fully wrap my head around a complex organization of hundreds of people and be able to learn what other teams are working on. It fundamentally broadens my aperture in knowing what is going on at dbt Labs and why - from answering quick policy questions to making sure I can keep track of the latest company objectives.

• The challenges: Unless you have incredibly strong document hygiene, you’re going to find messy and conflicting information. A simple question like “when is Coalesce 2025” can sometimes end up with 3 results - maybe we were initially thinking a different date and that date still lives in a document somewhere. Maybe someone just accidentally typed in the wrong date and left it there. The models need, as part of their context, not just all of your documentation, but signals as to which documentation is up to date, correct, organizationally approved.

LLMs + Metrics and structured data:

• What it is: All of the data that lives in your data platforms, your key metrics, customer information, business entities

• Why it’s great: I don’t want to be too dramatic here but being able to analyze a complex dataset using conversational analytics on top of a trustworthy interface feels like magic. Especially for data that you know well, it honestly feels like you are getting superpowers, with the answer to any question you might have available at the tip of your fingers.

• The challenges: What could possibly go wrong giving an LLM access to your data warehouse? Text-to-sql is good and getting better, I have no doubt that we will get to the point where, given a well structured problem and sufficient context about the underlying data, LLMs are going to be very successful at getting an answer that is reasonable and correct. But will it:

  • Be consistent across an organization? Will it be a single source of truth, based on vetted and well understood business concepts? Or will it be a very clever vibe coded 1200 line sql script that no one realistically is ever going to read.

  • Can you put it in the hands of your executives and have them trust the output? Not just as interesting anecdata - but as something that they can make decisions and take action off of?

  • Is it going to be governed? Will it know who the end user behind the query is, what data they should have access to and what they shouldn’t?

Each of these information sources add impressive and interesting capabilities to the underlying power of LLMs. When combined in a single interface, there is a combinatorial explosion of usefulness2 as the different information sources each unlock new capabilities.

Imagine you are the CEO of a restaurant chain planning whether to expand into a new territory. With access to these three information sources, you could prepare a deep research style report that:

• Understands the business context and strategy for a potential expansion based on your internal company documentation

• Has the ability to query and understand the actual data and financial metrics available to you via your data platform

• Searches the web for macro-level data and benchmarks about the location you’re planning to expand into

Sounds incredible right? This is totally doable, today - although it requires a bit of duct-taping systems together to make it work.

But it also sounds … daunting. Because each of the failure modes described above can cascade throughout this system. What if the model YOLOs a revenue definition that’s reasonable, but based on information from an out of date company document? What if it pulls benchmarked financial data from the web that ended up being wrong? The real world contains a whole lot of complexity and our systems need to be designed in a way to get the benefits here while having the appropriate guardrails to establish some sort of ground truth.

So why now

I want to return now to the question that I posed at the start of this - why were we all convinced that building systems for centralizing and managing your company data at scale was the right problem to solve directly before LLMs started to soar?

The answer lies in recognizing that what seemed like an accident of timing was actually a foundation being laid. The Modern Data Stack is not just about better dashboards —but about creating standardized and reliable workflows and interfaces across your entire data ecosystem that can power increasingly sophisticated use cases, at scale. It turns out this is just as necessary for AI as it is for humans.

We built the modern data stack to address fragmentation, improve data governance, and ensure consistent, reliable data. What we didn't fully realize at the time was that this was an essential piece of context for LLM applications as well.3

We finally have both data foundations and the AI models to make genuinely useful, reliable AI-driven and data-enriched workflows possible. The explosion of interest in AI didn't displace the need for the modern data stack—it just took some time for these systems to begin to speak to each other.4

Moving forward, we’re going to increasingly see the role of the data practitioner as providing governed, trustworthy data for LLMs and potentially, for using many of those same systems to enable safe, reliable deployment of AI systems.

I feel huge opportunity in this area. I also feel a lot of responsibility for us, as a community, to get this right. These systems are being experimented with, and in some cases deployed today and there is real institutional heft behind their rollout. We have the tools and the (forgive the pun) agency to make an impact on how that happens.

There is a tremendous amount of brainpower that reads this newsletter - people orchestrating the most complex data flows at the largest organizations in the planet and building the tooling that will enable it. There are a lot of unknown unknowns in terms of how we build the bridge from LLMs to our structured data, but I believe we’ve got the right set of humans in place here to begin meaningfully answering this question.

Let’s get after it. Want to talk about any of this? You know where to find me.

Last time I wrote I dove into a bunch of AI advancements that have happened over the past year. Reasoning models, chain of thought, inference-time compute, etc. And there’s more to explore there and I need to return to that series.

But for this week’s issue I want to pause on that and talk about AI from a different perspective. I want to think, as rationally as we can about an uncertain future, about how the job of the data engineer will change over the coming 1-2-3 years as a result of AI.

I am quite confident these changes will be massive. I think the word disrupt is not at all hyperbole—I think it will be hard to compare data engineering in 2024 and data engineering in 2028 and say “those are the same things.”

It just turns out that many of the tasks that data engineers do every day are tasks that AI can provide tremendous leverage in. I don’t know what the % efficiency metric will be—20%? 50%? 80%?—but I think it’s totally possible that it’s on the higher end of that range.

I think that will be both good for data engineers and good for the companies they work for. Data engineers will have more work to do than ever (Jevons paradox at work), but it will be more strategic, add more value to companies, and will likely see them get raises. Companies will get the higher-functioning, higher-ROI, more accessible data systems that have always seemed out of reach.

In this post I want to look at the specific tasks that data engineers spend their time on, and look at how addressable-or-not these tasks are with AI.

Let’s dive in.

The Tasks of a Data Engineer

AI doesn’t replace jobs, it automates tasks. So let’s look at the tasks that someone leveled a Senior Data Engineer most commonly spends time on today. Of course, it should go without saying, YMMV: there is no single canonical job description for a data engineer. But I think we can still get close enough to reason about.

What does a Senior Data Engineer spend their time on?

• Create technical artifacts

  • Landing new data. Building and maintaining automated data ingestion pipelines.

  • Transforming raw data into bronze, then silver, then gold layers. Includes authoring brand new pipelines as well as refactoring existing pipelines to handle new business requirements.

  • Defining metrics on top of transformed data.

  • Writing tests and documentation.

  • Monitoring costs of data infrastructure and refactor code to optimize performance characteristics.

  • Reviewing pull requests from peers.

  • Monitoring production jobs and declare incidents related to either pipeline failures or observability / quality issues. Investigate and resolve those incidents.

• Liaise with stakeholders and peers

  • Answer questions about currently-available data assets like “which data set should I use?” and “can I trust this?”

  • Collaboratively design changes to existing data assets to accommodate new requirements. Conversations like “what are the edge cases I need to know about when calculating cost of goods sold?”.

  • Stakeholder enablement and education.

  • Designing the overall architecture of the DAG, including modularization, team boundaries and ownership, modeling best practices, etc.

I’m sure you could find some other things to put on these lists, but I feel like they’re pretty representative. Feel free to tell me what I’m forgetting.

The Role of Frameworks and Tooling in an AI-centric World

Many of the above tasks are already doable with AI. And I want to talk more about that. But before I get there, it’s important to talk about frameworks, and how important frameworks are to an AI-centric world.

Claude 3.7 will write you almost any kind of code you could want. You can absolutely build a pipeline from the ground up, building ingestion, transformation, testing, etc. in Lisp. In Assembly. In the style of Guido van Rossum. Whatever. You could even imagine a world in which you had 1,000 distinct pipelines and every one was written in a different language or framework or set of conventions. All reading from and writing to a shared corpus of tabular data.

But: just because it is now conceivable to create such a codebase, is it a good idea?

The answer is: no. Obviously not. Just as a team of humans would have an impossible task of maintaining such a Frankenstein, the heterogeneity would make it intractable for LLMs as well.

This intuition pump is helpful to get us to an important conclusion: AI will be more effective as an accelerant when:

• a code base is fewer lines of code (less room for error)

• a code base is more consistent rather than less consistent: in languages, in coding conventions, in design

• a code base uses consistent CI/CD and other developer tooling

• a code base uses consistent and well-documented logging / observability

• a code base uses well-documented best practices also employed by a large community of users.

In general: code bases that are more concise, more homogeneous, and use standard tools that are well-documented in the model training data (i.e. the public internet) will be more comprehensible by AI systems.

One of the best ways to make all of these things true at the same time is to use frameworks and open standards. Claude 3.7 knows how to build reliably Airbyte ingestion pipelines because the framework is well documented and there are a lot of examples published. It’s also fantastic at writing dbt code for the same reasons. If you’re able to give it an environment where it can test its own code and validate downstream models as a part of its CoT—code quality goes up even further. Standardized frameworks also emit well-understood error messages, which pushes code quality up further.

In short: good frameworks, tooling, and standards are just as important for AI as they are for humans. And the wonderful thing about AI is: it is infinitely adaptable to whatever frameworks, tooling and standards tooling you want to use. No learning curves. Finally the promise of a consistent code base.

How many of these tasks are already doable?

Got it, frameworks are powerful in an AI world. Now let’s look at the individual tasks that data engineers spend time on and try to figure out how tractable they are.

In answering this question I am not going to assume massive improvements in model capability. Even with modest improvements I believe all of this will become true. What is fundamentally needed is productization of currently-available models directed at the specific needs of data engineers, not the invention of new frontier tech.

Creating Technical Artifacts

• Ingestion pipelines With nothing but Cursor you can already vibe code your way to a working ingestion pipeline from basically any data source with a publicly-available API. You can already add pagination and solve edge cases and inject instrumentation. It’s unclear, though, if this is actually what is needed. I still fundamentally don’t think most data movement code should be written and maintained within the walls of an individual company—AI or no, I still want to hire a vendor or support a community project. Data engineers shouldn’t be spending a lot of time on this problem today and likely shouldn’t be in the future either. When a custom build is required, AI can already do it well; try it yourself in Cursor today.

• Authoring new data transformation assets If you’re using dbt, data transformation is very soon to become heavily AI-enabled. Whether you’re building models, writing documentation and tests, or defining metrics, this is coming to you very soon. We demoed some of these capabilities at Coalesce and will have more to share on Wednesday at our dbt Developer Day. While we are certainly still in the early stages of where we ultimately want to get to, dbt Copilot is already very good at all of these authoring tasks and there is a very clear path to getting even better. Nick Shrock, in one of his best posts ever, called dbt and tools like it medium-code frameworks. It turns out that medium-code frameworks are extremely well-suited for AI. Having personally used dbt Copilot, I anticipate the time required to author new transformation code for data engineers will drop very significantly.

• Multi-file refactoring One thing that Cursor now does super-well is stage multi-file edits as a result of a single prompt. You could imagine a similar prompt in dbt: “refactor code in these two parts of the DAG to minimize duplication; combine models where appropriate.” Or: “A new field was added in this data source. Please pull that field all the way through to the DAG into [X] final model.” These types of refactoring tasks are low-creativity but highly time-intensive. Implementing them is product work, not research. The opportunity to get a handle on tech debt with tooling like this makes me giddy.

• Automated incident resolution Imagine providing the entire log output of a dbt run and the associated project code into a context window and getting back a diagnosis and proposed resolution. While we haven’t productized this experience yet, it’s not hard to experiment with this yourself hackathon-style. Imagine a world in which, following a pipeline failure, a full PR was queued up and run through CI, with a full report waiting for you and just ready to hit the merge button. We should anticipate this type of experience for data engineers in the not-too-distant future. How much time are you currently spending on break/fix? Slash it significantly.

I’m going to pause there because I’m at risk of boring you. Suffice it to say that I truly believe that a) much data engineering work has already been framework-ized, and b) AI will now make creation of, iteration on, and maintenance of these technical artifacts far more efficient. And for the aspects of data engineering that are not yet framework-ized (dbt or otherwise), there will be tremendous gravity towards pulling them into a framework because of the leverage that these types of high-quality AI experiences will provide.

Liaising with stakeholders and peers

There are countless people throughout the business who use data as a core part of their jobs, and data engineers are constantly fielding questions from them. I won’t re-list them all here, but if you’re a data engineer you know the drill. Forever, the hope of “self-service” has been the hope that these data users would not need to lean on data engineers in this way—these interactions inject friction and slowness that neither side wants.

This fully actualized self-service has never actually materialized, and the status quo has been frustratingly persistent. But I’m optimistic that we have more of a path today than ever.

The easiest thing to do for any technology vendor at the very onset of the AI era was to take all of the domain-specific context that you had and surface it to users in a chat interface. And we did the same thing. It was (and is) quite good—it does a great job of allowing users to ask business questions and answering them with semantic-layer-governed responses.

The problem with this approach is that users don’t actually want to interact with dozens of chat interfaces. They don’t want to remember to go to a given tool to get one type of answer and another tool for another type of answer. There will not be 30 chat experiences all with different context. There will be one…or maybe just a few. But likely a single dominant one.

This is how aggregators work. You likely don’t use a bunch of different search engines—you probably just use one, and it is probably Google. This is how chat will go as well.

The problem is, Google could scrape the web and respond to all queries based on that knowledge. But ChatGPT cannot know all of the information you want to ask it questions about (at least, yet). That lack of business context is the problem.

That’s where a context protocol comes in. A context protocol—a somewhat new topic in the public AI conversation—is a standardized way for services to provide additional context to models via an open protocol. The most promising one today is called MCP, but whether or not MCP wins, the awareness/excitement/support for this idea has developed a ton of momentum and I am fairly convicted that something like this will become real and widely-supported.

There will be a large number of context providers (every source of valuable enterprise context) and a large number of context consumers (different products with AI capabilities). There is no way to create point-to-point integrations to facilitate this. A protocol will be needed if we are going to see the right type of advancements, and I think it will happen.

Imagine that your license to ChatGPT enterprise or Claude Desktop or whatever already came with a connection to all of the metadata about every piece of structured data you had access to. What was there, how trustworthy it was, how suitable it was for the analysis you were describing, etc. I think that, very quickly, you would find yourself asking questions of your friendly AI rather than shoulder-tapping your colleague in data engineering.

That’s not to say that the existing relationship would go away, but I do think that this would represent a true reset of the working relationship between data engineers and downstream business stakeholders—one that both sides would benefit from.

Where does that leave us?

Over the past two years, critical innovations have been made in foundational AI technology. Chain of thought, reasoning models, inference-time compute, agentic workflows. These are the ingredients needed to build the AI-enabled data engineering future. But they are now here.

And open frameworks—from dbt to Spark to Airbyte to others—have become widely deployed. This makes it possible to create great framework-specific AI tooling, both for the commercial stewards of those frameworks (including us), but also by any other vendor.

The commercial incentive to innovate here is high, and there could not be more attention on delivering these types of benefits within companies of all sizes. This is going to happen, and data engineering as a profession is never going to be the same.

So what? Time to get a new job? Data engineers are obsolete?

Hardly. Data engineers, one of the hottest jobs of the last decade, will stay hot. But practitioners will be pushed in one of three directions: towards the business domain, towards automation, or towards the underlying data platform.

• Data Platform Engineers will become ever-more-important. They don’t spend their time building pipelines, but rather on the infrastructure that pipelines are built on. They are responsible for performance, quality, governance, uptime.

• Automation Engineers will sit side-by-side with data teams and take the insights coming out of data and build business automations around it. As a data leader recently told me: “I’m no longer in the business of insights. I’m in the business of creating action.”

• Data Engineers that are primarily obsessed with business outcomes will have ample opportunity to act as enablement and support for the insight-generation process, from owning and supporting datasets to liaising with stakeholders. The value to the business won’t change, but the way the job is done will.

You’ll hear a lot more from us on Wednesday about how we’re making this future a reality for dbt users. I’m excited to disrupt the decade-long status quo and build something better.

- Tristan