The State of Analytics Engineering survey is live
Share your voice on how data teams are navigating the agentic era. Take the survey and you’ll get early access to the 2027 State of Analytics Engineering Report.
Can data agents do real analysis? Shreya Shankar’s answer is that there is still a big gap. The best agent in her Data Agent Bench passed 61% of queries on the first try, and agents did significantly better when they had a semantic layer.
Shreya Shankar did her PhD in databases at UC Berkeley and joins Carnegie Mellon University as an assistant professor in 2027, where she will be part of the CMU Database Group. Her research sits where databases, AI agents, and human-computer interaction meet. She is the creator of DocETL, a system for LLM-powered data processing, and a co-author of the Data Agent Bench (DAB), a benchmark that tests whether AI agents can do the work of a junior data analyst. Her paper “Who Validates the Validators?” examines how developers build and trust LLM judges.
She joins Tristan Handy for a conversation about what happens when you ask AI agents to do analytics in a realistic environment: multiple databases, messy data, tribal knowledge, and free-text columns. The best agent in her benchmark passed 61% of queries on the first try, and agents that sat on top of a semantic layer did significantly better. They also get into evals, including why you can’t know what you want from an LLM judge until you have looked at a lot of outputs, and why there is still no equivalent of df.summary() for agent outputs.
Along the way: why data agents have no counterpart to the visual shortcut humans get from a chart, why humans are “actually awful at writing SQL” and SQL will stick around anyway, why agents want to select * from two databases and join them in pandas, why her defense of AI evals came down to what actually moves the quality and cost frontier, and what she plans to do about AI SQL queries that cost far too much.
Listen now: Spotify · Apple Podcasts · YouTube · Amazon Music · RSS
Please reach out at podcast@dbtlabs.com for questions, comments, and guest suggestions.
Three ideas from the episode
1. Data agents have no shortcut like the one visualization gives humans. Shreya describes an “inner loop” of tools like databases, which answer a question quickly once you know it, and an “outer loop” of tools like business intelligence, which help people figure out the right question in the first place. Charts work because people can see what’s interesting without spending reasoning effort. Agents have no analog of that yet, and Shreya’s open research question is what you can put in the prompt so an agent spends fewer reasoning tokens to get to the right answer.
2. Realistic benchmarks show a bigger gap than the clean ones do, and a semantic layer closes part of it. Existing benchmarks are either text-to-SQL, where the agent only writes a query, or table question answering, where the whole table goes into context. Shreya’s team built one with multiple databases, domain knowledge, and unstructured text to transform. The best agent got 61% pass@1. Giving agents a semantic layer significantly raised accuracy, which Shreya says speaks to the importance of the semantic layer, and of humans who understand what is in it.
3. You can’t judge LLM outputs well without looking at them first. In the “Who Validates the Validators?” study, developers’ criteria changed as they saw more outputs. Shreya calls this criteria drift. Two years later she says it still holds: to scale up LLM judges you have to read enough samples yourself to know what your rubric should be, and today that mostly means reading rows in a spreadsheet.
Everything referenced in this episode
Shreya Shankar's website: https://www.sh-reya.com
Data Agent Bench (DAB), "Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents": https://arxiv.org/abs/2603.20576 (code: https://github.com/ucbepic/DataAgentBench)
DocETL: https://www.docetl.org
"Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences": https://arxiv.org/abs/2404.12272
ChainForge: https://chainforge.ai/
"In Defense of AI Evals, for Everyone": https://www.sh-reya.com/blog/in-defense-ai-evals/
Key takeaways
Lightly edited for clarity and length.
Why did you put a note for AI agents on your website?
Shreya Shankar: I got a new job as a professor of computer science a couple of months ago, and all of a sudden I started getting an uptick in email requests. Even as a PhD student, people would email to say they wanted to do research with me. Once AI agents became popular it got easy to send a four or five paragraph email where every paragraph is a different paper of mine, and it’s all AI generated.
I was tired of reading those, so I put a note on my website. It took a couple of days, but within the first week the number of emails went down significantly. I used to get about five a day, and now I get about three a week. I’ve only gotten one email with the phrase the note asks for, so I don’t think I’ve made it a good prompt injection. My guess is that the agents mass-emailing professors are just not writing to me anymore, and I’m very happy with that.
I actually tell people now, just don’t use AI to write emails and you will get noticed. It’s never been easier to get a job or get noticed, because all you have to do is not use AI.
How does a machine learning engineer end up with a database PhD?
Shreya Shankar: I did my undergrad at Stanford during the deep learning boom, and did a couple of internships at Google Brain. I loved research, but I didn’t know what I would want to do a PhD in, so I went to industry. I was the first ML engineer at a data analytics company nobody has heard of, and what I found was that none of my problems were machine learning or AI problems. They were data engineering problems.
How do you do the data prep to harness machine learning at scale? That got me interested in databases and the idea that we should be able to build declarative systems and have the database engine optimize the queries, so data engineers don’t sit and hand-tune every one. I thought I wanted an AI PhD, but it turns out nobody in AI is interested in these problems, and everybody in databases is.
I’ve liked a lot about databases, especially the field’s open-mindedness to new applications. Everything has been rooted in relational databases, but people have thought about how to take the lessons and philosophy that work there and extend them to machine learning, ML-powered query processing, and now ETL.
Should data agents interact with data the way humans do?
Tristan Handy: Analysts have always done two things: answer questions somebody else generates, and, as they get more senior, find what’s interesting. Humans have a very limited working memory, so it seems like agents should be better than we are at looking at a big blob of data and telling you what’s interesting. And yet agents today interact with databases mostly the way humans do. We say, you’re good at SQL, go write some SQL.
Shreya Shankar: My framing is that there’s an inner loop of data tools, like databases, that are responsible for executing a question and getting the right answer quickly. Then there’s an outer loop of tools like business intelligence, which help the user figure out the right question, because they don’t know it in the first place.
One of the prime innovations there is visualization. It taps into our pre-attentive processing. We have a very strong visual cortex, so we can look at a plot and, without tapping into our reasoning tokens, see what the interesting question is. All of the good questions come from visualization, or at least it’s the seed. There’s no analog of this right now for data agents.
What people have agreed on is that tooling for agents is probably different from tooling for humans. Datadog has an MCP designed for agents, for example. But we need to go further and optimize these for token efficiency, so agents don’t read lots of tokens. They should also be able to ask a high-level question in a few tool calls, instead of listing a directory or running ten bash calls to figure out the query they want.
If I had to formalize the pre-attentive processing idea into a concrete research question, it would be: what do you provide in the prompt so that agents spend fewer reasoning tokens to get to the right answer? I don’t know if we have an answer to that.
SQL is the primary way we interact with relational data because it’s very good at modeling business data, not because humans are good at writing it. Humans are actually awful at writing SQL. But relational algebra is a good language for working with data, and it will probably stick around even as agents write all the queries for us.
What is a data agent, and what did the benchmark test?
Shreya Shankar: A data agent is some AI or entity interacting in the traditional business setting. There are many databases available, it interacts the way a human data analyst would, and it’s tasked with answering a question a human analyst would try to answer. I consider my data agent a junior analyst who uses all the same tools and has to work with all the same data I have. We scoped it to questions with a verifiable answer, like computing sales, or figuring out which customers to prioritize based on who wasn’t repeatedly buying our product.
We started the project last summer, or a little before, because agents seemed to be on the path to becoming analysts. They were already writing SQL, text-to-SQL had practically been solved, and AI was writing code. The obvious question was whether we can trust it to do data analysis. But it’s difficult to even get a sense of that, because the benchmarks out there are clean and stylized.
One category of text-to-SQL benchmarks has the agent author a query on your behalf, but as an analyst you’re not writing SQL all day. You’re stitching everything together and munging data from multiple databases. The other popular set, table question answering, puts your entire table in context. Data is bigger than what a model can hold, and people are not going to do that.
Can an LLM do relational algebra in context?
Shreya Shankar: I’m sure Fable can take in a whole table and come up with something interesting. I just think the majority of queries are relational algebra in some form, and so you should use a database to do it.
But whether an LLM can do relational algebra is a separate and intellectually interesting question, and in my opinion a more interesting benchmark than needle-in-a-haystack queries. Can you project a column? Filter? Join two tables in context? LLMs do attention over all pairs of tokens, so you’re getting the join for free in a very strange way that I don’t think we theoretically understand.
What makes an analytics environment realistic?
Shreya Shankar: We built our own benchmark, and it took a long time to figure out how. We talked to a lot of people and came up with four conditions analysts actually work under.
First, you have multiple databases or data warehouses in the environment, say a Postgres database and a Mongo database that stores unstructured customer data.
Second, an analyst needs to push computation down to the individual databases as much as possible, rather than loading both and joining them in pandas. That’s a bad idea, but data agents want to do it. Their instinct is to select star from the databases, load them into data frames, and join them.
Third, domain expertise has to be encoded somewhere, like what a stock ticker symbol means or what a rolling average is. You can encode some of that in a memory file, but it’s tribal knowledge.
Fourth, you often need some unstructured data transformation, where free-text columns need to be reasoned over to make a decision. Then we meticulously corrupted open source data to make sure those properties hold. That was the hard part. It took about eight months.
How did the agents actually perform?
Tristan Handy: A lot of the results I’ve been following were more positive than yours. I think you had a very high bar, and the models didn’t perform that well.
Shreya Shankar: We have an updated version of the paper that I need to put on arXiv. We expanded the query set and performance went up about twenty percentage points, which isn’t surprising given the transformation that Opus 4.6 basically had. That class of models suddenly became much better at writing code. We also started using the native harnesses, like Claude Code, because we didn’t want to get into the business of harness engineering ourselves.
Right now the best agent is at about 61% pass@1 over our queries. If you look at our leaderboard you’ll see numbers in the 80s, but a lot of people are overfitting to the task and encoding exactly how to answer in the prompt. As a benchmark admin it’s hard to tell whether someone is cheating, because it’s easy to cheat. They write a specific prompt per query, which isn’t feasible or realistic. Even with prompt-engineered Fable, which must cost people an arm and a leg, maybe ten thousand dollars to run the benchmark, people are getting around 85%. That’s pretty good, but I think there’s still a big gap.
Do data agents work better with a semantic layer?
Tristan Handy: Analytics engineers and data engineers structure your data. They make sure it’s in the same relational store and that your keys are mapped. They build context around it, maybe implement the semantic layer. Then they hand it to a data agent, maybe through an MCP server, and the agents perform pretty reasonably. Is that consistent with what you found?
Shreya Shankar: It’s consistent. We have different variants of the benchmark, including one where we provide a semantic layer, and that significantly increases accuracy. It speaks to the importance of having a good semantic layer. OpenAI and Anthropic both have blog posts on how they’ve built their own semantic layers, and it’s a P0 for them to get their agents to do data analysis. I also feel that the importance of humans knowing what’s going on in the semantic layer is very high, because when agents generate it, it becomes slop, it becomes muck, and the downstream impacts are pretty bad. Building the semantic layer was the hard part for most queries.
Tristan Handy: My general takeaway is that if you plug data agents into raw data and let it rip, you’re probably going to have a bumpy experience. If you plug them into gold layer tables with a semantic layer, you’re probably going to do okay.
Shreya Shankar: That’s very spot on, but the number of people who had very good gold tables was very low.
Tristan Handy: Most of the time it’s less a question of who has good data and who doesn’t. There’s an island of cleanliness and then a sea of raw data, so you use agents wisely.
What did “Who Validates the Validators?” find about LLM judges?
Shreya Shankar: The paper is more apt than I expected, given how fast this world moves. It came out when GPT-4 had just launched the week before I ran the user studies, and people weren’t really running LLM judges yet. We felt it was going to happen, because it’s the only scalable approach to evaluating hard, somewhat subjective criteria. The agenda was never that you shouldn’t use LLM judges. Two years later they’re used everywhere, in training and in reinforcement learning too. Without them you just can’t scale.
We looked at how to give developers confidence that the judges are doing what they asked, using ideas from active learning and interactive machine learning: get labels from the developer on the fly and synthesize judge prompts that align with how they would evaluate outputs. We prototyped it in ChainForge.
I thought it would be a flashy paper saying these ideas apply to LLM judges and it all works. It’s not going to work that simply, because human preference depends on what you’re seeing. People don’t know what they want until they see the outputs, and they have to see enough of them to develop a theory of how the model works before they feel confident. We call this criteria drift. You think you want something, you see many outputs, and it changes. Partly you’ve learned something about the problem domain, since we’re asking LLMs to do hard tasks. And partly you want to adapt your rubric to what LLMs do well.
If a model is very good at summarizing a facet of product reviews, I want to rely on that and change my rubric accordingly. That’s still true today. When we run LLMs over all of our data and want to judge the outputs, we have to look at enough samples ourselves to feel confident in the rubric before we trust that the judges are good.
Why defend AI evals?
Shreya Shankar: I spent two hours writing that post, so maybe I could have thought about it a little more. It was inspired by debates on Twitter. A new model had come out, everyone said it was really good, and the sentiment was that all the time spent on evals was dragging people down, so we should just rely on the models getting better. I think that’s silly on its face, but it spoke to a real frustration about how to do evals. People had probably invested in prompt engineering or automated prompt improvement to hill climb on their evals, and found that none of it panned out compared to simply waiting for the next model.
My takeaway is that the thing that pushes the Pareto frontier of quality and cost the most is a new model. Prompt engineering and harness improvements are great, but in the hands of the end user they’re limited. I think people felt a little bit lied to, because they thought they could push that frontier much further.
Tristan Handy: A year or so later, a lot of the gains are coming from the surrounding harness and your context. Do you agree?
Shreya Shankar: It’s twofold, and I don’t know how much is the harness and how much is the model. I can see a difference between Fable and Opus, probably using the same harness, and that difference is more striking to me than the difference between the Cursor harness and the Codex harness on the same GPT model. Someone will fight me on this. But there’s still a striking difference between models, and it’s jagged. For some tasks Fable is so much better, and for others, more rote ones, Sonnet may be good enough. You’re only ever seeing a small slice of the world of tasks.
How should data practitioners start doing evals?
Shreya Shankar: The concept of looking at your data is drilled into you as a data analyst or data scientist, and it’s paramount now, because AI generates stuff and you have to look at it to say anything intelligent about what the agent is doing. The new part, which is tricky, is that exploratory data analysis doesn’t cleanly map from structured data analytics to agents operating over unstructured data. There’s no equivalent of df.summary(), or “what are my outliers?” You can define those in your favorite data framework, but you can’t define them on your agent’s output. So we don’t have the tools yet to accelerate the patterns we know for working with data.
Imagine doing EDA with no pandas functions. You would load the table in Excel and look row by row and think, I don’t know what I’m doing. That’s the analog of how we work today. We look at a spreadsheet of LLM outputs row by row and hope for the best. We define a rubric, run it at scale with another LLM judge, and look at those results in a spreadsheet. I don’t have advice other than to look at things row by row, because unfortunately that’s all we’ve got right now. You can try your EDA tricks, like embedding your rows and clustering them, but it’s hard to say it will help. The embeddings might not be a good representation of the outputs, and they’re not interpretable, so it’s not as clean as clustering structured data. You’re going to have to write the code yourself, because there are no libraries for it. But look at your data, and try to do it in a principled way.
What are you working on next at CMU?
Shreya Shankar: I’m joining a database group I’ve admired for my whole PhD, the CMU Database Group, a very eclectic bunch of brilliant people. I’m excited to bring more AI and application-oriented work to the group, and to go deep into system internals and learn from everybody there. One big area is AI SQL. Snowflake, Google BigQuery, and Databricks have shipped a variant of SQL that lets you write AI operators in your queries, like filtering all your customer reviews for people who complained about the product in a particular way. It’s bringing the database companies a lot of revenue and everybody is thrilled, except for people who accidentally spend hundreds of thousands of dollars on their queries, which apparently happens all the time. I think it’s way too expensive, and we haven’t even scratched the surface of how to do query optimization here.
Right now, the way these query plans are executed is basically wrappers around ChatGPT. They run a prompt over every row of the database, sending a separate HTTP request to OpenAI for each one. We can do much better. A lot of the optimizations we play with are about reducing the number of LLM calls, like sampling when you’re doing aggregation, or using small models whenever possible. But that’s still the row-by-row approach. So one project is a custom inference engine for AI SQL, something like vLLM, where you do query planning and logical rewrites so that all of the AI operations are pre-fill oriented, with zero decode everywhere.
One barrier is that databases don’t run next to GPUs. That’s how the world has worked in the past. My hope is that AI changes this and it becomes important for a database to run next to a few GPUs, because they want to store more data or because AI agents are going to be in the database.
Tristan Handy: I really think there’s not enough work that bridges these two domains. The data people want the AI stuff, but the implementation is often trivial and naive. We need the database folks who think about query planners and database internals to get on the task.
Chapters
00:00 – Welcome
00:21 – The note for AI agents on Shreya’s website
01:29 – Where the note came from, and what it did to her inbox
04:19 – “Just don’t use AI to write emails”
04:34 – Stanford, Google Brain, and a data analytics startup
06:31 – Why data engineering problems led to a database PhD
07:15 – Why databases and AI sit in different departments
09:08 – The Berkeley PhD, ChatGPT, and DocETL
11:01 – Should agents interact with data differently than humans?
12:58 – The inner loop, the outer loop, and pre-attentive processing
14:33 – Designing tools and MCP servers for agents
16:11 – Why SQL sticks around
17:05 – What is a data agent?
19:06 – Why existing benchmarks fall short
20:50 – Can an LLM do relational algebra in context?
23:06 – Building a realistic benchmark: four properties
26:40 – Results: 61% pass@1, and why leaderboards mislead
28:44 – Analytics engineering meets data agents
30:14 – The semantic layer variant, and why humans must understand it
31:35 – Raw data versus gold tables
32:33 – “Who Validates the Validators?”
35:11 – How to build an LLM judge that works
37:38 – Meta, advertising, and closed-loop evals
39:18 – “In Defense of AI Evals”
41:31 – Model versus harness
42:56 – How practitioners should start with evals
46:08 – Headed to CMU
47:26 – AI SQL and why it costs too much
49:05 – A custom inference engine for AI SQL
50:21 – Wrap-up
This newsletter is sponsored by dbt Labs. Discover why more than 80,000 data teams use dbt to accelerate their data development.
Agents doing select * from two databases and joining in pandas get expensive fast on a warehouse. BigQuery on-demand bills by bytes scanned and a LIMIT doesn't lower that, so I'd set maximum bytes billed on every query the agent's tool runs before it gets near a wide fact table.