Something I'd like to see SDF enable in dbt is more ORM-like abstraction around the granularities and features derived through dbt models.
Today, accessing some column defined in a dbt model means referencing the full output table produced by that model - taking the whole thing as a dependency. Imagine if an ORM only let you define whole SQL queries. The default outcome is bottlenecks, and spaghetti lineages.
Theoretically, if you set every model's materialization to `ephemeral` you could immediately see better performance as the query plan for your DAG would be optimized automatically. This wouldn't actually work today, but I think SDF is positioned to make this more of a reality. This would open up development abstractions that are easier to use, and more performant by default.
@Jay Sobel i felt the same way so I compared drizzle to dbt for a few simple queries, i think the results are valid and they reinforce your point, I’d like your take as I’m not an expert in ORM but always felt like dbt lacked something in that direction
Something I'd like to see SDF enable in dbt is more ORM-like abstraction around the granularities and features derived through dbt models.
Today, accessing some column defined in a dbt model means referencing the full output table produced by that model - taking the whole thing as a dependency. Imagine if an ORM only let you define whole SQL queries. The default outcome is bottlenecks, and spaghetti lineages.
Theoretically, if you set every model's materialization to `ephemeral` you could immediately see better performance as the query plan for your DAG would be optimized automatically. This wouldn't actually work today, but I think SDF is positioned to make this more of a reality. This would open up development abstractions that are easier to use, and more performant by default.
@Jay Sobel i felt the same way so I compared drizzle to dbt for a few simple queries, i think the results are valid and they reinforce your point, I’d like your take as I’m not an expert in ORM but always felt like dbt lacked something in that direction
https://open.substack.com/pub/groupby1/p/a-modern-data-benchmark
hello
<h1>tester</h1>