Something I'd like to see SDF enable in dbt is more ORM-like abstraction around the granularities and features derived through dbt models.
Today, accessing some column defined in a dbt model means referencing the full output table produced by that model - taking the whole thing as a dependency. Imagine if an ORM only let you define whole SQL queries. The default outcome is bottlenecks, and spaghetti lineages.
Theoretically, if you set every model's materialization to `ephemeral` you could immediately see better performance as the query plan for your DAG would be optimized automatically. This wouldn't actually work today, but I think SDF is positioned to make this more of a reality. This would open up development abstractions that are easier to use, and more performant by default.
Something I'd like to see SDF enable in dbt is more ORM-like abstraction around the granularities and features derived through dbt models.
Today, accessing some column defined in a dbt model means referencing the full output table produced by that model - taking the whole thing as a dependency. Imagine if an ORM only let you define whole SQL queries. The default outcome is bottlenecks, and spaghetti lineages.
Theoretically, if you set every model's materialization to `ephemeral` you could immediately see better performance as the query plan for your DAG would be optimized automatically. This wouldn't actually work today, but I think SDF is positioned to make this more of a reality. This would open up development abstractions that are easier to use, and more performant by default.