Here are my thoughts on AI data workflow integration: number one use - reformatting and light cleaning. GPT works well for getting text out of an image, for reformatting code, for doing things you could do with regex but don’t feel like writing regex, for writing an annoyingly long case statement, pivoting or in pivoting data, etc. GPT hasn’t worked so well doing any kind of analysis with any complexity but it can do a little bit. My favorite use case that turned out pretty well was converting SQL between SQL Server syntax and Snowflake syntax. I would say it saved time but definitely wasn’t perfect.
Also - in the realm of dbt cleanup I’ll add implementing new dbt features. I know I have some projects from around 2019 that are still in production that I doubt anyone went back and added the concept of sources, to them much less any concept of metrics or semantic models, mesh, etc. AND I know I didn’t really document them very well. The use case was honestly version control for database views.. and that’s about it. Now I would add a lot more - but I suspect many companies may just be stuck at the “version control” stage. I say stuck - but still in a way better spot than before...
I see the same with GPT use cases - It's a good powertool for writing SQL, but it isn't good when you need to do context-heavy data analysis tasks. And it's not about knowing what each column of a table means, it's more about the amount of business context you need to produce a valuable analysis / data product.
OSS/hashicorp: see open tofu, open bao - terraform/vault were not big/complicated enough to be monopolized successfully, and the license allowed branching.
Here are my thoughts on AI data workflow integration: number one use - reformatting and light cleaning. GPT works well for getting text out of an image, for reformatting code, for doing things you could do with regex but don’t feel like writing regex, for writing an annoyingly long case statement, pivoting or in pivoting data, etc. GPT hasn’t worked so well doing any kind of analysis with any complexity but it can do a little bit. My favorite use case that turned out pretty well was converting SQL between SQL Server syntax and Snowflake syntax. I would say it saved time but definitely wasn’t perfect.
Also - in the realm of dbt cleanup I’ll add implementing new dbt features. I know I have some projects from around 2019 that are still in production that I doubt anyone went back and added the concept of sources, to them much less any concept of metrics or semantic models, mesh, etc. AND I know I didn’t really document them very well. The use case was honestly version control for database views.. and that’s about it. Now I would add a lot more - but I suspect many companies may just be stuck at the “version control” stage. I say stuck - but still in a way better spot than before...
I see the same with GPT use cases - It's a good powertool for writing SQL, but it isn't good when you need to do context-heavy data analysis tasks. And it's not about knowing what each column of a table means, it's more about the amount of business context you need to produce a valuable analysis / data product.
OSS/hashicorp: see open tofu, open bao - terraform/vault were not big/complicated enough to be monopolized successfully, and the license allowed branching.