I love this experiment. One thing that's important to remember - is that you experienced success within two hours in a very simple situation (as you stated). When you try to extrapolate this to a real customer environment, it can be 100x or even 1000x more complicated. It's not simply linear with the number of tables or models.
The reasons are all the usual issues: data is messy, businesses are complex, there isn't a single truth for how to calculate certain metrics, and so forth.
We've been working on it for over a year, a team of 17 people, and it's hard. But there's light at the end of the tunnel!
At Mammoth Growth we have build custom agents in Roo Code to build production code for our clients for the last few months. It’s real - but requires you shifting your effort to planning and verifying. We look forward to demoing at Coalesce.
These tools have gone from “what’s the right syntax for a concatenate function in Snowflake” to “here are some notes on my model along with a list of field names and mappings to foreign keys - can you build me an initial model and a fact table, along with some simple tests to prove that this is working? When you’re done, update an agent-context markdown file to keep your notes up to date.” And it works!
I would love for you to do a follow up post if you have time where you actually do validate the models and try to test some of the utility. I feel like a lot of posts about this stuff document the first stage (getting something up and running) but not the later parts, and the later parts are what we really need to understand now. How do we get from a working prototype to a usable system?
I hope this doesn't sound like a criticism because I actually love this post and I agree with you fully, I just think we're ready to start moving on to the next stage. Otherwise we'll get stuck forever in this stage where everyone is building prototypes that never make it to production, and we still can't prove out the real utility (or lack thereof).
I completely agree with you - the goal here is absolutely to find areas to take this into production. imo experiments like this are necessary but not sufficient - we've proven that it can pass the "toy project" experiment, next would be "pilot on a specific team or use case". Watch this space
I love this experiment. One thing that's important to remember - is that you experienced success within two hours in a very simple situation (as you stated). When you try to extrapolate this to a real customer environment, it can be 100x or even 1000x more complicated. It's not simply linear with the number of tables or models.
The reasons are all the usual issues: data is messy, businesses are complex, there isn't a single truth for how to calculate certain metrics, and so forth.
We've been working on it for over a year, a team of 17 people, and it's hard. But there's light at the end of the tunnel!
Completely agree!! The gap between "cool 2 hour demo" and "useful in production" can be massive. Would love to hear about what you've been working on.
At Mammoth Growth we have build custom agents in Roo Code to build production code for our clients for the last few months. It’s real - but requires you shifting your effort to planning and verifying. We look forward to demoing at Coalesce.
These tools have gone from “what’s the right syntax for a concatenate function in Snowflake” to “here are some notes on my model along with a list of field names and mappings to foreign keys - can you build me an initial model and a fact table, along with some simple tests to prove that this is working? When you’re done, update an agent-context markdown file to keep your notes up to date.” And it works!
I would love for you to do a follow up post if you have time where you actually do validate the models and try to test some of the utility. I feel like a lot of posts about this stuff document the first stage (getting something up and running) but not the later parts, and the later parts are what we really need to understand now. How do we get from a working prototype to a usable system?
I hope this doesn't sound like a criticism because I actually love this post and I agree with you fully, I just think we're ready to start moving on to the next stage. Otherwise we'll get stuck forever in this stage where everyone is building prototypes that never make it to production, and we still can't prove out the real utility (or lack thereof).
I completely agree with you - the goal here is absolutely to find areas to take this into production. imo experiments like this are necessary but not sufficient - we've proven that it can pass the "toy project" experiment, next would be "pilot on a specific team or use case". Watch this space