Optimism and "Green Shoots"
The one where I express a dangerous level of optimism about the economy, and what changes that could bring for the data industry.
The final episode of season 4 of the Analytics Engineering Podcast just dropped! In it, we got into what it’s like to collaborate with data at scale. Ben Flusberg (Cox Automotive) and Mortitz Heimpel (Siemens) share their experiences adopting a data mesh architecture and what that looks like at their organizations.
There’s also a fascinating discussion about internal charge-back models and how they impact the incentives of users of a data platform. We went surprisingly deep on this topic and I found it fascinating; I’d love to hear your thoughts.
I’ve been thinking a lot about the economy over the past couple of weeks, and about a post I wrote ~20 months ago now. In it, I described what I anticipated would become a significant change in the economy and what the impacts to data practitioners would be.
Given that the Fed will slow the economy by increasing rates (to combat rising inflation), will it push the economy into an outright recession? Maybe—smarter people than me certainly think so. And if that happens, will data budgets get downsized across the industry? This is hard to say, but it’s certainly reasonable to believe that some will.
Both of these things absolutely happened. I won’t give evidence on the Fed funds rate side; we’ve all been living through that. There are a million CIO surveys you can find over the past year and a half to illustrate the point on budgets; depending on which one you look at you’ll find that overall data spend (headcount + technology) has declined at a rate of ~5% YoY over this period, down from 5-10% YoY increases.
This represents a meaningful decline within a huge industry, but those averages hide how the change plays out in the wild. The number of data teams we work with who have experienced significant downsizing, or been laid off altogether, over the past 18 months is meaningful.
It’s hard to watch. Data practitioners haven’t done anything “wrong” here, it’s just the inevitable grinding forwards of the macroeconomic machine: starting with artificially low interest rates, continuing to COVID supply shocks, ensuing inflation, and the the fastest-ever rate-raising regime. The entire stock market dropped something like 20% in 2022 as these impacts played out in equity prices and conservatism flowed through to budgets.
So: the headcount and technology budgets for data teams have taken a real hit over the past couple of years. This has impacted so much about our industry.
Lower willingness / ability to adopt new ideas and technologies.
More time focused on cost-optimization projects.
Less practitioner time spent sharing work publicly or contributing to open source.
Fewer new data startups getting traction and funding. There are more acqui-hires than notable new companies getting founded. Just less energy and new ideas floating around.
Have you felt it? It’s been visceral for me—the impacts show up everywhere. It’s…honestly kind of a bummer. Data practitioners are, as a whole, people who love technology, who love progress and get excited about finding ways to get better. Sure, it’s fun to find a $100k optimization in your data platform spend every once in a while, but we are all aware of the fact that our organizations’ data problems are nowhere near solved, and holding off on forward-looking projects until budget frees up is always frustrating. And there’s a lot of that happening right now.
The question, though, is: for how long? If we’ve been in this world for ~20 months at this point, how long will it persist? Certainly, we shouldn’t expect to return to levels of ZIRP-fueled euphoria from 2021, but we should expect the tide to turn at some point, right?
This is the reason this whole topic has been on my mind of late. In the past couple of months there have been some signals that we might be moving into a new part of the cycle. While I don’t have up-to-date CIO survey data data to share, there are a couple of other useful places to look. The first is public company earnings. From Clouded Judgment:
Andy Jassy doubled down on some of the commentary he’s made in the last few weeks about cloud optimization headwinds starting to abate. On optimizations he said they’ve: “largely attenuated / not nearly at the same rate as before”… “feel good about deal growth in the last few months.” For many companies, seat based contracts have been right sized, consumption based usage has been right sized, and a lot of the software they could bundle into a larger vendor has been bundled. It appears 2024 will be more about building out new functionality vs “getting fit.”
In short: AWS is seeing investment in new innovation starting to outpace cost-savings efforts. From Snowflake’s earnings:
"Echoing commentary from other (software and cloud) vendors, Snowflake noted continued stabilization in consumption from customers, with September growth exceeding expectations," said William Blair analyst Jason Ader in a report. "This improvement in consumption is being driven by expansions within Snowflake's largest customers (9 of the top 10 customers grew sequentially) as they lap optimization initiatives and refocus on migrating away from legacy data warehouses."
Similar themes: an increased focus on future-oriented work vs. cost-optimization.
These are, of course, just ways to observe the behavior of hundreds of thousands of individual companies in aggregate. But what’s causing this behavior? Is it transient or indicative of a lasting trend?
The underlying story is simply that macro is stabilizing. The economic picture from 20 months ago:
artificially low interest rates, continuing to COVID supply shocks, ensuing inflation, and the the fastest-ever rate-rising regime
…has reversed, with interest rates and supply chains combining to return inflation within shouting distance of target levels. See below:
Core PCE, the Fed’s primary measure of inflation, is nearly back to the target range. We haven’t experienced a recession. Or unemployment above 4%. The above Twitter thread makes what is probably the strongest argument for a-soft-landing-is-already-happening that I’ve read, and Goldman Sachs is reporting that:
Members of the Fed have seemingly reinforced the market’s view that we may see potential rate cuts in coming months.
A reduction in the federal funds rate, of course, flows through to all parts of the economy, making it more attractive to invest for the future rather than simply find ways to optimize cash flow today. This impacts startup valuations and formation, but also corporate data budgets and projects.
And, meanwhile, the Bureau of Labor Statistics shared one of the most optimistic economic numbers I’ve seen for a while: Q3:23 saw a 5.2% increase in labor productivity. Given that labor productivity is one of the primary drivers of economic growth, this number demonstrates a high ceiling on the growth capacity of the economy. I bet it’s the impact of all those Microsoft Office Copilot licenses.
Whether or not you’ve started to feel this in your day-to-day life, the post-COVID economy actually may be sorting itself out (despite multiple ongoing wars and plenty of other global risk factors). It’s too early to say any of this with too much confidence, but it’s also irresponsible to fail to note the very significant positive trend lines we’re observing.
The question I’m left with, then, is about the future. IF companies, and investors, and startups, all move on from the current phase into the next phase…what does that next phase look like? Some guesses:
Modern cloud data platforms have solid customer counts, but if you look inside of a given large company you’ll likely find that only a modest % of all of their workloads are in the cloud. The number of companies that have 95% of their data workloads still on-prem is quite high. Or 95% of their data pipelines still flowing through technologies that were created 30 years ago. It is very common for cloud migration projects to start off with small, targeted investments. Over the past few years, the scale-out part of the project has paused to wait for budget. This will re-accelerate.
Additionally, there are huge user populations that are simply unaddressed by centralized data platforms today, left to fend for themselves in the land of “shadow IT”. This is bad for users, and it’s now untenably risky for companies given heightened data security challenges and compliance requirements. Expect these users to join the party.
The pace of org change picks up.
Slapping new technologies onto existing org structures is like taking electric motors and putting them inside of factories designed around steam: it produces very modest impact. In order to truly benefit from new technology, you need to re-shape your organization around it.
We’ve done this inside of our G&A function. Our operations for everything from finance to accounting to IT has massively changed in the past couple of years as we’ve made deep investments in automating the traditionally-manual processes involved in these back-office functions. We’ve re-shaped our team around this new reality.
From my viewpoint, the industry is quite immature along this dimension. But this is exactly the work of the deployment phase, and it’s work that will start to reaccelerate as organizations look towards the future again.
For a long time, data teams were constantly trying out new things. This made sense: a few new technologies had made massive improvements in productivity, and teams were looking for that next big uplift. But cuts to tooling budgets and reduced team sizes meant that experimentation was put on the back burner and teams executed with the tools and approaches they had.
My anticipation would be that a new environment will contain a modest but healthy amount of experimentation around tools. Some of these areas will likely be AI-enabled, including codegen for data pipelines and conversational analytical interfaces. Others will be focused around ideas that the industry has had for a little while but haven’t yet had their moment.
My mental model for this new world, should we be fortunate enough to find ourselves in it, is not a return to the ‘frenzy’ phase of 2020-2021. Instead, it’s the deployment phase. Practitioners putting technology to work, getting the real productivity benefits from it, and continuing to find adjacent technologies that help unlock the benefits of the core underlying platform.
The deployment phase is where most of the benefits of disruptive technologies actually show up. Widespread usage and system-level adaptation bring technologies previously reserved for early-adopters to the economy as a whole.
Of course, I have to close by saying: there are a million ways that we could get knocked off the path we seem to be on at the moment. And all of my default assumptions going into 2024 remain conservative. The above is not investment advice, it’s a thought experiment.
But still. I do think this data is pointing to something real, and I think we should keep our eyes open.