The FinOps Foundation’s State of FinOps 2023 survey finds that “getting engineers to take action on cost optimization” is once again the #1 FinOps challenge. The survey is not specific to cloud data environments, but virtually every challenge is highly relevant to data teams, arguably even more so.
Source: FinOps Foundation
Data engineers find it challenging “to take action on cost optimization” not because they’re obstinate (not usually), but because they simply don’t have the insights to optimize costs. In our experience, data engineers and Ops teams and everyone else wants to do the right thing (spend most cost-effectively); they just don’t have the information they need to do the right thing.
This is the issue MegaBankCorp ran into. Sammi tasked his engineering team leaders to have their people revisit what they have requested to “rent” from Databricks and AWS: measure what was asked for vs. what was actually needed. Just from spot-checking a couple of dozen jobs, he knew that a lot of the Spark jobs were calling for resources with more memory or compute power or storage than they needed to run successfully. Not to mention some persistent code inefficiencies that were making the jobs run longer than necessary.
What happened is that a couple of thousand engineers spent a lot of their time trying to do something they’re not good at, something they never trained for. They spun their wheels in time-consuming manual detective work and trial-and-error, hit-or-miss efforts (ironically, driving up the monthly cloud data bill), and still many never really cracked it. They were spending more time doing that than developing new applications.
For the individuals on Sammi’s teams to make the most cost-effective choices about configuring and running their particular jobs—which is where a lot of the money is being spent—it has to be easy for them to do the right thing. That means showing them what the right thing is. They don’t need more charts and graphs that, while showing a lot of useful information, still leave it in their shoulders to figure things out. They really just want to know how to fix things.
They really need AI.