Somewhere in your organization right now, a team's AI budget dashboard says they have $4,000 left this month, and their actual provider account will cut them off at $500. Both numbers are "correct." Neither of them knows about the other. Welcome to the two-layer budget problem.
Every platform that lets multiple teams spend real money against a shared LLM provider eventually has to answer an uncomfortable question: who actually stops the spending when it needs to stop? The honest answer is that you need two separate answers, at two separate layers, and the entire discipline here is refusing to let them merge into one.
The Two Layers, Named
The soft layer is your platform's own bookkeeping: per-team, per-project, or per-environment allocations that you track, display, and use to make decisions like "should this request be allowed" or "should we notify a budget owner." It's soft because it's advisory - a number your system computed and believes, not a number the money actually respects.
The hard layer is whatever your LLM provider (or your cloud billing account) will actually, physically enforce: a spend cap, a rate limit, a hard stop that doesn't care about your dashboards, your Slack alerts, or your feelings. It's hard because when it trips, requests fail. No appeal.
The tempting shortcut is to treat these as one thing - set your soft allocation and assume it's also the enforcement mechanism, or set a hard cap and assume your dashboard will always agree with it. Both shortcuts fail in the same way: eventually, the two numbers disagree, and now you have a problem that looks like a bug but is actually a design gap.
How They Drift Apart
Consider a scaling logistics company running an internal AI platform for forty engineering teams. The platform team builds a soft allocation system: each team gets a monthly token/dollar budget, tracked in a database, decremented as usage comes in. It's fast, it's queryable, it renders nice dashboards. Meanwhile, the actual bill is paid through a single shared provider account with its own cap, set by finance, who has never heard of the platform's internal allocation table and never will.
For a while, this works, because usage is well under both ceilings. Then one team ships a feature that triples their token consumption overnight. The soft layer flags it, sends a Slack alert, maybe throttles new requests. But the soft layer's throttle only stops new platform-mediated requests - it does nothing about background jobs, retried batch pipelines, or anything that doesn't pass through the platform's own gate. The hard cap, meanwhile, has no idea any of this internal bookkeeping exists. It just watches total spend on the account, and when it crosses the line, it cuts everyone off - including the twenty other teams who never came close to their soft allocation.
This is "trapped capacity" in its purest form: the soft layer believes there's $4,000 of headroom, distributed fairly across teams; the hard layer has already run out. Nobody can spend the capacity the dashboard says exists, because the dashboard was never wired to anything that actually enforces spend.
The inverse failure is just as common and arguably more infuriating: a hard cap set generously high (because nobody wants a false-positive outage), paired with a soft allocation system that's overly conservative and blocks legitimate requests well before the money would actually run out. Now you have real, paid-for capacity sitting unused because your own platform is more restrictive than the wallet backing it.
Why You Can't Just Merge Them
The obvious "fix" - just make the soft layer be the hard layer, i.e., give the platform's allocation table the actual power to cut off provider access - sounds appealing until you remember that the soft layer is, definitionally, a system you built, with your bugs in it. If your allocation-tracking service has an outage, a race condition, or a bad deploy, do you want that to be the only thing standing between your organization and an unbounded bill? Almost certainly not. You want the provider's own billing cap as a backstop that works even when your platform is on fire. That's the entire point of having two layers: the soft layer optimizes for good UX and fair distribution; the hard layer exists purely as a circuit breaker that doesn't trust your code.
The mental model that's worked for us isn't "which layer is correct" - it's "which layer is responsible for which failure mode." The soft layer's job is to make spending legible and fair before anything goes wrong. The hard layer's job is to make sure that when everything else goes wrong, the bill still has a ceiling. We won't pretend this is the only valid split - some teams we've talked to fold the hard cap into the platform entirely and accept the risk, and for a smaller org that might genuinely be the more pragmatic trade-off.
What's Worked for Us So Far
Treat the two layers as intentionally loosely coupled, with each one's ceiling set with headroom for the other's error margin. This is the version we've landed on after a couple of false starts - concretely:
- The hard cap should sit comfortably above the sum of soft allocations, with a buffer for burst usage that the soft layer hasn't caught up to yet - not equal to it.
- The soft layer should degrade gracefully and loudly when it can't confirm its own state (don't fail open silently; alert and slow down).
- Someone - a real owner, not a shared inbox - needs to reconcile the two on a schedule, because they will drift as usage patterns and provider pricing change.
- Never let a single team's soft-layer failure become a hard-layer incident for everyone else. If one team can trigger the account-wide cap, your soft layer isn't actually doing its job of containment.
Pitfalls
The most common mistake isn't technical, it's organizational: the soft layer is usually owned by the platform team, and the hard layer is usually owned by finance or a cloud/procurement team who set it once and forgot about it. If those two groups aren't talking, the two layers will silently diverge until an outage forces the conversation. Put a recurring reconciliation review on the calendar before you need one.
The second most common mistake is building elaborate self-service budgeting UX on the soft layer and then routing every exception - every "I need more budget" request - through email. That one's exactly the failure mode we'll dig into next: Self-Service Limits Need an Audit Trail, Not an Email Thread.
Where we're genuinely still not sure: how much buffer to build into the hard cap above the sum of soft allocations. We've picked a number that feels safe, but it's a guess dressed up as a policy, and if you've found a more principled way to size that buffer, we'd like to hear it.
Suggested hashtags: #EnterpriseAI #AIInfrastructure #AIPlatform #AIGovernance #PlatformEngineering #FinOps #CostManagement #CloudCosts #LLMOps