Every org building with AI right now is running an uncontrolled experiment, and the control group is "everyone who hasn't noticed yet".
That sentence sounds dramatic for a blog post about infrastructure. It isn't. If you're a modern, cloud-native company - no mainframes in the basement, no on-prem Active Directory (AD) forest from 2004 - you probably don't have shadow IT in the classic sense anymore. SaaS sprawl got tamed years ago with SSO and a procurement form. But shadow AI is sprawling right now, today, inside companies that think they've already solved this problem, because it doesn't look like the old problem. Nobody's expensing a rogue server. They're just calling an API from inside a script that lives in someone's personal repo, using a personal key, logged nowhere anyone will ever read.
This post is the "why" before the "how". The rest of this series is going to get deep into architecture - environments, identity federation, budget ledgers, agent memory tiers. None of that matters if you haven't first made the case, to yourself or to whoever signs off on headcount, that a platform is the right unit of investment instead of a pile of individually excellent point solutions.
The problem isn't that people are using AI. It's that you can't see it.
Here's the pattern, and if you've been at a scaling company for more than a year you've probably already lived a version of it:
A team needs to summarize customer tickets. They get an API key from a provider, wire it into a Lambda or a Cron job, and ship it in an afternoon. Great - fast, cheap, no platform team needed. Three months later, another team wants to draft outreach emails. They also get a key, because why would they know the first team already has one? Six months later, a third team builds an agent that reads internal documents and answers questions about them, using a third key, stored in a third place, with whatever permissions that particular cloud console defaulted to.
Now fast-forward a year. You have no idea:
- How much you're spending on AI in total, because it's spread across a dozen billing lines that nobody's centrally rolled up.
- Which teams have API keys with access to which data, because nobody centrally issued them.
- What happened when one of those keys leaked in a public repo for six hours before someone noticed - because nothing was logging who used it for what.
- Whether the "agent that reads internal documents" is reading documents it should not be reading, because nobody scoped its permissions to anything narrower than "read everything the service account can see".
Individually, every one of these teams did a reasonable thing. Collectively, you've rebuilt shadow IT, except the payload this time isn't an unsanctioned Dropbox account - it's a live pipe into a frontier model with your company's data flowing through it, and no fence around any of it.
Everyone is re-solving the same five problems, badly, in parallel
The deeper cost isn't the risk (though we'll get to the risk). It's the waste. Every team that wires up its own LLM integration ends up independently solving the same handful of problems:
- Auth - where does the API key live, who can rotate it, what happens when someone leaves the team.
- Secrets - is it in an env var, a config file that got committed by accident, a "temporary" hardcoded string that's now eighteen months old.
- Rate limiting and retries - what happens when the provider throttles you mid-incident, at the worst possible time.
- Logging - is there any record of what was sent to the model and what came back, for debugging or for the compliance team's inevitable question.
- Cost control - is there a ceiling on spend, or does a bug in a retry loop turn into a bill that surprises someone in finance.
None of this is exotic engineering. It's the same twenty lines of boilerplate, rewritten with subtly different bugs, by every team that touches an LLM. That's not "innovation happening at the edges" - that's the same tax being paid five times over, by five different teams, none of whom get to specialize in paying it well.
A platform means exactly one team gets good at auth, secrets, rate limiting, logging, and cost control for AI traffic - and everyone else gets to skip straight to the part where they actually build the thing they wanted to build. That's the entire pitch. Everything else in this series is detail.
The moment "just call the API directly" stops working
Imagine a 200-person, cloud-native fintech. For the first year of using LLMs, "each team gets its own key" is completely fine - genuinely, don't build a platform for three teams and a spend of a few thousand dollars a month, you will over-engineer yourself into irrelevance. The trigger points for when this stops being fine tend to cluster around a few recognizable moments:
- Team count. Somewhere around eight to twelve teams independently calling model providers, the fact that nobody can answer "which teams are using AI, and for what" stops being a curiosity and starts being an actual operational blind spot.
- Spend. When the AI line item is big enough that finance asks for a breakdown by team or product, and the honest answer is "we'd have to go ask each team", you have already lost the argument for staying decentralized.
- The first compliance question. Someone - a customer's security team, an internal auditor, a regulator - asks "what customer data has been sent to a third-party model, and under what data-processing agreement?" If the answer requires interviewing every engineering team individually, you don't have an answer. You have a fire drill.
- The first security incident. A leaked key, an over-permissioned agent, a prompt-injected tool call that did something it shouldn't have. The first one is a wake-up call. Whether it's also the last one depends entirely on whether you build the platform now or wait for the second one.
None of these triggers require you to be a Fortune 500 with two decades of technical debt. A three-year-old, forty-engineer company can hit all four in eighteen months if it's growing fast and shipping AI features aggressively - which, if you're reading this in 2026, you almost certainly are.
What "the platform" actually needs to be
To be clear about scope, because "AI platform" gets used to mean everything from "a Slack bot" to "a full agentic operating system": in our experience, the minimum useful version centralizes identity and secrets for AI traffic, gives you one place to see spend and enforce budget, gives every call an audit trail, and gives security a single surface to reason about instead of a dozen. That's the line we drew, anyway - it does not need to mean every team loses autonomy over what they build. The best version of this we've seen makes the golden path so much easier than the DIY path that nobody wants to go around it; coercion is a fallback, and a good platform mostly wins on convenience. Other orgs draw this line differently, sometimes centralizing more, sometimes less, and I am genuinely not certain mine is the optimal cut, but it served over ten thousand teams with more than a hundred thousand users.
The two forks you'll hit immediately
Once you've decided to build rather than let sprawl continue, you run into two forks almost immediately, and they shape everything downstream:
Which cloud and identity ecosystem does this platform live in? If your company runs on Google Workspace, your identity federation, your secrets management, and even your default model provider relationships look meaningfully different than if you're a Microsoft 365 / Azure shop. This isn't a minor implementation detail - it changes how agents authenticate, how tokens get exchanged, and which primitives you get for free versus which you have to build yourself.
How much do you centralize versus let teams self-serve? A platform that requires a ticket and a three-day wait to register a new model is a platform people will route around, and you're right back to shadow AI - just with extra resentment. A platform that lets any team spin up unlimited spend with zero guardrails is not a platform, it's a slightly nicer-looking version of the problem you started with.
Get the balance right, and the platform becomes the fastest way to ship an AI feature, not an obstacle to it. Get it wrong in either direction, and you'll find out - expensively.
The next post takes on the first fork directly: what actually changes in your architecture depending on whether you're building this on top of Google Workspace or Microsoft 365 / Azure, and where the two paths are more alike than the marketing decks would have you believe.