We learned this one the mildly stressful way, so let's save you the trouble: if your AI platform has exactly one environment, you don't have an environment strategy. You have a live production system that everyone is quietly terrified of touching, which is a different thing, and a worse one.
Traditional software engineering settled the dev/staging/production question decades ago. Nobody ships a code change straight from a laptop to the customer-facing server anymore - well, almost nobody, and the ones who do have a story they tell at conferences with a nervous laugh. But AI platforms are young enough that a lot of teams are still treating "the model registry" or "the agent config" as one flat, global thing, promoted nowhere, tested against nothing but hope.
What breaks without real environments
Picture a scaling logistics company running an internal AI platform. One team wants to try out a newly released model for a customer-support agent - genuinely reasonable, everyone wants to try the new thing. They register it. It's now sitting in the same model registry as every production model, addressable by every service that knows how to ask the registry for a model by name.
A week later, someone building a completely unrelated production feature queries "what models are available" and gets the experimental one back in the list, because nothing distinguished it. Or worse: the experimental registration shares a name or a routing rule with something already serving production traffic, and now an in-progress experiment is quietly eating into a live customer path. Nobody did anything wrong, exactly. There just wasn't a wall where a wall needed to be.
This is the core case for environments in an AI platform specifically: model registrations, agent configs, budget allocations - these are exactly the kind of thing you want to test, tune, and break in a space that provably cannot touch anything a customer is relying on right now.
The topology we landed on
We ended up with a fairly conventional shape - dev, staging, and production - with a few extra rules that turned out to matter more than the basic three-tier split itself. This isn't the only way to slice it, and if your org needs a fourth tier for something like a regulated-data environment, that's a very reasonable variation on the theme, not a deviation from it.
Model registrations are per-environment, not global. A model registered in dev is invisible to staging and production. It has to be explicitly, deliberately registered again in the next environment up - which sounds like friction, and is, on purpose. The alternative - one global registry with an environment tag on each entry - sounds more elegant right up until someone forgets to check the tag, or a query forgets to filter on it, and now you're back to the logistics-company scenario above.
Promotion is a deliberate action, not automatic. Nothing in staging silently becomes production just because it's been sitting there quietly and nobody complained. Someone - or some pipeline, with someone's sign-off baked in - has to say "yes, promote this."
The one governance rule that mattered more than the rest
Here's the rule that, in hindsight, did more work than anything else in the topology: exactly one customer-facing control plane may exist in production at a time.
It sounds almost too obvious to write down. It wasn't, in practice, because "control plane" is exactly the kind of thing a well-meaning team might want to stand up a second instance of - for a migration, for a regional split, for "just testing something in a prod-like way." Every one of those instincts is reasonable in isolation. The problem is that a second production control plane isn't a parallel universe, it's a second source of truth for things like budget enforcement and access control, and two sources of truth for the same real-world resource is a recipe for exactly the kind of split-brain problem you'd expect: two systems disagreeing about who's allowed to spend what, each confident it's right.
We don't think this rule is unique to AI platforms - it's really just "don't run two masters" wearing an AI-shaped hat - but it's easy to violate specifically because AI infrastructure gets built in a hurry, under pressure to ship something impressive, and "spin up a second instance real quick" feels lower-risk for infrastructure that's new enough that nobody's built the instincts around it yet.
What goes wrong without this
Skip the environment split, or skip the one-control-plane rule, and the failure modes we've seen (or narrowly avoided) tend to be some version of:
- An experimental model change in "not-quite-production" affects a customer-facing path, because nothing structurally prevented it from being reachable.
- Two systems each believe they're the authority on a resource's state, and reconciling them after the fact is a genuinely unpleasant afternoon.
- Nobody can confidently answer "is this specific model registration live in production right now," because the registry doesn't have a concept of "which environment am I actually in."
None of these are exotic failures. They're the AI-platform version of the exact bugs that dev/staging/prod solved for regular software fifteen years ago - it's just that we all had to relearn the lesson because the shiny new thing didn't feel like it needed the boring old discipline. It did.
We're fairly confident in the three-environment shape itself, less confident that our promotion process is the best version of it rather than just a working one - if you've built something more automated or more clever here, that's a comparison we'd actually want to have.
Next: post 4, on a mistake that's easy to make even inside a well-structured environment - letting a vendor's own model name leak into your API as if it were something you controlled.
Suggested hashtags: #EnterpriseAI #AIPlatform #PlatformEngineering #DevOps #SoftwareArchitecture #Environments #AIGovernance #MLOps #CloudArchitecture #TechLeadership