Architecture
Beyond the gateway: the AI control plane.
Every enterprise AI stack starts with a gateway. Eighteen months in, most are quietly rebuilding it as something stricter: a layer that doesn't just route each AI request but governs it, costs it and keeps a record of what it did. Network architecture has a name for that layer, and AI architecture is starting to use it: the control plane.
AI control plane
AI gateway
Request→Route→Model
When a gateway is enough
A gateway is the right answer to a smaller problem.
This isn't an argument that gateways are wrong. It's an argument that they're a different category of object from what comes next.
Past the threshold, a gateway still does real work, but it's one capability among several, not the whole story.
The threshold is quiet. It tends to be crossed when:
- Two or more teams use AI through the same infrastructure, with different budgets.
- A compliance, risk or audit function starts asking what the AI traffic actually did.
- LLM spend moves from a rounding error to a line item with an owner.
- A second model provider arrives, and choosing a provider becomes a policy question, not just a technical one.
The pivot
What an AI gateway doesn't carry.
| Concern | AI gateway or proxy | AI control plane |
|---|---|---|
| Primary job | Route requests | Govern requests |
| Cost | Logged, then forgotten | Ledgered, enforceable, per request |
| Compliance posture | Per-app paperwork, by hand | Horizontal evidence, by construction |
| Performance ceiling | Bound by middleware overhead | Bound by physics |
| Operational outcome | Tactical fix | Infrastructure asset |
The last row does the real work. A gateway is something a team owns and maintains: a tactical fix to a known routing problem. A control plane is something teams rely on and report into: an infrastructure asset with the same standing as a database or a service mesh, load-bearing, governed, and not part of any one team's surface area.
The argument
Three things a control plane does that a gateway doesn't.
The ledger is not a log
Most gateways log requests; some put a dashboard on the logs and call it cost visibility. That's observability, not governance. A control plane treats cost the way a financial system treats transactions: every request is charged to a budget, every budget has a holder, every holder has a threshold, and the threshold is enforced inline, before the next request goes through.
"We logged the spend" is a report Finance has to chase. "The spend stopped when the budget was hit" is a layer they can rely on.
Cost governance →Compliance is horizontal, not per app
When each team's app carries its own guardrails, audit log and access policy, governance is per-app paperwork. Every team does it slightly differently, auditors get a different artefact for every system, and a policy change means touching every codebase that ever made an AI call.
A control plane gives one boundary, one set of rules and one evidence stream. When ISO/IEC 42001 becomes a procurement requirement, the question is "show me the control plane's evidence", not "did every app implement the controls".
ISO/IEC 42001 →Performance is a property, not a tuning exercise
Anything in the path of every AI request can become the new bottleneck. A gateway written in Python carries overhead the runtime can't remove: garbage-collection pauses on the hot path, the GIL, repeated serialisation between middleware layers, network hops in and out of someone else's cloud.
A control plane built as a hot-path Rust process holds a deterministic overhead under 2 ms at the peak, where Python alternatives spike into seconds just as traffic is highest. Performance follows from the architecture you chose.
The benchmark →The benchmark is there as evidence of the architecture, not as a boast. One 8-core node, a stress-test peak rather than a sustained rate. Method and full results
The pivot, once more
Same SDKs. A different category of object underneath.
Moving from gateway to control plane isn't a rewrite. Applications keep using the OpenAI SDK, the Anthropic SDK, the Google ADK or whatever they were written against. What changes is what the SDK talks to: not a thin proxy one team maintains, but a layer every team relies on. Translation happens inside the control plane; the application is unaware.
So the most common objection, "we already have a gateway and can't afford to rewrite every app", is misframed. The applications don't change; the boundary they speak to does. The gateway was a tactical implementation of one job a control plane does. The control plane absorbs it and adds the ledger, the policy and the record the gateway never had.
Teams eighteen months into AI in production aren't, in general, arguing about whether a control plane is the right idea. They're deciding whether to keep building one badly themselves, or to buy one.
How we start together
Start with one team, on your own AI traffic.
- 01
Pick one app or team
Point it at the Control room. One configuration change; nothing else changes.
- 02
Watch for two weeks
Every call priced, checked and recorded. Rules start in observe mode.
- 03
See the evidence
What it cost, what it caught and what it would have stopped, from your own traffic.