Latency-critical AI

Your p99 is the number that loses you the auction.

In real-time bidding, trading and fraud scoring, every AI call sits inside a hard deadline measured in single-digit milliseconds. Vidai's overhead stays flat at the peak, while Python-based alternatives spike into seconds exactly when traffic is highest.

Latency against load slow fast low traffic peak Python-based alternative Vidai, flat
Illustrative shape. Measured on one 2013 Intel Xeon E3-1240 v3 node: Vidai holds a 1.95 ms median at 21,803 calls a second; a Python-based alternative reached a 197 ms median at its own peak.

Who this is for

Four models, one auction, one clock.

A real-time bidding auction closes in roughly 100 ms. Inside that, several AI models each run on their own far tighter budget. Miss any one of them and the bid is late, which means the bid is lost.
01< 5 ms

Bid shading and pricing

Predicting the optimal clearing price in a first-price auction so the buyer does not overpay. The tightest budget on the path: it runs on essentially every impression.

02< 10 ms

CTR and CVR prediction

Scoring click-through and conversion likelihood at bid time. The model that decides whether the impression is worth bidding on at all.

03< 15 ms

Pre-bid fraud and IVT

Catching bots, data-centre traffic and click farms before money is spent. Gradient-boosted trees over session signatures, run as a parallel ensemble.

04< 10 ms

Traffic shaping (sell side)

On the SSP, filtering bid requests a buyer is unlikely to want before they ever go over the wire, saving bandwidth and compute at both ends.

The budgets above are per model, inside the wider auction window. They are the reason the layer in front of these models cannot add a millisecond it cannot account for. Trading-adjacent inference, fraud scoring and live product surfaces face the same shape of deadline.

Agents multiply it

One agent task, ten calls or more, all on the clock.

Agents don't make one call. A single task plans, calls tools and checks its own work: ten model calls or more, mostly in sequence, each waiting for the one before. Whatever the layer in front of the models adds, the task pays it every time.

One task fans out

Ten calls through Vidai~20 ms
Through a Python-based alternative~2 s

Time the governing layer adds to a whole ten-call agent task, network included.†

The overhead adds up

Ten calls in sequence pay the layer's cost ten times. A gap that looks small on one call becomes the difference between a task that fits its window and one that doesn't.

The tail compounds

If one call in a hundred is slow, a ten-call task meets at least one slow call about one time in ten. For agents, the p99 of a call becomes everyday behaviour for the task.

Parallel steps wait for the slowest

When an agent fans out, the step finishes only when its last call returns. Flat overhead at the peak is what keeps every branch predictable.

†Ten calls in sequence at the medians of our published network test: 1.95 ms for Vidai at 21,803 calls a second, 197 ms for a Python-based proxy at its peak, on one 2013 Intel Xeon E3-1240 v3 (8 cores). Method and full results

The real problem

It's not the median. It's the peak.

Everything is fast at low traffic. Python-based middleware degrades non-linearly: the p99 spike lands exactly during the volume you care about, and a garbage-collection pause can stall a request mid-auction. "Usually fine" is not a latency guarantee when a 5 ms budget is on the line.
1.95 msMedian at the 21,803 calls/s peak: the floor the tail sits close to
23.5 msp95 at that same peak: the tail stays bounded
40 msp99 at that peak, where Python-based alternatives reach seconds
0Garbage-collection pauses on the hot path: the spike has no cause

Measured on a single 8-core node, an Intel Xeon E3-1240 v3 from 2013, chosen so the figure is a floor, not a best case. At their own peak, Python-based alternatives ran at a 197 ms median.

Full benchmark and method → The Rust and Python deep dive →

The architecture trend

The hot path is moving off the network hop.

At a million-plus queries a second, a microservice call for every model is dead weight: the network hop alone can spend the budget, and a garbage-collection pause blows the auction outright. The industry response is to compile and distil models so they run without a hop and without a collector. Vidai is built on the same physics, applied to the layer in front of those models.

In the path, not another hop

Vidai is not a microservice your bidder calls over the wire. It sits in the request path as one compiled Rust engine, so governing the traffic does not cost you a round trip.

It is the path, not the model

Your bid-shading, CTR and fraud models stay yours, wherever they run. Vidai is the governed layer around them: routing, guardrails, cost and fallback in one pass, not seven.

No collector on the hot path

A Rust engine has no garbage collector, so there is no pause to land mid-auction. The variance a collector introduces is simply not in the system.

The honest line

Vidai does not run inside your bidding loop. It removes the middleware hops and the latency variance around your models, so the budget is spent on inference, not plumbing.

Hot path, cold path

Not everything belongs on the clock.

The teams that win the latency game are strict about what runs inside the auction and what does not. Heavy work is pushed off the hot path, pre-computed and cached. Vidai is built for the part that genuinely has to be on the clock.
Cold path, pre-computedHeavy language work like contextual and brand-safety classification is too slow for the auction. It runs asynchronously and lands in an in-memory store.
Hot path, a fast lookupAt bid time the auction does a sub-millisecond read against that cache. The intelligence was computed earlier; the clock only sees a lookup.
Vidai governs the hot pathFor the calls that must happen inside the window, Vidai is the layer in the path that keeps the overhead flat and deterministic, at the peak as at idle.
Fail fast, never hangIf a downstream model is slow, a circuit breaker trips and the path falls back rather than blowing the deadline. A late bid is a lost bid; a fast fallback is not.

Why it holds the line

Deterministic by construction, not by tuning.

Rust hot path, no garbage collector

No collector pause mid-request. The overhead is the same at the peak as it is at idle, and that is the whole point.

One pass, not seven hops

Translation, routing, guardrails, cost and fallback happen in a single engine in the path, so there is no chain of libraries each adding its own millisecond.

In your path, on your infrastructure

No extra network hop out to someone else's cloud and back. The latency you measure is the latency you ship.

Cost control that doesn't cost latency

Per-call attribution and spend limits run inline at the same sub-2 ms, so governing the spend never slows the auction.

For platform engineering → Operations and fallback → Size your AI traffic →

How we start together

Put it on your critical path.

A 20-minute technical walkthrough: the latency profile under load, the p99 numbers, and how it behaves at your peak. Then a two-week pilot on your own traffic.
  1. 01

    Pick one app or team

    Point it at the Control room. One configuration change; nothing else changes.

  2. 02

    Watch for two weeks

    Every call priced, checked and recorded. Rules start in observe mode.

  3. 03

    See the evidence

    What it cost, what it caught and what it would have stopped, from your own traffic.