vLLM v0.32.0 puts agent routing inside the inference engine

What shipped

vLLM published v0.32.0 on Oct 18, 2026 — 814 commits from 363 contributors, 121 of them new to the project, per the project's release notes. The headline feature is a new structured-decision endpoint, POST /v1/systemone, that answers choice, boolean and scoring questions over a structured state without entering the standard autoregressive decode loop.

The rest of the release reads like an infrastructure operator's changelog: NVFP4 KV cache on more GPU architectures, FP8 KV cache as the GLM-5.x default, Model Runner V2 becoming the default on ROCm, faster startup and serving, and a security section headlined by a CORS default flip that will surprise anyone serving vLLM straight to a browser.

[1] [2]

The /v1/systemone endpoint

The interface is small: send a prompt plus a schema of type choice with a list of options, and the server returns the decision. The release notes enable it by default and note it also runs on Cloudflare's clef-omni, LiquidAI's d1-3B and DiffusionGemma — small models, the kind you would pick for a gating router.

The interesting part is what the engine does not do. There is no autoregressive decode loop, no five tokens of JSON to parse, no draft tokens from speculative decoding. The decision is evaluated in the engine process, on top of state whose KV blocks are already warm. That is the whole bet: routing stops being a round trip and becomes an in-place evaluation.

[1] [2] [3]

The round-trip tax

Picture the standard agent triage loop. A 12,000-token prompt lands — system instructions, workspace context, recent tool outputs, the latest user request. Before you spend a 70B model on it, you want a cheap routing decision: does this need shell execution, is it safe, which tool schema should be mounted?

The standard answer has been a separate routing call: prefill the same 12,000 tokens on a small model, stream five tokens of JSON, parse it, then fire the real request — and prefill those 12,000 tokens again if the prefix cache was evicted or the worker changed. You pay the prefill tax twice for one boolean question, and then you build Redis layers and sticky routing to stop your own stack from fighting itself.

Moving the decision inside the engine is a response to that specific pain. If the router evaluates where the KV blocks live, the prefix stays pinned in PagedAttention blocks and the follow-up call can hit the prefix cache instead of recomputing attention. How much you save depends on your traffic shape — but the tax it removes is real.

[3]

Low-precision KV cache everywhere

The second theme is memory density. NVFP4 KV cache now runs on SM8x, SM90 and SM12x (with XQA decode on SM12x), and GLM-5.x defaults to an FP8 KV cache on SM100, which the release notes say halves KV memory per rank while adding 2.3–5.5% throughput at a 7–11% time-to-first-token cost in the PR's benchmark. On ROCm, a 4-bit ultraquant cache claims about twice the density of FP8.

These are the project's own benchmark figures, not independent measurements, and the trade is always the same: less memory per token in exchange for some quality or latency risk you should measure on your own workload. But the direction is unmistakable — the scarce resource in serving is KV memory, and every release spends more effort compressing it.

[1]

The upgrade landmine: CORS

The change most likely to wake someone up is not a feature at all. vLLM 0.32 flips the --allowed-origins default from ["*"] to []: browsers will no longer let just any website call your model server by default. The change landed on Oct 11 and rode into 0.32 through the release candidates, so 0.31 — and every release back to 0.6 — still answers Access-Control-Allow-Origin: *.

If a web page on a different origin talks to your vLLM server directly — an internal chat UI, a playground on another port, a notebook frontend — the upgrade breaks it until you list the exact origins. Server-side callers are unaffected: curl, SDKs and gateways do not do CORS. Either way, run the preflight check against every endpoint you own before you upgrade, and keep the API key on.

[1] [4]

What it means for your stack

For agent builders, the signal is about where framework logic belongs. The boundary between inference server and agent framework used to be clean — the server streamed bytes, the framework handled retries, tool parsing and state machines. vLLM is now absorbing the parts of the framework that touch hot engine state: reasoning-block parsing, tool-call parsing, and now routing decisions.

The honest caveats still apply. The mechanism story — constrained sampling over option tokens, logprob scoring, cache retention — comes from a community explainer, not the release notes, and the release notes themselves render the middle question type as 'noul'. Treat the semantics as directionally right until you read the PR.

Operationally: if any browser talks to vLLM, set your origins before the upgrade, not after. If your agent router is a separate HTTP hop today, measure how often its prefix survives to the follow-up call — that number is what /v1/systemone is designed to eliminate. And as always, verify the vendor's throughput figures on your own traffic before resizing anything.

[3] [1] [4]