KV Cache Governance: How to Secure Prompt Caching in Multi-Tenant LLM Inference

If you run a large language model for more than one tenant, you are almost certainly sharing a cache between them. That sharing is why your inference is fast. It is also why one tenant can find out what another tenant asked.

This is not a bug in any particular product. It is the direct consequence of how modern inference engines are built, and the fix is a governance decision, not a patch.

Here is what the trade-off actually costs, and what to do about it.

What prompt caching does

Modern inference engines store the model's working memory the KV cache in fixed-size blocks rather than one contiguous slab per request. Once memory is organised that way, two requests that begin with the same text can point at the same physical blocks. The second request skips the work of recomputing them.

That is prefix caching, and the saving is large. In our measurements on an A100-SXM4-80GB running Qwen2.5-7B-Instruct over a 2,119-token shared prompt:

• 149.6 ms to first token when nothing was cached

• 32.8 ms when the prefix was already cached

• A 4.6x difference. On long system prompts and repeated document context, this is the single biggest lever on serving cost.

Why it leaks

The saving is the leak. Cached is fast, uncached is slow, and the difference is measurable from outside.

An attacker who can send requests to the same backend submits a guess. If the response starts noticeably faster, the guess matched something already in the cache — which means someone else recently sent it.

Two properties make this worse than it first sounds:

Structured prompts collapse the search space. If prompts follow a known template — a customer record, a claim form, an account summary — the attacker does not guess free text. They guess the value of one field. Confirming a single account number is one request, not a search.

The signal is strong at small sizes. Published research puts discrimination at near-perfect accuracy on prefixes as short as eight tokens.

The three numbers you should demand

Most vendor claims about cache isolation quote one number, chosen to flatter. Ask for three:

Configuration	TTFT	vs fully cached
Nothing cached	149.6 ms	—
Fully shared (leaky)	32.8 ms	baseline
Fully isolated	149.6 ms	4.6x slower

Measuring an isolation scheme against the uncached number makes it look almost free. Measuring against the fully cached number shows what your operators will actually feel, because fully cached is what they have today.

Anchoring on "uncached" also hides memory entirely — a system retaining nothing has no footprint to compare against.

Where the boundary goes

Full isolation is rarely the right answer. Most prompts are mostly boilerplate: a long shared system prompt and policy preamble, then a short tenant-specific tail. You can share the preamble and isolate only from the point where the prompt becomes specific to one principal.

This is where cache governance becomes a real decision, and where it goes wrong quietly.

Efficiency changes smoothly as you move the boundary. Security does not.

Every block you move the boundary later buys a little more speed, predictably. But protection does not degrade in step. It holds completely until the boundary passes the sensitive field — and then it fails completely.

The consequence is the dangerous part: there is no gradient warning you. Nothing gets slightly worse as you approach the edge. Latency improves right up to the moment protection disappears, and keeps improving afterwards. An operator tuning for latency, watching latency, walks straight past the line and sees only good news.

Efficiency is a slope. Security is a cliff. Most benchmarks plot only the slope.

Who sets the salt — the control that matters most

Inference engines expose a parameter (in vLLM, cache_salt) that mixes a value into the cache key so only requests carrying the same value can share blocks. Guidance usually focuses on making that value long and unpredictable.

That is the wrong thing to focus on.

The salt is an ordinary field in the request body. In a self-hosted deployment, the caller writes it. An adversary who sets their own salt to a victim's value recovers the shared blocks and the timing signal returns — and length does not help, because the adversary is not guessing the value. They are supplying it.

The control is architectural:

Derive the salt inside a component that authenticates the request. Compute it from the authenticated principal — for example HMAC(K, principal_id) with a server-held key — and discard any client-supplied value rather than honouring or merging it.

A salt the caller controls is not an isolation boundary, regardless of its length. This is the single highest-value control in cache governance, and it lives in your gateway, not in your inference engine.

How to prove it is working

Isolation fails silently. A misconfigured salt looks exactly like a working one if you only watch latency, because a cross-tenant cache hit and a same-tenant cache hit are indistinguishable by response time.

Read the engine's own counters instead. In vLLM those are vllm:prefix_cache_queries and vllm:prefix_cache_hits.

And assert both directions:

1. Two requests with different salts must not raise the hit counter.

2. Two requests with the same salt must raise it.

The second test is not optional. A check that only asserts the first passes when caching is broken entirely, and passes when your salt reaches nothing and no request ever hits. We had a measurement run silently invalidated by exactly this before adding the positive control.

A governance checklist

Work through these in order. The first two matter more than the rest combined.

Move salt derivation to an authenticated choke point. Strip client-supplied values at the gateway. Never accept a tenant's own salt as their boundary.

Decide where the boundary sits, deliberately and in writing. Which part of the prompt is shared, which is principal-specific, and who signs off when the prompt template changes.

Report three numbers, not one — uncached, fully cached, isolated — whenever anyone quotes the cost of isolation.

Monitor hit rate, not just latency, and alert on cross-boundary hits rather than on slow responses.

Test both directions in CI, on every path that accepts a salt — chat, completions, embeddings, tool turns, external cache tiers.

Keep evidence. A tamper-evident record of which principal's blocks were served to whom is what an auditor accepts. "We set a salt" is not evidence.

Re-check after every template change. A longer system prompt moves the sensitive field, and the boundary that was correct last month may now sit past it.

What you get free, and what you must build

Be clear about the split, because it determines what to buy and what to build.

Free, in the engine. The mechanism. vLLM ships cache_salt and documents it. Managed providers isolate at organisation or workspace level already.

Not in the engine, and never will be. Three things:

Deriving the salt. The engine cannot authenticate a principal; by the time a request arrives, the caller has already written the field.

Choosing the boundary. Knowing which part of a prompt is sensitive needs prompt structure, data classification, and policy — not inference machinery.

Proving it held. Counters do not attribute, retain, or produce an audit artifact.

Those three are governance functions. They sit above the engine, and they are where the actual work is.

Frequently asked questions

Does cache isolation slow down inference? Not per request. The cryptographic work is microseconds. The cost is lost cache reuse: up to 4.6x on time-to-first-token for a prefix that would otherwise have been served from another tenant's cache entry. Prefixes a tenant reuses under its own boundary are unaffected.

Is a longer, random salt enough? No. Entropy protects against guessing. It does nothing when the adversary supplies the value, which they can whenever the salt is accepted from the request body.

Do managed APIs have this problem? Major providers isolate prompt cache at organisation level, and some at workspace level. The exposure is concentrated in self-hosted deployments, where the operator owns the boundary and gets an all-or-nothing control by default.

How do I know my isolation is working right now? Send the same prompt twice under two different salts and confirm the prefix-cache hit counter does not move — then send it twice under the same salt and confirm it does. If you have not run the second test, you have not tested anything.

Further reading

Governing the KV Cache: Preventing Timing Side-Channel Leakage in Multi-Tenant LLM Inference — arXiv:2608.09225

vLLM security documentation, prefix cache timing side-channel section