Serving a language model involves repeating a great deal of work. System instructions, examples, and long documents often return at the beginning of many requests. Automatic prefix caching keeps the intermediate state of that processed section so the server does not have to compute it again. The savings can be substantial when many requests share a large foundation.
That optimization creates a question throughput cannot answer: who is allowed to share the same intermediate result? On August 1, vLLM merged cache_salt support into its Anthropic Messages-compatible API. The patch is small, but it closes a meaningful gap between interfaces. OpenAI-compatible routes already offered explicit cache isolation; clients using the Anthropic format did not have the same control.
What the cache reuses
Before generating an answer, a model transforms input tokens into attention states known as the key-value cache, or KV cache. When another request begins with the same sequence, the server can reuse previously computed blocks and skip part of the initial reading work.
This is not a store of finished answers. Generation still happens. The reused object is the computation for a common prefix. That makes the technique particularly useful for applications that repeat long instructions, stable document sets, or similar conversation structures.
vLLM identifies these blocks through hashes. Without an additional boundary, identical inputs can map to the same cache key. That improves reuse, but in a multi-user service it also means latency or hit patterns may reveal whether a sequence has already been processed. The project describes the concern as an attacker attempting to guess other users’ inputs. The claim is not that the cache directly returns their content. The concern is that shared performance can become a side signal about someone else’s activity.
Salt adds a boundary to the hash
cache_salt adds a secret, unpredictable value to the identity of a cached prefix. Two requests with identical text but different salts no longer share the same cache entry. Requests within one trusted boundary can still reuse work when they use the same salt.
The new /v1/messages implementation accepts an optional string, forwards it to the internal request representation already used by OpenAI routes, and reuses the engine’s existing validation. Omitting the field preserves the previous behavior because the default remains None. The token-counting endpoint is unchanged because it does not generate tokens or participate in prefix reuse.
The field description recommends a random value that is protected from third parties and long enough to be unpredictable, citing 256 bits as a reference. Salt selection therefore becomes an architectural decision. One installation-wide value offers little separation. A fresh value for every request maximizes isolation but destroys most reuse. For many products, the useful boundary will be a user, an organization, or a trusted session.
Compatible does not mean equivalent
The feature request behind the change described a practical mismatch. A team could rely on the Anthropic format for messages, tools, and system instructions, yet had to migrate the application to the OpenAI format solely to obtain deterministic cache isolation. The API looked compatible at its most visible layer while an important operational property existed on only one route.
This gap says something broader about servers that emulate familiar APIs. Compatibility is not merely accepting common fields and returning similar JSON. It includes cancellation, streaming, tools, token counting, errors, caching, and isolation guarantees. One missing extension can force a client migration or, worse, lead a team to assume a protection the route does not provide.
The new vLLM tests cover two direct contracts: a supplied salt reaches the internal request, and omission preserves the old default. That scope matches the patch. The contributor noted that the full suite could not be run locally without CUDA and PyTorch, relying on the project’s continuous integration. That limitation is worth retaining. The code was reviewed and merged, but operators should still validate the exact version they deploy.
What salt does not solve
cache_salt is not a replacement for authentication, authorization, encryption, or memory separation. It changes the logical identity of reusable blocks. If an application exposes the salt, groups unrelated users under one value, or lets clients choose arbitrary boundaries, the isolation property weakens.
There is a performance tradeoff as well. Narrower boundaries produce fewer cache hits and more initial reading work. The right decision is neither to maximize reuse nor to disable caching out of fear. It is to measure performance inside a separation model that matches the product and its risks.
Teams serving models can turn this change into four checks:
- Map which routes actually expose the required isolation controls.
- Define who shares a salt and where that value is stored.
- Measure cache-hit rate, initial-read latency, and consumption per boundary.
- Verify that client libraries preserve extended fields all the way to the server.
The lesson is compact: shared optimization needs explicit shared ownership. A cache can avoid millions of repeated calculations without blending every user’s signals. The boundary has to exist in the protocol, the server, and the application architecture, not only in the builder’s intent.