> ## Documentation Index
> Fetch the complete documentation index at: https://tensorfuse.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# How to enable and measure vLLM prefix caching

> Enable vLLM prefix caching, run repeated-prefix requests, check Prometheus cache-hit metrics, and compare warm and cold time to first token for agents and RAG.

An agent can resend the same instructions and tool definitions on every turn. Enable vLLM prefix caching to reuse attention state for the identical beginning of those prompts, then check cache-hit counters and time to first token. This saves repeated prefill computation when the prefix remains cached; generating each new answer still requires decode work.

<Note>
  Reviewed September 6, 2026 against vLLM 0.28.0. The example is a local verification workflow, not a Tensorfuse GPU benchmark or a guaranteed latency improvement.
</Note>

## Which part of the request gets reused?

The model processes input tokens during prefill, then generates output during decode. The key-value cache stores attention state. Cross-request prefix caching reuses compatible cached blocks when another request begins with the same token sequence.

Similar wording is insufficient: reuse depends on matching tokens and the engine's cache identity, including relevant model and adapter settings. See vLLM's [prefix-cache design](https://docs.vllm.ai/en/v0.28.0/design/prefix_caching/) for block hashing and isolation details.

## 1. Put stable content before changing content

For an agent that uses a fixed set of tools, a useful prompt layout is:

```text theme={null}
System instructions shared across requests
Stable tool definitions in a consistent order
Reusable document or conversation prefix
Current user request and request-specific context
```

A timestamp or random identifier at the beginning prevents later tokens from sharing the same prefix. Move changing metadata later when application semantics allow it. Keep the chat template and tool serialization consistent, preserve instruction priority, and evaluate answer quality after rearranging prompts.

For RAG, repeated questions about the same document can benefit from a shared document prefix. Retrieval that selects different passages on every request may have little reuse. These use cases and the prefill/decode distinction are covered in vLLM's [feature guide](https://docs.vllm.ai/en/v0.28.0/features/automatic_prefix_caching/).

## 2. Enable prefix caching on one server

Use a Linux NVIDIA GPU host with Docker, NVIDIA Container Toolkit, Python 3, and a CUDA 12.9-compatible driver. This example uses the small [Qwen2.5-1.5B-Instruct model](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) on an Ampere or newer supported NVIDIA GPU. Start on an otherwise idle GPU and keep port 8000 free.

```bash theme={null}
export INFERENCE_API_KEY="$(python3 -c 'import secrets; print(secrets.token_hex(32))')"
export VLLM_API_KEY="$INFERENCE_API_KEY"
docker run -d --rm --name qwen-prefix --gpus device=0 \
  --shm-size 2g \
  --env VLLM_API_KEY \
  -p 127.0.0.1:8000:8000 \
  -v qwen-model-cache:/root/.cache/huggingface \
  vllm/vllm-openai:v0.28.0-cu129 \
  --model Qwen/Qwen2.5-1.5B-Instruct \
  --served-model-name qwen-prefix \
  --dtype half --max-model-len 4096 --max-num-seqs 8 \
  --gpu-memory-utilization 0.8 \
  --enable-prefix-caching \
  --generation-config vllm \
  --host 0.0.0.0 --port 8000

docker logs -f qwen-prefix
```

Wait for application startup to complete, then press Ctrl-C to stop following logs. Confirm `enable_prefix_caching=True` in the engine configuration. The [v0.28.0 server reference](https://docs.vllm.ai/en/v0.28.0/cli/serve/) documents the flag and its disabling counterpart, `--no-enable-prefix-caching`.

The server reads [VLLM\_API\_KEY](https://docs.vllm.ai/en/v0.28.0/configuration/env_vars/) from its environment. Keep the published port on loopback. API-key authentication covers selected inference paths, not every vLLM route; put remote access behind a gateway that authenticates all exposed routes.

## 3. Send a cold prefix, then reuse it

Save this snippet as `prefix_check.py`, then run `python3 prefix_check.py` in the shell where you set `INFERENCE_API_KEY`. It uses only the standard library and prints client-observed time to the first nonempty streamed content chunk. It consumes the full response before the next request.

```python theme={null}
import json
import os
import time
import urllib.request
import uuid

base_url = "http://127.0.0.1:8000"
headers = {
    "Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
    "Content-Type": "application/json",
}
# A new salt makes each execution independent of an earlier run.
salt = uuid.uuid4().hex
document = "\n".join(
    f"Record {i}: the item price is 10 dollars and the quantity is 2."
    for i in range(100)
)
for label, question in [
    ("cold-prefix", "What is the quantity in record 10?"),
    ("warm-prefix", "What is the quantity in record 20?"),
    ("warm-prefix", "What is the quantity in record 30?"),
]:
    body = {
        "model": "qwen-prefix",
        "messages": [
            {"role": "system", "content": "Answer from the supplied records."},
            {"role": "user", "content": document + "\n\n" + question},
        ],
        "cache_salt": salt,
        "temperature": 0,
        "max_tokens": 32,
        "stream": True,
    }
    request = urllib.request.Request(
        base_url + "/v1/chat/completions",
        data=json.dumps(body).encode(), headers=headers,
    )
    start = time.perf_counter()
    first_content = None
    answer = []
    with urllib.request.urlopen(request, timeout=120) as response:
        for line in response:
            if not line.startswith(b"data: "):
                continue
            payload = line[6:].strip()
            if payload == b"[DONE]":
                break
            choices = json.loads(payload).get("choices", [])
            content = choices[0].get("delta", {}).get("content") if choices else None
            if content:
                if first_content is None:
                    first_content = time.perf_counter() - start
                answer.append(content)
    if first_content is None:
        raise RuntimeError("The response produced no content; inspect the server logs.")
    print(label, "first_content_seconds=", round(first_content, 4),
          "answer=", "".join(answer))
```

The first request must populate this salt's cache. The following requests share the document and change only the question. Complete blocks of that shared prefix can be reused, so a hit rate below 100% is expected. The first request may also include runtime warm-up effects; three requests demonstrate the mechanism and are insufficient for a performance conclusion.

## 4. Check cache counters and compare latency

Capture metrics before and after the requests, allowing the server's periodic metric update to complete:

```bash theme={null}
curl --fail-with-body http://127.0.0.1:8000/metrics |
  grep -E '^vllm:(prefix_cache_(hits|queries)_total|kv_cache_usage_perc)'
```

vLLM's [production metrics](https://docs.vllm.ai/en/v0.28.0/usage/metrics/) define prefix hits and queries as token counters. Their Prometheus samples use the [standard `_total` counter suffix](https://prometheus.github.io/client_python/instrumenting/counter/). Calculate the interval hit ratio from the increase in hits divided by the increase in queries for the same model and engine. A zero query increase means there was no sample to interpret.

For a Prometheus dashboard with this single model:

```promql theme={null}
sum(rate(vllm:prefix_cache_hits_total{model_name="qwen-prefix"}[5m]))
/
sum(rate(vllm:prefix_cache_queries_total{model_name="qwen-prefix"}[5m]))
```

Repeat with the flag `--no-enable-prefix-caching` after stopping and relaunching the container. At realistic load, compare repeated and unique prefixes, then repeat during replica startup. Keep prompt/output lengths and arrival rate comparable. Record p50/p95 time to first token, queue time, completed requests, and quality using the [benchmarking guide](/docs/guides/inference/benchmarking). The streaming snippet measures client delivery; the server's `vllm:time_to_first_token_seconds` histogram measures its own timing boundary.

## Why are there no hits, or no latency improvement?

Check whether requests reach the same replica, use the same salt, and retain identical token prefixes. Changing templates, tools, adapters, or early metadata can break reuse. Very short shared prefixes may not fill cache blocks, and competing requests can evict cached blocks.

A new replica begins without another replica's in-memory cache. Persistent model files on a Tensorfuse [volume](/docs/concepts/volumes) do not automatically create a shared KV cache. Distributed cache transfer and routing need their own implementation. Set tenant-specific cache salts through trusted application logic when isolating reuse between tenants.

If hit counters rise but total latency barely changes, inspect queue time and output length: decode can dominate. Continue with [speculative decoding](/docs/guides/inference/speculative-decoding) or the [inference cost reduction workflow](/docs/guides/inference/how-to/reduce-cost) based on the measured bottleneck. Stop the example with `docker stop qwen-prefix`.