> ## Documentation Index > Fetch the complete documentation index at: https://tensorfuse.io/docs/llms.txt > Use this file to discover all available pages before exploring further. # When does speculative decoding speed up LLM inference? > Evaluate speculative decoding in vLLM: understand draft verification, acceptance rates, memory overhead, and latency tradeoffs at real concurrency. When users wait for a long answer, generating each new token can dominate latency. Speculative decoding proposes several tokens and verifies them with the target model. It can reduce decode time when enough proposals are accepted, but its extra work and memory mean that speedups depend on the model and workload. ## Where does the saving come from? Ordinary autoregressive generation runs the target model repeatedly as the answer grows. A speculative method proposes candidate tokens using a draft model, additional prediction heads, or patterns in existing text. The target model checks those candidates and continues from the accepted sequence. The benefit depends on the cost of proposing and verifying tokens relative to ordinary decode. A high acceptance rate helps, but does not by itself prove lower latency. The [vLLM speculative-decoding guide](https://docs.vllm.ai/en/stable/features/speculative_decoding/) documents methods and version-specific requirements. ## Which method should you evaluate? | Method family | Where proposals come from | Check before testing | | ---------------------- | --------------------------------------- | ------------------------------------------------------ | | Draft model | A separate model predicts likely tokens | Compatibility, additional memory, draft execution cost | | EAGLE-style methods | A compatible trained drafting component | Exact target/draft pairing and engine support | | Multi-token prediction | Supported model prediction heads | Checkpoint and serving implementation | | Prompt lookup | Repeated token sequences in the prompt | Whether outputs repeat input text | Use the method documented for your exact checkpoint and engine release. Availability in one engine, model family, or hardware configuration does not establish availability in another. ## Compare it with ordinary decoding Start with a working endpoint and save its baseline configuration. Add speculation as the only change. Replay the same prompts with the same sampling parameters, output limits, and offered load. Measure end-to-end latency, time per output token, streaming stalls, throughput, memory use, errors, and acceptance metrics. Repeat at low concurrency and at the normal operating point: added draft work can compete with other requests when the GPU is busy. Run application correctness checks as well. Exact speculative sampling algorithms aim to preserve the target distribution; identical generated strings are not a universal expectation across execution settings and implementations. ## Distinguish the latency problems If long prompts delay the first token, investigate [prefix caching](/docs/guides/inference/prefix-caching) and prefill scheduling. If the first request waits for a model to load, investigate [cold starts](/docs/blogs/reducing_gpu_cold_start). Speculation primarily addresses the generation phase after the model is ready. Prefill/decode disaggregation is another approach: it separates prompt processing and generation across workers. It adds state-transfer and routing requirements, so evaluate it only after measurements show a reason to separate the stages. See the [NVIDIA inference recipes](https://docs.nvidia.com/llm-inference-quick-start-recipes/overview) for upstream deployment examples. Those recipes do not establish Tensorfuse-managed support. ## Related guides Compare latency and cost with speculation enabled and disabled. Connect engine tuning to deployment and scaling decisions.