Skip to main content
A tokens-per-second number cannot tell you whether an endpoint will keep up with your users. Benchmark LLM inference at several request rates, recording time to first token, streaming latency, completed requests, and failures. Compare configurations at the same quality and latency targets before translating resource cost into cost per successful request.

​
Record a reproducible configuration

Save the engine version and image digest, model and tokenizer revisions, GPU type/count, driver version, precision, context limit, server flags, replica count, and cache state. Keep an application correctness set alongside the load test. Time per output token averages decode time within a request; it does not capture every streaming stall. Name the metric and percentile when publishing a result.

​
Run a baseline against your endpoint

Use a separate load-generator environment with the vLLM CLI installed. The example follows the vLLM benchmark CLI. Check vllm bench serve --help in your pinned version for available flags. Set INFERENCE_URL to the HTTPS endpoint without /v1, MODEL_ID to the served model name, and INFERENCE_API_KEY to its API key. Set TOKENIZER_ID to the corresponding Hugging Face tokenizer repository if the served name is an alias.
The benchmark client reads OPENAI_API_KEY for authentication. Keeping the key out of --header also keeps it out of the parsed arguments printed by vLLM 0.28.0; see the client authentication implementation. These lengths, prompt count, and request rate are illustrative inputs, not a capacity recommendation. Early stopping and model-specific behavior can change actual output lengths; retain the measured token counts. Synthetic prompts also do not establish answer quality or realistic prefix reuse.

​
Increase load and replay realistic traffic

First hold the deployment at one warm replica to measure engine capacity. Increase the request rate in steps, running long enough at each level for stable results. Repeat runs and inspect client CPU and network use so that the load generator does not become the bottleneck. Then enable your intended autoscaling configuration and test bursts, scale-out, and a request after scale-to-zero. Include representative prompts, long inputs, long outputs, and repeated prefixes. Keep startup latency and steady-state latency in separate results.

​
Convert performance into cost

Choose a measurement window and sum its applicable compute, storage, network, and platform costs. Include idle replicas and failed work in the numerator. For example, a hypothetical 12windowwith6,000successfulrequestscosts12 window with 6,000 successful requests costs 0.002 per successful request. If only 4,000 meet the latency target, the cost is $0.003 per qualifying request. For token reporting, state whether the denominator is input tokens, output tokens, or both. Tokenizers differ, so cost per completed application task is often more useful when comparing models. Use actual billed rates for the AWS region and purchase option you test.

Compare vLLM and SGLang

Use the same workload and resource budget for each engine.

Investigate startup delays

Break down model loading and engine initialization.

​
Inference economics

Read cost per successful request for a worked cost comparison and GPU scale-to-zero economics for the idle-versus-restart calculation.