How to run a 4-bit quantized LLM with vLLM

By Samagra Sharma · Reviewed September 6, 2026 If model weights leave too little GPU memory for concurrent requests, weight quantization is one way to make room. Start with an existing quantized checkpoint, serve it with a compatible kernel, then compare quality and throughput against the original model. Smaller weights create an opportunity to reduce cost; they do not guarantee a faster or cheaper endpoint. This guide serves the official Qwen2.5-1.5B-Instruct-AWQ checkpoint. Its small size makes it useful for learning the workflow before testing a model that meets your application’s quality requirements.

Reviewed September 6, 2026 against vLLM 0.28.0 and the linked model configuration. These commands have not been benchmarked on a GPU by Tensorfuse.

1. Check hardware and choose the checkpoint

Use a Linux NVIDIA GPU host with Docker, NVIDIA Container Toolkit, Python 3, and a driver compatible with CUDA 12.9. An Ampere GPU such as A10 or an Ada GPU such as L4 is an appropriate architecture for this example. Consult the vLLM quantization compatibility table before substituting hardware. This CUDA image is separate from the AMD ROCm deployment path. The checkpoint’s configuration specifies AWQ, 4-bit weights, group size 128, and FP16 activations. vLLM reads this metadata automatically. Passing a quantization flag to an arbitrary full-precision checkpoint does not create this AWQ artifact.

nvidia-smi
export INFERENCE_API_KEY="$(python3 -c 'import secrets; print(secrets.token_hex(32))')"
export VLLM_API_KEY="$INFERENCE_API_KEY"
docker pull vllm/vllm-openai:v0.28.0-cu129

The image is pinned to the official CUDA 12.9 release. For a production experiment, also record the image digest and pin the model’s Hugging Face commit with --revision.

2. Start an authenticated local endpoint

docker run -d --rm --name qwen-awq --gpus device=0 \
  --shm-size 2g \
  --env VLLM_API_KEY \
  -p 127.0.0.1:8000:8000 \
  -v qwen-model-cache:/root/.cache/huggingface \
  vllm/vllm-openai:v0.28.0-cu129 \
  --model Qwen/Qwen2.5-1.5B-Instruct-AWQ \
  --served-model-name qwen-cost-test \
  --dtype half \
  --max-model-len 4096 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.8 \
  --no-enable-prefix-caching \
  --generation-config vllm \
  --host 0.0.0.0 --port 8000

docker logs -f qwen-awq

The server reads authentication from VLLM_API_KEY. Docker publishes port 8000 only to the host’s loopback interface. Keep it there during evaluation. vLLM’s API key does not protect every route, so remote deployments need an authenticated gateway and network controls; see the server authentication scope. The context limit includes input and output tokens. The concurrency and memory settings are initial bounds. Prefix caching is disabled to keep reuse from changing the comparison. The Docker guide describes container and shared-memory requirements.

3. Verify loading and send a request

Wait for startup to complete in the logs, then press Ctrl-C to stop following them; the detached container keeps running. Confirm that loading used the checkpoint’s AWQ configuration or a compatible automatically selected AWQ kernel, and record the logged model memory and KV-cache capacity.

curl --fail-with-body http://127.0.0.1:8000/v1/models \
  -H "Authorization: Bearer $INFERENCE_API_KEY"

curl --fail-with-body http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $INFERENCE_API_KEY" \
  -H 'Content-Type: application/json' \
  --data '{
    "model": "qwen-cost-test",
    "messages": [{"role": "user", "content": "Extract the invoice total as JSON: subtotal $40, tax $4, total $44."}],
    "temperature": 0,
    "max_tokens": 64
  }'

Expect a chat-completion response containing choices and usage. Check whether the answer identifies 44 as the total; a successful HTTP response alone does not establish task accuracy.

4. Compare against the original model

Stop this server with docker stop qwen-awq. Repeat the launch command with container name qwen-fp16 and model Qwen/Qwen2.5-1.5B-Instruct, keeping the remaining settings unchanged. Run the same held-out prompts against both endpoints sequentially.

Measure	Keep the comparison useful
Answer quality	Score extraction accuracy, valid JSON, and your actual application tasks. Record failures and retries.
GPU memory	Compare logged weight memory and available KV-cache capacity. Total `nvidia-smi` allocation also includes vLLM’s reserved cache.
Serving performance	Hold GPU, request rate, prompt lengths, and output limits constant. Record throughput and p95 latency.
Cost	Divide total serving cost by requests meeting both quality and latency requirements.

Four-bit weight storage uses one quarter of the bits per quantized value compared with 16-bit storage. Runtime memory also includes scales, unquantized tensors, activations, and the KV cache. Billing falls only when the measured improvement enables a cheaper configuration or a shorter job.

What if the quantized model fails or is slower?

If initialization runs out of memory, check other GPU processes and the startup memory breakdown. Reduce context or concurrency when it exceeds available KV-cache capacity. If weights do not fit, choose a smaller model or more memory. If kernels fail, verify the GPU architecture, driver, image, and checkpoint format together. A smaller checkpoint can still have lower throughput for your workload. Keep the full-precision model if it wins the quality-adjusted comparison. To create your own AWQ checkpoints, use the maintained LLM Compressor workflow linked by vLLM; the older AutoAWQ package is deprecated. While each configuration is running, use the benchmarking guide and inference cost reduction workflow to decide whether the memory savings change your deployment cost. Finish by stopping the remaining container with docker stop qwen-fp16.

Get Started

LLM Inference

Inference How-to Guides

Inference Hardware

Concepts

Operations

Troubleshooting

Enterprise Setup

Architecture

How to run a 4-bit quantized LLM with vLLM

1. Check hardware and choose the checkpoint

2. Start an authenticated local endpoint

3. Verify loading and send a request

4. Compare against the original model

What if the quantized model fails or is slower?

​1. Check hardware and choose the checkpoint

​2. Start an authenticated local endpoint

​3. Verify loading and send a request

​4. Compare against the original model

​What if the quantized model fails or is slower?

1. Check hardware and choose the checkpoint

2. Start an authenticated local endpoint

3. Verify loading and send a request

4. Compare against the original model

What if the quantized model fails or is slower?