> ## Documentation Index
> Fetch the complete documentation index at: https://tensorfuse.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# How to run a 4-bit quantized LLM with vLLM

> Serve Qwen with AWQ 4-bit weights in vLLM, check GPU memory use, compare answer quality with FP16, and measure whether quantization reduces inference cost.

By [Samagra Sharma](/docs/authors/samagra-sharma) · Reviewed September 6, 2026

If model weights leave too little GPU memory for concurrent requests, weight quantization is one way to make room. Start with an existing quantized checkpoint, serve it with a compatible kernel, then compare quality and throughput against the original model. Smaller weights create an opportunity to reduce cost; they do not guarantee a faster or cheaper endpoint.

This guide serves the official [Qwen2.5-1.5B-Instruct-AWQ checkpoint](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-AWQ). Its small size makes it useful for learning the workflow before testing a model that meets your application's quality requirements.

<Note>
  Reviewed September 6, 2026 against vLLM 0.28.0 and the linked model configuration. These commands have not been benchmarked on a GPU by Tensorfuse.
</Note>

## 1. Check hardware and choose the checkpoint

Use a Linux NVIDIA GPU host with Docker, NVIDIA Container Toolkit, Python 3, and a driver compatible with CUDA 12.9. An Ampere GPU such as A10 or an Ada GPU such as L4 is an appropriate architecture for this example. Consult the [vLLM quantization compatibility table](https://docs.vllm.ai/en/v0.28.0/features/quantization/) before substituting hardware. This CUDA image is separate from the [AMD ROCm deployment path](/docs/guides/inference/hardware/amd-rocm).

The checkpoint's [configuration](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-AWQ/blob/main/config.json) specifies AWQ, 4-bit weights, group size 128, and FP16 activations. vLLM reads this metadata automatically. Passing a quantization flag to an arbitrary full-precision checkpoint does not create this AWQ artifact.

```bash theme={null}
nvidia-smi
export INFERENCE_API_KEY="$(python3 -c 'import secrets; print(secrets.token_hex(32))')"
export VLLM_API_KEY="$INFERENCE_API_KEY"
docker pull vllm/vllm-openai:v0.28.0-cu129
```

The image is pinned to the [official CUDA 12.9 release](https://github.com/vllm-project/vllm/releases/tag/v0.28.0). For a production experiment, also record the image digest and pin the model's Hugging Face commit with `--revision`.

## 2. Start an authenticated local endpoint

```bash theme={null}
docker run -d --rm --name qwen-awq --gpus device=0 \
  --shm-size 2g \
  --env VLLM_API_KEY \
  -p 127.0.0.1:8000:8000 \
  -v qwen-model-cache:/root/.cache/huggingface \
  vllm/vllm-openai:v0.28.0-cu129 \
  --model Qwen/Qwen2.5-1.5B-Instruct-AWQ \
  --served-model-name qwen-cost-test \
  --dtype half \
  --max-model-len 4096 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.8 \
  --no-enable-prefix-caching \
  --generation-config vllm \
  --host 0.0.0.0 --port 8000

docker logs -f qwen-awq
```

The server reads authentication from [VLLM\_API\_KEY](https://docs.vllm.ai/en/v0.28.0/configuration/env_vars/). Docker publishes port 8000 only to the host's loopback interface. Keep it there during evaluation. vLLM's API key does not protect every route, so remote deployments need an authenticated gateway and network controls; see the [server authentication scope](https://docs.vllm.ai/en/v0.28.0/cli/serve/#--api-key).

The context limit includes input and output tokens. The concurrency and memory settings are initial bounds. Prefix caching is disabled to keep reuse from changing the comparison. The [Docker guide](https://docs.vllm.ai/en/v0.28.0/deployment/docker/) describes container and shared-memory requirements.

## 3. Verify loading and send a request

Wait for startup to complete in the logs, then press Ctrl-C to stop following them; the detached container keeps running. Confirm that loading used the checkpoint's AWQ configuration or a compatible automatically selected AWQ kernel, and record the logged model memory and KV-cache capacity.

```bash theme={null}
curl --fail-with-body http://127.0.0.1:8000/v1/models \
  -H "Authorization: Bearer $INFERENCE_API_KEY"

curl --fail-with-body http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $INFERENCE_API_KEY" \
  -H 'Content-Type: application/json' \
  --data '{
    "model": "qwen-cost-test",
    "messages": [{"role": "user", "content": "Extract the invoice total as JSON: subtotal $40, tax $4, total $44."}],
    "temperature": 0,
    "max_tokens": 64
  }'
```

Expect a chat-completion response containing `choices` and `usage`. Check whether the answer identifies 44 as the total; a successful HTTP response alone does not establish task accuracy.

## 4. Compare against the original model

Stop this server with `docker stop qwen-awq`. Repeat the launch command with container name `qwen-fp16` and model `Qwen/Qwen2.5-1.5B-Instruct`, keeping the remaining settings unchanged. Run the same held-out prompts against both endpoints sequentially.

| Measure             | Keep the comparison useful                                                                                                       |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Answer quality      | Score extraction accuracy, valid JSON, and your actual application tasks. Record failures and retries.                           |
| GPU memory          | Compare logged weight memory and available KV-cache capacity. Total `nvidia-smi` allocation also includes vLLM's reserved cache. |
| Serving performance | Hold GPU, request rate, prompt lengths, and output limits constant. Record throughput and p95 latency.                           |
| Cost                | Divide total serving cost by requests meeting both quality and latency requirements.                                             |

Four-bit weight storage uses one quarter of the bits per quantized value compared with 16-bit storage. Runtime memory also includes scales, unquantized tensors, activations, and the KV cache. Billing falls only when the measured improvement enables a cheaper configuration or a shorter job.

## What if the quantized model fails or is slower?

If initialization runs out of memory, check other GPU processes and the startup memory breakdown. Reduce context or concurrency when it exceeds available KV-cache capacity. If weights do not fit, choose a smaller model or more memory. If kernels fail, verify the GPU architecture, driver, image, and checkpoint format together.

A smaller checkpoint can still have lower throughput for your workload. Keep the full-precision model if it wins the quality-adjusted comparison. To create your own AWQ checkpoints, use the maintained [LLM Compressor workflow linked by vLLM](https://docs.vllm.ai/en/v0.28.0/features/quantization/auto_awq/); the older AutoAWQ package is deprecated.

While each configuration is running, use the [benchmarking guide](/docs/guides/inference/benchmarking) and [inference cost reduction workflow](/docs/guides/inference/how-to/reduce-cost) to decide whether the memory savings change your deployment cost. Finish by stopping the remaining container with `docker stop qwen-fp16`.