> ## Documentation Index
> Fetch the complete documentation index at: https://tensorfuse.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# How to serve LLMs on AWS Inferentia and Trainium with vLLM

> Run a TinyLlama API on Inf2 or Trn1 with a pinned Neuron container, check compilation and responses, and compare cost with GPU inference.

By [Samagra Sharma](/docs/authors/samagra-sharma) · Reviewed September 6, 2026

An inference service can keep its OpenAI-compatible API when moving to AWS Inferentia or Trainium, but it needs a different execution stack. Start with a supported Neuron container and a small model, verify compilation and generation, then measure the workload that determines your bill.

<Note>
  This upstream EC2 recipe uses vLLM 0.16.0 with Neuron SDK 2.30.0 and NxD Inference on Inf2 or Trn1. It has not been executed on Neuron hardware by Tensorfuse. Check the [Tensorfuse configuration reference](/docs/concepts/configuration) for documented platform resource options.
</Note>

## Which Neuron serving path should you use?

AWS currently documents two distinct integrations:

| Target                                  | Serving path                                                         |
| --------------------------------------- | -------------------------------------------------------------------- |
| Inferentia2 / Inf2 and Trainium1 / Trn1 | vLLM with NxD Inference; the pinned example below uses this path     |
| Trainium2 / Trn2 and Trainium3 / Trn3   | Also evaluate the newer vLLM Neuron plugin, introduced with SDK 2.31 |

AWS's [container support table](https://github.com/aws-neuron/deep-learning-containers#vllm-inference-neuronx) lists the instance types for each image. The [new plugin setup guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/vllm-neuron/docs/getting-started/setup-guide.html) explicitly targets Trn2 and Trn3. Keep its installation and flags separate from this NxD recipe. Inf1 uses a different software path and is outside this example.

## 1. Prepare the EC2 host

Use an Inf2 or Trn1 instance with Docker, SSH access, internet access for the container and model downloads, and a Neuron driver compatible with SDK 2.30.0. Follow the [Neuron 2.30 environment instructions](https://awsdocs-neuron.readthedocs-hosted.com/en/v2.30.0/deploy/environments/quickstart-deploy-dlc.html) when selecting the Neuron DLAMI. Record its AMI ID and your instance type.

On the host, check:

```bash theme={null}
neuron-ls
ls -l /dev/neuron0
docker version
```

Reserve the first device for this test. The command below exposes only `/dev/neuron0` and uses two tensor-parallel ranks. An [Inferentia2 device](https://aws.amazon.com/ai/machine-learning/inferentia/) and a [Trn1 device](https://awsdocs-neuron.readthedocs-hosted.com/en/v2.30.0/frameworks/torch/torch-neuronx/programming-guide/training/pytorch-neuron-programming-guide.html) each contain two NeuronCores. Verify those cores are available before launching a second model process.

## 2. Pull the pinned serving image

```bash theme={null}
NEURON_IMAGE='public.ecr.aws/neuron/pytorch-inference-vllm-neuronx:0.16.0-neuronx-py312-sdk2.30.0-ubuntu24.04'
docker pull "$NEURON_IMAGE"
NEURON_IMAGE=$(docker image inspect "$NEURON_IMAGE" \
  --format '{{index .RepoDigests 0}}')
printf '%s\n' "$NEURON_IMAGE"
```

Save this digest with your evaluation results. The explicit version follows AWS's [TinyLlama DLC quickstart](https://awsdocs-neuron.readthedocs-hosted.com/en/v2.30.0/deploy/environments/quickstart-deploy-dlc.html); it is a reproducible starting point, not a claim that SDK 2.30 is the newest release.

## 3. Launch TinyLlama

```bash theme={null}
docker run --name neuron-vllm \
  --device=/dev/neuron0 \
  --publish 127.0.0.1:8080:8080 \
  --volume neuron-hf-cache:/root/.cache/huggingface \
  --entrypoint vllm "$NEURON_IMAGE" serve \
  --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
  --host 0.0.0.0 --port 8080 \
  --tensor-parallel-size 2 \
  --max-model-len 128 \
  --max-num-seqs 4 \
  --block-size 32 \
  --num-gpu-blocks-override 16 \
  --additional-config '{"override_neuron_config":{"enable_bucketing":false}}'
```

This compact configuration is an installation check. Its 128-token limit includes the formatted input and generated output. The block settings and Neuron override follow the upstream example; benchmark production settings separately once this check works.

The process stays in the foreground so compilation and load errors remain visible. In a second terminal, `docker logs --follow neuron-vllm` shows the same logs. Wait for model initialization and API startup before sending traffic. Docker publishes the endpoint only on the EC2 host's loopback interface, so run the test there or use an SSH tunnel. Configure authenticated TLS access before making a remote service available to clients.

## 4. Verify a real completion

```bash theme={null}
curl --fail --silent --show-error http://127.0.0.1:8080/health
curl --fail --silent --show-error http://127.0.0.1:8080/v1/models
curl --fail --silent --show-error http://127.0.0.1:8080/v1/chat/completions \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    "messages": [{"role": "user", "content": "Name the capital of France."}],
    "temperature": 0,
    "max_tokens": 32
  }'
```

Expect a successful HTTP response with a nonempty assistant message and the TinyLlama model ID. This verifies the serving path, not your application's accuracy. Move to a supported application model and your own correctness tests before drawing a hardware conclusion.

## What usually needs fixing?

| Symptom                      | Check                                                                            |
| ---------------------------- | -------------------------------------------------------------------------------- |
| `/dev/neuron0` is missing    | Instance family, host driver installation, and `neuron-ls`                       |
| Insufficient NeuronCores     | Device exposure and other processes already using the two cores                  |
| Unrecognized Neuron argument | Image digest and whether the recipe belongs to NxD Inference or the newer plugin |
| Context-length rejection     | Prompt, chat-template tokens, and output budget must fit within 128 tokens       |
| First startup appears slow   | Compilation progress and the first actual error in the logs                      |

Separate first compilation, subsequent model loading, and steady-state requests in your measurements. AWS documents [Neuron's compilation cache](https://awsdocs-neuron.readthedocs-hosted.com/en/v2.30.0/about-neuron/arch/neuron-features/neuron-caching.html); verify which artifacts your serving integration reuses before assuming a fresh replica starts warm.

## How do you compare Neuron with GPU inference?

Use the [same benchmark workload](/docs/guides/inference/benchmarking), model quality requirement, and latency limits on both endpoints. Include the complete EC2 instance, startup and idle time, storage, and network charges. Exposing one device from a larger instance still leaves the entire instance bill in the numerator.

Follow the [cost reduction workflow](/docs/guides/inference/how-to/reduce-cost) and the [Inferentia versus Trainium explainer](/docs/guides/inference/hardware/aws-neuron) before selecting production capacity. Stop the evaluation with `docker stop neuron-vllm`, remove it with `docker rm neuron-vllm` when its logs and artifacts are no longer needed, and release unused EC2 capacity. Container cleanup alone does not end instance charges.