Deploy GPT-OSS 20B and 120B on AWS with vLLM

Deploy openai/gpt-oss-20b or openai/gpt-oss-120b as an authenticated chat endpoint in your AWS account using Tensorfuse and vLLM. This guide connects the model server, GPU allocation, API secret, and readiness probe, then shows how to query the endpoint. Benchmark your own workload before setting production capacity.

Before you deploy

Configure your cluster using Getting started. Confirm GPU quota and available capacity in your AWS region, then choose one model for the deployment directory. The Dockerfiles below pin the CUDA 12.9 variant of vLLM 0.28.0. Verify the cluster’s NVIDIA driver supports that image before deploying. Consult the upstream GPT-OSS recipe for model and hardware requirements.

These examples were checked against documentation and configuration syntax on September 6, 2026; the updated image has not been GPU-tested on a Tensorfuse cluster as part of this refresh. The GPU allocations below are starting configurations, not minimum requirements or throughput guarantees.

The model repositories are public. If your environment requires Hugging Face authentication, pass a token through Tensorfuse secrets, never through a Dockerfile.

1. Create an API secret

export VLLM_API_KEY="$(openssl rand -hex 32)"
tensorkube secret create vllm-token "VLLM_API_KEY=$VLLM_API_KEY" --env default

Keep the generated key in your password manager or secret store. The deployment injects it into the server environment. Clients need the same key in the Authorization header.

2. Choose a Dockerfile

Save one of the following as Dockerfile. Each starts a chat server on port 8000 and limits context to 8,192 tokens as an initial memory budget. Change that limit after measuring your prompt lengths, output lengths, and concurrency.

FROM vllm/vllm-openai:v0.28.0-cu129
EXPOSE 8000
ENTRYPOINT ["sh", "-c", "exec vllm serve openai/gpt-oss-20b --host 0.0.0.0 --port 8000 --max-model-len 8192 --api-key \"${VLLM_API_KEY:?Set VLLM_API_KEY through a secret}\""]

FROM vllm/vllm-openai:v0.28.0-cu129
EXPOSE 8000
ENTRYPOINT ["sh", "-c", "exec vllm serve openai/gpt-oss-120b --host 0.0.0.0 --port 8000 --tensor-parallel-size 8 --max-model-len 8192 --api-key \"${VLLM_API_KEY:?Set VLLM_API_KEY through a secret}\""]

The 120B example preserves this guide’s eight-GPU layout. Eight GPUs are not a minimum requirement: quantization, context, kernels, and throughput targets affect sizing. Keep the server’s tensor-parallel size aligned with the deployment GPU count when changing the layout.

3. Match the deployment configuration

Save the matching example as deployment.yaml in the same directory. port, the server’s --port, and the readiness probe all use 8000. See the configuration reference for the complete set of options.

gpus: 1
gpu_type: h100
port: 8000
secret:
  - vllm-token
min_scale: 0
max_scale: 1
readiness:
  httpGet:
    path: /health
    port: 8000

gpus: 8
gpu_type: h100
port: 8000
secret:
  - vllm-token
min_scale: 0
max_scale: 1
readiness:
  httpGet:
    path: /health
    port: 8000

These settings allow scale-to-zero and cap the example at one replica. A request after idle shutdown can wait for startup. Set a positive min_scale when a warm replica is needed, then tune concurrency and max_scale from load-test results.

4. Deploy and check readiness

tensorkube deploy --config-file ./deployment.yaml
tensorkube deployment list

Inspect startup logs if the service stays unready. Common causes include unavailable GPUs, image/driver incompatibility, model loading failures, and insufficient memory. Configure a custom domain with TLS before sending credentials over a network.

5. Send an authenticated chat request

Set INFERENCE_URL to your HTTPS endpoint without /v1. Set MODEL_ID to the model you deployed, and retain the API key from step 1. The chat endpoint applies the model’s chat template.

export INFERENCE_URL="https://YOUR_APP_HOST"
export MODEL_ID="openai/gpt-oss-20b"
: "${VLLM_API_KEY:?Load the API key created in step 1}"

curl --fail-with-body --no-buffer \
  --url "${INFERENCE_URL%/}/v1/chat/completions" \
  --header 'Content-Type: application/json' \
  --header "Authorization: Bearer $VLLM_API_KEY" \
  --data "{\"model\":\"$MODEL_ID\",\"messages\":[{\"role\":\"user\",\"content\":\"Explain why an inference service needs a readiness check.\"}],\"max_tokens\":512,\"stream\":true}"

For the 120B deployment, set MODEL_ID to openai/gpt-oss-120b. A successful streaming request returns server-sent events; the exact generated text depends on the model and sampling settings. With the openai Python package installed, use the same environment variables:

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["INFERENCE_URL"].rstrip("/") + "/v1",
    api_key=os.environ["VLLM_API_KEY"],
)

response = client.chat.completions.create(
    model=os.environ["MODEL_ID"],
    messages=[{"role": "user", "content": "Explain a readiness check."}],
    max_tokens=512,
)
print(response.choices[0].message.content)

Validate reasoning output and tool calls separately using the upstream recipe when your application requires them. Basic chat success does not establish parser compatibility or tool correctness. Also confirm an invalid API key is rejected by the chat endpoint before using it beyond your test environment.

Benchmark GPT-OSS latency and cost

Measure your input lengths, output lengths, request rate, and cache state.

Reduce repeated prompt work

Evaluate prefix reuse for agent instructions and conversation history.

Large Language Models

Image and Video Models

Audio Models

Miscellaneous

Deploy GPT-OSS 20B and 120B on AWS with vLLM

Before you deploy

1. Create an API secret

2. Choose a Dockerfile

3. Match the deployment configuration

4. Deploy and check readiness

5. Send an authenticated chat request

Benchmark GPT-OSS latency and cost

Reduce repeated prompt work

​Before you deploy

​1. Create an API secret

​2. Choose a Dockerfile

​3. Match the deployment configuration

​4. Deploy and check readiness

​5. Send an authenticated chat request

​Related guides

Benchmark GPT-OSS latency and cost

Reduce repeated prompt work

Before you deploy

1. Create an API secret

2. Choose a Dockerfile

3. Match the deployment configuration

4. Deploy and check readiness

5. Send an authenticated chat request

Related guides