Deploy Llama 4 Models on your AWS account

Llama 4 herd represents Meta’s newest generation of large language models, featuring Scout and Maverick variants.

These models introduce architecture innovations like Mixture of Experts (MoE) and Interleaved RoPE (iRoPE) that enable exceptional performance with massive context lengths while maintaining reasonable inference costs. In this guide, we’ll walk you through deploying these state-of-the-art models on your cloud account using Tensorfuse and vLLM v0.8.3.

​

Why Build with Llama 4

Llama 4 offers several compelling advantages that make it an excellent choice for production applications:

• Native Multimodality: Early fusion architecture seamlessly integrates text and images (up to 10 images per request)
• Massive Context Windows: Up to 10 million tokens for Scout, enabling multi-document summarization and reasoning over vast codebases
• Mixture of Experts (MoE) Architecture: More compute-efficient models that activate only a subset of parameters per token
• Interleaved RoPE (iRoPE): Novel attention mechanism that efficiently handles long sequences by alternating between global and local attention
• State-of-the-Art Performance: Competitive with or exceeding proprietary models like GPT-4o and Gemini 2.0
• Multilingual Support: Pre-trained on 200 languages, with over 100 languages having more than 1 billion tokens each
• Responsible AI: Meta’s advanced safety training and protections to prevent harmful, unsafe, and unethical outputs

Here’s a snapshot of benchmark scores for Llama 4:

​

Prerequisites

Before you begin, ensure you have configured Tensorfuse on your AWS account. If you haven’t done that yet, follow the Getting Started guide.

​

Deploying Llama 4 Models with Tensorfuse

Each Tensorkube deployment requires:

1. Your code (in this example, vLLM API server code from Docker image)
2. Your environment (as a Dockerfile)
3. A deployment configuration (deployment.yaml)

We will also add token-based authentication to our service, compatible with OpenAI client libraries.

​

Step 1: Setting up the secrets

tensorkube secret create hugging-face-secret HUGGING_FACE_HUB_TOKEN=hf_EkXXrzzZsuoZubXhDQ --env default

tensorkube secret create vllm-token VLLM_API_KEY=vllm-key --env default

​

Step 2: Prepare the Dockerfiles

Let’s create separate Dockerfiles for Scout and Maverick models:

FROM vllm/vllm-openai:v0.8.3

# Enable HF Hub Transfer
ENV HF_HUB_ENABLE_HF_TRANSFER=1

# Expose port 80
EXPOSE 80

# Entrypoint with API key
ENTRYPOINT [
  "python3", "-m", "vllm.entrypoints.openai.api_server",
  "--model", "meta-llama/Llama-4-Scout-17B-16E-Instruct",
  "--dtype", "bfloat16",
  "--trust-remote-code",
  "--tensor-parallel-size", "8",
  "--max-model-len", "1000000",
  "--port", "80",
  "--override-generation-config", "{\"attn_temperature_tuning\": true}",
  "--limit-mm-per-prompt", "image=10",
  "--kv-cache-dtype", "fp8",
  "--api-key", "${VLLM_API_KEY}"
]

We’ve configured the vLLM server with various CLI flags tailored to each model. For a comprehensive list of vLLM flags, refer to the vLLM documentation.

​

Step 3: Deployment Configuration

Create model-specific configuration files to optimize for each model’s requirements. This will be the same for both models.

gpus: 8
gpu_type: h100
secret:
  - hugging-face-secret
  - vllm-token
min-scale: 1
readiness:
    httpGet:
        path: /health
        port: 80

​

Step 4: Deploy your models

Deploy your services using these commands. Make sure there is only one Dockerfile in the directory (either Maverick or Scout).

tensorkube deploy --config-file ./deployment.yaml

​

Step 5: Accessing the deployed app

Voila! Your autoscaling production Llama 4 service is ready. Only authenticated requests will be served.

Once deployment is successful, check the status:

tensorkube deployment list

To test your deployment, replace YOUR_APP_URL with the endpoint from the command output and run:

curl --request POST
--url YOUR_APP_URL/v1/completions
--header 'Content-Type: application/json'
--header 'Authorization: Bearer vllm-key'
--data '{
"model": "meta-llama/Llama-4-Scout-17B-16E-Instruct",
"prompt": "Earth to Llama 4. What can you do?",
"max_tokens": 5000
}'

Since vLLM is compatible with the OpenAI API you can query the other endpoints present here.

You can also use the OpenAI Python SDK to query your deployment as shown below:

import openai

# Replace with your actual URL and token
base_url = "YOUR_APP_URL/v1"
api_key = "vllm-key"

client = openai.OpenAI(
    base_url=base_url,
    api_key=api_key
)

response = client.completions.create(
    model="meta-llama/Llama-4-Scout-17B-16E-Instruct",
    prompt="Hello, Llama 4! What can you do today?",
    max_tokens=200
)

print(response)

​

Multimodal Capabilities

Llama 4 shines with its early fusion multimodal architecture, allowing it to process text and images simultaneously. Here’s how to use multimodal capabilities with your deployment:

import openai
import base64

# Example 1: Single Image Chat Completion

# Load image as base64
with open("chart.jpg", "rb") as image_file:
    image_base64 = base64.b64encode(image_file.read()).decode('utf-8')

# Prepare the API client
client = openai.OpenAI(
    base_url="YOUR_APP_URL/v1",
    api_key="vllm-key"
)

# Create chat completion with the image
response = client.chat.completions.create(
    model="meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Analyze this chart and explain the key trends:"},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}}
            ]
        }
    ],
    max_tokens=300
)

print(response.choices.message.content)

# ------------------------------------------------------------------

# Example 2: Multiple Images Chat Completion

# Load multiple images as base64 and prepare image list
images = []
for filename in ["image1.jpg", "image2.jpg", "image3.jpg"]:
    with open(filename, "rb") as image_file:
        image_base64 = base64.b64encode(image_file.read()).decode('utf-8')
    images.append({"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}})

# Combine text and images into the message content
message_content = [
    {"type": "text", "text": "Compare these three images and tell me the differences:"}
]
message_content.extend(images)

# Prepare the API client (if needed, reusing the previous client instance)
client = openai.OpenAI(
    base_url="YOUR_APP_URL/v1",
    api_key="vllm-key"
)

# Make the API call with multiple images
response = client.chat.completions.create(
    model="meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8",
    messages=[
        {"role": "user", "content": message_content}
    ],
    max_tokens=500
)

print(response.choices.message.content)

​

Context Length Capabilities

Llama 4 models offer impressive context length capabilities across different hardware configurations:

These massive context windows enable entirely new use cases:

​

Advanced Configurations and Optimization Tips

​

Performance Optimization

1. FP8 KV Cache: Add --kv-cache-dtype fp8 to potentially double the usable context window and gain a performance boost with minimal accuracy impact:

   • Before optimization: ~90 tokens/sec on 8x H100
   • After optimization: ~180 tokens/sec on 8x H100

2. Long Context Accuracy: For contexts longer than 32K tokens, include --override-generation-config='{"attn_temperature_tuning": true}' to improve accuracy.

3. Continuous Batching: vLLM already implements continuous batching by default, maximizing throughput for multiple concurrent users.

4. Quantization: For Maverick, use the FP8 model variant which provides excellent performance with minimal accuracy drop.

​

Hardware Compatibility

1. A100 GPUs: The BF16 versions of both models work well on A100 GPUs but with reduced context lengths:

2. INT4 Quantization: For Scout, an INT4 quantization that allows the model to fit on a single H100 GPU is in development.

3. AMD MI300X: You can run Llama 4 on AMD MI300X GPUs by building vLLM from source, with nearly identical accuracy.

​

Key Architectural Innovations

Llama 4’s architecture enables efficient long-context inference through several innovations:

Mixture of Experts (MoE): Instead of activating all parameters for every token, Llama 4 models use a “router” to select which expert(s) should process each token:

• Scout has 16 experts (109B total parameters)
• Maverick has 128 experts (400B total parameters)
• Only 1-2 experts are activated per token (17B active parameters)
• This approach dramatically reduces computational costs while maintaining quality

Interleaved RoPE (iRoPE): Llama 4 alternates between global attention (without RoPE) and chunked local attention (with RoPE) in a 1:3 ratio:

• Global layers capture document-level patterns
• Local layers process detailed information within chunks
• This combination significantly reduces the quadratic complexity of attention

Early Fusion Multimodality: Rather than using separate encoders, Llama 4 integrates text and vision tokens directly into its core architecture:

• Images are processed through a vision encoder and then projected into the same embedding space as text tokens
• The model can attend to both text and image tokens simultaneously
• This enables deeper multimodal reasoning than late-fusion approaches

​

Conclusion

With this guide, you’ve successfully deployed Llama 4 models on serverless GPUs using Tensorfuse. These models represent the cutting edge of open-source AI, offering capabilities that rival or exceed proprietary alternatives at a fraction of the cost.

Whether you’re building a sophisticated chatbot, a multimodal content creation tool, or an enterprise knowledge system, Llama 4 provides the foundation for building AI applications that were previously only possible with proprietary models.

Click here to get started with Tensorfuse.

You can also explore the Tensorfuse examples repository for more deployment configurations and use cases.