Deploy Llama 4 Models on your AWS account

​

Why Build with Llama 4

• Native Multimodality: Early fusion architecture seamlessly integrates text and images (up to 10 images per request)
• Massive Context Windows: Up to 10 million tokens for Scout, enabling multi-document summarization and reasoning over vast codebases
• Mixture of Experts (MoE) Architecture: More compute-efficient models that activate only a subset of parameters per token
• Interleaved RoPE (iRoPE): Novel attention mechanism that efficiently handles long sequences by alternating between global and local attention
• State-of-the-Art Performance: Competitive with or exceeding proprietary models like GPT-4o and Gemini 2.0
• Multilingual Support: Pre-trained on 200 languages, with over 100 languages having more than 1 billion tokens each
• Responsible AI: Meta’s advanced safety training and protections to prevent harmful, unsafe, and unethical outputs

​

Prerequisites

​

Deploying Llama 4 Models with Tensorfuse

1. Your code (in this example, vLLM API server code from Docker image)
2. Your environment (as a Dockerfile)
3. A deployment configuration (deployment.yaml)

​

Step 1: Setting up the secrets

tensorkube secret create hugging-face-secret HUGGING_FACE_HUB_TOKEN=hf_EkXXrzzZsuoZubXhDQ --env default

tensorkube secret create vllm-token VLLM_API_KEY=vllm-key --env default

​

Step 2: Prepare the Dockerfiles

FROM tensorfuse/vllm-openai:v0.8.4-patched

    # Enable HF Hub Transfer
ENV HF_HUB_ENABLE_HF_TRANSFER=1

# Expose port 80
EXPOSE 80

# Entrypoint with API key
ENTRYPOINT python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --dtype bfloat16 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --max-model-len 1000000 \
  --port 80 \
  --override-generation-config '{"attn_temperature_tuning": true}' \
  --limit-mm-per-prompt image=10 \
  --kv-cache-dtype fp8 \
  --api-key ${VLLM_API_KEY}

FROM vllm/vllm-openai:v0.8.3

# Enable HF Hub Transfer
ENV HF_HUB_ENABLE_HF_TRANSFER=1

# Expose port 80
EXPOSE 80

# Entrypoint with API key
ENTRYPOINT python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --max-model-len 430000 \
  --port 80 \
  --override-generation-config '{"attn_temperature_tuning": true}' \
  --limit-mm-per-prompt image=10 \
  --api-key ${VLLM_API_KEY}

​

Step 3: Deployment Configuration

gpus: 8
gpu_type: h100
secret:
  - hugging-face-secret
  - vllm-token
min_scale: 1
readiness:
    httpGet:
        path: /health
        port: 80

​

Step 4: Deploy your models

tensorkube deploy --config-file ./deployment.yaml

​

Step 5: Accessing the deployed app

tensorkube deployment list

curl --request POST
--url YOUR_APP_URL/v1/completions
--header 'Content-Type: application/json'
--header 'Authorization: Bearer vllm-key'
--data '{
"model": "meta-llama/Llama-4-Scout-17B-16E-Instruct",
"prompt": "Earth to Llama 4. What can you do?",
"max_tokens": 5000
}'

curl --request POST \
--url YOUR_APP_URL/v1/completions \
--header 'Content-Type: application/json' \
--data '{
"model": "meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8",
"prompt": "Earth to Llama 4. What can you do?",
"max_tokens": 5000
}'

import openai

# Replace with your actual URL and token
base_url = "YOUR_APP_URL/v1"
api_key = "vllm-key"

client = openai.OpenAI(
    base_url=base_url,
    api_key=api_key
)

response = client.completions.create(
    model="meta-llama/Llama-4-Scout-17B-16E-Instruct",
    prompt="Hello, Llama 4! What can you do today?",
    max_tokens=200
)

print(response)

​

Multimodal Capabilities

import openai
import base64

# Example 1: Single Image Chat Completion

# Load image as base64
with open("chart.jpg", "rb") as image_file:
    image_base64 = base64.b64encode(image_file.read()).decode('utf-8')

# Prepare the API client
client = openai.OpenAI(
    base_url="YOUR_APP_URL/v1",
    api_key="vllm-key"
)

# Create chat completion with the image
response = client.chat.completions.create(
    model="meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Analyze this chart and explain the key trends:"},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}}
            ]
        }
    ],
    max_tokens=300
)

print(response.choices.message.content)

# ------------------------------------------------------------------

# Example 2: Multiple Images Chat Completion

# Load multiple images as base64 and prepare image list
images = []
for filename in ["image1.jpg", "image2.jpg", "image3.jpg"]:
    with open(filename, "rb") as image_file:
        image_base64 = base64.b64encode(image_file.read()).decode('utf-8')
    images.append({"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}})

# Combine text and images into the message content
message_content = [
    {"type": "text", "text": "Compare these three images and tell me the differences:"}
]
message_content.extend(images)

# Prepare the API client (if needed, reusing the previous client instance)
client = openai.OpenAI(
    base_url="YOUR_APP_URL/v1",
    api_key="vllm-key"
)

# Make the API call with multiple images
response = client.chat.completions.create(
    model="meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8",
    messages=[
        {"role": "user", "content": message_content}
    ],
    max_tokens=500
)

print(response.choices.message.content)

​

Context Length Capabilities

​

Advanced Configurations and Optimization Tips

​

Performance Optimization

1. FP8 KV Cache: Add --kv-cache-dtype fp8 to potentially double the usable context window and gain a performance boost with minimal accuracy impact:
   • Before optimization: ~90 tokens/sec on 8x H100
   • After optimization: ~180 tokens/sec on 8x H100
2. Long Context Accuracy: For contexts longer than 32K tokens, include --override-generation-config='{"attn_temperature_tuning": true}' to improve accuracy.
3. Continuous Batching: vLLM already implements continuous batching by default, maximizing throughput for multiple concurrent users.
4. Quantization: For Maverick, use the FP8 model variant which provides excellent performance with minimal accuracy drop.

​

Hardware Compatibility

1. A100 GPUs: The BF16 versions of both models work well on A100 GPUs but with reduced context lengths:

2. INT4 Quantization: For Scout, an INT4 quantization that allows the model to fit on a single H100 GPU is in development.
3. AMD MI300X: You can run Llama 4 on AMD MI300X GPUs by building vLLM from source, with nearly identical accuracy.

​

Key Architectural Innovations

• Scout has 16 experts (109B total parameters)
• Maverick has 128 experts (400B total parameters)
• Only 1-2 experts are activated per token (17B active parameters)
• This approach dramatically reduces computational costs while maintaining quality

• Global layers capture document-level patterns
• Local layers process detailed information within chunks
• This combination significantly reduces the quadratic complexity of attention

• Images are processed through a vision encoder and then projected into the same embedding space as text tokens
• The model can attend to both text and image tokens simultaneously
• This enables deeper multimodal reasoning than late-fusion approaches

​

Conclusion