Deploy Llama 4 Models on your AWS account

​

Why Build with Llama 4

• Native Multimodality: Early fusion architecture seamlessly integrates text and images (up to 10 images per request)
• Massive Context Windows: Up to 10 million tokens for Scout, enabling multi-document summarization and reasoning over vast codebases
• Mixture of Experts (MoE) Architecture: More compute-efficient models that activate only a subset of parameters per token
• Interleaved RoPE (iRoPE): Novel attention mechanism that efficiently handles long sequences by alternating between global and local attention
• State-of-the-Art Performance: Competitive with or exceeding proprietary models like GPT-4o and Gemini 2.0
• Multilingual Support: Pre-trained on 200 languages, with over 100 languages having more than 1 billion tokens each
• Responsible AI: Meta’s advanced safety training and protections to prevent harmful, unsafe, and unethical outputs

​

Prerequisites

​

Deploying Llama 4 Models with Tensorfuse

1. Your code (in this example, vLLM API server code from Docker image)
2. Your environment (as a Dockerfile)
3. A deployment configuration (deployment.yaml)

​

Step 1: Setting up the secrets

tensorkube secret create hugging-face-secret HUGGING_FACE_HUB_TOKEN=hf_EkXXrzzZsuoZubXhDQ --env default

tensorkube secret create vllm-token VLLM_API_KEY=vllm-key --env default

​

Step 2: Prepare the Dockerfiles

FROM tensorfuse/vllm-openai:v0.8.4-patched

    # Enable HF Hub Transfer
ENV HF_HUB_ENABLE_HF_TRANSFER=1

# Expose port 80
EXPOSE 80

# Entrypoint with API key
ENTRYPOINT [
  "python3", "-m", "vllm.entrypoints.openai.api_server",
  "--model", "meta-llama/Llama-4-Scout-17B-16E-Instruct",
  "--dtype", "bfloat16",
  "--trust-remote-code",
  "--tensor-parallel-size", "8",
  "--max-model-len", "1000000",
  "--port", "80",
  "--override-generation-config", "{\"attn_temperature_tuning\": true}",
  "--limit-mm-per-prompt", "image=10",
  "--kv-cache-dtype", "fp8",
  "--api-key", "${VLLM_API_KEY}"
]

​

Step 3: Deployment Configuration

gpus: 8
gpu_type: h100
secret:
  - hugging-face-secret
  - vllm-token
min-scale: 1
readiness:
    httpGet:
        path: /health
        port: 80

​

Step 4: Deploy your models

tensorkube deploy --config-file ./deployment.yaml

​

Step 5: Accessing the deployed app

tensorkube deployment list

curl --request POST
--url YOUR_APP_URL/v1/completions
--header 'Content-Type: application/json'
--header 'Authorization: Bearer vllm-key'
--data '{
"model": "meta-llama/Llama-4-Scout-17B-16E-Instruct",
"prompt": "Earth to Llama 4. What can you do?",
"max_tokens": 5000
}'

import openai

# Replace with your actual URL and token
base_url = "YOUR_APP_URL/v1"
api_key = "vllm-key"

client = openai.OpenAI(
    base_url=base_url,
    api_key=api_key
)

response = client.completions.create(
    model="meta-llama/Llama-4-Scout-17B-16E-Instruct",
    prompt="Hello, Llama 4! What can you do today?",
    max_tokens=200
)

print(response)

​

Multimodal Capabilities

import openai
import base64

# Example 1: Single Image Chat Completion

# Load image as base64
with open("chart.jpg", "rb") as image_file:
    image_base64 = base64.b64encode(image_file.read()).decode('utf-8')

# Prepare the API client
client = openai.OpenAI(
    base_url="YOUR_APP_URL/v1",
    api_key="vllm-key"
)

# Create chat completion with the image
response = client.chat.completions.create(
    model="meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Analyze this chart and explain the key trends:"},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}}
            ]
        }
    ],
    max_tokens=300
)

print(response.choices.message.content)

# ------------------------------------------------------------------

# Example 2: Multiple Images Chat Completion

# Load multiple images as base64 and prepare image list
images = []
for filename in ["image1.jpg", "image2.jpg", "image3.jpg"]:
    with open(filename, "rb") as image_file:
        image_base64 = base64.b64encode(image_file.read()).decode('utf-8')
    images.append({"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}})

# Combine text and images into the message content
message_content = [
    {"type": "text", "text": "Compare these three images and tell me the differences:"}
]
message_content.extend(images)

# Prepare the API client (if needed, reusing the previous client instance)
client = openai.OpenAI(
    base_url="YOUR_APP_URL/v1",
    api_key="vllm-key"
)

# Make the API call with multiple images
response = client.chat.completions.create(
    model="meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8",
    messages=[
        {"role": "user", "content": message_content}
    ],
    max_tokens=500
)

print(response.choices.message.content)

​

Context Length Capabilities

​

Advanced Configurations and Optimization Tips

​

Performance Optimization

1. FP8 KV Cache: Add --kv-cache-dtype fp8 to potentially double the usable context window and gain a performance boost with minimal accuracy impact:
   • Before optimization: ~90 tokens/sec on 8x H100
   • After optimization: ~180 tokens/sec on 8x H100
2. Long Context Accuracy: For contexts longer than 32K tokens, include --override-generation-config='{"attn_temperature_tuning": true}' to improve accuracy.
3. Continuous Batching: vLLM already implements continuous batching by default, maximizing throughput for multiple concurrent users.
4. Quantization: For Maverick, use the FP8 model variant which provides excellent performance with minimal accuracy drop.

​

Hardware Compatibility

1. A100 GPUs: The BF16 versions of both models work well on A100 GPUs but with reduced context lengths:

2. INT4 Quantization: For Scout, an INT4 quantization that allows the model to fit on a single H100 GPU is in development.
3. AMD MI300X: You can run Llama 4 on AMD MI300X GPUs by building vLLM from source, with nearly identical accuracy.

​

Key Architectural Innovations

Llama 4's Mixture-of-Experts Architecture

• Scout has 16 experts (109B total parameters)
• Maverick has 128 experts (400B total parameters)
• Only 1-2 experts are activated per token (17B active parameters)
• This approach dramatically reduces computational costs while maintaining quality

• Global layers capture document-level patterns
• Local layers process detailed information within chunks
• This combination significantly reduces the quadratic complexity of attention

• Images are processed through a vision encoder and then projected into the same embedding space as text tokens
• The model can attend to both text and image tokens simultaneously
• This enables deeper multimodal reasoning than late-fusion approaches

​

Conclusion