Built with developer experience in mind, Tensorkube simplifies the process of deploying serverless GPU apps. In this guide, we will walk you through the process of deploying Meta-Llama-3.2-11B-Instruct on your private cloud.

​
Prerequisites

Before you begin, ensure you have configured Tensorkube on your AWS account. If you haven’t done that yet, follow the Getting Started guide.

​
Deploying Meta-Llama-3.2-11B-Instruct with Tensorfuse

Each tensorkube deployment requires two things - your code and your environment (as a Dockerfile). While deploying machine learning models, it is beneficial if your model is also a part of your container image. This reduces cold-start times by a significant margin. To enable this, in addition to a FastAPI app and a dockerfile, we will also write code to download the model and place it in our image file. Learn more about llama 3.2 and gradio by visiting the docs. Also we are using a flask server with a mounted gradio app to serve the model requests. For more information about this you can refer the gradio apps here.

​
Download the model

We will write a small script that downloads the Llama-3.2-Vision-Instruct model from the Hugging Face model hub and saves it in the /models directory. Note: Since llama3.2 is a gated repo, you will need to request the authors of repo here for the access to the model.

download_model.py
import os

from huggingface_hub import snapshot_download
access_token = '<YOUR-HUGGINGFACE_TOKEN>'


if __name__=='__main__':
    # download the meta/llama3 model
    os.makedirs('./models', exist_ok=True)
    snapshot_download(repo_id="meta-llama/Llama-3.2-11B-Vision-Instruct", local_dir="models",ignore_patterns=["*.pt", "*.bin"], token=access_token)

​
Code files

We will write a small FastAPI app that loads the model and serves predictions by mounting the gradio app. The FastAPI app will have three endpoints - /readiness, /, and /gradio. Remember that the /readiness endpoint is used by Tensorkube to check the health of your deployments.

main.py
import torch
import transformers
from fastapi import FastAPI
import os

app = FastAPI()

model_dir = "models"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

pipeline = transformers.pipeline(
    "text-generation",
    model=model_dir,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(
    model_dir,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    local_files_only=True
)

@app.get("/")
async def root():
    is_cuda_available = torch.cuda.is_available()
    return {
        "message": "Hello World",
        "cuda_available": is_cuda_available,
    }

@app.get("/readiness")
async def readiness():
    return {"status": "ready"}


def generate_image(prompt, image):
    # Generate the image
    messages = [
        {"role": "user", "content": [
            {"type": "image"},
            {"type": "text", 
            "text": f"{prompt}"}
        ]}
    ]
    input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
    if image:
        inputs = processor(
            image, input_text, add_special_tokens=False, return_tensors="pt"
        ).to(model.device)
    else:
        inputs = processor( 
            input_text, add_special_tokens=False, return_tensors="pt"
        ).to(model.device)
    output = model.generate(**inputs, max_new_tokens=30)
    return processor.decode(output[0])

# Create the Gradio interface for the VLm model llama 3.2 instruct
interface = gr.Interface(
    fn=generate_image,
    inputs=[
        gr.Textbox(label="Prompt"),
        gr.Image(type="pil", label="Input Image")
    ],
    outputs="text",
    title="Llama 3.2 Instruct Image Generator using Tensorfuse",
    description="Generate images with VLM Model Llama 3.2 Instruct. Provide a prompt and optionally an input image.",
)

app = gr.mount_gradio_app(app, interface, path="/gradio")

​
Environment files (Dockerfile)

Next, create a Dockerfile for your FastAPI app. Given below is a simple Dockerfile that you can use:

Dockerfile
# Use the nvidia cuda base image
FROM nvidia/cuda:12.1.1-devel-ubuntu22.04

# Update and install required packages
RUN apt-get update && apt-get install -y \
    ffmpeg \
    python3.10 \
    python3.10-dev \
    python3-pip \
    && rm -rf /var/lib/apt/lists/*

# Set Python 3.10 as the default Python version
RUN ln -s /usr/bin/python3.10 /usr/bin/python

# Upgrade pip
RUN pip3 install --no-cache-dir --upgrade pip && pip install transformers torch accelerate fastapi huggingface_hub[cli] gradio hf_transfer

# Set working directory
WORKDIR /code

# Copy the code files
COPY main.py /code/main.py
COPY download_model.py /code/download_model.py

RUN HF_HUB_ENABLE_HF_TRANSFER=1 python download_model.py

EXPOSE 80

# Start a uvicorn server on port 80
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "80"]

​
Deploying the app

Llama-3.2-11b-instruct is now ready to be deployed on Tensorkube. Navigate to your project root and run the following command:

tensorkube deploy --gpus 1 --gpu-type a10g

Llama-3.2-11b-instruct is now deployed on your AWS account. You can access your app at the URL provided in the output or using the following command:

tensorkube list deployments

And that’s it! You have successfully deployed llama-3.2-11b-instruct on serverless GPUs using Tensorkube. 🚀

To test the app, visit the following link for the gradio deployment.


curl -X POST <YOUR_APP_URL_HERE>/gradio

You can also use the readiness endpoint to wake up your nodes in case you are expecting incoming traffic

curl <YOUR_APP_URL_HERE>/readiness
Deploy Mochi 1 Preview Video Model using ComfyUI on your AWSDeploying Pixtral-12B-Quantised-4-Bit on Serverless GPUs