Built with developer experience in mind, Tensorkube simplifies the process of deploying serverless GPU apps. In this guide, we will walk you through the process of deploying llama3.1-8b-instruct on your private cloud.

​
Prerequisites

Before you begin, ensure you have configured Tensorkube on your AWS account. If you haven’t done that yet, follow the Getting Started guide.

​
Deploying Llama-3.1-8B-Instruct with Tensorfuse

Each tensorkube deployment requires two things - your code and your environment (as a Dockerfile). While deploying machine learning models, it is beneficial if your model is also a part of your container image. This reduces cold-start times by a significant margin. To enable this, in addition to a FastAPI app and a dockerfile, we will also write code to download the model and place it in our image file.

​
Download the model

We will write a small script that downloads the llama3.1-8b-instruct model from the Hugging Face model hub and saves it in the /models directory.

download_model.py
import os

from huggingface_hub import snapshot_download
access_token = '<YOUR-HUGGINGFACE_TOKEN>'


if __name__=='__main__':
    # download the meta/llama3 model
    os.makedirs('./models', exist_ok=True)
    snapshot_download(repo_id="meta-llama/Meta-Llama-3.1-8B-Instruct", local_dir="models",ignore_patterns=["*.pt", "*.bin"], token=access_token)

​
Code files

We will write a small FastAPI app that loads the model and serves predictions. The FastAPI app will have three endpoints - /readiness, /, and /generate. Remember that the /readiness endpoint is used by Tensorkube to check the health of your deployments.

main.py
import torch
import transformers
from fastapi import FastAPI
import os

app = FastAPI()

model_dir = "models"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

pipeline = transformers.pipeline(
    "text-generation",
    model=model_dir,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

@app.get("/")
async def root():
    is_cuda_available = torch.cuda.is_available()
    return {
        "message": "Hello World",
        "cuda_available": is_cuda_available,
    }

@app.get("/readiness")
async def readiness():
    return {"status": "ready"}

# an inference endpoint for text generation
@app.post("/generate")
async def generate_text(data: dict):
    text = data.get("text")
    if not text:
        return {"error": "text field is required"}
    prompt = text
    
    messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": prompt}
    ]

    outputs = pipeline(
    messages,
    max_new_tokens=256,
    )

    response = outputs[0]["generated_text"][-1]["content"]
    return {"generated_text": response}

​
Environment files (Dockerfile)

Next, create a Dockerfile for your FastAPI app. Given below is a simple Dockerfile that you can use:

Dockerfile
# Use the nvidia cuda base image
FROM nvidia/cuda:12.1.1-devel-ubuntu22.04

# Update and install required packages
RUN apt-get update && apt-get install -y \
    python3.10 \
    python3.10-dev \
    python3-pip \
    && rm -rf /var/lib/apt/lists/*

# Set Python 3.10 as the default Python version
RUN ln -s /usr/bin/python3.10 /usr/bin/python

# Upgrade pip
RUN pip3 install --no-cache-dir --upgrade pip && pip install transformers && pip install torch fastapi uvicorn pydantic accelerate

# Set working directory
WORKDIR /code

# Copy the code files
COPY main.py /code/main.py
COPY download_model.py /code/download_model.py

# Run the downloader script to download the model
RUN python download_model.py

EXPOSE 80

# Start a uvicorn server on port 80
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "80"]

​
Deploying the app

Llama-3.1-8b-instruct is now ready to be deployed on Tensorkube. Navigate to your project root and run the following command:

tensorkube deploy --gpus 1 --gpu-type a10g

Llama-3.1-8b-instruct is now deployed on your AWS account. You can access your app at the URL provided in the output or using the following command:

tensorkube list deployments

And that’s it! You have successfully deployed llama-3.1-8b-instruct on serverless GPUs using Tensorkube. 🚀

To test it out you can run the following command by replacing the URL with the one provided in the output:


curl -X POST <YOUR_APP_URL_HERE>/generate -H "Content-Type: application/json" -d '{"text":"Name the intern at Crowstrike who pushed the buggy update"}'

You can also use the readiness endpoint to wake up your nodes in case you are expecting incoming traffic

curl <YOUR_APP_URL_HERE>/readiness
Deploying Int4 Quantized Llama-3.1-70B-Instruct on Serverless GPUsDeploying Stable Diffusion 3 Medium on Serverless GPUs