Built with developer experience in mind, Tensorkube simplifies the process of deploying serverless GPU apps. In this guide, we will walk you through the process of deploying Stable Diffusion 3 Medium on your private cloud.

​
Prerequisites

Before you begin, ensure you have configured Tensorkube on your AWS account. If you haven’t done that yet, follow the Getting Started guide.

​
Deploying Stable Diffusion 3 Medium with Tensorfuse

Each tensorkube deployment requires two things - your code and your environment (as a Dockerfile). While deploying machine learning models, it is beneficial if your model is also a part of your container image. This reduces cold-start times by a significant margin. To enable this, in addition to a FastAPI app and a dockerfile, we will also write code to download the model and place it in our image file.

​
Download the model

We will write a small script that downloads the stable-diffusion-3-medium model from the Hugging Face model hub and saves it in the /models directory.

download_model.py
import os

from huggingface_hub import snapshot_download
access_token = '<YOUR-HUGGINGFACE_TOKEN>'


if __name__=='__main__':
    # download the meta/llama3 model
    os.makedirs('./models', exist_ok=True)
    snapshot_download(repo_id="stabilityai/stable-diffusion-3-medium-diffusers", local_dir="models",ignore_patterns=["*.pt", "*.bin"], token=access_token)

​
Code files

We will write a small FastAPI app that loads the model and generate images from text prompts. The FastAPI app will have three endpoints - /readiness, /, and /generate. Remember that the /readiness endpoint is used by Tensorkube to check the health of your deployments.

main.py
import torch
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from diffusers import StableDiffusion3Pipeline
import io

app = FastAPI()

model_dir = "models"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

pipe = StableDiffusion3Pipeline.from_pretrained(model_dir, torch_dtype=torch.float16).to(device)


@app.get("/")
async def root():
    is_cuda_available = torch.cuda.is_available()
    return {
        "message": "Hello World",
        "cuda_available": is_cuda_available,
    }

@app.get("/readiness")
async def readiness():
    return {"status": "ready"}

# an inference endpoint for image generation
@app.post("/generate")
async def generate_image(data: dict):
    text = data.get("text")
    if not text:
        return {"error": "text field is required"}
    prompt = text

    image = pipe(
    prompt,
    negative_prompt="",
    num_inference_steps=28,
    guidance_scale=7.0,
    ).images[0]

    # Convert the image to a byte stream
    img_byte_arr = io.BytesIO()
    image.save(img_byte_arr, format='PNG')
    img_byte_arr.seek(0)

    return StreamingResponse(img_byte_arr, media_type='image/png')

​
Environment files (Dockerfile)

Next, create a Dockerfile for your FastAPI app. Given below is a simple Dockerfile that you can use:

Dockerfile
# Use the nvidia cuda base image
FROM nvidia/cuda:12.1.1-devel-ubuntu22.04

# Update and install required packages
RUN apt-get update && apt-get install -y \
    python3.10 \
    python3.10-dev \
    python3-pip \
    && rm -rf /var/lib/apt/lists/*

# Set Python 3.10 as the default Python version
RUN ln -s /usr/bin/python3.10 /usr/bin/python

# Upgrade pip
RUN pip3 install --no-cache-dir --upgrade pip && pip install transformers && pip install torch fastapi uvicorn pydantic accelerate protobuf sentencepiece diffusers

# Set working directory
WORKDIR /code

# Copy the code files
COPY main.py /code/main.py
COPY download_model.py /code/download_model.py

# Run the downloader script to download the model
RUN python download_model.py

EXPOSE 80

# Start a uvicorn server on port 80
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "80"]

​
Deploying the app

SD3-Medium is now ready to be deployed on Tensorkube. Navigate to your project root and run the following command:

tensorkube deploy --gpus 1 --gpu-type a10g

SD3-Medium is now deployed on your AWS account. You can access your app at the URL provided in the output or using the following command:

tensorkube list deployments

And that’s it! You have successfully deployed SD3-Medium on serverless GPUs using Tensorkube. 🚀

To test it out you can run the following command by replacing the URL with the one provided in the output:


curl -X POST -H "Content-Type: application/json" -d '{"text":"Generate an image of a cat holding Hello world signboard"}' <YOUR_APP_URL_HERE>/generate -o output_image.png

You can also use the readiness endpoint to wake up your nodes in case you are expecting incoming traffic

curl <YOUR_APP_URL_HERE>/readiness
Deploying Llama-3.1-8B-Instruct on Serverless GPUsDeploying Resnet on serverless GPUs