Deploying Int4 Quantized Llama-3.1-70B-Instruct on Serverless GPUs
Deploy serverless GPU applications on your AWS account
Built with developer experience in mind, Tensorkube simplifies the process of deploying serverless GPU apps. In this guide, we will walk you through the process of deploying 4-bit quantized version of llama3.1-70b-instruct on your private cloud.
Prerequisites
Before you begin, ensure you have configured Tensorkube on your AWS account. If you haven’t done that yet, follow the Getting Started guide.
Deploying Int4 Quantized Llama-3.1-70B-Instruct with Tensorfuse
Each tensorkube deployment requires two things - your code and your environment (as a Dockerfile). While deploying machine learning models, it is beneficial if your model is also a part of your container image. This reduces cold-start times by a significant margin. To enable this, in addition to a FastAPI app and a dockerfile, we will also write code to download the model and place it in our image file.
Download the model
We will write a small script that downloads the hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4 model from the Hugging Face model hub and saves it in the /models
directory.
Code files
We will write a small FastAPI app that loads the model and serves predictions. The FastAPI app will have three endpoints - /readiness
, /
, and /generate
. Remember that the /readiness
endpoint is used by Tensorkube to check the health of your deployments.
Environment files (Dockerfile)
Next, create a Dockerfile for your FastAPI app. Given below is a simple Dockerfile that you can use:
Deploying the app
Our model is now ready to be deployed on AWS. Navigate to your project root and run the following command:
Int4 Quantized Llama-3.1-70B-Instruct is now deployed on your AWS account. You can access your app at the URL provided in the output or using the following command:
And that’s it! You have successfully deployed the quantized llama-3.1-70b-instruct on serverless GPUs using Tensorkube. 🚀
To test it out you can run the following command by replacing the URL with the one provided in the output:
You can also use the readiness endpoint to wake up your nodes in case you are expecting incoming traffic