How to run LLM inference on a CPU with llama.cpp

By Samagra Sharma · Reviewed September 6, 2026 A small extraction or classification workload may fit on spare CPU capacity. To find out, build llama.cpp with GPU backends disabled, load a quantized model, and measure how many correct requests finish within your deadline. This guide serves Qwen2.5-1.5B-Instruct Q4_K_M through a local OpenAI-compatible API on Linux or macOS. The model is a small starting point for validating the serving path. Its ability to answer a smoke test does not establish quality for a production workload.

Reviewed September 6, 2026 against the pinned llama.cpp source and official Qwen model files. This guide has not been benchmarked on CPU hardware. It describes a standalone llama.cpp setup and does not imply additional Tensorfuse hardware support.

1. Build llama.cpp for CPU execution

Install Git, curl, CMake, and a C/C++ compiler. On Ubuntu, install build-essential, cmake, git, and curl with your package manager. On macOS, install the Xcode Command Line Tools and CMake. Then create a fresh checkout:

git clone --depth 1 --branch v0.4.0 \
  https://github.com/ggml-org/llama.cpp.git llama-cpp-cpu
cd llama-cpp-cpu

cmake -S . -B build-cpu \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_METAL=OFF \
  -DGGML_CUDA=OFF \
  -DGGML_HIP=OFF \
  -DGGML_VULKAN=OFF \
  -DGGML_SYCL=OFF \
  -DLLAMA_OPENSSL=OFF
cmake --build build-cpu --config Release --target llama-server -j 4
./build-cpu/bin/llama-server --version

This pins the v0.4.0 release. A fresh build directory matters because CMake remembers options from previous builds. Metal defaults on for supported Macs; disabling it makes this a CPU baseline. Other accelerator backends remain off in this fresh default configuration. See the upstream build instructions and backend options. OpenSSL support inside llama.cpp is disabled here because curl downloads the model separately and the server listens locally over HTTP.

2. Download one GGUF file

Run from the checkout directory:

mkdir -p models
curl --fail --location --retry 3 \
  'https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF/resolve/91cad51170dc346986eccefdc2dd33a9da36ead9/qwen2.5-1.5b-instruct-q4_k_m.gguf' \
  --output models/qwen2.5-1.5b-instruct-q4_k_m.gguf

This is the approximately 1.12 GB Q4_K_M file in Qwen’s official GGUF repository. The commit in the URL fixes the model revision. Quantization reduces weight storage; total RAM usage also includes the KV cache, compute buffers, the operating system, and other processes. Leave several GB of available memory for this initial short-context experiment and check actual usage after loading.

3. Start the server and make a request

./build-cpu/bin/llama-server \
  --model models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  --alias qwen-cpu \
  --n-gpu-layers 0 \
  --ctx-size 2048 \
  --parallel 1 \
  --threads 4 \
  --host 127.0.0.1 \
  --port 8080

Use four threads as an initial setting on a machine with at least four available CPU cores; lower it on smaller hosts. The 2,048-token context must accommodate the formatted input and generated output. One server slot makes the initial latency measurement easier to interpret. In a second terminal:

curl --fail http://127.0.0.1:8080/health
curl --fail http://127.0.0.1:8080/v1/models

curl --fail --show-error http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen-cpu",
    "messages": [{
      "role": "user",
      "content": "Classify this message as billing or technical. Reply with one label: I was charged twice."
    }],
    "temperature": 0,
    "max_tokens": 16
  }'

Expect a successful health response after loading, qwen-cpu in the models response, and text inside choices[0].message.content. Check the label itself for correctness. The alias and endpoints follow the pinned server API. Keep the unauthenticated example on loopback and stop it with Ctrl+C when finished.

4. Find the cost and latency boundary

Replay representative requests, including longer inputs and failure cases. Compare warm and first-request latency, accepted requests per minute, and peak memory. Increase thread count one setting at a time while keeping the model, prompts, and output limits fixed. More threads can lose performance through memory contention or competition with other services. Compare CPU and GPU using the same task deadline and quality threshold:

cost per accepted request =
  total serving cost during the test window
  / requests meeting both quality and latency requirements

Include the full VM charge for a dedicated cloud CPU instance. For shared capacity, document its cost allocation and the impact on neighboring workloads. For owned hardware, include electricity and hardware allocation. Count retries and fallback requests. A lower hourly price can still produce a higher cost per useful result when throughput is too low.

What if the CPU run fails?

Symptom	Next check
Build cannot find a compiler	Install the platform’s compiler tools and rerun configuration.
Server cannot load the model	Check the curl exit status, file size, and exact file path.
Requests exceed context	Shorten the input or raise `--ctx-size` after checking available RAM.
Added threads make requests slower	Reduce threads and check competing load, memory bandwidth, and socket placement.
The small model misses your quality target	Compare a larger model or higher precision before accepting a cost result.

Use the benchmarking guide for a controlled comparison. On a Mac, test MLX on the Apple GPU separately. Read the CPU and Apple silicon explainer and inference cost checklist before selecting the next backend.

Get Started

LLM Inference

Inference How-to Guides

Inference Hardware

Concepts

Operations

Troubleshooting

Enterprise Setup

Architecture

How to run LLM inference on a CPU with llama.cpp

1. Build llama.cpp for CPU execution

2. Download one GGUF file

3. Start the server and make a request

4. Find the cost and latency boundary

What if the CPU run fails?

​1. Build llama.cpp for CPU execution

​2. Download one GGUF file

​3. Start the server and make a request

​4. Find the cost and latency boundary

​What if the CPU run fails?

1. Build llama.cpp for CPU execution

2. Download one GGUF file

3. Start the server and make a request

4. Find the cost and latency boundary

What if the CPU run fails?