By Samagra Sharma · Reviewed September 6, 2026
A small extraction or classification workload may fit on spare CPU capacity. To find out, build llama.cpp with GPU backends disabled, load a quantized model, and measure how many correct requests finish within your deadline. This guide serves Qwen2.5-1.5B-Instruct Q4_K_M through a local OpenAI-compatible API on Linux or macOS.
The model is a small starting point for validating the serving path. Its ability to answer a smoke test does not establish quality for a production workload.
Reviewed September 6, 2026 against the pinned llama.cpp source and official Qwen model files. This guide has not been benchmarked on CPU hardware. It describes a standalone llama.cpp setup and does not imply additional Tensorfuse hardware support.
1. Build llama.cpp for CPU execution
Install Git, curl, CMake, and a C/C++ compiler. On Ubuntu, install build-essential, cmake, git, and curl with your package manager. On macOS, install the Xcode Command Line Tools and CMake. Then create a fresh checkout:
This pins the v0.4.0 release. A fresh build directory matters because CMake remembers options from previous builds. Metal defaults on for supported Macs; disabling it makes this a CPU baseline. Other accelerator backends remain off in this fresh default configuration. See the upstream build instructions and backend options.
OpenSSL support inside llama.cpp is disabled here because curl downloads the model separately and the server listens locally over HTTP.
2. Download one GGUF file
Run from the checkout directory:
This is the approximately 1.12 GB Q4_K_M file in Qwen’s official GGUF repository. The commit in the URL fixes the model revision. Quantization reduces weight storage; total RAM usage also includes the KV cache, compute buffers, the operating system, and other processes. Leave several GB of available memory for this initial short-context experiment and check actual usage after loading.
3. Start the server and make a request
Use four threads as an initial setting on a machine with at least four available CPU cores; lower it on smaller hosts. The 2,048-token context must accommodate the formatted input and generated output. One server slot makes the initial latency measurement easier to interpret.
In a second terminal:
Expect a successful health response after loading, qwen-cpu in the models response, and text inside choices[0].message.content. Check the label itself for correctness. The alias and endpoints follow the pinned server API. Keep the unauthenticated example on loopback and stop it with Ctrl+C when finished.
4. Find the cost and latency boundary
Replay representative requests, including longer inputs and failure cases. Compare warm and first-request latency, accepted requests per minute, and peak memory. Increase thread count one setting at a time while keeping the model, prompts, and output limits fixed. More threads can lose performance through memory contention or competition with other services.
Compare CPU and GPU using the same task deadline and quality threshold:
Include the full VM charge for a dedicated cloud CPU instance. For shared capacity, document its cost allocation and the impact on neighboring workloads. For owned hardware, include electricity and hardware allocation. Count retries and fallback requests. A lower hourly price can still produce a higher cost per useful result when throughput is too low.
What if the CPU run fails?
Use the benchmarking guide for a controlled comparison. On a Mac, test MLX on the Apple GPU separately. Read the CPU and Apple silicon explainer and inference cost checklist before selecting the next backend. 
