CPU versus GPU inference for local open-source models

When running open-source language models locally, the choice of hardware represents the most critical technical and financial decision. While the central processing unit (CPU) is already standard in virtually every existing server or workstation, a graphics processing unit (GPU) often demands substantial investments in dedicated PCIe cards, additional cooling capacity, and heavy-duty power supplies. The fundamental difference between both processing methods does not merely come down to raw compute power, but primarily to the way memory bandwidth, cache hierarchies, and parallel instructions are handled. In this article, we dissect the bottlenecks, compare performance across prefill and decode cycles, examine the impact of multi-channel architecture, and calculate the tipping point between affordable CPU utilization and necessary GPU acceleration.

The Fundamentals of Inference: Compute versus Memory Bandwidth

To understand why hardware choices have such a direct impact on the end-user experience, we need to break down the computational cycle of a Large Language Model (LLM). AI inference in autoregressive transformer models consists of two distinct phases: the prefill phase (where the input prompt is processed) and the decode phase (where the model generates the response token by token). These two phases impose fundamentally different demands on the underlying silicon architecture.

During the prefill phase, all input tokens are analyzed simultaneously. This step consists of matrix-matrix multiplications (General Matrix Multiply, or GEMM) and is heavily compute-intensive (compute-bound). This is where a processor featuring thousands of parallel compute cores and specialized matrix units, such as Nvidia's Tensor Cores, excels. The entire input context can be processed in a single computational pass, keeping the execution units fully saturated throughout.

During the subsequent decode phase, however, the dynamics change completely. To predict exactly one new token, the system must transfer virtually all of the model's parameters and weights from main memory to the compute units once. This step involves a matrix-vector multiplication (GEMV). Because arithmetic intensity (the number of floating-point operations per transferred byte) is extremely low here, the processor almost immediately becomes limited by memory bandwidth (memory-bound). The compute cores spend most of their time waiting for the required parameters to arrive across the memory bus.

The theoretical maximum throughput in tokens per second during this decode phase can be reliably approximated for single-batch inference using the following mathematical formula:

Doorvoersnelheid (tokens/s) ≈ Geheugenbandbreedte (GB/s) / Modelgrootte in werkgeheugen (GB)

When an 8B model quantized to 4-bit precision occupies roughly 5.5 GB of RAM on a standard system with dual-channel DDR5 memory (effective bandwidth of about 75 GB/s), the absolute theoretical ceiling is around 13.6 tokens per second. In practice, kernel overhead and context management reduce this to approximately 9 to 11 tokens per second. On a dedicated graphics card with GDDR6X memory (1000 GB/s bandwidth), this same model shoots up to well over 100 tokens per second. This mechanism provides the hard physical explanation for the speed difference between CPU and GPU.

Memory Architecture: DDR5 versus GDDR6, HBM, and Unified Memory

The massive contrast in memory bandwidth between standard system RAM and specialized video memory explains why pure CPU inference can feel slow in interactive applications. Standard desktop and server RAM is primarily designed for low random access times (low latency) with complex, sequential CPU instructions, not for moving massive, contiguous data streams simultaneously.

The table directly demonstrates why consumer CPUs with dual-channel memory struggle with larger language models. However, as soon as we move to octa-channel server configurations, CPU bandwidth begins to seriously make a difference. To determine exactly how much system memory or VRAM a chosen model architecture requires, including the necessary KV cache, it is advisable to beforehand calculate required VRAM from parameters so that memory bottlenecks do not occur during peak loads.

Compute power and vector instructions: AVX-512, AMX versus Tensor Cores

While the decode phase is primarily limited by memory bandwidth, processing long input prompts places high demands on raw compute power instead. During the prefill phase, the entire attention matrix must be calculated across all input tokens. This is where specialized instruction sets prove their value.

Modern CPUs leverage advanced vector and matrix instructions. With AVX2 and AVX-512, modern AMD and Intel processors can perform multiple 8-bit or 16-bit calculations simultaneously within a single clock cycle. Intel AMX (Advanced Matrix Extensions), present on modern Xeon processors, takes this a step further by dedicating hardware blocks to 2D matrix multiplications. This allows a high-end CPU to deliver significantly better performance than previous processor generations.

Nevertheless, the gap with graphics hardware remains enormous. A modern consumer GPU features thousands of streaming multiprocessors and hundreds of specialized Tensor Cores, pushing raw compute throughput for INT4 and FP16 operations into the hundreds to thousands of TFLOPS. This difference translates directly into the Time-to-First-Token (TTFT). Where a CPU takes several seconds over a 4000-token document to generate the first response token, a GPU completes that exact same task in under a hundred milliseconds.

The impact of quantization on CPU and GPU architectures

Quantization compresses model weights from their original 16-bit floating-point format (FP16 or BF16) into more compact precisions such as 8-bit, 4-bit, or even 2-bit integers. This compression yields two decisive benefits: the model fits into a substantially smaller memory footprint, and total data transfer across the memory bus per token generated is drastically reduced.

The choice of underlying hardware platform largely dictates which quantization format is preferred. For CPU inference, the GGUF format, developed within the open-source llama.cppecosystem, is the undisputed standard. GGUF kernels are specifically engineered to maximize CPU vector instructions (such as AVX-512) and system RAM without unnecessary overhead.

On dedicated GPUs, other formats often deliver significantly higher performance. Formats such as AWQ (Activation-aware Weight Quantization) and EXL2 (ExLlamaV2) are specifically optimized for the memory and compute hierarchy of GPUs. Consult the comparative guide on choosing quantization formats like GGUF, AWQ, and EXL2 to make the right trade-off between compute platform and file format.

Using 4-bit quantization, powerful models ranging from 7 to 14 billion parameters can be deployed effortlessly on hardware-constrained devices. How compact models perform within local environments without enterprise accelerators is covered extensively in the guide on small on-device models.

Hybrid Inference: Layer Offloading and PCIe Bottlenecks

When a model is just slightly too large for a graphics card's available video memory, hybrid inference via layer offloading provides a practical intermediate solution. Frameworks such as llama.cpp and Ollama make it possible to allocate an exact number of transformer model layers to the fast VRAM of the GPU, while the remaining layers are processed by the CPU and system RAM.

The principle behind layer offloading is straightforward to configure via the command-line interface:

# Starten van een 32-lagen model met gedeeltelijke GPU-offloading
# 24 lagen draaien in GPU VRAM (-ngl 24), de overige 8 lagen blijven in RAM
./llama-cli -m models/meta-llama-3-8b-instruct.Q4_K_M.gguf -ngl 24 -c 4096 -p "Vraag:"

During each token generation step, activation tensors are transferred back and forth between the transformer layers across the PCIe bus. Although the GPU layers are computed at lightning speed, an unavoidable latency penalty occurs during the transition to the CPU layers. The final generation speed is bottlenecked by the bandwidth of the system memory and the throughput of the PCIe slot (roughly 31.5 GB/s for a PCIe 4.0 x16 connection or 63 GB/s for PCIe 5.0).

Consequently, hybrid inference performs slower than a full GPU setup, but it delivers a substantial speedup compared to pure CPU execution and prevents processes from crashing due to out-of-memory errors.

Measurement Methodology and Reproducible Benchmark Setup

To determine performance differences objectively and reproducibly, benchmarks must be executed under standardized conditions. Simply timing an arbitrary chat prompt yields skewed results due to cache variability and varying response lengths. A reliable benchmark setup strictly decouples prefill latency from decode speed.

Standardized measurements utilize the internal benchmarking tools of llama.cpp, strictly fixing both the prompt length (e.g., exactly 512, 1024, or 2048 tokens) and the number of tokens to generate (e.g., 128 or 256 tokens):

# Gestandaardiseerde benchmark voor TTFT en tokengeneratie
./llama-bench -m models/llama-3-8b-instruct.Q4_K_M.gguf -p 512,2048 -n 128 -t 16

Key variables that must be controlled during measurements include:

1. Memory Channels: Verify via BIOS that RAM is actively running in multi-channel mode (dual-, quad-, or octa-channel).
2. Thermal Throttling: Ensure adequate cooling to prevent processors from downclocking under sustained loads.
3. Context Size: Keep the KV cache size identical across test runs, as an expanding context generates additional memory traffic.

Practical Comparison and Benchmark Results

The benchmark results below provide a representative overview of typical hardware configurations running a mainstream 8B model and a heavier 70B model, both quantized to 4-bit precision (Q4_K_M).

The figures clearly show that for small models, a consumer CPU can offer a reading speed that is just sufficient for single-user use (above 10 tokens per second), but falls short for interactive chatbots handling long documents. To make a balanced choice between throughput, latency, and model complexity, the overview on the balance between model size, latency, and accuracy provides additional guidelines.

Edge Cases and Exceptions: Concurrency, Batching, and Apple Silicon

When evaluating hardware, looking solely at single requests is not enough. In practice, specific scenarios arise where the performance dynamics shift.

Multiple Concurrent Users (Concurrency): As soon as a local system needs to handle multiple requests simultaneously via continuous batching, the computational intensity of the decode phase changes. Because multiple tokens are computed at the same time for different users, the operation shifts from matrix-vector (GEMV) to matrix-matrix (GEMM). GPUs scale exceptionally well in this scenario thanks to their massive parallel capacity. CPUs, by contrast, quickly become bottlenecked under concurrent requests, causing latency per user to increase linearly.

Long Contexts and KV Cache Pressure: With prompts exceeding 16,000 tokens, the size of the KV cache increases drastically. A model that typically fits into 6 GB can suddenly demand 12 to 16 GB of memory with a full context window. If this exceeds the VRAM limit of a graphics card, the GPU pipeline crashes unless offloading to CPU memory or advanced KV cache quantization is used.

The Apple Silicon Exception: Apple's Unified Memory Architecture (UMA) is in a category of its own. Because the CPU, GPU, and Neural Engine share direct access to the same wide memory pool with bandwidths of up to 800 GB/s, 70B parameter models can be loaded entirely into system memory without the physical VRAM constraints of standard PCIe add-in cards. This makes it an attractive platform for workstations used for local testing of large models.

Explicit Weaknesses and Operational Limitations

Every hardware choice involves clear trade-offs that must be factored into infrastructure design beforehand:

Limitations of CPU Inference:

The Time-to-First-Token on a CPU for documents with thousands of tokens is unacceptably slow for real-time user interfaces. Furthermore, power efficiency per token under heavy load is significantly worse: because a CPU computation takes up to ten times longer than on a GPU, the processor remains under full load and draws power for a much longer period.

Limitations of GPU Inference:

Graphics cards have a hard VRAM limit. If a model, including context, does not fit entirely into video memory, execution immediately fails with an Out-Of-Memory error message. Furthermore, high-end GPUs require heavy power supplies (often 850W to 1500W), generate substantial heat and noise, and require servers with specific PCIe layouts providing sufficient physical clearance for airflow.

Costs, power consumption, and TCO of local hardware

Beyond the initial purchase price of components, the Total Cost of Ownership (TCO) over a two- to three-year period is decisive. A server with multiple high-end graphics accelerators requires a substantial upfront investment, but delivers significantly more completed tokens per kilowatt-hour.

When an enterprise server operates 24/7 handling continuous AI workloads, a 700-watt GPU cluster can consume thousands of kilowatt-hours of electricity annually. For incidental or asynchronous background workloads (such as indexing internal documents overnight), it may actually be more cost-effective to utilize an existing CPU server, eliminating the need to purchase additional hardware.

When local model servers are deployed to integrate internal enterprise systems and microservices, a standardized API architecture is essential. How to reliably expose local models via a dedicated gateway with load balancing and routing can be found in the architecture guide on hosting local models behind a custom API.

Conclusion and decision framework for hardware selection

The trade-off between CPU and GPU inference is not a matter of principle, but a technical balance between latency requirements, concurrent user load, model size, and available budget. The decision rules below summarize the operational choices:

Choose CPU inference when:

The processing task can run asynchronously in the background (such as nightly summaries, email classification, or periodic data extraction); the model is compact (1B to 8B parameters with 4-bit quantization); there is strictly only one active user at a time; or when powerful server hardware with octa-channel DDR5 memory is already available, making additional investments unnecessary.

Choose GPU or Unified Memory acceleration when:

Real-time interaction is required, such as interactive chatbots or live assistants where responses must appear immediately; large prompts (more than 2000 tokens) are processed that require a fast Time-to-First-Token; multiple concurrent requests are handled; or when medium to large models (14B, 32B to 70B parameters) need to run with a sustained throughput above 25 tokens per second.