llama.cpp Adds CPU Expert Offload Cache for MoE Models

What Happened

A llama.cpp pull request introduces llama_moe_cache, a CPU-side expert offload cache designed for Mixture-of-Experts models. The implementation runs expert layers that miss the GPU residency window on CPU rather than stalling or failing, and ships with a tuning script for calibrating cache size and offload behavior against specific hardware. The change targets large MoE checkpoints running on memory-constrained consumer and workstation configurations.

Why It Matters

MoE architectures front-load a structural problem: total parameter count determines memory footprint, but only a fraction of experts activate per token. Prior CPU offload paths in llama.cpp handled dense layers acceptably but degraded badly on MoE because expert routing produces unpredictable, sparse access patterns across the parameter space. Caching missed experts in a CPU-resident tier converts that unpredictability from a hard failure mode into a latency tradeoff. Operators can now size a cache proportional to their VRAM budget and accept measured per-token CPU cost instead of being locked out of checkpoints entirely. This expands the viable deployment envelope for open-weight MoE models — including recent large sparse releases — to single-GPU and mixed CPU/GPU nodes where full residency was previously impossible.

Technical Details

The cache intercepts expert tensors evicted from GPU memory and retains them in pinned host memory, with the tuning script exposing cache capacity and eviction parameters for workload-specific calibration. Routing sparsity in MoE means hit rates depend heavily on batch size and token distribution: small batches with diverse routing paths will miss more often than batched inference with correlated expert selection, so the tuning script's role is to map cache size against observed hit rate for a given model and prompt distribution. The design assumes PCIe-class interconnect; CPU-side expert execution adds per-token latency proportional to miss rate multiplied by expert compute time on host cores. Expect this to be viable for interactive use only at moderate miss rates, and more suited to throughput-oriented or batch workloads where latency amortizes. Integration is within the existing llama.cpp offload framework — no new runtime dependencies, but cache memory competes directly with KV cache and model weights for host RAM, which constrains usable context length on memory-tight systems.

Operational Impact

Builders running 30B–120B-class sparse models on 24–48GB GPUs gain a path to deployment without multi-GPU provisioning, at the cost of accepting a documented latency floor. Capacity planning shifts: instead of asking "does the model fit," operators now ask "what miss rate does the workload tolerate, and what host RAM does that imply" — a quantifiable tradeoff the tuning script makes measurable. Batch and offline inference pipelines benefit most, since miss-rate penalties amortize across concurrent sequences; real-time chat deployments should benchmark before committing. The host RAM budget becomes a first-class deployment constraint alongside VRAM, and node specs that previously sat idle (large-RAM, modest-GPU boxes) become viable inference targets for MoE workloads.