MTP (Multi-Token Prediction) support merged into llama.cpp

What Happened

Multi-Token Prediction (MTP) support has been merged into llama.cpp via pull request #22673, confirmed by the project maintainers and corroborated across r/LocalLLaMA discussion threads. The merge places MTP in the main branch of the codebase, meaning it will ship with subsequent standard builds rather than requiring a separate fork or patch.

MTP is a speculative decoding technique that allows a model to predict multiple tokens in a single forward pass, rather than the traditional one-token-per-step autoregressive loop.

Why It Matters

Token generation latency in autoregressive inference scales roughly linearly with the number of sequential forward passes required to produce a sequence. MTP reduces that count by producing several predicted tokens per step, and the framework then validates those predictions — a mechanism structurally similar to speculative decoding but operating inside the model rather than requiring a separate draft model.

For operators running local inference on constrained hardware, this shifts the performance ceiling without requiring hardware upgrades, quantized model swaps, or architectural changes to the serving stack. The practical beneficiary is anyone whose workload is bottlenecked by tokens-per-second rather than memory bandwidth or model quality: interactive chat, code completion, agent loops, and streaming applications where perceived latency is a direct function of forward-pass count.

Because the merge lands in mainline llama.cpp, downstream consumers — Ollama, LM Studio, llama-cpp-python, and various agent frameworks — inherit the capability on their next sync, though with varying latency depending on how their abstraction layers expose decoding options.

Technical Details

MTP was introduced in models such as Meta's DeepSeek-V3 derivatives and DeepSeek's own V3/R1 architectures, which trained dedicated MTP heads that output predictions for tokens at offsets of 1 through N. These heads add parameters and compute at training time, but at inference they enable the model to propose a draft sequence that the base model then verifies in parallel.

The llama.cpp implementation wires MTP into the sampling and verification path, so compatible GGUF models with MTP heads present can engage the technique; models trained without MTP heads cannot, regardless of runtime flags. Gains depend on acceptance rate — how often the predicted tokens match what the base model would have produced — which varies by model, prompt distribution, and temperature. Warmer sampling typically lowers acceptance and erodes the speedup.

Integration requirements are minimal: a compatible model file, a recent llama.cpp build, and appropriate configuration. The feature does not change memory footprint materially for the base model, though MTP heads and their associated KV cache entries add some overhead that scales with the number of predicted tokens per step.

Operational Impact