PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

What Happened

Researchers introduced PagedWeight, a serving technique for Mixture-of-Experts LLMs that applies dynamic, quality-aware weight quantization on a per-token basis. Rather than committing to a single static precision level across the model, PagedWeight assigns precision tiers according to each token's measured sensitivity to quantization error, keeping high-impact tokens at higher fidelity while demoting low-impact ones. The scheme targets the memory and compute bottlenecks that dominate MoE inference, where only a subset of experts activate per token but the full weight footprint must remain resident.

Why It Matters

MoE architectures trade parameter count for conditional compute, but serving them still requires holding far more weights in memory than dense models of equivalent active size. Static quantization—the conventional lever—forces a single accuracy-versus-cost decision across the entire model, which means operators either over-provision hardware to preserve quality on sensitive tokens or accept uniform degradation to cut costs. Quality-aware per-token quantization breaks that coupling, letting the runtime spend precision where it affects output and reclaim memory where it does not. For operators, this translates directly into lower cost per inference, higher achievable batch sizes, and better cache utilization without a corresponding fidelity penalty. It also reduces the incentive to reserve headroom for worst-case quality demand, which is where much of the current infrastructure spend accumulates.

Technical Details

PagedWeight quantizes weights at runtime according to a per-token quality budget, with precision tiers mapped to paged weight storage so that experts can be fetched at the granularity the token requires. The mechanism depends on page-level memory management, meaning the serving stack must track precision state alongside weight residency rather than treating quantized weights as a fixed model artifact. Integration requirements include a scheduler aware of per-token precision, kernel support for mixed-precision expert dispatch, and a cache policy that can evict or promote weight pages by tier. Limitations follow from the same design: precision decisions add per-token overhead, and heavily skewed sensitivity distributions may force more high-tier fetches than the average case suggests. Reported gains concentrate in memory footprint and batch throughput rather than raw FLOP reduction, so the benefit is largest where memory bandwidth and KV-cache pressure are the binding constraints.

Operational Impact

Scheduling and load-balancing layers inherit a new runtime variable: per-token precision state. Batching logic that previously grouped requests by sequence length or expert affinity must now consider precision tier distribution, since a batch mixing many high-tier tokens behaves differently from one that is mostly low-tier. Static quantization pass selection—today's default workflow for fitting MoE models onto fixed hardware—becomes obsolete, replaced by adaptive precision policies tuned per deployment. Caching shifts from a simple residency problem to a tiered promotion problem, which changes eviction heuristics and can improve hit rates if tuned correctly. The net effect is that capacity planning moves from model-level provisioning toward runtime policy tuning, and the cost of serving a marginal request depends on its token-level quality profile rather than a fixed per-model constant.