PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

PagedWeight introduces a dynamic quantization scheme for Mixture-of-Experts LLMs that adjusts weight precision per token based on quality impact, reducing memory and compute without uniform degradation.

For operators running large MoE models, per-token quality-aware quantization directly lowers per-inference cost and latency while preserving output fidelity. This shifts the trade-off between throughput and accuracy from static model-level configuration to runtime granularity, enabling more efficient cache utilization and higher batch sizes. The approach reduces the pressure to over-provision hardware for peak quality demands, meaning operators can serve more requests on the same infrastructure. Operationally, scheduling and load balancing layers must now account for per-token precision state, adding a new variable to inference pipelines. Expect workflows that previously relied on static quantization pass selection to become obsolete in favor of real-time adaptive precision.