On-Policy Distillation in LLMs: From Noisy Teacher to Self-Improvement

On-policy distillation, where a student model learns from its own sampled outputs under teacher supervision, shows measurable gains in training stability and self-correction behavior, per this paper’s controlled experiments. The finding contradicts the assumption that off-policy teacher data is strictly superior for distillation.

Operationally, this signals a shift toward compute-efficient fine-tuning loops. Builders currently relying on static, pre-generated teacher datasets for RLHF or SFT can reduce dependency on expensive offline data pipelines. The practical implication: distillation becomes a closed-loop process—sampling, scoring, and updating occur in a single forward pass cycle, lowering memory and storage overhead for dataset versioning. This makes iterative model improvement cheaper and faster, especially for small teams with constrained GPU budgets. A second-order effect is that evaluation harnesses must adapt, as standard static benchmark sets may no longer capture the dynamics of self-improvement. Expect tooling for online reward models and adaptive sampling to become more critical than curated corpora.