A new study tackles a practical question for large language model training: does on-policy distillation (OPD) — where a student learns from its own rollouts under dense, token-level supervision from a frozen teacher — actually reduce the compute needed to reach a given accuracy compared to reinforcement learning, specifically GRPO (Group Relative Policy Optimization)? The researchers compare the two approaches across several dimensions: task difficulty, student and teacher model size, domain, and optimization choices, measuring held-out accuracy against training FLOPs.
The headline finding is nuanced. When the teacher is an externally post-trained model, OPD generally underperforms GRPO in accuracy achieved per unit of compute, even when that teacher has itself been further improved via GRPO. OPD also tends to suffer from response-length inflation, which further erodes its practical efficiency. This challenges the intuitive assumption that dense, per-token feedback should automatically translate into faster learning.
The picture shifts substantially when the teacher is the student's own GRPO checkpoint. In that setting, OPD recovers the checkpoint's accuracy on math tasks using roughly 50 times fewer distillation FLOPs than GRPO would need to reach the same level. This efficiency gain holds consistently with a larger student model, on coding tasks, and in a second model family, suggesting a robust pattern rather than an isolated artifact.
The study extends this analysis to a more realistic multi-domain scenario, where separate GRPO checkpoints each excel in a specific domain but no single one covers every capability. A multi-teacher OPD approach consolidates these domain specialists into one student model, which matches each expert's accuracy at a distillation cost representing only 5.2% (across two domains) to 8.0% (across four domains) of the specialist training compute. Counting both specialist training and distillation together, the full pipeline matches the performance of a jointly trained GRPO model while using 42% less total compute.
The findings offer a useful, concrete takeaway for teams designing large-scale training pipelines: on-policy distillation is not a universally cheaper alternative to reinforcement learning, but it proves particularly valuable as a consolidation step after domain-specific specialist training, rather than as a direct substitute for GRPO.