On-Policy Self-Distillation without Any Supervision

2026-08-06 · 17:18 UTC ·Yijiang Li, Bingyang Wang, Yijun Liang... ·1 min read

Brief

Unsupervised On-Policy Self-Distillation (U-OPSD) eliminates external supervision by leveraging a model's own generations: multiple rollouts produce a majority-vote pseudo-solution, a teacher is conditioned on the shortest pseudo-solution, and distillation is applied to prefixes of the longest incorrect completion. On math/problem benchmarks with Qwen3 4B/8B, U-OPSD yields substantial gains (e.g., +8.5%/+10.7% non-thinking) and matches or surpasses OPSD and GRPO.

Why it matters

U-OPSD (Unsupervised On-Policy Self-Distillation) achieves true self-distillation without external supervision by using internal consistency: sample multiple rollouts, form a pseudo-solution via majority vote under a self-consistency threshold, condition a teacher on the shortest pseudo-solution, and distill into prefixes of the model's longest incorrect completion.

Key details

On benchmarks AIME24, AIME25, HMMT25, MATH500, and AMC23 with Qwen3 models, U-OPSD improves base-model accuracy in non-thinking mode by 8.5% (4B) and 10.7% (8B) and outperforms OPSD by an average of 3.2% (4B) and 2.3% (8B).

In thinking mode U-OPSD is on par or better than supervised methods: it beats OPSD by 0.9% at 4B and matches OPSD at 8B, while surpassing GRPO by 0.7% (4B) and 1.1% (8B).

Abstract

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose Unsupervised On-Policy Self-Distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo-solution by majority vote under a self-consistency threshold. It then conditions a teacher distribution on the shortest pseudo-solution and distills it into prefixes of the model's longest incorrect completion, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT), such as OPSD and GRPO. On AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at the 4B and 8B scales, respectively, and outperforms OPSD by an average of 3.2% and 2.3%. In thinking mode, U-OPSD remains on par with OPSD, outperforming it by 0.9% at 4B and matching it at 8B, while surpassing GRPO by 0.7% and 1.1%, respectively.