1. [Publications](/index.php/publications)
2. When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better

 # When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better

  ![Publication image](/sites/default/files/styles/wide/public/default_images/default.jpeg?itok=TfIobf92 "Publication image")

 On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs? We show that a simple alternative, Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency. Across 17 teacher-student pairs ranging from 1.5B to 235B parameters, Semi-OPD outperforms OPD in 14 cases, with up to +13.6% accuracy and 11.4x training speedup. We further find that the choice between OPD and Semi-OPD depends on the alignment between the initial teacher and student, quantified by an output-token overlap ratio: OPD is beneficial only when the two are highly aligned with high overlap ratios. Our deeper investigation suggests that effective distillation requires on-policyness w.r.t. both the student and the teacher. For misaligned pairs, student rollouts can become increasingly off-policy w.r.t. the teacher as context length grows, weakening the distillation signal. In contrast, Semi-OPD is often more stable, as it distills on shorter contexts while covering full trajectories and exposing the student to more teacher-preferred tokens. Beyond proposing Semi-OPD as an efficient alternative, our work motivates the community to rethink when to use OPD and to study stronger OPD variants with meaningful teacher-student pairs.

 ## Authors

Siyan Zhao (NVIDIA)

[Yonggan Fu](/index.php/person/yonggan-fu)

[Jindong Jiang](/index.php/person/jindong-jiang)

Shih-Yang Liu (NVIDIA)

[Song Bian](/index.php/person/song-bian)

[Byung-Kwan Lee](/index.php/person/byung-kwan-lee)

Sharath Turuvekere Sreenivas (NVIDIA)

Wenliang Dai (NVIDIA)

[Hanrong Ye](/index.php/person/hanrong-ye)

Aditya Grover (NVIDIA)

[Pavlo Molchanov](/index.php/person/pavlo-molchanov)

 ## Publication Date

Sunday, December 6, 2026

 ## Published in

[ArXiv](https://arxiv.org/abs/2610.11291)

 ## Research Area

[Artificial Intelligence and Machine Learning ](/index.php/research-area/machine-learning-artificial-intelligence)

[Natural Language Processing](/index.php/research-area/natural-language-processing)