Home / AI Research / Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts
REAI Research
Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts
by illinois · Zhenrui Yue, Gongxiaohui Chen, Huimin Zeng, Honglei Zhuang, Jiashuo Sun, Zhen Han
Frontier post-training follows a common recipe: train one expert per domain with reinforcement learning (RL), then consolidate the experts into one model via multi-teacher on-policy distillation (MOPD). Such consolidation is expensive: the student samples in every environment again, an expert scores every sampled token, and the student inherits the drift of the final experts. Yet we argue that the experts' RL runs have already produced what consolidation needs: rollouts with their sampling probabilities and rewards, in the order the experts learned from them. We propose multi-expert replay policy distillation (MRPD), which replays these training logs through the student in the experts'…
Upvotes5
Comments2
Published17 Nov 2026