AI RESEARCH ANALYSIS

Mixture-of-Experts (MoE) Models: A Unique Challenge

For Mixture-of-Experts (MoE) LLMs, the stability challenges of RL are further amplified. MoE models dynamically select and activate a subset of expert parameters for each token. This expert routing mechanism introduces additional complexity:

• Amplified Training-Inference Discrepancy: Inconsistent expert routing between training and inference engines, even with the same model parameters, can lead to different outputs and thus unstable importance sampling.
• Enhanced Policy Staleness: Policy updates not only change model parameters (θ) but can also shift the routed experts (e), significantly altering the policy and impacting stability.

To counteract this, the paper introduces Routing Replay. This technique stabilizes MoE training by fixing the routed experts during policy optimization, allowing the model to be optimized more like a dense network. Two implementations are discussed:

• Vanilla Routing Replay (R2): Mitigates policy staleness by replaying routed experts determined by the rollout policy in the training engine.
• Rollout Routing Replay (R3): Reduces training-inference discrepancy by uniformly replaying routed experts determined by the rollout policy in the inference engine, also impacting policy staleness.

These methods are crucial for unlocking the potential of MoE architectures in demanding enterprise applications, ensuring their training remains robust and effective.

Key Techniques for Stable RL Training

The research empirically validates several techniques essential for stabilizing RL training, especially when combined in off-policy settings:

• Importance Sampling (IS) Correction: Identified as an inherent component of the first-order approximation, the IS weight corrects for training-inference discrepancy and is critical for stability. Omitting it leads to rapid training collapse.
• Clipping Mechanism (e.g., PPO): Prevents aggressive policy updates by stopping gradients for certain tokens. This effectively restrains policy staleness, a key contributor to instability, especially when off-policy updates are introduced.
• Routing Replay (R2 & R3): For MoE models, both R2 and R3 are shown to be essential. R2 performs better for small degrees of off-policiness, while R3 becomes necessary and superior under larger off-policiness by reducing training-inference discrepancies and policy staleness.

These combined strategies—particularly importance sampling, clipping, and Routing Replay for MoE models—form a robust recipe for achieving high training stability and performance, enabling enterprise AI teams to reliably enhance LLM capabilities.

Beyond Cold-Start: The Power of Stable Training

A significant finding of this research is that with a stable RL training recipe in place, the choice of cold-start initialization becomes less critical. Experiments demonstrated that models initialized with different cold-start data consistently achieve comparable final performance after prolonged, stable RL training.

This insight shifts the focus for enterprise AI development: instead of optimizing heavily for specific cold-start conditions, resources can be concentrated on developing robust and stable RL training methodologies. Once training is stabilized, models—whether on-policy or off-policy—consistently converge to similar peak performance. This underscores the paramount importance of sustained stable training for successfully scaling LLM capabilities, ensuring that computational investments yield predictable and high-quality results.