Enterprise AI Analysis

From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Space

While Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances LLM reasoning by optimizing the conditional distribution P(y|x), its potential is fundamentally bounded by the base model's existing output distribution. This paper introduces PreRL (Pre-train Space RL) to address this by optimizing the marginal distribution P(y), thereby encoding reasoning ability and preserving broad exploration capacity through reward-driven online updates. We theoretically and empirically validate PreRL's strong gradient alignment with P(y|x) and uncover Negative Sample Reinforcement (NSR) as a critical mechanism. NSR-PreRL rapidly prunes incorrect reasoning spaces, stimulating endogenous reflective behaviors (increasing transition thoughts by 14.89x and reflection thoughts by 6.54x). Leveraging these insights, we propose Dual Space RL (DSRL), a Policy Reincarnation strategy that initializes models with NSR-PreRL to expand the reasoning horizon before transitioning to standard RL for fine-grained optimization. Extensive experiments demonstrate that DSRL consistently outperforms strong baselines, proving that pre-train space pruning effectively steers the policy toward a refined correct reasoning subspace.