Reinforcement Learning

To mitigate plasticity loss, various approaches have been proposed (Section 2). Among these, a particularly promising family of methods is based on periodically resetting network parameters [6, 14, 22, 25]. Resets are effective because they restore the network to a well-conditioned, highly plastic initialization that is gradually lost during training. As networks adapt to specific tasks or data distributions, they accumulate pathologies—such as dormant neurons, increasing weight magnitudes, and reduced rank—that impair their ability to learn from new data [6]. Resetting the parameters removes these accumulated effects and reinitializes the network to conditions resembling its original, plastic initialization. Nikishin et al. [22] empirically demonstrated that resetting a network can substantially improve performance by renewing its ability to learn and exploit data.

AltNet makes a structural departure from prior reset-based approaches. It prevents recently-reset networks from acting in the environment until they have received sufficient training. In contrast, Standard Resets [22] expose the reset network directly to the environment, making immediate performance collapse inevitable. RDE [14] employs ensembles with a Q-value-weighted gating policy to reduce the likelihood that a reset agent acts prematurely, but still allows recently reset networks to act. AltNet, on the other hand, guarantees that only trained networks interact with the environment. In AltNet, reset networks first train passively before taking over. Empirically, the resulting mechanism is more robust and simpler: AltNet avoids post-reset performance drops across replay ratios and achieves higher and more stable returns.

In reinforcement learning, agents learn through direct interaction with the environment, which is often slow and expensive in real-world domains such as robotics or healthcare applications. This makes sample efficiency—learning as much as possible from limited interactions—a central concern. A common strategy to improve sample efficiency is to increase the replay ratio (RR), defined as the number of gradient updates performed per environment step [8, 9, 27]. Higher replay ratios allow agents to reuse past experiences more extensively, thereby extracting additional learning signals from limited data. However, increasing RR also linearly increases computational cost and, beyond a point, can degrade performance due to overfitting to outdated experiences [8, 22].

As shown in Figure 4, increasing the replay ratio from 1 to 8 improves SAC's performance by allowing more updates per sample, but performance is substantially lower when RR = 32. AltNet, by contrast, achieves superior performance even at RR = 1 (see Figure 8) and RR = 4 (see Figure 4), surpassing SAC trained at much higher replay ratios. This shows that unlike SAC, AltNet achieves higher performance and greater sample efficiency at lower replay ratios, and consequently reduced computational overhead. We further establish that AltNet's improved performance is not due to increased capacity from the additional network: reducing the total number of parameters across the two networks to match a single SAC network yields nearly identical performance (Figure 5).