Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, John Schulman

OpenAI

{alex, vickipfau, csh, oleg, joschu}@openai.com

Abstract

In this report, we present a new reinforcement learning (RL) benchmark based on the Sonic the Hedgehog^TM video game franchise. This benchmark is intended to measure the performance of transfer learning and few-shot learning algorithms in the RL domain. We also present and evaluate some baseline algorithms on the new benchmark.

1             Motivation

In the past few years, it has become clear that deep reinforcement learning can solve difficult, high-dimensional problems when given a good reward function and unlimited time to interact with the environment. However, while this kind of learning is a key aspect of intelligence, it is not the only one. Ideally, intelligent agents would also be able to generalize between tasks, using prior experience to pick up new skills more quickly. In this report, we introduce a new benchmark that we designed to make it easier for researchers to develop and test RL algorithms with this kind of capability.

Most popular RL benchmarks such as the ALE [1] are not ideal for testing generalization between similar tasks. As a result, RL research tends to “train on the test set”, boasting an algorithm’s final performance on the same environment(s) it was trained on. For the field to advance towards algorithms with better generalization properties, we need RL bench- marks with proper splits between “train” and “test” environments, similar to supervised learning datasets. Our benchmark has such a split, making it ideal for measuring cross-task generalization.

One interesting application of cross-task generalization is few-shot learning. Recently, supervised few-shot learning algorithms have improved by leaps and bounds [2]–[4]. This progress has hinged on the availability of good meta-learning datasets such as Omniglot [5] and Mini-ImageNet [6]. Thus, if we want better few-shot RL algorithms, it makes sense to construct a similar kind of dataset for RL. Our benchmark is designed to be a meta-learning dataset, consisting of many similar tasks sampled from a single task distribution. Thus, it is a suitable test bed for few-shot RL algorithms.

Beyond few-shot learning, there are many other applications of cross-task generalization that require the right kind of benchmark. For example, you might want an RL algorithm to learn how to explore in new environments. Our benchmark poses a fairly challenging exploration problem, and the train/test split presents a unique opportunity to learn how to explore on some levels and transfer this ability to other levels.

2           Related Work

Our Gym Retro project, as detailed in Section 3.1, is related to both the Retro Learning En- vironment (RLE) [7] and the Arcade Learning Environment (ALE) [1]. Unlike these projects, however, Gym Retro aims to be flexible and easy to extend, making it straightforward to create a huge number of RL environments.

Our benchmark is related to other meta-learning datasets like Omniglot [5] and Mini- ImageNet [6]. In particular, our benchmark is intended to serve the same purpose for RL as datasets like Omniglot serve for supervised learning.

Our baselines in Section 4 explore the ability of RL algorithms to transfer between video game environments. Several prior works have reported positive transfer results in the video game setting:

• Parisotto et al. [8] observed that pre-training on certain Atari games could increase a network’s learning speed on other Atari games.
• Rusu et al. [9] proposed a new architecture for transfer learning called progressive networks, and showed that it could boost learning speed across a variety of previously unseen Atari games.
• Pathak et al. [10] found that an exploratory agent trained on one level of Super Mario Bros. could be used to boost performance on two other levels.
• Fernando et al. [11] found that their PathNet algorithm increased learning speed on average when transferring from one Atari game to another.
• Higgins et al. [12] used an unsupervised vision objective to produce robust features for a policy, and found that this policy was able to transfer to previously unseen vision tasks in DeepMind Lab [13] and MuJoCo [14].

In previous literature on transfer learning in RL, there are two common evaluation tech- niques: evaluation on synthetic tasks, and evaluation on the ALE. The former evaluation technique is rather ad hoc and makes it hard to compare different algorithms, while the latter typically reveals fairly small gains in sample complexity. One problem with the ALE in particular is that all the games are quite different, meaning that it may not be possible to get large improvements from transfer learning.

Ideally, further research in transfer learning would be able to leverage a standardized benchmark that is difficult like the ALE but rich with similar environments like well-crafted synthetic tasks. We designed our proposed benchmark to satisfy both criteria.

3           The Sonic Benchmark

This section describes the Sonic benchmark in detail. Each subsection focuses on a different aspect of the benchmark, ranging from technical details to high-level design features.

3.1           Gym Retro

Underlying the Sonic benchmark is Gym Retro, a project aimed at creating RL environments from various emulated video games. At the core of Gym Retro is the gym-retro Python package, which exposes emulated games as Gym [15] environments. Like RLE [7], gym-retro uses the libretro API¹ to interface with game emulators, making it very easy to add new emulators to gym-retro.

The gym-retro package includes a dataset of games. Each game in the dataset consists of a ROM, one or more save states, one or more scenarios, and a data file. Here are high-level descriptions of each of these components:

• ROM – the data and code that make up a game; loaded by an emulator to play that game.

• Save state – a snapshot of the console’s state at some point in the game. For example, a save state could be created for the beginning of each level.

• Data file – a file describing where various pieces of information are stored in console memory. For example, a data file might indicate where the score is located.

• Scenario – a description of done conditions and reward functions. A scenario file can reference fields from the data file.

3.2          The Sonic Video Game

In this benchmark, we use three similar games: Sonic The Hedgehog^TM, Sonic The Hedgehog^TM2, and Sonic 3 & Knuckles. All of these games have very similar rules and controls, although there are subtle differences between them (e.g. Sonic 3 & Knuckles includes some extra controls and characters). We use multiple games to get as many environments for our dataset as possible.

Each Sonic game is divided up into zones, and each zone is further divided up into acts. While the rules and overarching objective remain the same throughout the entire game, each zone has a unique set of textures and objects. Different acts within a zone tend to share these textures and objects, but differ in spatial layout. We will refer to a (ROM, zone, act) tuple as a “level”.

The Sonic games provide a rich set of challenges for the player. For example, some zones include platforms that the player must jump on in order to open doors. Other zones require the player to first jump on a lever to send a projectile into the air, then wait for the projectile to fall back on the lever to send the player over some sort of obstacle. One zone even has a swing that the player must jump off of at a precise time in order to launch Sonic up to a higher platform. Examples of these challenges are presented in Figure 1.

3.3          Games and Levels

Our benchmark consists of a total of 58 save states taken from three different games, where each of these save states has the player at the beginning of a different level. A number of acts from the original games were not used because they contained only boss fights or because they were not compatible with our reward function.

We split the test set by randomly choosing zones with more than one act and then randomly choosing an act from each selected zone. In this setup, the test set contains mostly objects and textures present in the training set, but with different layouts.

The test levels are listed in the following table:

3.4          Frame Skip

The step() method on raw gym-retro environments progresses the game by roughly  ¹th of a second. However, following common practice for ALE environments, we require the use of a frame skip [16] of 4. Thus, from here on out, we will use timesteps as the main unit of measuring in-game time. With a frame skip of 4, a timestep represents roughly  ¹th of a second. We believe that this is more than enough temporal resolution to play Sonic well.

Moreover, since deterministic environments are often susceptible to trivial scripted solutions [17], we require the use of a stochastic “sticky frame skip”. Sticky frame skip adds a small amount of randomness to the actions taken by the agent; it does not directly alter observations or rewards.

Like standard frame skip, sticky frame skip applies n actions over 4n frames. However, for each action, we delay it by one frame with probability 0.25, applying the previous action for that frame instead. The following diagram shows an example of an action sequence with sticky frame skip:

3.5          Episode Boundaries

Experience in the game is divided up into episodes, which roughly correspond to lives. At the end of each episode, the environment is reset to its original save state. Episodes can end on three conditions:

• The player completes a level successfully. In this benchmark, completing a level corre- sponds to passing a certain horizontal offset within the level.
• The player loses a life.
• 4500 timesteps have elapsed in the current episode. This amounts to roughly 5 minutes of in-game time.

The environment should only be reset if one of the aforementioned done conditions is met. Agents should not use special APIs to tell the environment to start a new episode early.

Note that our benchmark omits the boss fights that often take place at the end of a level. For levels with boss fights, our done condition is defined as a horizontal offset that the agent must reach before the boss fight. Although boss fights could be an interesting problem to solve, they are fairly different from the rest of the game. Thus, we chose not to include them so that we could focus more on exploration, navigation, and speed.

3.6          Observations

A gym-retro environment produces an observation at the beginning of every timestep. This observation is always a 24-bit RGB image, but the dimensions vary by game. For Sonic, the screen images are 320 pixels wide and 224 pixels tall.

3.7           Actions

At every timestep, an agent produces an action representing a combination of buttons on the game console. Actions are encoded as binary vectors, where 1 means “pressed” and 0 means “not pressed”. For Sega Genesis games, the action space contains the following buttons: B, A, MODE, START, UP, DOWN, LEFT, RIGHT, C, Y, X, Z.

A small subset of all possible button combinations makes sense in Sonic. In fact, there are only eight essential button combinations:

{{}, {LEFT}, {RIGHT}, {LEFT, DOWN},

{RIGHT, DOWN}, {DOWN}, {DOWN, B}, {B}}

The UP button is also useful on occasion, but for the most part it can be ignored.

3.8         Rewards

During an episode, agents are rewarded such that the cumulative reward at any point in time is proportional to the horizontal offset from the player’s initial position. Thus, going right always yields a positive reward, while going left always yields a negative reward. This reward function is consistent with our done condition, which is based on the horizontal offset in the level.

The reward consists of two components: a horizontal offset, and a completion bonus. The horizontal offset reward is normalized per level so that an agent’s total reward will be 9000 if it reaches the predefined horizontal offset that marks the end of the level. This way, it is easy to compare scores across levels of varying length. The completion bonus is 1000 for reaching the end of the level instantly, and drops linearly to zero at 4500 timesteps. This way, agents are encouraged to finish levels as fast as possible².

Since the reward function is dense, RL algorithms like PPO [18] and DQN [16] can easily make progress on new levels. However, the immediate rewards can be deceptive; it is often necessary to go backwards for prolonged amounts of time (Figure 2). In our RL baselines, we use reward preprocessing so that our agents are not punished for going backwards. Note, however, that the preprocessed reward still gives no information about when or how an agent should go backwards.

3.9          Evaluation

In general, all benchmarks must provide some kind of performance metric. For Sonic, this metric takes the form of a “mean score” as measured across all the levels in the test set. Here are the general steps for evaluating an algorithm on Sonic:

1. At training time, use the training set as much or as little as you like.

• At test time, play each test level for 1 million timesteps. Play each test level separately; do not allow information to flow between test levels. Multiple copies of each environment may be used (as is done in algorithms like A3C [19]).
• For each 1 million timestep evaluation, average the total reward per episode across all episodes. This gives a per-level mean score.
• Average the mean scores for all the test levels, giving an aggregate metric of performance.

Figure 2: A trace of a successful path through the first part of Labyrinth Zone, Act 2 in Sonic The Hedgehog^TM. In the initial green segment, the agent is moving rightwards, getting positive reward. In the red segment, the agent must move to the left, getting negative reward. During the orange segment, the agent is once again moving right, but its cumulative reward is still not as high as it was after the initial green segment. In the final green segment, the agent is finally improving its cumulative reward past the initial green segment. For an average player, it takes 20 to 30 seconds to get through the red and orange segments.

The most important aspect of this procedure is the timestep limit for each test level. In the infinite-timestep regime, there is no strong reason to believe that meta-learning or transfer learning is necessary. However, in the limited-timestep regime, transfer learning may be necessary to achieve good performance quickly.

We aim for this version of the Sonic benchmark to be easier than zero-shot learning but harder than ∞-shot learning. 1 million timesteps was chosen as the timestep limit because modern RL algorithms can make some progress in this amount of time.

4           Baselines

In this section, we present several baseline learning algorithms and discuss their performance on the benchmark. Our baselines include human players, several methods that do not make use of the training set, and a simple transfer learning approach consisting of joint training followed by fine tuning. Table 1 gives the aggregate scores for each of the baselines, and Figure 3 compares the baselines’ aggregate learning curves.

4.1           Humans

For the human baseline, we had four test subjects play each test level for one hour. Before seeing the test levels, each subject had two hours to practice on the training levels. Table 7 in Appendix C shows average human scores over the course of an hour.

Table 1: Aggregate test scores for each of the baseline algorithms.

4.2          Rainbow

Deep Q-learning (DQN) [16] is a popular class of algorithms for reinforcement learning in high-dimensional environments like video games. We use a specific variant of DQN, namely Rainbow [20], which performs particularly well on the ALE.

We retain the architecture and most of the hyper-parameters from [20], with a few small changes. First, we set V_max = 200 to account for Sonic’s reward scale. Second, we use a replay buffer size of 0.5M instead of 1M to lower the algorithm’s memory consumption. Third, we do not use hyper-parameter schedules; rather, we simply use the initial values of the schedules from [20].

Since DQN tends to work best with a small, discrete action space, we use an action space containing seven actions:

{{LEFT}, {RIGHT}, {LEFT, DOWN}, {RIGHT, DOWN}

{DOWN}, {DOWN, B}, {B}}

We use an environment wrapper that rewards the agent based on deltas in the maximum x-position. This way, the agent is rewarded for getting further than it has been before (in the current episode), but it is not punished for backtracking in the level. This reward preprocessing gives a sizable performance boost.

Table 2 in Appendix C shows Rainbow’s scores for each test level.

4.3          JERK: A Scripted Approach

In this section, we present a simple algorithm that achieves high rewards on the benchmark without using any deep learning. This algorithm completely ignores observations and instead looks solely at rewards. We call this algorithm Just Enough Retained Knowledge (JERK). We note that JERK is loosely related to The Brute [21], a simple algorithm that finds good trajectories in deterministic Atari environments without leveraging any deep learning.

Algorithm 1 in Appendix A describes JERK in detail.    The main idea is to explore using a simple algorithm, then to replay the best action sequences more and more frequently as training progresses. Since the environment is stochastic, it is never clear which action sequence is the best to replay. Thus, each action sequence has a running mean of its rewards. Table 3 in Appendix C shows JERK’s scores for each test level. We note that JERK actually performs better than regular PPO, which is likely due to JERK’s perfect memory and its tailored exploration strategy.

4.4          PPO

Proximal Policy Optimization (PPO) [18] is a policy gradient algorithm which performs well on the ALE. For this baseline, we run PPO individually on each of the test levels.

For PPO we use the same action and observation spaces as for Rainbow, as well as the same reward preprocessing. For our experiments, we scaled the rewards by a small constant factor in order to bring the advantages to a suitable range for neural networks. This is similar to how we set V_max for Rainbow. The CNN architecture is the same as the one used in [18] for Atari.

We use the following hyper-parameters for PPO:

4.5          Joint PPO

While Section 4.4 evaluates PPO with no meta-learning, this section explores the ability of PPO to transfer from the training levels to the test levels. To do this, we use a simple joint training algorithm³, wherein we train a policy on all the training levels and then use it as an initialization on the test levels.

During meta-training, we train a single policy to play every level in the training set. Specifically, we run 188 parallel workers, each of which is assigned a level from the training set. At every gradient step, all the workers average their gradients together, ensuring that the policy is trained evenly across the entire training set. This training process requires hundreds of millions of timesteps to converge (see Figure 4), since the policy is being forced to learn a lot more than a single level. Besides the different training setup, we use the same hyper-parameters as for regular PPO.

Once the joint policy has been trained on all the training levels, we fine-tune it on each test level under the standard evaluation rules. In essence, the training set provides an initialization that is plugged in when evaluating on the test set. Aside from the initialization, nothing is changed from the evaluation procedure used for Section 4.4.

Figure 4: Intermediate performance during the process of joint training a PPO model. The x-axis corresponds to timesteps into the joint training process. The zero-shot curves were densely sampled during training, while the fine-tuning curves were sampled periodically.

Table 5 in Appendix C shows Joint PPO’s scores for each test level. Table 9 in Appendix D shows Joint PPO’s final scores for each training level. The resulting test performance is superior to that of Rainbow, and is roughly 100% better than that of regular PPO. Thus, it is clear that some kind of useful information is being transferred from the training levels to the test levels.

4.6          Joint Rainbow

Since Rainbow outperforms PPO with no joint training, it is natural to ask if Joint Rainbow analogously outperforms Joint PPO. Surprisingly, our experiments indicate that this is not the case.

To train a single Rainbow model on the entire training set, we use a multi-machine training setup with 32 GPUs. Each GPU corresponds to a single worker, where each worker has its own replay buffer and eight environments. The environments are all “joint environments”, meaning that they sample a new training level at the beginning of every episode. Each worker runs the algorithm described in Algorithm 2 in Appendix A.

Besides the unusual batch size and distributed worker setup, all the hyper-parameters are kept the same as for the regular Rainbow experiment.

Table 6 in Appendix C shows the performance of fine-tuning on every test level. Table 8 in Appendix D shows the performance of the jointly trained model on every training level.

5            Discussion

We have presented a new reinforcement learning benchmark and used it to evaluate several baseline algorithms. Our results leave a lot of room for improvement, especially since our best transfer learning results are not much better than our best results learning from scratch. Also, our results are nowhere close to the maximum achievable score (which, by design, is somewhere between 9000 and 10000).

Now that the benchmark and baseline results have been laid out, there are many directions to take further research. Here are some questions that future research might seek to answer:

• How much can exploration objectives help training performance on the benchmark?
• Can transfer learning be improved using data augmentation?
• Is it possible to improve performance on the test set using a good feature representation learned on the training set (like in Higgins et al. [12])?
• Can different architectures (e.g. Transformers [23] and ResNets [24]) be used to improve training and/or test performance?

While we believe the Sonic benchmark is a step in the right direction, it may not be sufficient for exploring meta-learning, transfer learning, and generalization in RL. Here are some possible problems with this benchmark, which will only be proven or disproven once more work has been done:

• It may be possible to solve a Sonic level in many fewer than 1M timesteps without any transfer learning.
• Sonic-specific hacks may outperform general meta-learning approaches.
• Exploration strategies that work well in Sonic may not generalize beyond Sonic.

• Mastering a Sonic level involves some degree of memorization. Algorithms which are good at few-shot memorization may not be good at other tasks.

References

• M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, “The Arcade Learning Environment: An evaluation platform for general agents,” Journal of Artificial Intelligence Research, vol. 47, pp. 253–279, Jun. 2013.
• A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap, “Meta-learning with memory-augmented neural networks,” in International conference on machine learning, 2016, pp. 1842–1850.
• C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” 2017. eprint: arXiv:1703.03400.
• N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel, “Meta-learning with temporal convolutions,” 2017. eprint: arXiv:1707.03141.
• B. M. Lake, R. Salakhutdinov, J. Gross, and J. B. Tenenbaum, “One shot learning of simple visual concepts,” in Conference of the Cognitive Science Society (CogSci), 2011.
• O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra, “Matching networks for one shot learning,” in Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, Eds., Curran Associates, Inc., 2016, pp. 3630–3638.
• N. Bhonker, S. Rozenberg, and I. Hubara, “Playing SNES in the Retro Learning Environment,” 2016. eprint: arXiv:1611.02205.
• E. Parisotto, J. L. Ba, and R. Salakhutdinov, “Actor-Mimic: Deep multitask and transfer reinforcement learning,” 2015. eprint: arXiv:1511.06342.
• A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “Progressive neural networks,” 2016. eprint: arXiv:1606.04671.
• D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-driven exploration by self- supervised prediction,” 2017. eprint: arXiv:1705.05363.
• C. Fernando, D. Banarse, C. Blundell, Y. Zwols, D. Ha, A. A. Rusu, A. Pritzel, and D. Wierstra, “PathNet: Evolution channels gradient descent in super neural networks,” 2017. eprint: arXiv:1701.08734.
• I. Higgins, A. Pal, A. A. Rusu, L. Matthey, C. P. Burgess, A. Pritzel, M. Botvinick, C. Blundell, and A. Lerchner, “DARLA: Improving zero-shot transfer in reinforcement learning,” 2017. eprint: arXiv:1707.08475.
• C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Ku¨ttler, A. Lefrancq, S. Green, V. Vald´es, A. Sadik, J. Schrittwieser, K. Anderson, S. York, M. Cant, A. Cain, A. Bolton, S. Gaffney, H. King, D. Hassabis, S. Legg, and S. Petersen, “DeepMind Lab,” 2016. eprint: arXiv:1612.03801.
• E. Todorov, T. Erez, and Y. Tassa, “MuJoCo: A physics engine for model-based control.,” in IROS, IEEE, 2012, pp. 5026–5033.
• G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “OpenAI Gym,” 2016. eprint: arXiv:1606.01540.
• V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015.
• M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. J. Hausknecht, and M. Bowling, “Revisiting the Arcade Learning Environment: Evaluation protocols and open problems for general agents,” CoRR, vol. abs/1709.06009, 2017.
• J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy opti- mization algorithms,” 2017. eprint: arXiv:1707.06347.
• V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” 2016. eprint: arXiv: 1602.01783.
• M. Hessel, J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: Combining improvements in deep reinforcement learning,” 2017. eprint: arXiv:1710.02298.
• M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, “The arcade learning environment: An evaluation platform for general agents,” in Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015, pp. 4148–4152.
• A. Nichol and J. Schulman, “On first-order meta-learning algorithms,” 2018. eprint: arXiv: 1803.02999.
• A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L- . Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 6000–6010.
• K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Pro- ceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770– 778.

A             Detailed Algorithm Descriptions

B            Plots for Multiple Seeds

In this section, we present per-algorithm learning curves on the test set. For each algorithm, we run three different random seeds.

C             Scores on Test Set

Table 2: Detailed evaluation results for Rainbow.

Table 3: Detailed evaluation results for JERK.

Table 4: Detailed evaluation results for PPO.

Table 5: Detailed evaluation results for Joint PPO.

Table 6: Detailed evaluation results for Joint Rainbow.

Table 7: Detailed evaluation results for humans.

D                        Scores on Training Set

Table 8: Final performance for the joint Rainbow model over the last 10 episodes for each environment. Error margins are computed using the standard deviation over three runs.

Table 9: Final performance for the joint PPO model over the last 10 episodes for each environment. Error margins are computed using the standard deviation over two runs.