# Goal Misgeneralization in Procgen

This is a fork of the [procgen benchmark](https://github.com/openai/procgen) that implements modifications for the paper Goal Misgeneralization in Deep Reinforcement Learning.

## Changes in this Fork

### New Environment: `maze_afh`

A new maze variant with `random_percent` support for OOD detection experiments (similar to coinrun's `random_percent` flag).

- `random_percent=0`: Cheese placed in top-right corner (like `maze_aisc`)
- `random_percent=100`: Cheese placed randomly (like `maze`)
- Intermediate values: Probabilistic mix
- Exposes `info["randomize_goal"]` to indicate placement type (useful for OOD ground truth)

**Modified files:**
- `procgen/src/games/maze_afh.cpp`: New game implementation
- `procgen/env.py`: Added to `ENV_NAMES` and `EXPLORATION_LEVEL_SEEDS`
- `procgen/CMakeLists.txt`: Added to build

## Descriptions of the modified environments

* `coinrun_aisc`: Like `coinrun`, but the coin is placed randomly on ground level instead of at the far right end.
* `coinrun`: Added a flag `--random_percent`, which places the coin randomly in a given percentage of environments. Default 0.
* `heist_aisc_many_chests`: A heavily modified `heist`. Doors are now 'chests' (they do not prevent the agent from passing). Every key can open every chest. The agent is rewarded for opening chests. This version generates twice as many chests as keys. 
* `heist_aisc_many_keys`: Same as `heist_aisc_many_chests`, but instead has twice as many keys as chests.
* `maze_aisc`: Like maze, but the cheese is always to be found in the top right corner.
* `maze_afh`: Like maze, but with `--random_percent` support (similar to coinrun). When `random_percent=0`, cheese is placed in the top right corner (like `maze_aisc`). When `random_percent=100`, cheese is placed randomly (like `maze`). Intermediate values give a probabilistic mix. Exposes `info["randomize_goal"]` to indicate placement type for each level.
* `maze_yellowgem`: like maze, but the goal is a yellow gem.
* `maze_redgem_yellowstar`: like maze, but two objects are placed in the maze: a red gem, and a yellow star. The objective is the red gem.
* `maze_yellowstar_redgem`: Identical to `maze_yellowstar_redgem`, but the objective is instead the yellow star.


For both 'Keys and Chests' environments we added two options:
* `--key_penalty`: integer. Every time the agent picks up a key it loses `options.key_penalty / 10` reward.
* `--step_penalty`: integer time penalty. Each step, `options.step_penalty / 1000` is subtracted from the reward.

For more information on the standard environments see the original repository.

## Seed Lists

This fork adds support for passing a list of level seeds to control which levels are played. All parallel environments share a single seed pool.

### Usage

```python
from procgen import ProcgenGym3Env

# Sequential mode (default): seeds used in order, stops when exhausted
env = ProcgenGym3Env(
    num=4,
    env_name="coinrun",
    level_seeds=[100, 200, 300, 400, 500],
    level_seeds_mode="sequential"
)

# Container mode: random draw from pool, refills when empty, cycles forever
env = ProcgenGym3Env(
    num=4,
    env_name="coinrun",
    level_seeds=[100, 200, 300, 400, 500],
    level_seeds_mode="container"
)

# Random mode: sample with replacement, cycles forever
env = ProcgenGym3Env(
    num=4,
    env_name="coinrun",
    level_seeds=[100, 200, 300, 400, 500],
    level_seeds_mode="random"
)
```

### Modes

#### Sequential Mode (`level_seeds_mode="sequential"`)

Seeds are consumed in exact order across all parallel environments.

- When an environment resets, it gets the next available seed from the shared list
- When all seeds are exhausted:
  - Environments finish their current level
  - After completion, they "freeze" and return their final observation repeatedly
  - `info["seeds_exhausted"]` is set to `True` for frozen environments
  - Frozen environments return `done=False` and `reward=0`

Useful for: evaluation on a fixed set of test levels where each level should be played exactly once.

#### Container Mode (`level_seeds_mode="container"`)

Seeds are randomly drawn from a pool without replacement. When the pool is empty, it refills with all original seeds.

- When an environment resets, it randomly picks a seed from remaining seeds in the container
- Each seed is used exactly once per cycle
- When all seeds have been used, the container refills and the cycle repeats
- Never exhausts - cycles forever

Useful for: training where you want to sample from a specific set of levels but in random order, ensuring each level is seen once per epoch.

#### Random Mode (`level_seeds_mode="random"`)

Seeds are randomly sampled with replacement from the full list.

- When an environment resets, it randomly picks a seed from the full list of seeds
- Seeds can be repeated before all seeds have been seen
- Never exhausts - cycles forever

Useful for: training where you want to sample from a specific set of levels with replacement, without the constraint that each level must be seen exactly once per epoch.

### Example (Sequential Mode)

```python
from procgen import ProcgenGym3Env
import numpy as np

env = ProcgenGym3Env(num=3, env_name="coinrun", level_seeds=[1, 2, 3, 4])

# Seed distribution across resets:
# Reset 1: env[0]=seed 1, env[1]=seed 2, env[2]=seed 3
# Reset 2: env[0]=seed 4, env[1]=frozen, env[2]=frozen

while True:
    action = np.array([env.ac_space.sample() for _ in range(3)])
    env.act(action)
    rew, obs, first = env.observe()
    info = env.get_info()
    
    # Check which environments have exhausted their seeds
    if all(info["seeds_exhausted"]):
        print("All environments exhausted")
        break
```

### Notes

- When `level_seeds` is provided, it takes precedence over `start_level`/`num_levels`
- Seed acquisition is thread-safe across parallel environments
- In sequential mode, all environments can continue stepping after exhaustion; the user decides when to stop

## Installation

Below we reproduce the instructions to install from source, copied from the [original repo](https://github.com/openai/procgen).

---

First make sure you have a supported version of python:

```
# run these commands to check for the correct python version
python -c "import sys; assert (3,6,0) <= sys.version_info <= (3,9,0), 'python is incorrect version'; print('ok')"
python -c "import platform; assert platform.architecture()[0] == '64bit', 'python is not 64-bit'; print('ok')"
```

If you want to change the environments or create new ones, you should build from source.  You can get miniconda from https://docs.conda.io/en/latest/miniconda.html if you don't have it, or install the dependencies from [`environment.yml`](environment.yml) manually.  On Windows you will also need "Visual Studio 15 2017" installed.

```
git clone git@github.com:openai/procgen.git
cd procgen
conda env update --name procgen --file environment.yml
conda activate procgen
pip install -e .
# this should say "building procgen...done"
python -c "from procgen import ProcgenGym3Env; ProcgenGym3Env(num=1, env_name='coinrun')"
# this should create a window where you can play the coinrun environment
python -m procgen.interactive
```

The environment code is in C++ and is compiled into a shared library exposing the [`gym3.libenv`](https://github.com/openai/gym3/blob/master/gym3/libenv.h) C interface that is then loaded by python.  The C++ code uses [Qt](https://www.qt.io/) for drawing.
