Chopper Command does not always terminate when out of lives

[brett-daley]

Chopper Command sometimes does not terminate when the agent runs out of lives, instead continuing with an empty battlefield until exactly 21,600 timesteps have elapsed (i.e. 108k frames with frameskip=5, or 30 minutes of gameplay). See below for code.

This might be related to #262, where BattleZone and Yars' Revenge also do not terminate sometimes. @mgbellemare suggests a 30-minute time limit in these cases, which is already implemented by ALE. Is this also standard practice for Chopper Command? Or is there a severely different bug here?

Minimum working example. (Tested on Linux, Python 3.8.10, ale-py 0.7.2)
The code executes a random policy for 3 episodes and records videos in the video/ directory. The 2nd episode does not terminate when all lives are lost and gets stuck until the time limit is reached, which can be seen in the video.

import gym
from gym.wrappers import Monitor

env = gym.make('ALE/ChopperCommand-v5')
env = Monitor(env, 'video', force=True, video_callable=lambda e: True)
env.seed(0)
env.action_space.seed(0)

for i in range(3):
    state = env.reset()
    done = False
    t = 0
    while not done:
        t += 1
        state, _, done, _ = env.step(env.action_space.sample())
        if done:
            print(f"Episode {i} terminated after {t} steps")

Output:

A.L.E: Arcade Learning Environment (version +a7a216c)
[Powered by Stella]
Episode 0 terminated after 422 steps
Episode 1 terminated after 21600 steps
Episode 2 terminated after 597 steps

[JesseFarebro]

Sadly it's not related. Thanks to your example I can confirm this is because of the following change. I'm not sure why this was added, I'll try to figure this out. From what I can tell RAM address 0xC2 in Chopper Command seems to indicate whether the left side of the map was loaded or not. If the chopper goes far enough to the left that RAM address is set to 0 and the call to isTerminal() will return false.

[brett-daley]

Thanks for looking into this. I don't have any experience with the underlying ALE implementation, but it seems like it would be sufficient to just check for lives == 0 and then raise done (which was the previous behavior before that change you linked). Let me know if you figure it out and/or if there's any way I can help.

[JesseFarebro]

I agree it should be sufficient. This check was added when we were adding game modes/difficulties into the ALE so I just want to be confident there's no collateral if we remove this check. I'll keep you updated once I have more info.

[JesseFarebro]

@brett-daley this issue will be fixed in the next release of the ALE (< 1 week hopefully).

[JesseFarebro]

This is fixed as of 0.7.4. Thanks, @brett-daley! 🥳