[BUG] async_reset crashes when called multiple times

[jbuckman]

Describe the bug

When we call async_reset multiple times, we get a crash.

The use case here is that I want to run many parallel episodes with the async interface, and then once all the episodes are complete, start some new episodes again.

To Reproduce

``

from ipdb import set_trace
import envpool
import numpy as np

if __name__ == '__main__':
    EPISODES_PER_GRADIENT = 1
    BATCH_SIZE = 1

    ## Initialize environments
    env = envpool.make("CartPole-v1", env_type="gym", num_envs=EPISODES_PER_GRADIENT, batch_size=BATCH_SIZE)

    for round in range(10000000):
        print(f"==> Round {round: 4}.", end="\r")
        ## Reset all environments
        env.async_reset()

        ## Play exactly 100 episodes
        for i in range(100):
            state, rew, done, info = env.recv()
            env_id = info["env_id"]
            env.send(np.random.randint(env.action_space.n, size=BATCH_SIZE), env_id)

$ python3.9 test2.py 
terminate called after throwing an instance of 'std::out_of_range'
  what():  StateBuffer out of storage
Aborted

Expected behavior

Each environment should reset, no crashing.

System info

Linux, python3.9, using anaconda, installed via pip install.

import envpool, numpy, sys
print(envpool.__version__, numpy.__version__, sys.version, sys.platform)

0.5.3.post1 1.22.3 3.9.12 (main, Apr  5 2022, 06:56:58) 
[GCC 7.5.0] linux

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have read the documentation (required)
• I have provided a minimal working example to reproduce the bug (required)

[Trinkle23897]

This is expected behavior. async_reset can only be called once at the beginning. All the rest of work is handled by send and recv with auto-reset mechanism.

You can change the code to what I mentioned in #112. That should work.

[Trinkle23897]

import envpool
import numpy as np

if __name__ == '__main__':
    N = 100

    # Initialize environments
    env = envpool.make("CartPole-v1", env_type="gym", num_envs=N)

    for round in range(1000):
        print(f"==> Round {round: 4}.", end="\r")
        env_id = np.arange(N)
        # Reset all environments
        obs = env.reset(env_id)
        done = np.array([False] * N)

        # Play exactly 100 episodes
        while not np.all(done):
            action = np.random.randint(env.action_space.n, size=len(env_id))
            state, rew, done, info = env.step(action, env_id)
            env_id = info["env_id"][~done]

[mavenlin]

One thing we could improve here is to throw a verbose exception instead of the internal error.

As for your use case, I think it is achievable by creating a new envpool instance.