env.step API updated in gym 0.25.0

[bitsandscraps]

• I have marked all applicable categories:
  • exception-raising bug
  • RL algorithm bug
  • documentation request (i.e. "X is missing from the documentation.")
  • new feature request
• I have visited the source website
• I have searched through the issue tracker for duplicates
• [] I have mentioned version numbers, operating system and environment, where applicable:

  import tianshou, gym, torch, numpy, sys
  print(tianshou.__version__, gym.__version__, torch.__version__, numpy.__version__, sys.version, sys.platform)

Gym 0.25.0 introduced a new step API that returns a tuple of length 5(observation, reward, terminated, truncated, info) instead of 4(observation, reward, done, info).
I think tianshou's setup.py should restrict the gym version to <= 0.24.1 until it supports the new API.

[Trinkle23897]

As long as openai/gym#3019 is not merged, it's okay not to restrict the gym version.

[bitsandscraps]

My bad. The new version of gym prints warnings when calling gym.make without setting the new_step_api to True, though.

[pseudo-rnd-thoughts]

In gym version 0.25, we have implemented the new termination truncation step api which will raise a warning if users use the old done step api (which is turned on by default). However, v0.25 should be completely backwards compatible.

We are planning on removing default support for the old done step API in v0.26 (the planned release is the next week). To use the old step API, we provide a conversion function and wrapper for users if they need it.

[YouJiacheng]

@Trinkle23897 Now gym#3019 merged. Maybe restrict <= 0.25.2

[pseudo-rnd-thoughts]

I believe that @Markus28 is planning on making a v26 upgrade for Tianshou