[RLlib] [MultiAgentEnv Refactor #2] Change space types for BaseEnvs and MultiAgentEnvs

[avnishn]

Environment's spaces should always be of the format:

{agent_id_1: space_1, agent_id_2 : space_2} in the multi-agent case, and gym.Space for non-multi-agent vectorized environments.

This differs from a change I made recently where my spaces mapped {env_id : env.space}. I misunderstood that when a base environment has multiple sub environments, this can only mean that the environment has been vectorized, and not that multiple different types of environments have been initialized in the BaseEnv.

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[gjoliver]

this is a pure software engineering thing, any reason we don't do this in init but here instead, in a utility function?

[avnishn]

Users will inherit from this class, however they may not necessarily call this function inside of their init functions. If I added this function to init, it may get called before the user has defined their observation/action spaces.

Although calling it here is messy, it allows for a nice external functionality, where if a user defines their spaces in the preferred format, they do not need to implement any of these sampling or checking methods on their own.

[sven1977]

Incomplete docstring.

[avnishn]

yerp sorry.

[avnishn]

Users will inherit from this class, however they may not necessarily call this function inside of their init functions. If I added this function to init, it may get called before the user has defined their observation/action spaces.

Although calling it here is messy, it allows for a nice external functionality, where if a user defines their spaces in the preferred format, they do not need to implement any of these sampling or checking methods on their own.

[avnishn]

yerp sorry.