[Proposal] Migrating the base IsaacEnv

[Mayankm96]

Proposal

Gymnasium is the maintained version of OpenAI Gym that is now handled by Farama Foundation. It is currently the definition used in libraries such as rllib and tianshou, while others like stable-baselines3 will also update soon.

More recently, there has been an interesting proposal on RL environment definition in the framework torchrl which would fit well for our targetted applications.

This thread aims to serve as a voting/staging on what would be the best environment definition to use.

Motivation

Migrating to Gym 0.28

The current Gym definition (from 0.21.0) is outdated and depends on older libraries such as importlib-meta==4.1 package. This creates conflicts with new updated packages and it would be best to switch to the new Gym definition.

Related issues: OIGE #28, Sb3 #1327

Once SB3 also upgrades to Gymnasium, then it would be best to update everything to use the latest definitions.

More information on Gymnasium:

• Migration guide: https://gymnasium.farama.org/content/migration-guide/
• Compatibility guide: https://gymnasium.farama.org/content/gym_compatibility/

Migrating to TorchRL definition

The environment definition EnvBase which relies heavily on using tensordict.

The advantages of their base class are:

• tensordict makes it easier to work with given the scaling possible with Isaac Sim and should be more efficient/flexible than dict of torch.Tensor
• It allows having complex datatypes (continuous/discrete, images/proprioception) for both observation and action spaces. This helps generalize to a wider audience (such as multi-agent learning)

Related Issues: torchrl #883

Effect on the remaining framework

Since there are wrappers for other RL frameworks (RL-Games and RSL-RL), this probably won't cause any breaking changes on that side. We will just need to adapt them based on the chosen IsaacEnv definition.

Checklist

• I have checked that there is no similar issue in the repo (required)

[romesco]

A third option to consider:

Simply define it as a Protocol. Something like EnvCompatible.

Over the years, I have run into countless issues I can trace back to the "kitchen sink" structure of gym. Don't get me wrong, I am extremely grateful for the step towards standardization that it catalyzed, but it's clear that when it was originally formulated, it was meant to be a standalone repository. I'm always looking to shy away from taking on unneeded dependencies since this can do a lot of harm to the lifecycle of the codebase.

I believe in the modern days, gym is used more like an interface to ensure that libraries for RL (both on the algorithmic side and on the dynamics side) agree on the correct "protocol" of methods that should be made available.

The best part is using a Protocol definition is also backward compatible as well as compatible with libraries that depend on gymnasium or whichever other flavor.

Something as simple as:

from typing import Protocol, runtime_checkable, Any, SupportsFloat, Optional

@dataclass
class StepReturn:
    observation: Any
    reward: SupportsFloat
    terminated: bool
    truncated: bool
    #info: Info | None

@dataclass
class ResetReturn:
    observation: Any
    #info: Info | None

@runtime_checkable
class EnvCompatible(Protocol):
    def step( self,  action: Any ) -> StepReturn:
        """Step."""

    def reset( self, seed: Optional[Any], options: Optional[Any]) -> ResetReturn:
        """Reset."""

Edit: You can also make something IsaacEnvCompatible or RLGamesEnvCompatible to specify additional behavior that must be satisfied. Multiple-implementation of protocols is seamless since there is no MRO to contend with (as is the case with abstract base classes).

Anyways, just something to think about.