Support ToyText env

[ace-han]

Problem Description

Launching via python cleanrl/ppo.py --seed 1 --env-id FrozenLake-v1 will failed.

Since ToyText envs are extremely simple and easy to learn, it's better to support them.

Checklist

• I have installed dependencies via poetry install (see CleanRL's installation guideline.
• I have checked that there is no similar issue in the repo.
• I have checked the documentation site and found not relevant information in GitHub issues.

Current Behavior

python cleanrl/ppo.py --seed 1 --env-id FrozenLake-v1 will yield below

/Users/ace/Library/Caches/pypoetry/virtualenvs/cleanrl-TlZg4FDS-py3.10/lib/python3.10/site-packages/tyro/_fields.py:307: UserWarning: The field wandb_entity is annotated with type <class 'str'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/Users/ace/Library/Caches/pypoetry/virtualenvs/cleanrl-TlZg4FDS-py3.10/lib/python3.10/site-packages/tyro/_fields.py:307: UserWarning: The field target_kl is annotated with type <class 'float'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
Traceback (most recent call last):
  File "/Users/ace/workspace/ace/cleanrl/cleanrl/ppo.py", line 167, in <module>
    agent = Agent(envs).to(device)
  File "/Users/ace/workspace/ace/cleanrl/cleanrl/ppo.py", line 104, in __init__
    layer_init(nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64)),
  File "/Users/ace/Library/Caches/pypoetry/virtualenvs/cleanrl-TlZg4FDS-py3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 99, in __init__
    self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs))
TypeError: empty(): argument 'size' failed to unpack the object at pos 2 with error "type must be tuple of ints,but got numpy.float64"
(cleanrl-TlZg4FDS-py3.10

Expected Behavior

The program proceeds without exceptions

Possible Solution

It looks like nn.Linear(np.array(envs.single_observation_space.shape).prod(), 64) not working well with Discret space