Add API tests for all supported environments #18
Conversation
| print() | ||
| if terminated: | ||
| break | ||
| finally: |
There was a problem hiding this comment.
Should we not catch the exception and tell the user?
There was a problem hiding this comment.
Any exception would halt the program (and the stack trace would be printed to stderr). The try-finally block is there to ensure that the environment is closed.
| from gymnasium import spaces | ||
| from gymnasium.utils.env_checker import check_env | ||
| from gymnasium.wrappers.flatten_observation import FlattenObservation | ||
|
|
||
| import miniwob # noqa: F401 | ||
| from miniwob.tests.utils import ALL_TESTABLE_MINIWOB_ENVS | ||
|
|
||
|
|
||
| class TestGymAPI: |
There was a problem hiding this comment.
This looks completely fine, Im just more use to functional style testing rather than class based testing.
Is @pytest.fixture(params=...) equal to @pytest.mark.parameterize("env_id", ALL_...)?
There was a problem hiding this comment.
Yes, it should do the same thing.
miniwob/tests/utils.py
Outdated
| @@ -0,0 +1,111 @@ | |||
| """Test utilities.""" | |||
|
|
|||
| ALL_TESTABLE_MINIWOB_ENVS = [ | |||
There was a problem hiding this comment.
Damn that is alot of environments, is there anyway to make this programmatic? i.e., loop over the environment registry (gymnasium.registry) for all env id with miniwob in the env_id (more proper would be to check namespace)
There was a problem hiding this comment.
I have changed to reading from the registry.
|
|
||
| # Throw errors on warnings, except for some specific warnings. | ||
| filterwarnings = [ | ||
| "error", |
There was a problem hiding this comment.
Why are we filtering errors?
There was a problem hiding this comment.
The "error" in filterwarnings means to turn warnings into errors (source). I have edited the comment to clarify this.
|
Sorry for not responding earlier. I have addressed the comments. |
|
Thanks for the PR, looks good |
Description
For all registered environments (excluding FlightWoB), added tests for the following:
gymnasium.utils.env_checker.check_env, which checks the observation space and seed determinism (exceptclick-pie, where the UI position can jitter by a few pixels).Also added a utility for inspecting the observation space. Tasks were modified to pass the tests.
Addressed part of issue #15.
Type of change
Please delete options that are not relevant.
Checklist:
pre-commitchecks withpre-commit run --all-files(seeCONTRIBUTING.mdinstructions to set it up)