Evaluation worker feature #192
Conversation
|
@ViktorM FYI |
|
@alex-petrenko will it work with IG on one gpu? as I remember I cannot create second IG on same gpu anyway? |
|
@Denys88 it worked fine on my 1080Ti provided there's enough memory, although I only tried on my machine. |
|
@ViktorM this is the version we'll need to use for the demo |
|
@alex-petrenko please let me know if you are going to add a few more changes or I can just merge it and refactor later. |
|
@Denys88 I think it's solid and works reliably. We were able to use it with both IGE and Omniverse IsaacGym. If you don't want the file monitor thing (watchdog) to be in the main list of dependencies, you can remove it from setup py and add a warning that it should be installed under the evaluation section in the code. @Denys88 not sure about #195 - this is something @ArthurAllshire should know more about |
|
@Denys88 is it good to go? |
|
@ViktorM not yet. need to test envpool and ray vecenvs first. and update readme. |
| os.makedirs(self.eval_checkpoint_dir, exist_ok=True) | ||
|
|
||
| patterns = ["*.pth"] | ||
| from watchdog.observers import Observer |
There was a problem hiding this comment.
can we move this logic to the separate file?
This adds a new feature that allows real-time evaluation and visualization of agents during the training session.
The evaluation worker is supposed to run in a separate process from the training session and thus enables evaluation on a small number of agents (i.e. 1 or 64) which still leaves enough resources to train on thousands of agents.
This will be triggered in isaacgymenvs using a new flag. Alternatively player can be started with
evaluation=True dir_to_monitor=/some/experiment/dir/containing/checkpointsto monitor an existing training session.Player with evaluation=True will continuously monitor the experiment dir for new checkpoints, load them, and visualize a new policy.