[Bug Report] TensorBoard logging issues with `rl_games` library

[SkoveBC]

Problem

I need to use TensorBoard to log rewards and other metrics while training a custom task using the rl_games library. The task involves a UR5e robot arm with a processing tool mounted on its flange, which is required to reach multiple waypoints. I have copied rl_games_ppo_cfg.yaml from the Isaac-Reach-UR10-v0 task. The training runs successfully, but unfortunately, there are issues with the logging frequency when using the PPO configuration from rl_games, but not for rsl_rl or skrl library.

Description

These are the logs from one training run with rl_games PPO configuration:

I have altered the following parameters and set them to save_best_after: 10 and save_frequency: 10:

IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/reach/config/ur_10/agents/rl_games_ppo_cfg.yaml

Lines 70 to 71 in 75b6715

	save_best_after: 200
	save_frequency: 100

For some reason, the change in metrics happens periodically at epoch 313. These are the logs from the terminal around the change in metrics logging. Apparently, the best reward remains constant until epoch 313 is reached:

This change is also shown in this video for the timestep at 626:

record_sudden_change.mp4

Attempts to fix this error

I have also checked whether this logging issue occurs for the rl_rsl library using rsl_rl_ppo_cfg.py with max_iterations=500, but there the logging does work as intended:

The same for the skrl library with skrl_ppo_cfg.yaml:

UR10 Reach task

This is the Tensorboard logging output from the Isaac-Reach-UR10-v0 task showing two metrics. If I zoom in one logged metric, I can see that the update happens after every 15 epochs:

System Info

• Commit: 75b6715
• Isaac Sim Version: 5.0.0
• OS: Ubuntu 22.04.5 LTS
• GPU: Quadro RTX 6000
• CUDA: 13.0
• GPU Driver: 580.65.0

Checklist

• I have checked that there is no similar issue in the repo
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

[RandomOakForest]

Thank you for posting this and adding all the plots. The team will review it.

[SkoveBC]

Hi @RandomOakForest, thank you for replying.

I have continued to try to fix the issue and the behaviour can now be reproduced:

I have found out that the TensorBoard logging frequency in rl_games can be controlled by some of these parameters from this code section:

IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/reach/reach_env_cfg.py

Lines 204 to 212 in 90dda53

	def __post_init__(self):
	"""Post initialization."""
	# general settings
	self.decimation = 2
	self.sim.render_interval = self.decimation
	self.episode_length_s = 12.0
	self.viewer.eye = (3.5, 3.5, 3.5)
	# simulation settings
	self.sim.dt = 1.0 / 60.0

This combination results in an logging period update of 15 steps, as shown in the last screenshot of my prior post, when starting the task with this command inside the docker container:

isaaclab -p scripts/reinforcement_learning/rl_games/train.py --task Isaac-Reach-UR10-v0 --headless --num_envs 1024 --max_iterations 500

I had set the parameters for my custom task in the following way:

• self.decimation = 4
• self.sim.dt = 1 / 500.0
• self.episode_length_s = 60

This combination of parameters results in an TensorBoard logging period of 312 epochs, for my custom task as well as Isaac-Reach-UR10-v0 if I use the above listed values.

By increasing sim.dt, and decreasing self.decimation and self.episode_length_s, the logging frequency can be increased. However, I want to keep these parameters as they are, since I need a high policy and physics frequency for better path-following accuracy, and a longer episode length because reaching multiple waypoints with a slow TCP speed requires more time. The training results are successful, but I need to monitor the metrics, especially when using hyperparameter tuning or population-based training with the rl_games library.

Are there any other parameters I can adjust to increase the TensorBoard logging frequency and view the logging results similar to the rsl_rl and skrl library training runs which are posted in the prior comment?

[ozhanozen]

@SkoveBC any chances #3531 could be related to this?

[SkoveBC]

@ozhanozen, thank you for referring to this issue and for helping out. I applied the changes from the fix, but unfortunately, they did not affect the logging frequency in TensorBoard when training with rl_games with the settings which I do use.

[SkoveBC]

I discovered that the logging frequency in TensorBoard when using rl_games depends on how many epochs are required to complete one episode, since metrics are only logged once per episode.

The number of epochs per episode is determined by the formula:

epochs_per_episode = episode_length_s / (sim.dt * decimation * horizon_length)

Using my current settings:

60.0 / ((1/500) * 4 * 24) = 312.5 epochs per episode

Using the original settings:

12.0 / ((1/60) * 2 * 24) = 15 epochs per episode

To increase logging frequency, I need to increase then the horizon_length parameter, which reduces the number of epochs per episode but comes with a trade-off of increased training time per epoch.

Edit
The update will happen only when a termination is triggered. For this case, the TerminationTerm mdp.time_out is used:

IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/reach/reach_env_cfg.py

Lines 163 to 167 in ad4ce34

	@configclass
	class TerminationsCfg:
	"""Termination terms for the MDP."""
	time_out = DoneTerm(func=mdp.time_out, time_out=True)