[Question] How to customize the loss calculation for PPO

[olmoulin]

❓ Question

Hi,
For a research purpose, I need to modify the loss calculation for PPO in order to take into account another parameter ... I have modified the PPO.py file as follow : loss = (policy_loss + self.ent_coef * entropy_loss + self.vf_coef * value_loss + gen_loss), gen_loss being my new parameter ... but this has absolutely no impact on the training ... not sure, but am I making something wrong there ? thanks a lot in adance.

Checklist

• I have checked that there is no similar issue in the repo
• I have read the documentation
• If code there is, it is minimal and working
• If code there is, it is formatted using the markdown code blocks for both code and stack traces.

[araffin]

Probably a duplicate of #1881

[olmoulin]

This is what I have applied, and it does run properly my code, but it seems that no matter how much you modify the loss calculation it has no impact ... am I missing something ?
Can you please then remove the duplicate tag on it. thanks

[araffin]

This is what I have applied,

then this should be fine.

how much you modify the loss calculation it has no impact

No impact on the behavior?
You should be able to log the total loss and see some difference (while fixing the seed).
You can also check the parameters value.