[Question] Batch Size Selection for a Finite MDP

[DavidLudl]

❓ Question

Hello.

I would like to ask if I have a finite MDP, where each episode has a same fixed timestep $T$. Then during the training, do I have to choose batch size with $n\times T$? Or any other batch size is also OK?

Thank you for your time,

Best regards,

Checklist

• I have checked that there is no similar issue in the repo
• I have read the documentation
• If code there is, it is minimal and working
• If code there is, it is formatted using the markdown code blocks for both code and stack traces.

[araffin]

Then during the training, do I have to choose batch size with n × T ? Or any other batch size is also OK?

What algorithm are you using?
I guess you are talking about on policy algorithm (A2C/PPO) and the n steps parameter?

In that case, you can use any number of steps (this parameter can impact performance though).

[DavidLudl]

I am using PPO. They parameter I want to ask is the batch_size. Should I set the batch_size (default 64) in PPO algorithm to n*T ?

[araffin]

The mini batch size can be kept as is, it is only about the gradient step.

[DavidLudl]

Thank you, now I understand.