Add ddpg_continuous_action.py docs
#137
Conversation
|
This pull request is being automatically deployed with Vercel (learn more). 🔍 Inspect: https://vercel.com/vwxyzjn/cleanrl/CCjtcjP4CWC3iPu2nuVKoyG9EuMD |
|
|
|
Hey @yooceii @dosssman, the PR is ready for review. Could you take a look at https://cleanrl-git-ddpg-docs-vwxyzjn.vercel.app/rl-algorithms/ddpg/? Thank you |
docs/rl-algorithms/ddpg.md
Outdated
| * `losses/qf1_loss`: the MSE between the Q values at timestep $t$ and the target Q values at timestep $t+1$, which minimizes temporal difference. | ||
| * `losses/actor_loss`: implemented as `-qf1(data.observations, actor(data.observations)).mean()`; it is the *negative* average Q values calculated based on the 1) observations and the 2) actions computed by the actor based on these observations. By minimizing `actor_loss`, the optimizer updates the actors parameter using the following gradient (Lillicrap et al., 2016, Equation 6)[^1]: | ||
|
|
||
| $$ \mathbb{E}_{s_{t} \sim \rho^{\beta}}\left[\left.\left.\nabla_{a} Q\left(s, a \mid \theta^{Q}\right)\right|_{s=s_{t}, a=\mu\left(s_{t}\right)} \nabla_{\theta_{\mu}} \mu\left(s \mid \theta^{\mu}\right)\right|_{s=s_{t}}\right]$$ |
There was a problem hiding this comment.
What's the
There was a problem hiding this comment.
I didn't add too much explanation for this because I referenced equation 6 from the original paper. If you think this could confuse the readers, I can remove it and just say equation 6 without copying it in our docs.
There was a problem hiding this comment.
But the equation 6 is DPG's update method.
I think DDPG uses $$ \frac{1}{N}\sum_i\left.\left.\nabla_{a} Q\left(s, a \mid \theta^{Q}\right)\right|{s=s_{i}, a=\mu\left(s_{i}\right)} \nabla_{\theta_{\mu}} \mu\left(s \mid \theta^{\mu}\right)\right|{s_{i}} $$
as shown in
There was a problem hiding this comment.
You're absolutely right. Thanks for the couch and I have just fixed it.
|
Hey, @yooceii thanks for reviewing the PR :) Let me know if there are other issues. |
|
Given that all the comments @yooceii are addressed, I am merging the PR as is but happy to open follow-up PRs if anything else is needed. |
This RP adds docs for
ddpg_continuous_action.py.Checklist for
ddpg_continuous_action.py:pre-commit run --all-filespasses (required).--capture-videoflag toggled on (required).mkdocs serve.width=500andheight=300).