Refactor documentation

[vwxyzjn]

What is the problem

The current documentation requires more work. First, some of the implemented algorithms such as Apex-DQN, TD3, and SAC are not documented at https://docs.cleanrl.dev. Second, even the documented algorithm such as PPO does not have complete documentation: for example, the ppo_atari_envpool.py is not really documented. Third, there doesn't seem to be a single-source place to put documentation.

Going forward, I'd like to impose a specific documentation style and improve the overall workflow, which will also help #117.

Proposed solution

I was thinking maybe we can put a documentation link at the beginning of each file. For example, we could add these two lines at ppo.py.

https://github.com/vwxyzjn/cleanrl/blob/c8faef93fc8dbc9528183840ab75b8962df7b9c4/cleanrl/ppo.py#L1-L7

And this link of https://cleanrl-553u0zazz-vwxyzjn.vercel.app/rl-algorithms/ppo/#ppopy will point to the corresponding documentation that has

• Brief overview of the algorithm
• Original paper and relevant resources
• Short description of what ppo.py specifically does
• Explanation of important implementation details
• Experiment results (and how they compare to the original paper or/and other reference implementations)
• Learning curves
• Tracked experiments

Which roughly looks like below (haven't added the tracked experiments)

ppodemo.mp4

List of files needed to add documentation

• Add ppo.py documentation #120
• Deprecate apex_dqn_atari.py #136
• Add ddpg_continuous_action.py docs #137
• Add dqn_atari.py documentation #124
• SAC Documentation - Benchmarks - Minor code tweaks #146
• Add documentation for td3_continuous_action.py #141
• Add docs for dqn.py #157
• Add docs for c51.py and c51_atari.py #159
• Add PPO documentation #163
• Support Pettingzoo Multi-agent Atari envs with PPO  #188