Update docs (custom policy, type hints)

[araffin]

Description

• Add custom feature extractor example
• Add full custom policy example for on-policy algorithms
• Re-enable autodoc type hint extension (looks much nicer with the latest version) and fix some issues due to that

You can see the difference with those links:
current doc: https://stable-baselines3.readthedocs.io/en/master/modules/td3.html#parameters
this PR: https://stable-baselines3.readthedocs.io/en/doc-custom-policy/modules/td3.html#parameters

Motivation and Context

• I have raised an issue to propose this change (required for new features and bug fixes)

Addresses part of #10 and #166
closes #144
closes #168

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)

Checklist:

• I've read the CONTRIBUTION guide (required)
• I have updated the changelog accordingly (required).
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.
• I have reformatted the code using make format (required)
• I have checked the codestyle using make check-codestyle and make lint (required)
• I have ensured make pytest and make type both pass. (required)

Note: we are using a maximum length of 127 characters per line

[Miffyli]

Looking good and indeed much cleaner docs! Some comments though:

• Does the example run as expected (wondering because of [bug] Unexpected keyword argument "use_sde" in custom policy #168)?
• With hints automatically included from type hinting, we should clean up docstrings from type hints. I think it should be included with this update or Missing Documentation #166.
• Type hints in docs would be cleaner if parentheses were also formatted with code-like style, but I figure this goes to sphinx side where modifications can be difficult.

[araffin]

Does the example run as expected (wondering because of #168)?

actually, the current example runs but is simpler than #168
that's why i do not close #166
i will probably add a full advance usage today then.

for your last two points, i made the same remarks to myself but i would address them in a separate PR to keep that one small ;)

[Miffyli]

In that case LGTM! I added the two remarks to respective issues as TODOs.

[pengzhi1998]

Sorry to bother you.

I'm constructing a different network architecture with self-attention which is not a sequential network though. While the actor and critic networks are not sharing the parameters.

I'm wondering whether I could use advanced custom policy to construct my attention block for the PPO network with the function _build_mlp_extractor as shown in the example? Or, for this purpose, I should use a custom feature extractor?

I really appreciate your great help!

[Miffyli]

@pengzhi1998 Hey. We unfortunately do not have time to offer custom tech support for custom scenarios. For your scenario, you should use the fully custom policy (which you found already) to define things from ground-up. That is the most customizable way: modifying feature extractor can be limiting.

PS: next time please open a new issue for questions instead of continuing a closed PR :)

[pengzhi1998]

Thank you!! And so sorry for keeping pestering you and for this inconvenience. I'll start a new issue.

Have a great day!