[Feature Request] TRPO needed

[GoingMyWay]

🚀 Feature

I found there is no TRPO in this repo, however, in https://github.com/hill-a/stable-baselines, it has. Will you add it?

Motivation

• TRPO is a very important baseline.

Pitch

Add TRPO from https://github.com/hill-a/stable-baselines

### Checklist

• I have checked that there is no similar issue in the repo (required)

[araffin]

Hello,

I found there is no TRPO in this repo, however, in https://github.com/hill-a/stable-baselines, it has. Will you add it?

TRPO is planned (see roadmap #1), however there is no clear deadline, especially as PPO is usually a competitive alternative to ti. We would welcome a PR that adds it ;) (and that must replicate previous results, as we do for the contrib repo: https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/blob/master/CONTRIBUTING.md)

[GoingMyWay]

Hello,

I found there is no TRPO in this repo, however, in https://github.com/hill-a/stable-baselines, it has. Will you add it?

TRPO is planned (see roadmap #1), however there is no clear deadline, especially as PPO is usually a competitive alternative to ti. We would welcome a PR that adds it ;) (and that must replicate previous results, as we do for the contrib repo: https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/blob/master/CONTRIBUTING.md)

Thanks, I see.

[jdchang1]

Hi, I am actually in the process of implementing TRPO and was wondering how/where to discuss certain design choices? One question I am wondering about is the ActorCriticPolicy design. Currently the Adam optimizer is set to optimize both actor/critic params but in TRPO we would want self.optimizer to just update the critic. This can be done with a custom policy but seems wasteful given how much code repetition there would be. Thanks!

[Miffyli]

@jdchang1 Hey! Adding new algos is always welcome! I recommend looking at contrib repo for a place to add the algorithm. There you are free to do more ad-hoc solutions without fear of breaking or bloating up the core repository (this one), so you can create a new version of the ActorCriticPolicy that does not update both parts. Feel free to open a new issue discussing adding TRPO there :)

[araffin]

closing in fabor of Stable-Baselines-Team/stable-baselines3-contrib#38