reccurent policy

[yiwc]

is there a easy way to implement a recurrent policy now in the bate BS3?

I notice these is not such policy like lstm. But, we are in urgently need that.

Thanks!

[Miffyli]

Closing as duplicate of #1 and #160.

LSTM/Recurrent support is planned for v1.1. Adding support for them is not as trivial as in supervised learning so it will require modifications all around the algorithms.

[sycz00]

@Miffyli
Hey I hope its okay to post that into this thread and not one of the others.
I know that recurrent policies are not available in SB3 atm. I am working on a project for which I need some sort of recurrent structure, since its a POMDP env. My question is: do you think it could work using a TCN in the end of my custom feature extractor to learn long term dependencies ? because as far as I've understood, a TCN does not need BPTT.
Greetings:)

[Miffyli]

@sycz00 TCN (temporal convolution network, I assume?) could work for that case. There is now also an experimental LSTM support for PPO here. You need to experiment to find out what works for your case :)

[sycz00]

Yes thats what I mean. Wow thats amazing, thank you. Iam gonna look into this
It seems to be not working so far ? thu-ml/tianshou#486

[sycz00]

pip install git+https://github.com/Stable-Baselines-Team/stable-baselines3-contrib
does not contain the ppo_lstm branch ?

[Miffyli]

@sycz00 You need to checkout feat/ppo-lstm branch, not ppo_lstm.

[sycz00]

@Miffyli Thanks:)

It looks like MultiInputPolicy is not available yet ? cause I got dict observation space. For now I am trying it with just one image.

[araffin]

It looks like MultiInputPolicy is not available yet ? cause I got dict observation space. For now I am trying it with just one image.

it is not (yet) but you can add the support, it is mostly a question of copy-pasting what is in the replay buffer code...

[sycz00]

Okay good:) does the LSTM only exists for the actor ? if yes, why ? :)