transferrable models

[blurLake]

Hi, is there a way to save a model only with the internal layers of the NN? Or is it possible to modify a trained model (the zip file), to let it retrain with different action dimension?

I am thinking the following example, say we have env_1 with action dimension 10, and env_2, which is more complicated version of env_1, with action dimension 20. One model is trained with env_1. Can we modify the model (zip file) and use it as initial value to train on env_2 afterwards?

Thank you very much!

[Miffyli]

You can do this with original stable-baselines using get_parameters and load_parameters, and with bit of manual tinkering. You need manually create the mismatching parameter arrays for env_2 agent and update correct parameters with ones from env_1 agent. E.g. if only the last fully-connected layer changes, you need to manually crate final_params = np.zeros((N, 20)) and then assign the weights of original parameters to it final_params[:, :10] = original_params. If you want to modify save .zip files, the format is specified here.

Similar support is planned / partially working in SB3, but still needs to go through a check and review.

[bektaskemal]

So, is exporting saved model as Pytorch model not supported yet in SB3? Is there a way to get model parameters as in stable-baselines' get_parameters function?

[araffin]

So, is exporting saved model as Pytorch model not supported yet in SB3?

It is for policy. (but not properly documented yet). There are nn.Module anyway.

[bektaskemal]

Thanks for reply. I have a related question. I was considering exporting actor network only but I noticed that model.predict and model.actor.predict returns different values. Is it expected behavior?

model = SAC('MlpPolicy', 'CartPole-v1', verbose=1)
env = gym.make ('CartPole-v1')
obs = env.reset()
print(model.predict(obs))
print(model.actor.predict(obs))

Returns: (array([-0.20559013], dtype=float32), None)
(array([-0.9510132], dtype=float32), None)

Edit: My mistake. It works as expected when both is called as deterministic.

[araffin]

Similar support is planned / partially working in SB3, but still needs to go through a check and review.

done in #138 (will merge today)