-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[MRG] StableBaselines Agent #148
Conversation
It seems tests are failing because |
Yes I think it is ok to add it as optional requirement because I think the people that install torch will also want to install stablebaseline. |
@omardrwch mentioned that maybe we can create a dummy SB3 algorithm class and avoid the import issues. I will look into this. Edit: Actually, it was a recommendation to use only one SB3 algorithm in the tests - done! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, this looks nice !
I have just a small comment on the test and some comment on the doc.
Concerning the doc, could you move the link to the stablebaselines tuto (and also the link to gym tuto) from the index page to the user guide ? I think it would make more sense there. Also could you add the wrapper in the API (file docs/api.rst) ?
rlberry/utils/tests/test_sb_agent.py
Outdated
Does not check that the fit is additive, because that doesn't seem to be | ||
always the case with StableBaselines3. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this seems to be due to the callback that can stop the fit early (e.g. https://stable-baselines.readthedocs.io/en/master/guide/callbacks.html _on_step
method). Would it make sense to redefine the default stablebaselines callback in order to avoid stopping early ? And is this early stopping a problem for Hyperparameter optimization ? (also @omardrwch on this)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, I just added the agent to the API and I've rewritten the documentation page for it so it looks more like a tutorial. Soon, I will move the links as you asked.
About the fit additive thing, it is kind of a lie: I'm pretty sure that SB3 should support additive fitting, it just doesn't work when I test it. I actually spent some time doing the following experiment:
- Create two identical agents (same seed too);
- Train one for X steps;
- Compare their policy logits on a fixed trajectory (policy logits are not sampling anything so we eliminate a source of randomness here).
And what I got was a bit difficult to explain, but I will summarize here:
- For our agents, the agents have the same logits when X is small (<10) but for bigger values, it is clear that they are different.
- For SB3 agents... There doesn't seem to be a lot of consistency: sometimes they have identical results even though X = 100 and sometimes they have different results when they should be equal. The additive fit constraint is only respected sometimes. I also found out that adding more training steps can change results, for example, agents trained for 100 and 200 steps give different results but agents trained for 500 and 600 steps give the same results...
In the end, I shelved the whole thing and ignored the test! I think that there are some interesting questions in checking if RL agents are almost equal, namely:
- How to set an appropriate budget different such that the algorithms can be differentiated? Budget usually refers to the amount of collected experience, but the agent may spend some time collecting experience before updating its weights;
- How does the environment affect the testing? I had some problems with small environments such as
Chain-v0
where the agent learns to always go right very quickly. As a result, differentiating between two trained agents is difficult with policy samples since the policies are so similar. Additionally, how should we design an environment for such tests? - Should we compare the policies on which states? Currently, we use only initial states, but I also had problems when using environments that had simple optimal policies on the initial states. I ended up using states obtained by running one of the policies one time.
This investigation was part of my looking into the deep RL agents, and I noticed some things that we could improve in the seeding (namely seeding the external seeds on the __init__
method of torch
agents to make sure that the random weight initialization is the same). However, even though I spent quite some time aggressively seeding everything, I still cannot get identical training runs (same episode lengths, returns) which may indicate that there are still things to correct.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some comments on this
- When you are doing your tests, do you use one thread, not parallelized ? Parallelization may have some unexpected consequences and this is why I used non-parallelized agents in the tests.
- For the stablebaselines algo I think this is linked to the early stopping they perfom, which seems to me a bit ad hoc. I don't know how well we know what is the tradeoff behind this kind of early stopping. It is not clear for me, which is why I think we may want to avoid this early stopping if we are searching for stable/reliable/reproducible results.
- How does the environment affect the testing?
I think you are right, we one way would be to test on an environment where there are a lot of possible action so that at first the agent have to basically do pure exploration. Maybe we could design an ad-hoc environment withfiniteMDP
class to do that. Another way would be to test the agent on a bandit environment (bandit environments may have a lot of possible actions) - Generally you are right that the budget is something not very precise. It is by design in rlberry to have something flexible but it makes testing a bit awkward. One way to solve this would be to specify a set of fit arguments (budget, n_simulations...) for each algorithm in the test, it would be a bit tedious but still doable.
About fixing external seed in __init__
I think there may be some caveat linked to the parallelization and the spawning of new seeds in AgentManager
and multimanager
. It would be great if all the seeding in rlberry could be handled by rlberry and not manually by using set_external_seed
but I am not sure how easy it is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just some quick responses:
- Unfortunately, the tests are indeed all single-threaded;
- I didn't realize that they applied early stopping by default, this might explain it indeed;
- I thought about doing something similar to what you suggested. Namely, I wanted to write some tools/environments to make these "almost equal" tests;
- I think that it is best to change the budget per test. The current value works well for the basic agents but I am 80% sure it won't work for all Deep RL algorithms.
- That's a good point about using global seeds, but unfortunately
torch
doesn't have Seeder objects so we have to use it for the Torch Agents and the StableBaselinesAgent.
I will take a look into this early stopping issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After looking into it, I think the default is no early stopping in stablebaselines (I had to go back several inherited modules in stablebaselines, this is quite the maze) you are right. The problem seems to be seomewhere else.
On the other hand, the learn method has a loop that says
for update in range(1, total_timesteps // self.n_batch + 1):
which would not give a total of total_timesteps
iterations unless self.n_batch
divides total_timesteps
. Maybe the problem comes from here ?
On another note, I added a test to test |
I don't really know what we should do. i mean, tensorboard is not huge (5Mb) so I think we could just add it to the requirements and be done with it. |
@TimotheeMathieu On another note, I think PR #144 broke my git pre-commit setup - now flake8 stops me because of the unused imports on the |
No this is not due to PR #144, precommit already checked the
but I did not test it. Either way, you can say that you don't want pre-commit to check with |
No this is not due to PR #144, precommit already checked the
but I did not test it. Either way, you can say that you don't want pre-commit to check with |
@omardrwch What do you think about this? I'm tempted to just add it to the requirements for now. Also, I'll look into the |
@TimotheeMathieu So I rerun some reproducibility experiments with the SB3 A2C algorithm and our new agent, here are the results: Difference ExperimentsIn these experiments, I choose a batch size X and a budget difference Y and analyze the results for every (X, Y) pair. Additionally, you can specify the base budget B. In summary, this experiment:
The figure below shows the results with B = 0: Note that we would expect a significant error rate for all experiments where Y is not zero. For completeness, here are the results for B = 100 and B = 1000: As you can see, using a higher base budget only attenuates the errors. Additive Fit ExperimentsThis experiment is similar to the previous setting. Instead of varying the budget difference, we vary the total budget T. This experiment:
The results are shown below: Note that we would expect all error rates to be zero since both agents are trained for the same budget. The additive fit seems to always commit some mistakes with T > 0 for all batch sizes. It seems like it is something we can't rely on, but if you have any further suggestions, please let me know. Edit: I also uploaded my code for these results here. |
I did some experiments and I think it may be because of
I obtain the following add graph: and I guess this is OK, with batch size 1 the fit is additive (which is what we want). |
@TimotheeMathieu |
Just an update: I've been able to (partially) fix the additive fit issues for the agent. I encountered one last problem: it seems that it is difficult to guarantee additive fits when a model is saved and then loaded during training. First, the standard This is relevant because our Edit: I am also pretty sure that the |
Sorry, I'm a bit (very) late in this PR. Regarding the tensorboard requirement, I think it's ok to add it for now. I think the SB3 wrapper is a very important feature, so testing it should be worth the extra requirement. Later we could implement @TimotheeMathieu 's suggestion (of creating a directory for additional heavier tests) when needed (e.g. also for a more comprehensive testing of our agents, to make sure that they match some benchmarks). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok for me. We can make separate PRs for the tests and seeder later. Thanks.
I'm finishing work on the StableBaselines wrapper that I was working on before. Unfortunately, SB3 algorithms don't seem to reliably satisfy
check_fit_additive
, so I removed it from the tests.Let me know what you think!