Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

通用的经验表示格式，结合经验池机制优化数据流 #31

Open
StepNeverStop opened this issue Jan 2, 2021 · 1 comment
Open

通用的经验表示格式，结合经验池机制优化数据流 #31

StepNeverStop opened this issue Jan 2, 2021 · 1 comment
Assignees
@StepNeverStop
Labels
enhancement New feature or request optimization Better performance or solution

Comments

@StepNeverStop
Copy link
Owner

No description provided.

@StepNeverStop StepNeverStop self-assigned this Jan 2, 2021
@StepNeverStop StepNeverStop added the enhancement New feature or request label Jan 2, 2021
StepNeverStop added a commit that referenced this issue Jan 2, 2021
@StepNeverStop
refactor(data format): use custom namedtuple to transfer data (#31)
38a7a7c 
- rename `indexs.py` to `specs.py`
- add 3 namedtuple named `ModelObservations`, `Experience` and `SingleModelInformation`
StepNeverStop added a commit that referenced this issue Jan 3, 2021
@StepNeverStop
refactor(rls): implement nametuple-style experience and buffer (#31)
cd5804c 
- add `BatchExperiences`, `ModelObservations`, `NamedTupleStaticClass` in `specs.py`
- fix bugs of `dpg` and `ClippedNormalActionNoise`
- refactor ExperienceReplay
- remove redundant function
@StepNeverStop
Copy link
Owner Author

  • 适配gym
  • 测试LSTM是否正确开启训练
  • 修复On-policy算法

StepNeverStop added a commit that referenced this issue Jan 3, 2021
@StepNeverStop
fix(rnn): fix data shape when using rnn training (#31)
76f7333
StepNeverStop added a commit that referenced this issue Jan 3, 2021
@StepNeverStop
perf(rnn): change data dimension from 3D to 2D when using rnn (#31)
08532ef
StepNeverStop added a commit that referenced this issue Jan 4, 2021
@StepNeverStop
refactor(on-policy): refactor on-policy experience mechanism (#31)
e9de81f 
- rename `memories` to `BATCH` for better reading
- implement new type of `DataBuffer` to adapt namedtuple `BatchExperiences`
- fix PPO
StepNeverStop added a commit that referenced this issue Jan 4, 2021
@StepNeverStop
fix(ICM): fix intrinsic curiosity mechanism of on-policy (#31)
d8ba3f3
StepNeverStop added a commit that referenced this issue Jan 4, 2021
@StepNeverStop
fix(on-policy): fix on-policy algorithms (#31)
a810d79 
- fix `pg`, `a2c`, `aoc`, `ppoc`, `trpo`
StepNeverStop added a commit that referenced this issue Jan 4, 2021
@StepNeverStop
refactor(train): change default dimension of reward and done to 2D (#31)
0a54a23
StepNeverStop added a commit that referenced this issue Jan 4, 2021
@StepNeverStop
refactor(train): change default dimension of reward and done to 2D (#31…
2e291ab 
…, #32)
StepNeverStop added a commit that referenced this issue Jan 4, 2021
@StepNeverStop
fix(gym): fix gym training (#31)
eda5daf
StepNeverStop added a commit that referenced this issue Jan 4, 2021
@StepNeverStop
fix(train): fix discrete action training and vector obs normalization (…
8496a46 
…#31)
StepNeverStop added a commit that referenced this issue Jan 4, 2021
@StepNeverStop
perf(train): improve the tolerance of data dimension storage (#31, close
706006d 
 #32)

- remove redundant calculation
- add length-equal check function for on-policy algorithm storing valid data
- optimize `trpo`
StepNeverStop added a commit that referenced this issue Jan 4, 2021
@StepNeverStop
style: rename some identifiers (#31)
9bd1ad4
StepNeverStop added a commit that referenced this issue Jan 5, 2021
@StepNeverStop
refactor(data): TAG 2.0.0 (#25, #33, open #31)
dc5391d 
- support multi-vector and multi-visual input
- optimize `gym` and `unity` wrapper
- fix `ActorCriticValueCts`
- tag 2.0.0
- add `ObsSpec`
- refactor `SingleAgentEnvArgs` and `MultiAgentEnvArgs`
- remove `self.s_dim`, use `self.concat_vector_dim` instead
- stop using vector input normalization temporarily
@StepNeverStop StepNeverStop reopened this Jan 5, 2021
@StepNeverStop StepNeverStop added the optimization Better performance or solution label Jan 6, 2021
StepNeverStop added a commit that referenced this issue Jul 2, 2021
@StepNeverStop
fix(multi-agents): implemented n-step replay buffer for multi-agents …
dbc6136 
…training. (#41,#25,#31)

1. change variable name from "is_lg_batch_size" to "can_sample"
2. optimized unity wrapper
3. optimized multi-agents replay buffers
StepNeverStop added a commit that referenced this issue Jul 4, 2021
@StepNeverStop
fix&feat: optimized training framework. (#41, #34, #33, #31, #25)
0a8d247 
1. fixed n-step replay buffer
2. reconstruct representation net
3. remove 'use_stack'
4. implement multi-agent algorithms with shared parameters
5. optimized agent network
StepNeverStop added a commit that referenced this issue Jul 6, 2021
@StepNeverStop
perf(data): remove namedtuple and NamedTuple, use dataclasses.@DataClass
a0e5be7 
 instead. (#31, #33)
StepNeverStop added a commit that referenced this issue Jul 11, 2021
@StepNeverStop
Chang RLs from TF2 to PyTorch, replace almost 90% function and variab…
f325a21 
…les to torch. (#43, #31)
StepNeverStop added a commit that referenced this issue Jul 11, 2021
@StepNeverStop
fix&perf: added function wrapper 'iTensor_oNumpy'. (#43, #34, #31)
b1e6dc4
StepNeverStop added a commit that referenced this issue Jul 29, 2021
@StepNeverStop
v4.1.0 perf: implemented unified env interface.(#34, #31, #26, #25, #40)
941e14d 
1. added `test.yaml` for quickly verify RLs
2. change folder name from `algos` to `algorithms` for better reading
3. removed single agent recoder, all algorithms(sarl&marl) using  `SimpleMovingAverageRecoder`
4. removed `GymVectorizedType` in `common/specs.py`
5. removed `common/train/*`, and implement unified training interface in `rls/train`
6. reconstructed `make_env` function in `rls/envs/make_env`
7. optimized function `load_config`
8. moved `off_policy_buffer.yaml` to `rls/configs/buffer`
9. removed configurations like `eval_while_train`, `add_noise2buffer` etc.
10. optimized environments' configuration files
11. optimized environment wrappers and implemented unified env interface for `gym` and `unity`, see `env_base.py`
12. updated dockerfiles
13. updated README
StepNeverStop added a commit that referenced this issue Aug 25, 2021
@StepNeverStop
perf: reconstruct repo(#47, #25, #46, #34, #31, #33, #39, #41, #45, #26)
67b8979
StepNeverStop added a commit that referenced this issue Sep 3, 2021
@StepNeverStop
perf(buffer): optimized replay buffer. (#31)
6081ec0 
use `Once` to control buffer that only be builded one time.
StepNeverStop added a commit that referenced this issue Sep 4, 2021
@StepNeverStop
v0.0.1 perf(env): optimized env wrapper and data storing. (#31,#34,#25)
7d5e00f 
*. redefine version to v0.0.1
1. removed package `supersuit`
2. implemented class `MPIEnv`
3. implemented class `VECEnv`
4. optimized env wrappers, implemented `render` method to `gyms` environment.
5. reconstructed some of returns of `env.step` from `obs` to `obs_fa` and `obs_fs`.
  - `obs_fa` is used to choose action based by agent/policy. For the cross point of episode i and i+1, `obs_fa` represents $observation_{i+1}^{0}$, otherwise it keeps same with `obs_fs` which represents $observation_{i}^{t}$.
  - `obs_fs` is used to be stored in buffer. For the cross point of episode i and i+1, `obs_fs` represents $observation_{i}^{T}$, otherwise it keeps same with `obs_fa`.
6. optimzed `rssm` related based on mentioned `obs_fs`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@StepNeverStop StepNeverStop

Labels
enhancement New feature or request optimization Better performance or solution
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@StepNeverStop