feature(rjy): add mamujoco env and related configs #153
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
puyuan1996
added
environment
|
以 independent learning 的形式接一下sampled efficientzero算法，验证环境的逻辑 |
puyuan1996
reviewed
Dec 4, 2023
| """ | ||
| Overview: | ||
| The modified Multi-agentMuJoCo environment with continuous action space for LightZero's algorithms. | ||
| """ |
There was a problem hiding this comment.
There was a problem hiding this comment.
There was a problem hiding this comment.
PR的description里面增加这个PR的简要描述
There was a problem hiding this comment.
原来DI-engine这里似乎还没replay，等把其他改完我再测试一下
puyuan1996
changed the title
puyuan1996
requested changes
Jun 12, 2024
| # split a full batch into slices of mini_infer_size: to save the GPU memory for more GPU actors | ||
| slices = int(np.ceil(transition_batch_size / self._cfg.mini_infer_size)) | ||
| network_output = [] | ||
| for i in range(slices): | ||
| beg_index = self._cfg.mini_infer_size * i | ||
| end_index = self._cfg.mini_infer_size * (i + 1) | ||
| m_obs = torch.from_numpy(value_obs_list[beg_index:end_index]).to(self._cfg.device).float() | ||
| m_obs = to_dtype(to_device(to_tensor(value_obs_list[beg_index:end_index]), self._cfg.device), torch.float) |
There was a problem hiding this comment.
为什么要这样修改呢？之前的方法在多智能体下面会有报错吗？你现在的写法是在单/多智能体下都能与预期一致吗
|
|
||
| # calculate the target value | ||
| m_obs = default_collate(m_obs) |
| target_values.append(0) | ||
| target_value_prefixs.append(value_prefix) | ||
| target_values.append(np.zeros_like(value_list[0])) | ||
| target_value_prefixs.append(np.array([0,])) |
There was a problem hiding this comment.
单/多智能体运行都是正常的吗？测试一下mamujoco hopper和lunarlander-cont
lzero/mcts/buffer/game_segment.py
Outdated
| pad_frames = np.array([stacked_obs[-1] for _ in range(pad_len)]) | ||
| stacked_obs = np.concatenate((stacked_obs, pad_frames)) | ||
| pad_frames = [stacked_obs[-1] for _ in range(pad_len)] | ||
| stacked_obs += pad_frames |
There was a problem hiding this comment.
单/多智能体运行都是正常的吗？测试一下mamujoco hopper和lunarlander-cont
| @@ -0,0 +1,540 @@ | |||
| from typing import Optional, Tuple | |||
There was a problem hiding this comment.
从SampledEfficientZeroModelMLP继承，只改写不同的method，增加overview 阐述具体的不同
lzero/worker/muzero_collector.py
Outdated
| @@ -388,8 +398,12 @@ def collect(self, | |||
| ready_env_id = ready_env_id.union(set(list(new_available_env_id)[:remain_episode])) | |||
| remain_episode -= min(len(new_available_env_id), remain_episode) | |||
|
|
|||
| stack_obs = {env_id: game_segments[env_id].get_obs() for env_id in ready_env_id} | |||
| stack_obs = {env_id: game_segments[env_id].get_obs()[0] for env_id in ready_env_id} | |||
| if __name__ == "__main__": | ||
| from zoo.multiagent_mujoco.entry import train_sez_independent_mamujoco | ||
|
|
||
| train_sez_independent_mamujoco([main_config, create_config], seed=seed, max_env_step=max_env_step) |
There was a problem hiding this comment.
目前 mamujoco 上的实验状态是？写在description里面吧，以及相对单agent的核心算法overview
| ) -> 'Policy': # noqa | ||
| """ | ||
| Overview: | ||
| The train entry for MCTS+RL algorithms, including MuZero, EfficientZero, Sampled EfficientZero, Gumbel Muzero. |
There was a problem hiding this comment.
更新overview，阐述清楚主要的改动代码
There was a problem hiding this comment.
这个与原有的 train_muzero的主要区别是？如果区别不大，尽量复用原有的代码哈
| @@ -0,0 +1,132 @@ | |||
| from easydict import EasyDict | |||
| import os | |||
| os.environ["CUDA_VISIBLE_DEVICES"] = '6' | |||
added 2 commits
June 14, 2024 15:28
puyuan1996
reviewed
Jun 25, 2024
| @POLICY_REGISTRY.register('sampled_efficientzero') | ||
| class SampledEfficientZeroPolicy(MuZeroPolicy): | ||
| @POLICY_REGISTRY.register('sampled_efficientzero_ma') | ||
| class SampledEfficientZeroMAPolicy(SampledEfficientZeroPolicy): |
puyuan1996
reviewed
Jun 25, 2024
| class SampledEfficientZeroMAPolicy(SampledEfficientZeroPolicy): | ||
| """ | ||
| Overview: | ||
| The policy class for Sampled EfficientZero proposed in the paper https://arxiv.org/abs/2104.06303. |
There was a problem hiding this comment.
更新注释，只重写需要修改的method哈，大部分应该是不用重写的。
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
MAmujoco Environment Integration: I have added support for the MAmujoco environment and successfully adapted it for use with LightZero. For detailed information about the MAmujoco environment, please refer to the original repository at MaMuJoCo Environments.
Independent Learning Pipeline: A new independent learning pipeline has been introduced to the project. This pipeline is currently integrated with the existing codebase and can be activated by setting the 'multi_agent' parameter accordingly.
These updates aim to enhance the project's functionality and scalability, providing a robust framework for multi-agent learning scenarios.