[hybrid] Fix model parallel non-distributed param broadcast #36186
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
wangxicoding
requested review from
JZ-LIANG,
gongweibao,
sandyhouse and
fuyinno4
JZ-LIANG
reviewed
Sep 29, 2021
| rings.append(self.dp_ring_id) | ||
|
|
||
| # need sync non distributed param in mp group | ||
| if self.mp_ring_id is not None: |
There was a problem hiding this comment.
放另一个地方会好一些吧？ mp 的初始化同步为什么会放到 offload 中实现？
There was a problem hiding this comment.
因为需要先把参数给广播好，然后再插入cast、memcpy op，否则会造成各个卡的fp16参数和offload变量不一致
There was a problem hiding this comment.
也可以直接在_initialization_broadcast里再写一段逻辑专门处理offload和optimize_cast需要先广播参数的需求，可能麻烦一些，不过从模块化角度来说，确实要好一些。
There was a problem hiding this comment.
我之后再专门搞个逻辑处理处理这个需求吧。
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Bug fixes
PR changes
Others
Describe
1、混合并行中，mp的非distributed的参数需要保持各个mp rank一致，PR修复mp的非distributed参数未广播的问题。
2、开启optimize_offload或(optimize_cast+optimize_sharding)时，会将param设置为 非persistable，在program.clone时，会出错。鉴于该问题目前只在hybrid中存在，PR通过将非persistable param重新生成为var的方式修复该问题。