feat: large-scale EP(part 1: Add MNNVL MoE A2A support) #3504
Conversation
|
/bot run |
|
PR_Github #2049 [ run ] triggered by Bot |
juney-nvidia
changed the title
|
PR_Github #2049 [ run ] completed with state |
|
/bot run |
|
PR_Github #2113 [ run ] triggered by Bot |
|
PR_Github #2113 [ run ] completed with state |
|
/bot run |
dongxuy04
force-pushed
the
user/dongxuy/mnnvl_a2a_github
branch
from
24a44a8 to
163b322
Compare
|
/bot run |
|
PR_Github #2385 [ run ] triggered by Bot |
|
PR_Github #2385 [ run ] completed with state |
|
/bot run |
|
PR_Github #2408 [ run ] triggered by Bot |
|
PR_Github #2408 [ run ] completed with state |
dongxuy04
force-pushed
the
user/dongxuy/mnnvl_a2a_github
branch
from
163b322 to
bfda091
Compare
|
/bot run |
|
PR_Github #2768 [ run ] triggered by Bot |
|
PR_Github #2768 [ run ] completed with state |
|
/bot run --add-multi-gpu-test --disable-fail-fast |
|
PR_Github #2773 [ run ] triggered by Bot |
|
PR_Github #2773 [ run ] completed with state |
|
/bot run --disable-fail-fast |
dongxuy04
force-pushed
the
user/dongxuy/mnnvl_a2a_github
branch
from
bfda091 to
89ec4a5
Compare
dongxuy04
force-pushed
the
user/dongxuy/mnnvl_a2a_github
branch
from
b928430 to
b81aae4
Compare
dongxuy04
force-pushed
the
user/dongxuy/mnnvl_a2a_github
branch
from
b81aae4 to
dd0c525
Compare
|
/bot run |
|
PR_Github #3227 [ run ] triggered by Bot |
|
PR_Github #3227 [ run ] completed with state |
|
/bot run |
|
PR_Github #3269 [ run ] triggered by Bot |
|
PR_Github #3269 [ run ] completed with state |
Signed-off-by: Dongxu Yang <[email protected]>
Signed-off-by: Dongxu Yang <[email protected]>
Signed-off-by: Dongxu Yang <[email protected]>
Signed-off-by: Dongxu Yang <[email protected]>
Signed-off-by: Dongxu Yang <[email protected]>
Signed-off-by: Dongxu Yang <[email protected]>
Signed-off-by: Dongxu Yang <[email protected]>
Signed-off-by: Dongxu Yang <[email protected]>
dongxuy04
force-pushed
the
user/dongxuy/mnnvl_a2a_github
branch
from
dd0c525 to
2070dca
Compare
|
/bot run |
|
PR_Github #3341 [ run ] triggered by Bot |
|
PR_Github #3341 [ run ] completed with state |
juney-nvidia
changed the title
|
|
||
| struct GroupSharedBuffer | ||
| { | ||
| int groupIndiceBuffer[GROUP_MAX_INDICE_COUNT]; |
There was a problem hiding this comment.
Note that it should either be Index or Indices(with a "s").Indexis the singular andIndices` is the plural.
| public: | ||
| static constexpr int GROUP_COUNT_PER_BLOCK = 8; | ||
| static_assert(GROUP_COUNT_PER_BLOCK <= 8, "GROUP_COUNT_PER_BLOCK must be less than or equal to 8"); | ||
| static constexpr int WARP_PER_GROUP = 2; |
There was a problem hiding this comment.
It should be WARPS_PER_GROUP or WARP_COUNT_PER_GROUP.
| TLLM_CHECK_WITH_INFO( | ||
| blockCountPerChannel <= smCount, "GPU should support at lease one channel, usableSmCount=%d", smCount); | ||
| int perferredChannel = smCount / 2 / blockCountPerChannel; // use half SMs for communication | ||
| int channelCount = std::max(perferredChannel, 1); // at lease one channel |
There was a problem hiding this comment.
In the comment, it must be at least
| static int computeMoeCommChannelCount(int epSize) | ||
| { | ||
| int smCount = getMaxUsableSmCount(); | ||
| int blockCountPerChannel = (epSize + GROUP_COUNT_PER_BLOCK - 1) / GROUP_COUNT_PER_BLOCK; |
There was a problem hiding this comment.
Isn't there a function in TRT-LLM to compute this (something like divUp or ceilDiv are common names).
4 participants
Add MNNVL MoE AllToAll support for large scale expert parallism.