Skip to content

Enable GTP topology for heterogeneous MIMO training - #6264

Closed
yashaswikarnati wants to merge 1 commit into
NVIDIA:pull-request/6234from
yashaswikarnati:ykarnati/mimo-hetero-gtp-integration
Closed

Enable GTP topology for heterogeneous MIMO training#6264
yashaswikarnati wants to merge 1 commit into
NVIDIA:pull-request/6234from
yashaswikarnati:ykarnati/mimo-hetero-gtp-integration

Conversation

@yashaswikarnati

Copy link
Copy Markdown
Contributor

What

  • Resolve dense and expert weight-shard counts against the language TP axes.
  • Materialize GTP/EGTP dimensions and all required module-owned process groups in the heterogeneous grid.
  • Account for GTP data lanes in bridge fan-out, mock data, token normalization, and batch sizing.
  • Seed dense and expert GTP ranks explicitly without MPU-global process-group reads.
  • Reuse each module's ProcessGroupCollection during optimizer construction.
  • Keep encoder and projection configs non-GTP.
  • Extend the 20-layer mock recipe to support the multi-node topology required by GTP8/EGTP2.

Dependency

Stacked on #6234 by @fanshiqing. Checkpoint semantics are intentionally excluded and follow in a separate PR.

Validation

  • behavioral argument and process-group topology tests
  • Python compile checks
  • shell syntax check
  • import sorting
  • git diff --check
  • full 100-iteration save/load validation is tracked by the integration PR

Signed-off-by: ykarnati <ykarnati@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@copy-pr-bot
copy-pr-bot Bot force-pushed the pull-request/6234 branch 2 times, most recently from 07b0066 to 382835b Compare August 5, 2026 02:50
@copy-pr-bot
copy-pr-bot Bot deleted the branch NVIDIA:pull-request/6234 August 5, 2026 19:21
@copy-pr-bot copy-pr-bot Bot closed this Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants