Skip to content

feat(pipeline): carry mHC residual streams between stages - #5943

Closed
FDecaYed wants to merge 1 commit into
NVIDIA:pull-request/5936from
FDecaYed:codex/hybrid-mhc-pipeline
Closed

feat(pipeline): carry mHC residual streams between stages#5943
FDecaYed wants to merge 1 commit into
NVIDIA:pull-request/5936from
FDecaYed:codex/hybrid-mhc-pipeline

Conversation

@FDecaYed

Copy link
Copy Markdown
Contributor

What

Teach the PP/VPP schedules to carry expanded mHC residual streams across real stage-to-stage edges:

  • active edges use num_residual_streams times hidden_size;
  • first receive, last send, PP=1, non-mHC, and variable-shape paths retain existing behavior;
  • interleaved VPP keeps one uniform expanded shape for its active edges;
  • MultiModulePipelineCommunicator remains compatible when no single pp_group exists;
  • no new global process-group reads are introduced.

Stack

Provenance

Testing

  • isort, Black --check, ruff, pylint (10/10), compileall, and git diff --check passed.
  • CPU/single-process shape tests are included; local execution is blocked because this host lacks Torch. Real PP/VPP behavior remains for stack GPU CI.

Expand only active PP/VPP communication edges to num_residual_streams times hidden_size while preserving boundary, single-stage, non-mHC, and variable-shape behavior.

Signed-off-by: Deyu Fu <Deyu.Foo@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 21, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@FDecaYed

Copy link
Copy Markdown
Contributor Author

/ok to test f12259e

Copy link
Copy Markdown
Contributor Author

Closing this draft until Round 2. Its logical successor is P3 (mHC PP/VPP transport), rebuilt on main after P2 merges. It will not remain a stacked-review PR. Tracking PR: #5795.

@copy-pr-bot
copy-pr-bot Bot deleted the branch NVIDIA:pull-request/5936 July 23, 2026 13:49
@copy-pr-bot copy-pr-bot Bot closed this Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant