Skip to content

Backport v1 MFSDP full-iteration CUDA graph safety - #29

Closed
lhb8125 wants to merge 2 commits into
shjwudp:mfsdp_refactorfrom
lhb8125:codex/mfsdp-v1-fullcg-backport
Closed

Backport v1 MFSDP full-iteration CUDA graph safety#29
lhb8125 wants to merge 2 commits into
shjwudp:mfsdp_refactorfrom
lhb8125:codex/mfsdp-v1-fullcg-backport

Conversation

@lhb8125

@lhb8125 lhb8125 commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • preserve v1 Megatron-FSDP gradient references required by full-iteration CUDA graph replay
  • disable the optimizer-step parameter all-gather that crosses the full-iteration capture boundary
  • preserve non-FSDP-unit parameter bucket storage across AllGatherPipeline reset
  • keep preserved buckets eligible for a fresh all-gather before reuse

This is a focused backport of behavior already merged on dev through NVIDIA#4663 and NVIDIA#4717. It is independent from the FSDP v2 stable-storage and integration PRs.

The first commit is mechanical formatting required by the current formatter; the second commit contains the behavior change.

Validation

  • Python 3.12 py_compile on all changed files
  • CHECK_ONLY=true BASE_REF=mfsdp_refactor bash tools/autoformat.sh
  • isolated v1-on-v1 baseline 20260703-014718-6375 / 2265910 at 337d07a35 completed 30/30 finite iterations plus 32 validation and 32 test iterations; warmup-8 mean/median were 1032.31/1058.90 TFLOP/s/GPU, with peak allocated/reserved/device-used 90,809/117,334/127,586 MB
  • first required-set v1-on-v2 run 20260703-013644-1ba9 / 2265881 completed the same workload at 1027.27/1051.95 TFLOP/s/GPU, with peak memory 89,885/110,590/120,842 MB; final loss/grad norm were finite and aligned
  • final head d7177a1aa v1 control 20260703-020830-a4a1 / 2266229 completed 30/30 finite iterations plus 32 validation and 32 test iterations; peak memory was 89,885/119,370/129,622 MB
  • the final run had a transient slowdown in iterations 9-17; its stable tail (iterations 18-30) was 1026.50/1051.00 TFLOP/s/GPU, matching the first same-node control

@lhb8125

lhb8125 commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

Closing this PR because it is not a dependency of the FSDP v2 integration. The validated v1 baseline already contains upstream NVIDIA#4663 and NVIDIA#4717; relative to that baseline, NVIDIA#5636 is the only additional fix required for the target v1 full-iteration CUDA graph configuration. We will keep this branch/commit only as a temporary experiment dependency when running v1 full-CG on the older mfsdp_refactor base, until that base is updated to current dev or otherwise includes NVIDIA#4663 and NVIDIA#4717.

@lhb8125 lhb8125 closed this Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant