Backport v1 MFSDP full-iteration CUDA graph safety - #29
Closed
lhb8125 wants to merge 2 commits into
Closed
Conversation
Collaborator
Author
|
Closing this PR because it is not a dependency of the FSDP v2 integration. The validated v1 baseline already contains upstream NVIDIA#4663 and NVIDIA#4717; relative to that baseline, NVIDIA#5636 is the only additional fix required for the target v1 full-iteration CUDA graph configuration. We will keep this branch/commit only as a temporary experiment dependency when running v1 full-CG on the older mfsdp_refactor base, until that base is updated to current dev or otherwise includes NVIDIA#4663 and NVIDIA#4717. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is a focused backport of behavior already merged on
devthrough NVIDIA#4663 and NVIDIA#4717. It is independent from the FSDP v2 stable-storage and integration PRs.The first commit is mechanical formatting required by the current formatter; the second commit contains the behavior change.
Validation
py_compileon all changed filesCHECK_ONLY=true BASE_REF=mfsdp_refactor bash tools/autoformat.sh20260703-014718-6375/2265910at337d07a35completed 30/30 finite iterations plus 32 validation and 32 test iterations; warmup-8 mean/median were1032.31/1058.90 TFLOP/s/GPU, with peak allocated/reserved/device-used90,809/117,334/127,586 MB20260703-013644-1ba9/2265881completed the same workload at1027.27/1051.95 TFLOP/s/GPU, with peak memory89,885/110,590/120,842 MB; final loss/grad norm were finite and alignedd7177a1aav1 control20260703-020830-a4a1/2266229completed 30/30 finite iterations plus 32 validation and 32 test iterations; peak memory was89,885/119,370/129,622 MB1026.50/1051.00 TFLOP/s/GPU, matching the first same-node control