fix(ci): pin linear CUDA-graph sizing distribution for gpt_grpo_*_8b_throughput - #5242
Closed
Connor-XY wants to merge 1 commit into
Closed
fix(ci): pin linear CUDA-graph sizing distribution for gpt_grpo_*_8b_throughput#5242Connor-XY wants to merge 1 commit into
Connor-XY wants to merge 1 commit into
Conversation
…throughput gpt_grpo_tp4_pp1_dp2_8b_throughput (+ _cudagraphs variant) fail their mem-allocated-bytes guardrail: peak ~69.2GB vs ~60.9GB golden (+13.6%, tol 10%). lm-loss passes -- training is numerically correct; only peak memory grew. Bisected the regression to a single commit (last-pass bdcaf26 -> first-fail 16b7194, the only commit between): PR NVIDIA#3509 "Change the cudagraph distribution from linearly to exponentially-decreasing + grid for mixed prefill", which made `exponential` the default sizing distribution and added a mixed-prefill grid. Set --inference-dynamic-batching-cuda-graph-sizing-distribution=linear to pin the pre-NVIDIA#3509 behavior for these tests. Caveat: these recipes use --inference-dynamic-batching-num-cuda-graphs 1, so the sizing distribution mainly selects the single captured graph size; NVIDIA#3509 also added the mixed-prefill grid (--inference-dynamic-batching-cuda-graph-mixed-prefill-count, default 16) which may be a separate memory driver. If `linear` alone does not bring peak memory under the 10% bound in CI, the follow-up lever is reducing the mixed-prefill-count (or the inference team confirms exponential+grid is the intended default and the golden is rebaselined). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Connor-XY
force-pushed
the
yxu1/grpo-cudagraph-linear-distribution
branch
from
June 9, 2026 22:32
8d52f83 to
7207238
Compare
Connor-XY
marked this pull request as ready for review
June 9, 2026 22:34
Contributor
Author
|
/ok to test 7207238 |
Contributor
Author
|
Superseded by #5253. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Pin the linear CUDA-graph sizing distribution for the GRPO throughput tests:
gpt_grpo_tp4_pp1_dp2_8b_throughputgpt_grpo_tp4_pp1_dp2_8b_cudagraphs_throughputvia
--inference-dynamic-batching-cuda-graph-sizing-distribution: linear.Why
Both fail their
mem-allocated-bytesguardrail in CI: peak ~69.2 GB vs ~60.9 GB golden (+13.6%, tol 10%).lm-losspasses — training is numerically correct; only peak memory grew.Bisected (scanned every GRPO run May 18→Jun 5) to a single commit — last-pass
bdcaf267→ first-fail16b71941, the only commit between:It made
exponentialthe default sizing distribution and added a mixed-prefill grid. This pins the pre-#3509 (linear) behavior for these tests.Caveat (validate on CI)
These recipes use
--inference-dynamic-batching-num-cuda-graphs 1, so the sizing distribution mainly selects the single captured graph size; #3509 also added the mixed-prefill grid (--inference-dynamic-batching-cuda-graph-mixed-prefill-count, default 16), which may be a separate memory driver. Iflinearalone doesn't bring peak memory under the 10 % bound, the follow-up lever is reducingmixed-prefill-count— or the inference team confirms exponential+grid is the intended default and the golden is rebaselined. (Couldn't reproduce locally — GRPO needs the Qwen3-8B ckpt + RL rollout harness.)🤖 Generated with Claude Code