test: add async scheduling inference coverage - #6104
Merged
lmcafee-nvidia merged 5 commits intoJul 30, 2026
Conversation
Add gpt_dynamic_inference_tp1_pp1_583m_async_sched, exercising the new --inference-dynamic-batching-async-sched-mode async path end-to-end with greedy sampling and comparing generated tokens against golden values. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: shanmugamr1992 <shanmugamr1992@gmail.com> Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com>
Register the gpt_dynamic_inference_tp1_pp1_583m_async_sched test case on dgx_gb200 via a new gb200/gpt-dynamic-inference.yaml recipe (nodes: 1, gpus: 4) and capture platform-specific golden values. Generated tokens match the H100 golden (deterministic greedy sampling); throughput is GB200-specific. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: shanmugamr1992 <shanmugamr1992@gmail.com> Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com>
Adds inference performance tests exercising dynamic-batching async prefill scheduling on both H100 (DP=8) and GB200 (DP=4) for the 583M checkpoint. - run_perf_test.sh: read ASYNC_SCHED config key; when true, launch the server with --inference-dynamic-batching-async-sched-mode async. Also pass --skip-prompt-log-probs because the server tool force-sets return_log_probs=True (echo/loglikelihood support), which would set materialize_only_last_token_logits to False and violate the async-scheduling requirement. - gpt_583m_perf_async_sched (H100) and gpt_583m_perf_async_sched_gb200_4gpu (GB200) test cases with platform-keyed baselines captured on cw-dfw / oci-hsg. - Register both in the existing h100/gpt-perf-dp8 and gb200/gpt-perf-dp4 recipes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: shanmugamr1992 <shanmugamr1992@gmail.com> Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com>
The initial GB200 batch-32 capture (1002 tok/s, TPOT 31.9) was a noisy cold outlier; a fresh capture puts it at ~1217 tok/s (TPOT 26.3), in line with the non-async DP=4 baseline. Re-recording avoids a too-low floor that would push normal runs near the +20% improvement ceiling. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: shanmugamr1992 <shanmugamr1992@gmail.com> Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com>
Contributor
Author
|
/ok to test b15d040 |
balasaajay
approved these changes
Jul 29, 2026
Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com>
Contributor
Author
|
/ok to test d9e177d |
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/30514662280 |
svcnvidia-nemo-ci
pushed a commit
to dimapihtar/Megatron-LM
that referenced
this pull request
Aug 4, 2026
Signed-off-by: shanmugamr1992 <shanmugamr1992@gmail.com> Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com> Co-authored-by: shanmugamr1992 <shanmugamr1992@gmail.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds H100 and GB200 functional and performance coverage for the async dynamic-inference scheduling path introduced in #5939.
gpt_dynamic_inference_tp1_pp1_583m_async_sched, a deterministic 583M GPT functional test using--inference-dynamic-batching-async-sched-mode async, with platform-specific H100 and GB200 golden values.gpt_583m_perf_async_sched(H100, DP=8) andgpt_583m_perf_async_sched_gb200_4gpu(GB200, DP=4), with checked-in baselines across multiple batch sizes.ASYNC_SCHEDperformance-test setting that launches the server in async mode and skips prompt log probabilities, which the benchmark does not request.This clean branch preserves the test artifacts authored by @shanmugamr1992 while rebasing the work directly onto current
mainafter #5939 merged.Performance
Same 583M checkpoint, parallelism, and synthetic dataset (ISL=512, OSL=128) per platform; only the dynamic-batching scheduling mode differs.
Throughput (tokens/sec)
Time per output token (ms/token)
The observed differences are within run-to-run noise for this small dense model and provide regression baselines rather than a speedup claim.
Test plan