tests: update golden values for inference_optimized_swiglu and moe_resume_gb200 (MCORE-610) - #6231
Conversation
…sume_gb200 Fixes MCORE-610: logprob[5] for gpt_dynamic_inference_tp4_pp1_ep4_16B_inference_optimized_swiglu_logitsmatch exceeded abs_tol=0.001 by ~0.00279 due to numerical drift after PR NVIDIA#5700 (inference optimized qwen moe) and subsequent commits (bs-invar MoE NVIDIA#4871). Also refreshes lm_loss golden values for gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer on GB200 which drifted in the same window. Downloaded from GitLab pipeline 60908036 (--only-failing). Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
|
This PR has been automatically converted to draft because all PRs must start as drafts. When you are ready for review, click Ready for Review to begin the review process. This will:
See the contribution guide for more details. |
|
/ok to test 759d782 |
…en values The download script pulled a string "nan" at iteration-time step 1 (warmup noise), which fails the check_golden_values.py non-finite validator in CI. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>
…en-values' into fix/update-swiglu-logitsmatch-golden-values
…itsmatch-golden-values
|
/ok to test b3bf87e |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/31077632130 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/31135944404 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/31142375875 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/31163439809 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/31173379036 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/31176659552 |
3 participants
What does this PR do?
Refreshes golden values for two failing functional tests to fix MCORE-610.
Root cause
gpt_dynamic_inference_tp4_pp1_ep4_16B_inference_optimized_swiglu_logitsmatchwas added as a regression guard in PR #5700 (merged Jul 31). Commits landing the same day and shortly after — notably "Bs invar MoE" (#4871) — caused numerical drift in the MoE inference path, producinglogprob[5] = -1.79449vs. the golden-1.79170(diff ~0.00279, exceedingabs_tol=0.001by 179%).Changes
gpt_dynamic_inference_tp4_pp1_ep4_16B_inference_optimized_swiglu_logitsmatch/golden_values_dev_dgx_h100.json— updated logprobs to match current inference output; fixes MCORE-610gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_gb200.json— refreshes lm_loss step values that drifted in the same windowDownloaded from GitLab pipeline 60908036 with
--only-failing.