fix(ci): AUT-957 support golden checks in merge queue by svcnemo-autobot · Pull Request #5989 · NVIDIA/Megatron-LM

svcnemo-autobot · 2026-07-23T07:37:43Z

Background

Merge-group run 29983603449 failed in linting: get-pr-info is skipped for merge_group, while the golden-value step evaluated fromJSON on its empty output and made the workflow template invalid. The PR event path itself passed in run 29902228424.

What changed

Run golden-value validation for both PR pushes and merge groups.
Resolve the base from merge_group.base_ref, with safe PR metadata fallback.
Normalize the merge-group refs/heads/ prefix before fetching and diffing.
Carry the pending ci: Add validation for updated golden-value JSON files in CI workflow #5959 golden-value validator because it is not yet on main.

Details

.github/workflows/cicd-main.yml now avoids parsing an empty get-pr-info output and uses the merge-group event payload directly. tools/check_golden_values.py is unchanged from #5959.

Tested

uv sync --locked --only-group linting
BASE_REF=main CHECK_ONLY=true SKIP_DOCS=false bash tools/autoformat.sh
Black, isort, pylint, and ruff on tools/check_golden_values.py
Finite, non-finite, and malformed JSON cases
PR-event validator and lint passed in run 29902228424

- Introduced a new step in the CI workflow to validate golden-value JSON files for NaN and infinity values. - Added a new script, `check_golden_values.py`, to perform the validation checks on the specified files. - The validation step is triggered during pull request pushes, ensuring that any updates to golden values are properly checked before merging. Signed-off-by: [Your Name] <your.email@example.com> Co-authored-by: [Co-author Name] <coauthor.email@example.com> Signed-off-by: Ajay Balasa <abalasa@nvidia.com> Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

github-actions · 2026-07-23T07:38:10Z

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

Add the oncall reviewer (optional reviewer)
Add required review teams based on your changes

See the contribution guide for more details.

copy-pr-bot · 2026-07-23T07:38:12Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

svcnemo-autobot · 2026-07-23T07:39:55Z

/ok to test a6eb587

The linting job's new 'Validate updated golden values' step (check_golden_values.py, added on main via #5989) rejects any non-finite value in golden-value JSON files that differ from the base branch. Three golden files were refreshed on main (#5953 BERT H100, #5816 MoE GB200) and taken into this sync; each carries a 'nan' at iteration-time.values['1'] — the warmup iteration. The functional comparison harness (python_test_utils/common.py) evaluates iteration-time only over a steady-state window (range(5,21) or range(30,46)), so the warmup iteration is never read. Replacing the warmup 'nan' with iteration 2's finite value is inert for the comparison but satisfies the finiteness gate. All of main's refreshed measured data is preserved (one line changed per file). Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>

Two Phase-3 CI fixes for the 23_07_2026 nightly sync: 1. Golden-value finiteness (linting job, tools/check_golden_values.py from #5989): replace the warmup-iteration "nan" at iteration-time.values["1"] in three main-refreshed golden files with iteration 2's finite value. The value stays outside the steady_window used by the functional comparison harness, so it is inert for accuracy comparison; only the new finiteness gate (which runs on files differing from origin/dev) was tripping. - tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp2/golden_values_dev_dgx_h100.json - tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp4_vp2/golden_values_dev_dgx_h100.json - tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2/golden_values_dev_dgx_gb200.json 2. hybrid->dynamic context-parallel rename (Pip install-test ImportError): megatron/training/training.py was taken from main (skill override list), but main's callers reference get_hybrid_data_context_parallel_groups / args.hybrid_context_parallel while dev's parallel_state.py defines get_dynamic_data_context_parallel_groups and arguments.py defines dynamic_context_parallel. Renamed all four references to match dev's API, matching dev's own training.py. Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>

Three Phase-3 CI fixes for the 23_07_2026 nightly sync: 1. Golden-value finiteness (linting job, tools/check_golden_values.py from #5989): replace the warmup-iteration "nan" at iteration-time.values["1"] in three main-refreshed golden files with iteration 2's finite value. The value stays outside the steady_window used by the functional comparison harness, so it is inert for accuracy comparison; only the new finiteness gate (which runs on files differing from origin/dev) was tripping. - tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp2/golden_values_dev_dgx_h100.json - tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp4_vp2/golden_values_dev_dgx_h100.json - tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2/golden_values_dev_dgx_gb200.json 2. hybrid->dynamic context-parallel rename (Pip install-test ImportError): megatron/training/training.py was taken from main (skill override list), but main's callers reference get_hybrid_data_context_parallel_groups / args.hybrid_context_parallel while dev's parallel_state.py defines get_dynamic_data_context_parallel_groups and arguments.py defines dynamic_context_parallel. Renamed all four references to match dev's API, matching dev's own training.py. 3. Duplicate PackedSeqParams field (build-docs --fail-on-warning: 'Duplicate item ...PackedSeqParams.pad_between_seqs [autodoc2.dup_item]'). Merge accident in megatron/core/packed_seq_params.py: dev added 'pad_between_seqs: Optional[bool]' (plus dev-only cp_partition_mode) and main added 'pad_between_seqs: bool' at a different position; the merge kept both. Removed main's duplicate, kept dev's Optional[bool] (superset). All 14 dataclass fields now present exactly once. Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>

The main->dev merge adopts main's new golden-value validation (tools/check_golden_values.py + the linting job's "Validate updated golden values" step, added by #5989 / AUT-957 on 2026-07-23). That step runs check_golden_values.py on every golden file that differs from the PR base and rejects any NaN/Inf. The merge pulled main's versions of three golden files that predate the check (refreshed by #5953 on 07-22 and #5816 on 07-20), so they still encode the warmup-step iteration timer as the legacy sentinel "1": "nan". main's own post-check convention (e.g. the MoE goldens refreshed by #5916 AFTER the check landed, which passed CI) is to omit the step-1 iteration-time cell entirely and set start_step: 2, rather than store "nan". Eleven files already in the corpus have exactly this shape. Bring the three stale files into that convention: in the iteration-time block only, drop "1": "nan" and set start_step 1->2. All other metrics and all other steps are byte-for-byte unchanged. This regresses no data: step-1 iteration-time is never read by the functional comparator (it only evaluates the steady-window steps 5-21 / 30-46) nor by compare_golden_values_kl.py (which skips non-finite via math.isfinite). Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>

Golden values: drop the stale step-1 iteration-time NaN sentinel in three functional-test golden files (bert_mcore_tp1_pp2, bert_mcore_tp1_pp4_vp2, deepseek_proxy_fsdp_ep2_fsdp2) and set start_step=2, matching main's own post-check convention. The new-on-main tools/check_golden_values.py (AUT-957, #5989) rejects non-finite sentinels; these files predate the check and were never refreshed under it. No numeric data changed — step-1 iteration-time is a warmup-timer artifact that every comparator already skips. Dropped imports (3-way merge kept main's usages but dev's narrower import line): - dynamic_engine.py: restore InferenceBatchDimensions, CUDAGraphBatchDimensionBuilder from batch_dimensions_utils (main #5939). - dynamic_context.py: restore MAX_INTERMEDIATE_OFFSETS_PER_REQUEST from mamba_slot_allocator. - optimizer.py: restore 'from itertools import chain' for the GTP-backfill helper main added. A tree-wide pylint E0602/E0601 sweep over all merge-changed files confirms no other dropped-import merge accidents remain (all residual findings are pre-existing pylint flow-analysis false positives present on dev and/or main). Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>

Golden values: drop the stale step-1 iteration-time NaN sentinel in three functional-test golden files (bert_mcore_tp1_pp2, bert_mcore_tp1_pp4_vp2, deepseek_proxy_fsdp_ep2_fsdp2) and set start_step=2, matching main's own post-check convention. The new-on-main tools/check_golden_values.py (AUT-957, #5989) rejects non-finite sentinels; these files predate the check and were never refreshed under it. No numeric data changed — step-1 iteration-time is a warmup-timer artifact that every comparator already skips. Dropped imports (3-way merge kept main's usages but dev's narrower import line): - dynamic_engine.py: restore InferenceBatchDimensions, CUDAGraphBatchDimensionBuilder from batch_dimensions_utils (main #5939). - dynamic_context.py: restore MAX_INTERMEDIATE_OFFSETS_PER_REQUEST from mamba_slot_allocator. - optimizer.py: restore 'from itertools import chain' for the GTP-backfill helper main added. API mismatch from override-from-main file (the hybrid→dynamic context-parallel rename dev landed and main had not): - training.py: main's version imported/called get_hybrid_data_context_parallel_groups, but the merged parallel_state.py (dev's) defines only get_dynamic_data_context_parallel_groups (a byte-identical rename). Point both the import and the get_batch_on_this_cp_rank call at the dynamic name, matching dev's own pretrain_hybrid.py. args.dynamic_context_parallel was already correct in the merged training.py. A tree-wide pylint E0602/E0601 sweep plus a static def-level import resolver over all 611 megatron/*.py files confirm no other dropped-import or renamed-symbol merge accidents remain; residual static findings are pre-existing false positives (guarded try/except loss_scaler import, compiled helpers_cpp extension, and pylint flow-analysis FPs present on dev and/or main). Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>

Signed-off-by: [Your Name] <your.email@example.com> Signed-off-by: Ajay Balasa <abalasa@nvidia.com> Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com> Co-authored-by: Ajay Balasa <abalasa@nvidia.com> Co-authored-by: [Co-author Name] <coauthor.email@example.com>

…iles The main-only linting check tools/check_golden_values.py (added by #5989, after the last main->dev sync) rejects any NaN/Inf token in golden-value JSON files that changed relative to the PR base (dev). Three files carried a leading "iteration-time"."values"."1" = "nan" warmup placeholder that was present on both dev and main and inherited by this merge: - bert/bert_mcore_tp1_pp2/golden_values_dev_dgx_h100.json - bert/bert_mcore_tp1_pp4_vp2/golden_values_dev_dgx_h100.json - moe/deepseek_proxy_fsdp_ep2_fsdp2/golden_values_dev_dgx_gb200.json The runtime iteration-time comparison (tests/functional_tests/python_test_utils/ common.py) evaluates only a steady-state window (steps 5-20 or 30-45) reduced to a single median, so step 1 is never compared. Replacing the step-1 warmup placeholder with a finite 0.0 is therefore comparison-neutral and unblocks the lint gate without altering any evaluated golden value. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>

Two Phase-3 fixes for the 30_07_2026 main->dev sync: 1) megatron/core/inference/engines/dynamic_engine.py — the merge dropped two main-only lines (CG admission gating, #4870), which dev did not have, causing the Pip/UV import checks to fail with 'NameError: name InferenceBatchDimensions is not defined' at class definition time and breaking import of the whole megatron.core.inference package: - restored 'from megatron.core.inference.batch_dimensions_utils import (CUDAGraphBatchDimensionBuilder, InferenceBatchDimensions)' (both symbols are used: lines 1663/1784/1815/1836/1845) - restored 'self.cuda_graph_all_prefills = inference_config.cuda_graph_all_prefills' (read at line ~1759; inference_config field exists in config.py). The file is now byte-identical to origin/main. 2) Three golden-value JSON files carried an inherited 'iteration-time'.'values'.'1' = 'nan' warmup placeholder rejected by the new main-only lint tools/check_golden_values.py (#5989). The runtime iteration-time comparison only evaluates a steady-state window (steps 5-20 or 30-45), so step 1 is never compared; replaced with a finite 0.0: - bert/bert_mcore_tp1_pp2/golden_values_dev_dgx_h100.json - bert/bert_mcore_tp1_pp4_vp2/golden_values_dev_dgx_h100.json - moe/deepseek_proxy_fsdp_ep2_fsdp2/golden_values_dev_dgx_gb200.json Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>

…en warmup values Three Phase-3 fixes for the 30_07_2026 main->dev sync: 1) megatron/core/inference/engines/dynamic_engine.py — the merge dropped two main-only lines (CG admission gating, #4870) that dev lacked, causing 'NameError: name InferenceBatchDimensions is not defined' at class definition and breaking import of the whole megatron.core.inference package. Restored the batch_dimensions_utils import (CUDAGraphBatchDimensionBuilder, InferenceBatchDimensions; both used) and 'self.cuda_graph_all_prefills = inference_config.cuda_graph_all_prefills'. File is now byte-identical to origin/main. 2) megatron/training/training.py — taking main's version reintroduced the old symbol name 'get_hybrid_data_context_parallel_groups', which dev renamed to 'get_dynamic_data_context_parallel_groups' in megatron/core/parallel_state.py. This broke 'Check imports for megatron.training' with 'ImportError: cannot import name get_hybrid_data_context_parallel_groups'. Renamed both the import (line 95) and the call site (hybrid_cp_group_func=..., line ~2499) to the dev name, matching origin/dev. A full stale-import audit of megatron/training found no other renamed symbols. 3) Three golden-value JSON files carried an inherited 'iteration-time'.'values'.'1' = 'nan' warmup placeholder rejected by the new main-only lint tools/check_golden_values.py (#5989). The runtime iteration-time comparison only evaluates a steady-state window (steps 5-20 or 30-45), so step 1 is never compared; replaced with a finite 0.0: - bert/bert_mcore_tp1_pp2/golden_values_dev_dgx_h100.json - bert/bert_mcore_tp1_pp4_vp2/golden_values_dev_dgx_h100.json - moe/deepseek_proxy_fsdp_ep2_fsdp2/golden_values_dev_dgx_gb200.json Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>

Signed-off-by: [Your Name] <your.email@example.com> Signed-off-by: Ajay Balasa <abalasa@nvidia.com> Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com> Co-authored-by: Ajay Balasa <abalasa@nvidia.com> Co-authored-by: [Co-author Name] <coauthor.email@example.com>

Signed-off-by: [Your Name] <your.email@example.com> Signed-off-by: Ajay Balasa <abalasa@nvidia.com> Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com> Co-authored-by: Ajay Balasa <abalasa@nvidia.com> Co-authored-by: [Co-author Name] <coauthor.email@example.com> Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>

balasaajay and others added 2 commits July 23, 2026 07:07

fix(ci): validate goldens in merge queue

a6eb587

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

svcnemo-autobot requested a review from a team as a code owner July 23, 2026 07:37

svcnvidia-nemo-ci marked this pull request as draft July 23, 2026 07:38

copy-pr-bot Bot temporarily deployed to public July 23, 2026 07:38 Inactive

copy-pr-bot Bot temporarily deployed to test July 23, 2026 07:38 Inactive

ko3n1g marked this pull request as ready for review July 23, 2026 07:39

ko3n1g approved these changes Jul 23, 2026

View reviewed changes

svcnvidia-nemo-ci requested a review from a team July 23, 2026 07:39

svcnvidia-nemo-ci added the Approved All necessary approvals have been made label Jul 23, 2026

ko3n1g merged commit bb5647a into NVIDIA:main Jul 23, 2026
27 of 30 checks passed

ko3n1g mentioned this pull request Jul 23, 2026

ci: Add validation for updated golden-value JSON files in CI workflow #5959

Closed

6 tasks

github-actions Bot added the complexity: low label Jul 23, 2026

philipcmonk mentioned this pull request Jul 23, 2026

Add statistics logging for params and activations #5492

Closed

6 tasks

svcnvidia-nemo-ci mentioned this pull request Aug 13, 2026

chore: nightly sync main into dev (13_08_2026) #6522

Closed

Navigation Menu

Uh oh!

fix(ci): AUT-957 support golden checks in merge queue - #5989

fix(ci): AUT-957 support golden checks in merge queue#5989
ko3n1g merged 2 commits into
NVIDIA:mainfrom
svcnemo-autobot:fix/golden-values-event-context

svcnemo-autobot commented Jul 23, 2026

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

copy-pr-bot Bot commented Jul 23, 2026

Uh oh!

svcnemo-autobot commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

svcnemo-autobot commented Jul 23, 2026

Background

What changed

Details

Tested

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

copy-pr-bot Bot commented Jul 23, 2026

Uh oh!

svcnemo-autobot commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants