Skip to content

test(release): add release goldens for deepseekv3/nemotron3 and set tp2pp2 exit-interval - #4932

Merged
ko3n1g merged 3 commits into
NVIDIA:mainfrom
ko3n1g:ko3n1g/test/release-golden-values
May 25, 2026
Merged

test(release): add release goldens for deepseekv3/nemotron3 and set tp2pp2 exit-interval#4932
ko3n1g merged 3 commits into
NVIDIA:mainfrom
ko3n1g:ko3n1g/test/release-golden-values

Conversation

@ko3n1g

@ko3n1g ko3n1g commented May 22, 2026

Copy link
Copy Markdown
Contributor
Claude summary

Adds release-test golden values and one config update.

Changes

  • New tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release/golden_values_dev_dgx_h100.json — generated from convergence tensorboard logs at /lustre/fsw/portfolios/coreai/projects/coreai_dlalgo_mcore/release-testing/mcore-v0.17.1/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release/1/tensorboard for 47683 training iters.
  • New tests/functional_tests/test_cases/nemotron/nemotron3_super_release_g200/golden_values_dev_dgx_gb200.json.
  • Modified tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp2pp2emp16etp1cp1_gb_200_release/model_config.yaml — added --exit-interval: 47683 to match the convergence target.

How the goldens were generated

ssh cs-cw-dfw-login-01 'cd ~/mlm_gv && PYTHONPATH=$HOME/mlm_gv uv run --no-project \
  --with tensorboard --with click --with pydantic --with numpy \
  python -m tests.functional_tests.python_test_utils.get_test_results_from_tensorboard_logs \
    --logs-dir /lustre/fsw/portfolios/coreai/projects/coreai_dlalgo_mcore/release-testing/mcore-v0.17.1/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release/1/tensorboard \
    --train-iters 47683 \
    --output-path ~/mlm_gv/golden_values_dev_dgx_h100.json \
    --is-convergence-test'

@ko3n1g
test(release): add release goldens for deepseekv3/nemotron3 and set t…
e378947 
…p2pp2 exit-interval

- add golden_values_dev_dgx_h100.json for mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release (47683 iters)
- add golden_values_dev_dgx_gb200.json for nemotron/nemotron3_super_release_g200
- set --exit-interval: 47683 for mixtral/deepseekv3_proxy_flex_tp2pp2emp16etp1cp1_gb_200_release

Signed-off-by: oliver könig <okoenig@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented May 22, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ko3n1g

ko3n1g commented May 22, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@ko3n1g
ko3n1g marked this pull request as ready for review May 22, 2026 13:04
@svcnvidia-nemo-ci
svcnvidia-nemo-ci requested a review from a team May 22, 2026 13:04
@ko3n1g
fix
244c140 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ko3n1g enabled auto-merge May 22, 2026 13:49
@ko3n1g
ko3n1g added this pull request to the merge queue May 22, 2026
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented May 22, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/26292087055

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 22, 2026
@ko3n1g
ko3n1g added this pull request to the merge queue May 22, 2026
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented May 22, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/26294512496

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 22, 2026
@ko3n1g
ko3n1g added this pull request to the merge queue May 25, 2026
@svcnvidia-nemo-ci

svcnvidia-nemo-ci commented May 25, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/26387793918

Merged via the queue into NVIDIA:main with commit be2b2cd May 25, 2026
72 of 73 checks passed
@ko3n1g
ko3n1g deleted the ko3n1g/test/release-golden-values branch May 25, 2026 07:41
santhnm2 pushed a commit to santhnm2/Megatron-LM that referenced this pull request May 26, 2026
@ko3n1g @santhnm2
test(release): add release goldens for deepseekv3/nemotron3 and set t…
767a0a0 
…p2pp2 exit-interval (NVIDIA#4932)

Signed-off-by: oliver könig <okoenig@nvidia.com>
Victarry pushed a commit to yanring/Megatron-LM that referenced this pull request May 27, 2026
Dennis Liu
Merge remote-tracking branch 'origin/main' into pr3431-conflict-fix
535f9d9 
* origin/main: (50 commits)
  Drain predecessor reduce-scatter at dispatch time (NVIDIA#4940)
  ci: Add allow_failure flag to gpt and moe recipes that are failing in nightlies (NVIDIA#4905)
  fix(tests): initialize num_microbatches calculator in vision cudagraph tests (NVIDIA#4986)
  test: re-enable test_pp2_create_cudagraphs_first_stage on TE 2.15+ (NVIDIA#4985)
  ci: Add support for MBridge job gating based on PR labels  (NVIDIA#4926)
  test(ci): re-enable 8experts2parallel_multi_dist_optimizer_instances_1node (NVIDIA#4984)
  test: re-enable paged stashing MoE tests (NVIDIA#4978)
  Fix elastification unwrap_model import (NVIDIA#4972)
  Avoid offsetting functional test master port (NVIDIA#4973)
  test: enable NVTE_CUTEDSL_FUSED_GROUPED_MLP via pytest fixture (NVIDIA#4931)
  chore(beep boop 🤖): Bump  (main) (2026-05-25)
  test(release): add release goldens for deepseekv3/nemotron3 and set tp2pp2 exit-interval (NVIDIA#4932)
  Fix `get_batch` return order to ignore BlendedDataset provenance fields (NVIDIA#4952)
  ci: restore perf test torchrun logs (NVIDIA#4951)
  Various training utils (NVIDIA#4872)
  ci: Update training script paths in BERT and T5 (NVIDIA#4939)
  [MXFP8/FP4-param-gather] Post processing after forced param AG in eval (NVIDIA#4562)
  Fix mxfp8 param gather numerical issue when DP overlap is off (NVIDIA#4800)
  Add TEFusedDenseMLP for Dense+Grouped GEMM fusion on SM100+ (NVIDIA#4318) (NVIDIA#4786)
  Fix paged stashing test submodules lookup (NVIDIA#4925)
  ...

# Conflicts:
#	megatron/training/training.py
janEbert pushed a commit to janEbert/Megatron-LM that referenced this pull request Jun 2, 2026
@ko3n1g @janEbert
test(release): add release goldens for deepseekv3/nemotron3 and set t…
ba5a84b 
…p2pp2 exit-interval (NVIDIA#4932)

Signed-off-by: oliver könig <okoenig@nvidia.com>
mathemakitten pushed a commit to mathemakitten/Megatron-LM that referenced this pull request Jun 12, 2026
@ko3n1g @mathemakitten
test(release): add release goldens for deepseekv3/nemotron3 and set t…
26cc83f 
…p2pp2 exit-interval (NVIDIA#4932)

Signed-off-by: oliver könig <okoenig@nvidia.com>
yhgalaxy pushed a commit to yhgalaxy/Megatron-LM that referenced this pull request Jun 17, 2026
@ko3n1g @yhgalaxy
test(release): add release goldens for deepseekv3/nemotron3 and set t…
b80d0a8 
…p2pp2 exit-interval (NVIDIA#4932)

Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: yhgalaxy <yhgalaxy@outlook.com>
jon-barker pushed a commit to jon-barker/Megatron-LM that referenced this pull request Jul 10, 2026
@ko3n1g Jon Barker
test(release): add release goldens for deepseekv3/nemotron3 and set t…
c6afcba 
…p2pp2 exit-interval (NVIDIA#4932)

Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Jon Barker <jbarker@aws-cmh-slurm-1-vscode-02.cm.cluster>
terminator123 pushed a commit to 021ai/Megatron-LM that referenced this pull request Aug 3, 2026
@ko3n1g @terminator123
test(release): add release goldens for deepseekv3/nemotron3 and set t…
4506f57 
…p2pp2 exit-interval (NVIDIA#4932)

Signed-off-by: oliver könig <okoenig@nvidia.com>
svcnvidia-nemo-ci pushed a commit to dimapihtar/Megatron-LM that referenced this pull request Aug 4, 2026
@ko3n1g @dimapihtar
test(release): add release goldens for deepseekv3/nemotron3 and set t…
b820098 
…p2pp2 exit-interval (NVIDIA#4932)

Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Dmytro Pykhtar <dpykhtar@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chtruong814 chtruong814 chtruong814 approved these changes

Assignees

No one assigned

Labels

complexity: high

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ko3n1g @svcnvidia-nemo-ci @chtruong814