Skip to content

feat: long convergence resiliency for release tests - #4335

Merged
ko3n1g merged 5 commits into
NVIDIA:mainfrom
ko3n1g:ko3n1g/feat/long-convergence-resiliency
Apr 16, 2026
Merged

feat: long convergence resiliency for release tests#4335
ko3n1g merged 5 commits into
NVIDIA:mainfrom
ko3n1g:ko3n1g/feat/long-convergence-resiliency

Conversation

@ko3n1g

@ko3n1g ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR improves the resiliency and observability of long-running release test pipelines.

Changes

  • ft_launcher for all release tests — adds LAUNCHER: ft_launcher to all 16 _release `model_config.yaml` files so release runs use `ft_launcher` instead of `torchrun`
  • Slack alerts during release runs — adds `send_slack_alert()` to `launch_jet_workload.py` that calls `notify.py` at three checkpoints (training finished, pipeline failed/retrying, max attempts exhausted); release-only, all other test types are unaffected
  • Golden values — adds/updates golden value files for bert, gpt, mixtral, and t5 release test cases from pipeline 47719739

Affected test cases (ft_launcher)

Model Test case
BERT `bert_release`, `bert_release_sm`
GPT `gpt3_15b_8t_release`, `gpt3_15b_8t_release_gb200`, `gpt3_15b_8t_release_sm`, `gpt3_15b_8t_release_sm_gb200`
Mixtral `deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release{,_sm}`, `deepseekv3_proxy_flex_tp2pp2emp16etp1cp1_gb_200_release{,_sm}`, `mixtral_8x22b_tp2pp8ep8vpp1_release`, `mixtral_8x7b_alltoall_tp2pp4ep4_release{,_sm}`, `mixtral_8x7b_tp1pp4ep8vpp8_release`
T5 `t5_release`, `t5_release_sm`

Example diff (ft_launcher)

```yaml
TEST_TYPE: "release"
+LAUNCHER: ft_launcher
MODEL_ARGS:
```

Slack alert flow

```
release run iteration
├── training finished → notify.py (pipeline-context: "<test_case> | iteration=N | attempt=N | training finished")
├── pipeline failed, retry → notify.py (pipeline-context: "<test_case> | iteration=N | attempt=N | pipeline failed, retrying")
└── max attempts exhausted → notify.py (pipeline-context: "<test_case> | iteration=N | attempt=N | max attempts exhausted")
```

Test plan

  • CI release pipelines pass with ft_launcher
  • Slack alerts fire at the correct checkpoints

Add LAUNCHER: ft_launcher to all 16 release-type model_config.yaml
files so that release runs use ft_launcher instead of torchrun.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Apr 16, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ko3n1g

ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@svcnvidia-nemo-ci svcnvidia-nemo-ci added this to the Core 0.16 milestone Apr 16, 2026
@ko3n1g
ko3n1g marked this pull request as ready for review April 16, 2026 10:36
@svcnvidia-nemo-ci
svcnvidia-nemo-ci requested a review from a team April 16, 2026 10:36
Add golden_values_dev_dgx_gb200.json for the
deepseekv3_proxy_flex_tp2pp2emp16etp1cp1_gb_200_release test case.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g

ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

Downloaded from GitLab pipeline 47719739. Skipped files with only 100
steps (not representative of full release runs).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g

ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

Add send_slack_alert() to launch_jet_workload.py that calls notify.py
at three checkpoints exclusive to release-type tests: training finished,
pipeline failed (retrying), and max attempts exhausted.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g

ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@ko3n1g ko3n1g changed the title feat: enable ft_launcher for all release test cases feat: long convergence resiliency for release tests Apr 16, 2026
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g

ko3n1g commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@ko3n1g
ko3n1g enabled auto-merge April 16, 2026 12:36
@ko3n1g
ko3n1g added this pull request to the merge queue Apr 16, 2026
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/24514520678

Merged via the queue into NVIDIA:main with commit f098fe8 Apr 16, 2026
116 of 123 checks passed
@ko3n1g
ko3n1g deleted the ko3n1g/feat/long-convergence-resiliency branch April 16, 2026 16:14
yangbofun pushed a commit to xlm-research/Megatron-LM that referenced this pull request May 22, 2026
Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
yhgalaxy pushed a commit to yhgalaxy/Megatron-LM that referenced this pull request Jun 17, 2026
Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: yhgalaxy <yhgalaxy@outlook.com>
jon-barker pushed a commit to jon-barker/Megatron-LM that referenced this pull request Jul 10, 2026
Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Jon Barker <jbarker@aws-cmh-slurm-1-vscode-02.cm.cluster>
terminator123 pushed a commit to 021ai/Megatron-LM that referenced this pull request Aug 3, 2026
Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants