Add nightly sync workflow from main to dev by Phlip79 · Pull Request #4165 · NVIDIA/Megatron-LM

Phlip79 · 2026-04-06T21:36:08Z

Summary

Adds a GitHub Actions workflow that syncs main into dev using Claude Code Action. Currently manual-dispatch only (workflow_dispatch) for testing. A cron schedule will be added once validated.

Test workloads can be found here.

How it works

Closes stale sync PRs — any open PR with a main2dev/* branch is closed
Checks if sync is needed — skips if main has no new commits ahead of dev
Claude Code merges and resolves conflicts:
- git merge -X theirs to prefer main on textual conflicts
- Targeted overrides for specific files with known semantic conflicts
- Preserves dev-only features (new classes, tests, modules) that don't conflict with main
Creates a draft PR on branch main2dev/dd_mm_yyyy
Triggers CI via /ok to test <sha> comment using secrets.PAT
Iterates on failures — Claude reads CI logs, fixes issues, pushes, and re-triggers until all tests pass
Marks PR ready for review when CI is green

Design decisions

Main wins on conflicts, dev features preserved: -X theirs for merge conflicts, but dev-only additions (new classes, modules) are kept through normal git merge
Targeted overrides: Only specific files with confirmed semantic conflicts are replaced with main's version (training.py, pyproject.toml, etc.)
No retry limit: Claude keeps fixing until CI passes (6-hour job timeout is the only guard)
Claude Opus 4.6: Most capable model for complex merge resolution and test fixes

Automates a daily sync of main into dev at 4:00 AM China time (20:00 UTC) on weekdays. The workflow: - Closes any previous unmerged sync PR before creating a new one - Merges main into dev, preferring main on all conflicts (-X theirs) - Takes main's pyproject.toml and uv.lock for lock-file consistency - Creates a draft PR labeled "nightly-sync" for human review Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

copy-pr-bot · 2026-04-06T21:36:13Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

After merging main into dev, also take main's versions of key integration files (training.py, initialize.py, utils.py, data_samplers.py, layer_wise_optimizer.py) that have deep semantic dependencies on conflict-resolved modules. This prevents runtime AttributeError and TypeError from auto-merged dev code referencing APIs removed in main. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

This workflow belongs on the main branch (PR NVIDIA#4165), not the sync PR. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Instead of a fragile hardcoded file list, the workflow now takes main's version for ALL files that exist in both branches after the merge. This prevents semantic conflicts where auto-merged dev code references APIs that main renamed/removed. Dev-only new files are preserved. Also adds a black formatting step to fix trailing whitespace issues that appear when main's files differ from dev's formatting baseline. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The sync workflow now also restores files that exist in main but were deleted or renamed in dev (e.g. hybrid_cp_schedule.py). Without this, main's code that imports these files will fail with ModuleNotFoundError. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Redesigned the main-to-dev nightly sync to use Claude Code Action for: - Merge conflict resolution (prioritizing main) - Iterative CI failure fixing (no retry limit) - Automatic "/ok to test" CI triggering - Marking PR ready for review when all tests pass Key changes: - Schedule: daily at 20:00 UTC (all days) - Branch pattern: main2dev/dd_mm_yyyy - Uses secrets.PAT for push/PR/CI operations - Uses svcnvidia-nemo-ci as git committer - Closes previous unmerged sync PRs by branch pattern - Claude loops indefinitely: fix → push → trigger CI → wait → check → repeat Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Key changes: - Remove blanket "take main for all shared files" — preserves dev features (HyperConnection, CheckpointManager, etc.) - Only override specific files with confirmed semantic conflicts - Tell Claude to read CLAUDE.md and build-and-test skill first - Goal-oriented prompt instead of prescriptive shell commands - Add concurrency group to prevent overlapping runs - Add Agent to allowedTools for complex fixes - Teach Claude to surgically restore missing classes rather than replacing entire files - Remove trigger_phrase (not needed for direct_prompt) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Remove the cron schedule. The workflow can be re-enabled with a schedule trigger once the manual dispatch has been validated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Phlip79 · 2026-04-07T23:29:09Z

/claude review

claude

LGTM

Phlip79 · 2026-04-07T23:32:43Z

/ok to test d929db9

…nt step Add data_schedule.py as override candidate, run pylint on megatron/core files after formatting, poll CI every 2min with per-job failure checking, and improve guidance for files with divergent main/dev classes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Phlip79 · 2026-04-08T18:32:59Z

/ok to test 9826d45

Phlip79 · 2026-04-08T22:09:07Z

/ok to test 93581f7

Signed-off-by: oliver könig <okoenig@nvidia.com>

FDecaYed · 2026-04-10T13:09:40Z

The general setup looks great. Here are some issues and suggestions:

In order to review what I do was git show --remerge-diff and only check the conflicts that agent help resolved. If the PR is not a single merge commit but also couple fixes on top, the command to generate conflict solving diff is more complex but still same idea apply. I think this can both be one of the output to the PR so people can check, but agent can maybe utilize it in the process(not sure how yet).
For above diff, the raw diff is still pretty long. For now I've been mainly inspecting changes under megatron/core, but maybe some other places need to be inspect too. Also maybe claude can organize it into file or categories, or even figure out the related commit/PR.
to improve this, we can also let it try merge a thing we manually merge and tested(maybe last sync, or we create another smaller one with less PR). this provide another directly check what the agent is doing

As of the real PR it attempted, #4213, I found couple (potential) issues

the file got deleted by dev got added back by the merge. some are file/tests dev just removed since it is unrelated to dev features and that simplify testing. But there are also some real feature related deletion where the pull added file back. I think this is just git merge not tracking deleted file correctly
similar thing happened in code level. lets say dev have PR1 added something, and PR2 follow up and changes it on top. but currently only PR1 is merged into main. with --theirs merge, PR1 now overwrite dev back and things in PR2 might be loss or break. this is also why sometimes we develop a series of PR so it is logically step by step small reviewable change, but we prefer to merge these squashed into main in single PR.
the 2 error tests. at a glance I also don't see why it is passing in dev but failing after pull main. all related code path(both core and test) seems identical. and after checking dev CI I think agent's conclusion is kind of true. these were broken tests in dev but dev CI doesn't run it. Since now we favor main during merge for package version and test setup, these are now exposed

Changes based on review of PR NVIDIA#4213 and NVIDIA#4165: 1. Don't blindly restore files deleted by dev — check whether main's code actually imports them before restoring. Dev may have intentionally removed files to simplify testing. 2. Include remerge-diff in PR body so reviewers can inspect only the conflict resolutions without wading through the full merge diff. 3. Add warning about squashed-merge chains: when dev has PR1→PR2→PR3 but only PR1 is on main, -X theirs may overwrite PR2/PR3's follow-up changes. Claude should check git log on dev for follow-up commits before favoring main. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Tighten the nightly sync workflow prompt and skill so the bot cannot mark the PR ready while non-exempt checks are pending or failing. - Replace the loose Phase 4 gate with a strict "all terminal, all green" rule: every non-exempt required check in statusCheckRollup must be COMPLETED + {SUCCESS, SKIPPED, NEUTRAL}. Queued or in_progress is never acceptable. - Explicitly ban background tasks (Bash/Agent run_in_background, ScheduleWakeup, &, nohup, disown, setsid, tail -f on background output) and explain why: the GitHub Actions step process owns the shell and is destroyed on exit, so backgrounded work cannot resume. - Anchor CI polling to `gh pr view --json statusCheckRollup` — the actions/runs/.../jobs endpoint alone misses external status contexts (GitLab CI, copy-pr-bot, etc.), which was the failure mode on run 24800621116. - Distinguish outer loop (agent's sequence of tool calls) from inner loop (single blocking Bash call); provide a validated bash template that normalizes CheckRun and StatusContext entries, uses Nemo_CICD_Test as a sentinel to close the empty-rollup edge case, and classifies against the exempt regex. - Align exempt list with actual check names seen on PR NVIDIA#4159: approval (codeowners-approval, check-approval, multi-approval-bot-summary, is-not-external-contributor), coverage (Coverage (unit-test), Coverage_Fake), docs (build-docs / Build docs, build-docs-summary).

# Conflicts: # pyproject.toml # skills/nightly-sync/SKILL.md # uv.lock

This branch targets main and should only add the nightly-sync workflow and skill. It should not touch pyproject.toml, uv.lock, or Dockerfile.ci.dev. The prior commit 1844bb0 (take dev's versions) was correct for the sync bot's runtime behavior (merging main->dev) but wrong to include in a PR to main — dev's dep pins, Python version floor, and transformer-engine range must not propagate back to main. Restore main's versions here.

The first sync run (PR NVIDIA#4436) violated two invariants that the prior prompt treated as soft guidance: 1. .github/CODEOWNERS was modified (+45/-1) — the -X theirs merge pulled main's detailed CODEOWNERS on top of dev's one-liner. 2. gh pr ready was invoked with a known non-exempt failure (gpt/gpt3_mcore_te_tp2_pp2_mhc) AND ~54 functional tests still QUEUED on H100/GB200. Changes: - State the CODEOWNERS rule as a HARD invariant in the Files to Override section: never modify, restore via `git checkout origin/dev -- .github/CODEOWNERS`, verify diff is empty. - Add a new "Pre-push invariant checks" bash block that runs before every push (initial and fix-push) and exits 1 if CODEOWNERS diverges from origin/dev.

Reviewers skim the PR stats line to gauge the real code surface, but the total is dominated by golden-value JSON and uv.lock churn. Add a Phase 2 instruction to compute and include Python-only additions / deletions (via 'git diff --numstat origin/dev...HEAD -- *.py') in the PR body.

GitHub Actions cron is UTC-only and does not follow DST, so the job runs at 1 PM PST during the winter months.

Phlip79 · 2026-04-23T23:43:22Z

/ok to test 0d8ad45

opus[1m] is the Claude Code rolling alias for the latest Opus model with the 1M-context variant. It auto-resolves to the current Opus (4.7 today, 4.8+ automatically). Avoids having to bump this workflow on each Opus release.

ko3n1g · 2026-04-28T09:07:00Z

/ok to test ddab623

svcnvidia-nemo-ci · 2026-04-28T10:10:31Z

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/25046906403

Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Philip Petrakian <> Co-authored-by: oliver könig <okoenig@nvidia.com>

Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Philip Petrakian <> Co-authored-by: oliver könig <okoenig@nvidia.com> Signed-off-by: yhgalaxy <yhgalaxy@outlook.com>

Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Philip Petrakian <> Co-authored-by: oliver könig <okoenig@nvidia.com> Signed-off-by: Jon Barker <jbarker@aws-cmh-slurm-1-vscode-02.cm.cluster>

Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Philip Petrakian <> Co-authored-by: oliver könig <okoenig@nvidia.com>

Phlip79 changed the title ~~ci: add nightly sync workflow from main to dev~~ Add nightly sync workflow from main to dev Apr 6, 2026

Phlip79 added a commit to Phlip79/Megatron-LM that referenced this pull request Apr 6, 2026

chore: remove accidentally added workflow file

f07c00b

This workflow belongs on the main branch (PR NVIDIA#4165), not the sync PR. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Phlip79 and others added 6 commits April 7, 2026 04:58

ci: use claude-opus-4-6 model for nightly sync

bdacf5f

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ci: make nightly sync manual-only for initial testing

d929db9

Remove the cron schedule. The workflow can be re-enabled with a schedule trigger once the manual dispatch has been validated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Phlip79 changed the title ~~Add nightly sync workflow from main to dev~~ Add manual sync workflow from main to dev Apr 7, 2026

claude Bot approved these changes Apr 7, 2026

View reviewed changes

Phlip79 marked this pull request as ready for review April 7, 2026 23:32

Phlip79 requested a review from a team as a code owner April 7, 2026 23:32

svcnvidia-nemo-ci requested a review from a team April 7, 2026 23:32

svcnvidia-nemo-ci added this to the Core 0.16 milestone Apr 7, 2026

svcnvidia-nemo-ci added the complexity: medium label Apr 7, 2026

copy-pr-bot Bot temporarily deployed to test April 7, 2026 23:33 Inactive

copy-pr-bot Bot temporarily deployed to test April 8, 2026 18:34 Inactive

Update workflow

93581f7

copy-pr-bot Bot temporarily deployed to test April 8, 2026 22:09 Inactive

ci: add push trigger to register workflow for dispatch

6fcd721

Signed-off-by: oliver könig <okoenig@nvidia.com>

Phlip79 added 6 commits April 22, 2026 23:03

Merge NVIDIA/Megatron-LM add-nightly-sync-workflow

048f2ab

# Conflicts: # pyproject.toml # skills/nightly-sync/SKILL.md # uv.lock

ci: temp push trigger

9f488fa

ci: remove push trigger

2a75639

Phlip79 mentioned this pull request Apr 23, 2026

chore: nightly sync main into dev (22_04_2026) #4436

Merged

Phlip79 marked this pull request as ready for review April 23, 2026 23:31

Phlip79 requested a review from ko3n1g April 23, 2026 23:31

svcnvidia-nemo-ci requested a review from a team April 23, 2026 23:31

svcnvidia-nemo-ci added the complexity: medium label Apr 23, 2026

Phlip79 added 2 commits April 23, 2026 23:39

ci: schedule nightly sync to run daily at 21:00 UTC (2 PM PDT)

0d8ad45

GitHub Actions cron is UTC-only and does not follow DST, so the job runs at 1 PM PST during the winter months.

copy-pr-bot Bot temporarily deployed to test April 23, 2026 23:44 Inactive

ci: use rolling opus[1m] alias and raise --max-turns to 1500

ddab623

opus[1m] is the Claude Code rolling alias for the latest Opus model with the 1M-context variant. It auto-resolves to the current Opus (4.7 today, 4.8+ automatically). Avoids having to bump this workflow on each Opus release.

ko3n1g approved these changes Apr 28, 2026

View reviewed changes

ko3n1g enabled auto-merge April 28, 2026 09:07

svcnvidia-nemo-ci added the Approved All necessary approvals have been made label Apr 28, 2026

copy-pr-bot Bot temporarily deployed to test April 28, 2026 09:07 Inactive

ko3n1g added this pull request to the merge queue Apr 28, 2026

Merged via the queue into NVIDIA:main with commit 9c15290 Apr 28, 2026
69 checks passed

Phlip79 changed the title ~~Add manual sync workflow from main to dev~~ Add nightly sync workflow from main to dev Apr 28, 2026

Navigation Menu

Uh oh!

Add nightly sync workflow from main to dev - #4165

Add nightly sync workflow from main to dev#4165
ko3n1g merged 62 commits into
NVIDIA:mainfrom
Phlip79:add-nightly-sync-workflow

Phlip79 commented Apr 6, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Apr 6, 2026

Uh oh!

Phlip79 commented Apr 7, 2026

Uh oh!

claude Bot left a comment

Uh oh!

Phlip79 commented Apr 7, 2026

Uh oh!

Phlip79 commented Apr 8, 2026

Uh oh!

Phlip79 commented Apr 8, 2026

Uh oh!

FDecaYed commented Apr 10, 2026

Uh oh!

Phlip79 commented Apr 23, 2026

Uh oh!

ko3n1g commented Apr 28, 2026

Uh oh!

svcnvidia-nemo-ci commented Apr 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Phlip79 commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How it works

Design decisions

Uh oh!

copy-pr-bot Bot commented Apr 6, 2026

Uh oh!

Phlip79 commented Apr 7, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Phlip79 commented Apr 7, 2026

Uh oh!

Phlip79 commented Apr 8, 2026

Uh oh!

Phlip79 commented Apr 8, 2026

Uh oh!

FDecaYed commented Apr 10, 2026

Uh oh!

Phlip79 commented Apr 23, 2026

Uh oh!

ko3n1g commented Apr 28, 2026

Uh oh!

svcnvidia-nemo-ci commented Apr 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Phlip79 commented Apr 6, 2026 •

edited

Loading