Skip to content

ci: add retry loop to apt-get update to handle transient mirror sync failures - #4209

Merged
ko3n1g merged 1 commit into
NVIDIA:mainfrom
ko3n1g:ko3n1g/ci/apt-get-retry
Apr 14, 2026
Merged

ci: add retry loop to apt-get update to handle transient mirror sync failures#4209
ko3n1g merged 1 commit into
NVIDIA:mainfrom
ko3n1g:ko3n1g/ci/apt-get-retry

Conversation

@ko3n1g

@ko3n1g ko3n1g commented Apr 8, 2026

Copy link
Copy Markdown
Contributor

Problem

The Install uuidgen step in .github/actions/action.yml (and Install GH CLI steps in cicd-main.yml) fail when the Ubuntu package mirror is mid-sync, returning File has unexpected size and exit code 100. This is a pure infrastructure flake — example failure: https://github.com/NVIDIA/Megatron-LM/actions/runs/24125565433/job/70393731027?pr=4195

Fix

Wrap apt-get update in a 3-attempt retry loop with a 10s back-off before each retry. If all three attempts fail, the step still exits non-zero as before.

for i in 1 2 3; do
  apt-get update && break
  echo "apt-get update failed (attempt $i/3), retrying in 10s..."
  sleep 10
done
apt-get install -y uuid-runtime

Files changed

  • .github/actions/action.ymlInstall uuidgen step (used by unit-test jobs)
  • .github/workflows/cicd-main.yml — both Install GH CLI steps (used by container-build job)

🤖 Generated with Claude Code

@ko3n1g
ko3n1g requested a review from a team as a code owner April 8, 2026 14:35
@svcnvidia-nemo-ci
svcnvidia-nemo-ci marked this pull request as draft April 8, 2026 14:35
@github-actions

github-actions Bot commented Apr 8, 2026

Copy link
Copy Markdown
Contributor

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

  1. Add the oncall reviewer (optional reviewer)
  2. Add required review teams based on your changes

See the contribution guide for more details.

@copy-pr-bot

copy-pr-bot Bot commented Apr 8, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

… sync failures

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ko3n1g force-pushed the ko3n1g/ci/apt-get-retry branch from 3a94050 to d60647f Compare April 8, 2026 14:44
@ko3n1g
ko3n1g marked this pull request as ready for review April 14, 2026 13:55
@svcnvidia-nemo-ci
svcnvidia-nemo-ci requested a review from a team April 14, 2026 13:55
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the Approved All necessary approvals have been made label Apr 14, 2026
@ko3n1g
ko3n1g enabled auto-merge April 14, 2026 14:06
@ko3n1g
ko3n1g added this pull request to the merge queue Apr 14, 2026
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/24407944547

Merged via the queue into NVIDIA:main with commit bcec618 Apr 14, 2026
95 of 108 checks passed
@ko3n1g
ko3n1g deleted the ko3n1g/ci/apt-get-retry branch April 14, 2026 16:27
yangbofun pushed a commit to xlm-research/Megatron-LM that referenced this pull request May 22, 2026
…failures (NVIDIA#4209)

Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
yhgalaxy pushed a commit to yhgalaxy/Megatron-LM that referenced this pull request Jun 17, 2026
…failures (NVIDIA#4209)

Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: yhgalaxy <yhgalaxy@outlook.com>
jon-barker pushed a commit to jon-barker/Megatron-LM that referenced this pull request Jul 10, 2026
…failures (NVIDIA#4209)

Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Jon Barker <jbarker@aws-cmh-slurm-1-vscode-02.cm.cluster>
terminator123 pushed a commit to 021ai/Megatron-LM that referenced this pull request Aug 3, 2026
…failures (NVIDIA#4209)

Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Approved All necessary approvals have been made complexity: low

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants