Skip to content

ci: retry transient network fetch failures in lint workflow - #36563

Merged
mateo-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_lint_fetch_retry
Aug 11, 2026
Merged

ci: retry transient network fetch failures in lint workflow#36563
mateo-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_lint_fetch_retry

Conversation

@mateo-berri

@mateo-berri mateo-berri commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • One runner network blip fails the whole required lint check
  • The job died in 12 seconds before linting anything
  • A human had to notice and press re-run by hand

How it solves it:

  • Wraps each network fetch step in a 3-attempt retry
  • Backs off 15 then 30 seconds, like the uv setup retries

User Flow

Before: one runner network blip fails the required lint check before any lint runs, and the PR stays blocked until someone re-runs the job by hand

  1. They push a commit to their open PR and watch https://github.com/BerriAI/litellm/pull/{number}/checks
  2. The required "LiteLLM Linting / lint" check turns red after about 12 seconds
  3. The check's log ends at the very first fetch with "fatal: unable to access 'https://github.com/BerriAI/litellm/': server certificate verification failed", before any lint ran
  4. Their code was never the problem, but the failing required check blocks merge
  5. They press "Re-run failed jobs" and wait another 7 minutes for the same commit to pass

After: the same blip is absorbed inside the job and the check goes green in one run

  1. They push a commit to their open PR and watch https://github.com/BerriAI/litellm/pull/{number}/checks
  2. The same fetch hits the same certificate error, and the step log shows it pausing 15 seconds and trying again
  3. The retry succeeds and the run continues into the actual lint gates
  4. The required "LiteLLM Linting / lint" check turns green in one run, with no manual re-run

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Reproduction, captured at d608948 on PR #36543: https://github.com/BerriAI/litellm/actions/runs/31518654317/job/93869992555 is the real transient failure that motivated this. The required lint job died 12 seconds in, on its first network step

Fetch gate base (merge-base with target branch)
  git fetch --no-tags --depth=1 origin "$MERGE_BASE"
  fatal: unable to access 'https://github.com/BerriAI/litellm/': server certificate verification failed. CAfile: none CRLfile: none
  ##[error]Process completed with exit code 128.

Retry semantics, captured at e9156af with the exact helper line from the workflow: a command that fails once then succeeds is absorbed, and a command that always fails still fails the step after three attempts

$ bash -ec '
retry() { "$@" || { sleep 15; "$@"; } || { sleep 30; "$@"; }; }
attempts=/tmp/attempts; rm -f $attempts
flaky() { echo x >> $attempts; [ $(wc -l < $attempts) -ge 2 ] && echo SUCCESS-OUTPUT; [ $(wc -l < $attempts) -ge 2 ]; }
OUT=$(retry flaky)
echo "captured: $OUT after $(wc -l < $attempts) attempts"'
captured: SUCCESS-OUTPUT after 2 attempts

$ bash -ec 'retry() { "$@" || { "$@"; } || { "$@"; }; }; retry false; echo should-not-print'; echo "exit: $?"
exit: 1

This PR's own required lint check ran the retry-wrapped workflow at e9156af and passed in one run: https://github.com/BerriAI/litellm/actions/runs/31523584763. The happy path is unchanged with zero added latency, since the retries only engage after a failure

Type

🚄 Infrastructure

Caveats (if any)

  • Only this workflow's fetch steps are covered
  • actions/checkout's own network calls are not retried

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

@mateo-berri
mateo-berri requested a review from a team August 11, 2026 18:36
@greptile-apps

greptile-apps Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR makes the lint workflow more resilient to transient network failures by adding three-attempt retries with 15- and 30-second backoff.

  • Retries the GitHub compare API request and merge-base fetch used by lint gates.
  • Retries base fetching for the non-gating budget-ratchet job.
  • Retries the full-history fetch required by the ggshield secret scan.

Confidence Score: 5/5

The PR appears safe to merge because exhausted retries still fail their steps and successful retries preserve the existing fetch and lint behavior.

The retry helper performs at most three attempts, returns the final command’s failure status when all attempts fail, and leaves the existing merge-base and secret-scan sequencing intact.

Important Files Changed

Filename Overview
.github/workflows/test-linting.yml Adds bounded retries around three existing network-fetch paths while preserving final failure propagation and downstream validation.

Reviews (1): Last reviewed commit: "ci: retry transient network fetch failur..." | Re-trigger Greptile

@codecov

codecov Bot commented Aug 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@mateo-berri
mateo-berri enabled auto-merge August 11, 2026 19:19
@mateo-berri
mateo-berri merged commit 953bcad into litellm_internal_staging Aug 11, 2026
79 checks passed
@mateo-berri
mateo-berri deleted the litellm_lint_fetch_retry branch August 11, 2026 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants