Skip to content
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions .github/workflows/retry_daily_ci.yml

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blanket-retrying all failed jobs can silently mask real failures — a test that fails due to a genuine bug but passes on retry due to non-determinism would go unnoticed. I'd prefer we either scope retries to known-transient failure patterns (dependency resolution, docker pulls, credential issues) or evaluate each test to see if each of the test is non-deterministic and add retry for those test. As-is, this optimizes for green CI at the cost of CI trustworthiness.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The next revision adds skip list functionality which can be used to not retry a given suite. Whenever we have fuzz tests in this library, we shall use that skip list.
Except for dedicated fuzz tests, retry should be fine for general use.

Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# This workflow automatically re-runs failed jobs from the Daily CI and PR CI.
# It triggers once when either workflow completes, and if any jobs failed,
# it re-runs only the failed jobs. This handles transient failures like
# Maven Central outages, Docker pull errors, or credential issues.
# It only retries once to avoid infinite loops.
name: Retry Failed CI

on:
workflow_run:
workflows: ["Daily CI", "PR CI"]
types:
- completed

jobs:
retry:
if: github.event.workflow_run.conclusion == 'failure'
runs-on: ubuntu-latest
permissions:
actions: write
steps:
- name: Re-run failed jobs
uses: actions/github-script@v7
with:
script: |
// Check if this run was already a retry to avoid infinite loops
const run = await github.rest.actions.getWorkflowRun({
owner: context.repo.owner,
repo: context.repo.repo,
run_id: context.payload.workflow_run.id,
});
if (run.data.run_attempt > 1) {
console.log('Already a retry (attempt ' + run.data.run_attempt + '). Skipping.');
return;
}
console.log('Re-running failed jobs for run ' + context.payload.workflow_run.id);
await github.rest.actions.reRunWorkflowFailedJobs({
owner: context.repo.owner,
repo: context.repo.repo,
run_id: context.payload.workflow_run.id,
});
Loading