Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions .github/workflows/retry_daily_ci.yml

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blanket-retrying all failed jobs can silently mask real failures — a test that fails due to a genuine bug but passes on retry due to non-determinism would go unnoticed. I'd prefer we either scope retries to known-transient failure patterns (dependency resolution, docker pulls, credential issues) or evaluate each test to see if each of the test is non-deterministic and add retry for those test. As-is, this optimizes for green CI at the cost of CI trustworthiness.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The next revision adds skip list functionality which can be used to not retry a given suite. Whenever we have fuzz tests in this library, we shall use that skip list.
Except for dedicated fuzz tests, retry should be fine for general use.

Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# This workflow automatically re-runs failed jobs from the Daily CI and PR CI.
# It triggers once when either workflow completes, and if any jobs failed,
# it re-runs only the failed jobs — but ONLY if no failures are in the
# skip list below. If any failure matches the skip list (e.g., fuzz tests),
# the retry is skipped to avoid masking non-deterministic test failures.
# It only retries once to avoid infinite loops.
name: Retry Failed CI

on:
workflow_run:
workflows: ["Daily CI", "PR CI"]
types:
- completed

jobs:
retry:
if: github.event.workflow_run.conclusion == 'failure'
runs-on: ubuntu-latest
permissions:
actions: write
steps:
- name: Check failures and retry if appropriate
uses: actions/github-script@v7
with:
script: |
const runId = context.payload.workflow_run.id;

// Check if this is already a retry to avoid infinite loops
const run = await github.rest.actions.getWorkflowRun({
owner: context.repo.owner,
repo: context.repo.repo,
run_id: runId,
});
if (run.data.run_attempt > 1) {
console.log('Already a retry (attempt ' + run.data.run_attempt + '). Skipping.');
return;
}

// Jobs that should NOT be retried. These are non-deterministic tests
// (e.g., fuzz tests) where a retry could mask a real failure.
// Use job name prefixes/substrings to match.
const skipPatterns = [

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assumes non deterministic job name will always include "fuzz". String matching on job name is too fragile.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer an allowlist (opt in retries) over a denylist (opt out retries).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are in situation where a any random test fails in first run and passes in second run masking the bug for the first run, we are doomed. This behavior MUST always be an exception or an explicit choice where we add fuzz tests. The skip list are generally made for exception cases and which is what is implemented here.

'fuzz',
];

// Get all jobs for this run
const jobs = await github.paginate(
github.rest.actions.listJobsForWorkflowRun,
{ owner: context.repo.owner, repo: context.repo.repo, run_id: runId }
);

const failedJobs = jobs.filter(j => j.conclusion === 'failure');
console.log(`Found ${failedJobs.length} failed job(s):`);
failedJobs.forEach(j => console.log(` - ${j.name}`));

// Check if any failed job matches the skip list
const skipped = failedJobs.filter(job => {
return skipPatterns.some(pattern =>
job.name.toLowerCase().includes(pattern.toLowerCase())
);
});

if (skipped.length > 0) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a unit test and a fuzz test fails, non of them will be retried. I prefer retrying unit test and skipping fuzz test.

console.log('Failures in skip-listed jobs found. Skipping retry:');
skipped.forEach(j => console.log(` - ${j.name}`));
return;
}

console.log('No skip-listed failures. Re-running failed jobs...');
await github.rest.actions.reRunWorkflowFailedJobs({
owner: context.repo.owner,
repo: context.repo.repo,
run_id: runId,
});
Loading