-
Notifications
You must be signed in to change notification settings - Fork 20
chore(java): retry failed workflow #2229
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| # This workflow automatically re-runs failed jobs from the Daily CI and PR CI. | ||
| # It triggers once when either workflow completes, and if any jobs failed, | ||
| # it re-runs only the failed jobs — but ONLY if no failures are in the | ||
| # skip list below. If any failure matches the skip list (e.g., fuzz tests), | ||
| # the retry is skipped to avoid masking non-deterministic test failures. | ||
| # It only retries once to avoid infinite loops. | ||
| name: Retry Failed CI | ||
|
|
||
| on: | ||
| workflow_run: | ||
| workflows: ["Daily CI", "PR CI"] | ||
| types: | ||
| - completed | ||
|
|
||
| jobs: | ||
| retry: | ||
| if: github.event.workflow_run.conclusion == 'failure' | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| actions: write | ||
| steps: | ||
| - name: Check failures and retry if appropriate | ||
| uses: actions/github-script@v7 | ||
| with: | ||
| script: | | ||
| const runId = context.payload.workflow_run.id; | ||
|
|
||
| // Check if this is already a retry to avoid infinite loops | ||
| const run = await github.rest.actions.getWorkflowRun({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| run_id: runId, | ||
| }); | ||
| if (run.data.run_attempt > 1) { | ||
| console.log('Already a retry (attempt ' + run.data.run_attempt + '). Skipping.'); | ||
| return; | ||
| } | ||
|
|
||
| // Jobs that should NOT be retried. These are non-deterministic tests | ||
| // (e.g., fuzz tests) where a retry could mask a real failure. | ||
| // Use job name prefixes/substrings to match. | ||
| const skipPatterns = [ | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This assumes non deterministic job name will always include "fuzz". String matching on job name is too fragile.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd prefer an allowlist (opt in retries) over a denylist (opt out retries).
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If we are in situation where a any random test fails in first run and passes in second run masking the bug for the first run, we are doomed. This behavior MUST always be an exception or an explicit choice where we add fuzz tests. The skip list are generally made for exception cases and which is what is implemented here. |
||
| 'fuzz', | ||
| ]; | ||
|
|
||
| // Get all jobs for this run | ||
| const jobs = await github.paginate( | ||
| github.rest.actions.listJobsForWorkflowRun, | ||
| { owner: context.repo.owner, repo: context.repo.repo, run_id: runId } | ||
| ); | ||
|
|
||
| const failedJobs = jobs.filter(j => j.conclusion === 'failure'); | ||
| console.log(`Found ${failedJobs.length} failed job(s):`); | ||
| failedJobs.forEach(j => console.log(` - ${j.name}`)); | ||
|
|
||
| // Check if any failed job matches the skip list | ||
| const skipped = failedJobs.filter(job => { | ||
| return skipPatterns.some(pattern => | ||
| job.name.toLowerCase().includes(pattern.toLowerCase()) | ||
| ); | ||
| }); | ||
|
|
||
| if (skipped.length > 0) { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If a unit test and a fuzz test fails, non of them will be retried. I prefer retrying unit test and skipping fuzz test. |
||
| console.log('Failures in skip-listed jobs found. Skipping retry:'); | ||
| skipped.forEach(j => console.log(` - ${j.name}`)); | ||
| return; | ||
| } | ||
|
|
||
| console.log('No skip-listed failures. Re-running failed jobs...'); | ||
| await github.rest.actions.reRunWorkflowFailedJobs({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| run_id: runId, | ||
| }); | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Blanket-retrying all failed jobs can silently mask real failures — a test that fails due to a genuine bug but passes on retry due to non-determinism would go unnoticed. I'd prefer we either scope retries to known-transient failure patterns (dependency resolution, docker pulls, credential issues) or evaluate each test to see if each of the test is non-deterministic and add retry for those test. As-is, this optimizes for green CI at the cost of CI trustworthiness.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The next revision adds skip list functionality which can be used to not retry a given suite. Whenever we have fuzz tests in this library, we shall use that skip list.
Except for dedicated fuzz tests, retry should be fine for general use.