Skip to content

fix(ci): cap E2E lanes and auto-retry transient main failures once #10229

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yiliang114 wants to merge 1 commit into from
Closed

fix(ci): cap E2E lanes and auto-retry transient main failures once #10229

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/.size-baseline
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
31677 desktop-release.yml
2038 docs-page-action.yml
10005 dsw-swe-verified-release.yml
2061 e2e-flaky-rerun.yml
12340 e2e.yml
11394 finalize-release.yml
16647 live-host-release.yml
Expand Down
46 changes: 46 additions & 0 deletions .github/workflows/e2e-flaky-rerun.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Re-runs the failed lanes of a post-merge E2E run once before leaving main
# red. The recurring 'Main CI failed: E2E Tests' issues are overwhelmingly
# transient model-endpoint connectivity flakes, not regressions: run 32966197395
# (2026-08-26) failed a Linux shard with `connect ETIMEDOUT` to the endpoint
# and passed the identical tree on attempt 2. One automatic re-run turns those
# green again; a failure that survives both attempts stays red and still
# raises the Main CI Failure issue.
Comment on lines +6 to +7

@qwen-code-ci-bot qwen-code-ci-bot Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Critical] The Main CI Failure issue bot races this watcher, so a first-attempt flake still files and autofix-routes a "Main CI failed: E2E Tests" issue even though the re-run heals it. main-ci-failure-issue.yml subscribes to the same workflow_run completion, and its analyze guard checks only conclusion == 'failure' && head_branch == 'main' && event == 'push' — run_attempt appears nowhere in that file (verified at this commit). On a push to main where one shard flakes with connect ETIMEDOUT — the exact incident this header cites (run 32966197395) — the single attempt-1 completion event fires both watchers at once: this one starts attempt 2, while the issue bot downloads the attempt-1 logs, files the issue, applies type/bug + status/ready-for-agent + autofix/approved, and assigns the autofix bot. When attempt 2 turns green ~20-40 minutes later, nothing closes or de-routes the issue (the bot only acts on conclusion == 'failure', and no workflow in .github/ closes issues), so an autofix agent is dispatched at a healthy tree — on every absorbed flake. This comment says only "a failure that survives both attempts" raises the issue; that is false of the wiring as it stands, and before this change a filed issue at least coincided with a red main.

Witness (live state observed at this commit):

run 32966197395 → conclusion: success, run_attempt: 2, event: push, name: "E2E Tests"
issue 10186     → state: OPEN, labels: [type/bug, status/ready-for-agent], assignee: qwen-code-dev-bot

Coordinate the two responders: gate the issue bot's analyze path on the retried attempt for E2E push runs — e.g. append && (github.event.workflow_run.name != 'E2E Tests' || github.event.workflow_run.run_attempt > 1) in main-ci-failure-issue.yml, leaving SDK Python and scheduled "Qwen Code CI" untouched since neither has a rerun watcher — and decide the fallback explicitly for the case where this watcher itself never fires (expired PAT, API outage), so an attempt-1 failure is not silently unfiled. If first-attempt filing is intended to stay, correct this comment and the "reaches the issue bot untouched" note instead. main-ci-failure-issue.yml is outside this diff, so a linked follow-up is fine. If the gate is added, extend scripts/tests/main-ci-failure-issue-workflow.test.js to pin the new attempt-exclusion clause — removing the clause must turn that test red.

— qwen3.8-max via Qwen Code /review (v0.22.2)


name: 'E2E Flaky Rerun'

on:
workflow_run:
workflows: ['E2E Tests']
types: ['completed']
branches: ['main']

permissions:
contents: 'read'

jobs:
rerun-failed-lanes:
name: 'Re-run failed E2E lanes (once)'
# First-attempt push runs only: the attempt-2 completion event carries
# run_attempt == 2, so the re-run cannot re-trigger itself, and a failure
# that survives both attempts reaches the issue bot untouched. Scheduled
# and dispatched runs stay manual: they exist to surface flakiness, not to
# gate main.
if: |-
${{ github.repository == 'QwenLM/qwen-code' && github.event.workflow_run.conclusion == 'failure' && github.event.workflow_run.event == 'push' && github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.run_attempt == 1 }}
# Deliberately hosted, NOT the ECS pool — same reasoning as
# main-ci-failure-issue.yml#file_issue: a recovery job must not queue
# behind the infrastructure it is recovering from.
Comment on lines +30 to +32

@qwen-code-ci-bot qwen-code-ci-bot Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] The deliberate runs-on: 'ubuntu-latest' choice is asserted only in this comment — the new test pins the trigger, the guard, and the PAT usage, but not runs-on, so a sweep moving the job to the repo's ECS pool passes every test. Verified by mutation at this commit: swapping in the repo's own ECS fromJSON expression kept the suite 3/3 green, and that expression appears in 44 places across other workflows, so such a sweep is plausible. If the ECS pool is the reason the E2E run failed, the recovery job queues behind the dead pool it is recovering from, the re-run never fires, and main stays red with nothing reporting the watcher's stall. Pin the choice in scripts/tests/e2e-flaky-rerun-workflow.test.js:

expect(job['runs-on']).toBe('ubuntu-latest');

Acceptance check: with that assertion in place, change runs-on in this workflow and confirm the suite goes red.

— qwen3.8-max via Qwen Code /review (v0.22.2)

runs-on: 'ubuntu-latest'
timeout-minutes: 5
steps:
# Checks out nothing and runs no repository code: the job holding the
# bot PAT only calls the Actions API.
- name: 'Re-run the failed jobs'
env:
# GitHub writes use CI_BOT_PAT; keep the generated GITHUB_TOKEN
# read-only.
GH_TOKEN: '${{ secrets.CI_BOT_PAT }}'
REPO: '${{ github.repository }}'
RUN_ID: '${{ github.event.workflow_run.id }}'
run: |-
gh run rerun --failed "${RUN_ID}" --repo "${REPO}"
11 changes: 11 additions & 0 deletions .github/workflows/e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,11 @@ jobs:
e2e-test-linux:
name: 'E2E Test (Linux) - ${{ matrix.sandbox }} - shard ${{ matrix.shard }}'
runs-on: 'ubuntu-latest'
# Healthy lanes finish in ~17min. Without a cap, a lane that loses
# connectivity to the model endpoint burns every test's full 300s timeout
# (plus retries) before reporting — the 2026-08-26 run took over 4 hours
# to fail. Cap the lane so a hung endpoint surfaces within the hour.
timeout-minutes: 60
# Skip on fork PRs: forks have no access to repository secrets
# (OPENAI_*, DOCKERHUB_*), so the matrix would fail unconditionally
# and show misleading red status. Same-repo PRs run normally.
Expand Down Expand Up @@ -150,6 +155,8 @@ jobs:
e2e-test-macos:
name: 'E2E Test - macOS - shard ${{ matrix.shard }}'
runs-on: 'macos-latest'
# Same cap rationale as e2e-test-linux.
timeout-minutes: 60
# Skip on fork PRs (no secrets) — see e2e-test-linux above.
if: |-
${{ github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository }}
Expand Down Expand Up @@ -206,6 +213,8 @@ jobs:
isolated-nightly:
name: '${{ matrix.label }} (nightly)'
runs-on: 'ubuntu-latest'
# Same cap rationale as e2e-test-linux.
timeout-minutes: 60
if: |-
${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }}
continue-on-error: true
Expand Down Expand Up @@ -264,6 +273,8 @@ jobs:
web-shell-browser-regression:
name: 'web-shell Browser Regression'
runs-on: 'ubuntu-latest'
# Same cap rationale as e2e-test-linux.
timeout-minutes: 60
if: |-
${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }}
steps:
Expand Down
53 changes: 53 additions & 0 deletions scripts/tests/e2e-flaky-rerun-workflow.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/

import { readFileSync } from 'node:fs';
import { describe, expect, it } from 'vitest';
import { parse } from 'yaml';

describe('e2e flaky rerun workflow', () => {
const workflow = readFileSync(
'.github/workflows/e2e-flaky-rerun.yml',
'utf8',
);
const yml = parse(workflow);
const job = yml.jobs['rerun-failed-lanes'];

it('watches completed E2E Tests runs on main only', () => {
expect(workflow).toContain('workflow_run:');
expect(yml.on.workflow_run.workflows).toEqual(['E2E Tests']);
expect(yml.on.workflow_run.types).toEqual(['completed']);
expect(yml.on.workflow_run.branches).toEqual(['main']);
});

it('retries only first-attempt failed push runs, exactly once', () => {
// The attempt-2 completion event carries run_attempt == 2: without that
// guard the re-run could re-trigger itself forever, and with it a failure
// that survives both attempts reaches the issue bot untouched. Scheduled
// and dispatched runs exist to surface flakiness, not to gate main, so
// push is the only event that earns a retry. Pin the whole clause so a
// connective mutation fails here.
Comment on lines +31 to +32

@qwen-code-ci-bot qwen-code-ci-bot Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] This comment promises that a connective mutation fails here, but the five toContain substring checks below cannot catch one: flipping any && in the guard to || keeps every substring present, so the suite stays green. Verified by mutation at this commit — flipping the last connective to head_branch == 'main' || run_attempt == 1 left the suite 3/3 green, while the guard then parses as (A && B && C && D) || E (GitHub's && binds tighter than ||) and fires on any run with run_attempt == 1 — green runs, PR-branch runs, scheduled runs — letting the watcher re-trigger itself on every failed completion and defeating the "exactly once" invariant this test is named for. Pin the whole expression exactly instead (the |- block scalar parses to this string verbatim):

expect(job.if).toBe(
  "${{ github.repository == 'QwenLM/qwen-code' && github.event.workflow_run.conclusion == 'failure' && github.event.workflow_run.event == 'push' && github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.run_attempt == 1 }}",
);

Acceptance check: with the exact match in place, flip any && to || in e2e-flaky-rerun.yml's guard and confirm "retries only first-attempt failed push runs, exactly once" goes red.

— qwen3.8-max via Qwen Code /review (v0.22.2)

expect(job.if).toContain("github.repository == 'QwenLM/qwen-code'");
expect(job.if).toContain(
"github.event.workflow_run.conclusion == 'failure'",
);
expect(job.if).toContain("github.event.workflow_run.event == 'push'");
expect(job.if).toContain("github.event.workflow_run.head_branch == 'main'");
expect(job.if).toContain('github.event.workflow_run.run_attempt == 1');

@qwen-code-ci-bot qwen-code-ci-bot Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] This pin is a prefix substring, so a value mutation to run_attempt == 1<digit> keeps the test green while silently killing the retry. Verified by mutation at this commit: changing the guard to run_attempt == 10 left the suite 3/3 green — and since run_attempt starts at 1 and only increments on re-runs, == 10 never matches, so the auto-rerun this PR adds dies silently (e.g. after a stray edit or autofix typo) with the suite green and no other signal. The four sibling checks above are terminated by a closing quote; this one is not. The exact-match fix proposed in the other finding on this test subsumes this hole, or pin the clause on its own:

Suggested change
expect(job.if).toContain('github.event.workflow_run.run_attempt == 1');
expect(job.if).toMatch(/github\.event\.workflow_run\.run_attempt == 1(?![0-9])/);

Acceptance check: with the tightened pin, mutate the workflow guard to run_attempt == 10 and confirm "retries only first-attempt failed push runs, exactly once" goes red.

— qwen3.8-max via Qwen Code /review (v0.22.2)

});

it('re-runs only the failed jobs with the bot PAT, not repository code', () => {
// The job holding the PAT must stay a pure Actions-API call: no checkout,
// no repository code. `--failed` keeps the green lanes' results so the
// re-run covers exactly the lanes that flaked.
expect(workflow).not.toContain('actions/checkout');
expect(workflow).toContain("GH_TOKEN: '${{ secrets.CI_BOT_PAT }}'");
expect(workflow).toContain(
'gh run rerun --failed "${RUN_ID}" --repo "${REPO}"',
);
expect(workflow).toContain("RUN_ID: '${{ github.event.workflow_run.id }}'");
});
});
16 changes: 16 additions & 0 deletions scripts/tests/e2e-workflow.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,4 +32,20 @@ describe('e2e workflow', () => {
expect(group).toContain('github.event_name');
expect(group).toContain('github.head_ref || github.ref_name');
});

it('caps every lane so a hung model endpoint cannot stall the run', () => {
// A lane that loses connectivity to the model endpoint burns every test's
// full 300s timeout before reporting — the 2026-08-26 run took over
// 4 hours to fail while healthy lanes finish in ~17 minutes. Dropping a
// cap would silently reintroduce that, so every lane's timeout is pinned.
const lanes = [
'e2e-test-linux',
'e2e-test-macos',
'isolated-nightly',
'web-shell-browser-regression',
];
Comment on lines +41 to +46

@qwen-code-ci-bot qwen-code-ci-bot Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] The pin hand-enumerates today's four job ids, so it silently stops covering any lane added later. Verified by mutation at this commit: appending a fifth job to e2e.yml without timeout-minutes left the suite green. That is not speculative — e2e.yml's own header talks about promoting lanes to merge_group once a stable subset is carved out — and an uncapped future lane would reintroduce the hours-long stall this PR exists to prevent while this test stays green. Quantify over the workflow's jobs instead:

for (const [lane, job] of Object.entries(yml.jobs)) {
  expect(job['timeout-minutes'], lane).toBe(60);
}

(allow named exceptions explicitly if a future job legitimately needs no cap). Acceptance check: with the quantified loop, delete timeout-minutes from any lane — current or future — and confirm the test goes red.

— qwen3.8-max via Qwen Code /review (v0.22.2)

for (const lane of lanes) {
expect(yml.jobs[lane]['timeout-minutes'], lane).toBeGreaterThan(0);

@qwen-code-ci-bot qwen-code-ci-bot Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] toBeGreaterThan(0) pins only that a cap exists, not the 60-minute value this PR sets — the comment above says "every lane's timeout is pinned", but mutating timeout-minutes: 60 to any positive value (say 240) passes this test. Verified by mutation at this commit: bumping e2e-test-linux to 240 left the suite green, while the documented property — a hung endpoint surfaces within the hour, the 2026-08-26 failure mode this PR exists to remove — silently degrades. Only deleting the key entirely is caught today.

Suggested change
expect(yml.jobs[lane]['timeout-minutes'], lane).toBeGreaterThan(0);
expect(yml.jobs[lane]['timeout-minutes'], lane).toBe(60);

Acceptance check: with the pin tightened, change any lane's timeout-minutes in e2e.yml and confirm "caps every lane so a hung model endpoint cannot stall the run" goes red.

— qwen3.8-max via Qwen Code /review (v0.22.2)

}
});
});
Loading