fix(ci): retry transient sandbox image build failures in E2E (#10355) - #10359
Merged
Conversation
Collaborator
Author
E2E report — issue #10355 (Main CI failed: E2E Tests on bc6f1a0)DiagnosisRun 33139344576 (attempt 1) failed with exactly one red job: Evidence that this is a transient pool-runner environment failure, not a code defect:
No authenticated log access exists in this environment (the logs endpoint requires a token), so the exact docker-side error line is unknown; the isolation evidence above is what carries the diagnosis. FixAdded one bounded retry to the sandbox image build in the E2E workflow, mirroring the pattern already landed for the release image build after incident #9898: the first attempt runs with The retry shape is pinned by six new cases in the existing e2e-workflow test (continue-on-error present on the first attempt, step id stable, retry gated on the outcome expression and not Not changed
Verification
中文说明E2E 报告 — issue #10355（主分支 CI 失败：E2E Tests，提交 bc6f1a0）诊断运行 33139344576（第 1 次尝试）只有一个任务变红： 以下证据表明这是一次池内运行器环境的瞬时故障，而非代码缺陷：
本环境没有带鉴权的日志访问权限（logs 接口需要 token），因此 docker 侧的具体报错行未知；上述隔离证据是诊断的依据。 修复为 E2E 工作流中的沙箱镜像构建增加了一次有界重试，沿用发布镜像构建在 #9898 事件后已落地的模式：第一次尝试带 重试结构由现有 e2e-workflow 测试中新增的六个用例钉住（第一次尝试带 continue-on-error、步骤 id 稳定、重试以 outcome 表达式而非 未改动
验证
🧠 Handled by Qwen Code · model/模型 |
github-actions
Bot
added
the
review/self-reported
Collaborator
|
✅ Qwen Triage finished — CI landed green on ✅ Qwen Triage 已完成 —— |
yiliang114
approved these changes
Aug 28, 2026
Collaborator
|
Thanks for the PR! Template looks good ✓ Problem: Observed and independently verified. Run 33139344576 attempt 1 failed with exactly one red job — Direction: CI reliability on main is squarely in scope, and this extends an established pattern rather than inventing one: the identical bounded-retry shape has been running in the release image build ( Size: not applicable — no core paths touched. 74 added lines across two files: workflow YAML (+20) and its guard test (+54), zero deletions. Approach: right-sized. A first attempt with Risk: no high-risk path matches; no elevated risk signals. Moving on to code review. 🔍 中文说明感谢贡献！ 模板完整 ✓ 问题：已观测并独立核实。运行 33139344576 第 1 次尝试恰有一个红色任务—— 方向：main 分支的 CI 可靠性完全在范围内，而且这是扩展既有模式而非新造：相同的有界重试形态自 #9898 事件以来已在发布镜像构建（ 规模：不适用——未触及核心路径。两个文件共新增 74 行：工作流 YAML（+20）与守卫测试（+54），零删除。 方案：范围恰当。第一次尝试带 风险：未命中高风险路径；无升级风险信号。 进入代码审查 🔍 — Qwen Code · qwen3.8-max Reviewed at |
Collaborator
Code reviewNo blockers. The change reproduces the bounded-retry shape that has been running in the release image build since #9898, and every load-bearing detail is right:
Independent baseline: for "a transient docker build failure turns a whole E2E shard red on main", the approach I would have chosen is this exact two-step pattern rather than a shell-level retry loop — it is already proven in Testing evidence (this PR's own CI, read via API — no PR code executed)CI on the reviewed commit is still settling: the One gap named honestly: no pre-merge lane can exercise the retry path firing on a real transient failure — it cannot be forced in PR CI by construction, and the sandboxed lanes drive the built product, not workflow YAML, so neither applies. What settles the claim pre-merge is that the YAML shape is pinned by the six guard tests and the identical shape is proven in production by the release image build; the first live confirmation is the next main E2E run (red first attempt, green retry, green shard), exactly as the PR describes. Final CI results for
One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行（取最新一次运行），省略 skipped，失败项排在最前。 中文说明代码审查无阻塞问题。改动复刻了 #9898 以来发布镜像构建中已在运行的有界重试形态，每个关键细节都正确：
独立基线：面对"瞬时 docker 构建失败让 main 上整个 E2E 分片变红"，我会选择的就是这个两步模式而非 shell 层重试循环——它已在 测试证据（通过 API 读取本 PR 自身的 CI——未执行任何 PR 代码）被审提交上的 CI 仍在进行： 如实说明一个缺口：任何合并前通道都无法演练重试路径在真实瞬时故障下触发——按构造它无法在 PR CI 中强制发生，而沙箱通道驱动的是构建产物而非工作流 YAML，两者均不适用。合并前能支撑结论的是：YAML 形态由六个守卫测试钉住，相同形态已由发布镜像构建在生产验证；首次实际确认是下一次 main 的 E2E 运行（第一次尝试红、重试绿、分片绿），与 PR 描述完全一致。 — Qwen Code · qwen3.8-max Reviewed at |
Collaborator
|
Confidence: 5/5 — verified incident, production-proven pattern, minimal diff, every load-bearing property pinned by tests. The failure this fixes is real and independently verified: run 33139344576 attempt 1 died with exactly one red job — docker shard 3/3 at One honesty note: the older recurrence (run 33066706127) is a weaker signature than the PR implies — that run failed all eight E2E jobs — but the same step did die in its docker shard 2/3, and the primary incident stands on its own evidence. CI on the reviewed commit is still in flight ( 中文说明置信度：5/5 —— 事件已核实、模式已在生产验证、diff 最小、每个关键属性都有测试钉住。 本修复针对的故障真实存在且已独立验证：运行 33139344576 第 1 次尝试恰有一个红色任务——docker 分片 3/3 死在 一点诚实说明：较早的复发（运行 33066706127）特征弱于 PR 的表述——那次运行八个 E2E 任务全部失败——但同一确实死在了其 docker 分片 2/3，且主事件本身的证据已经足够。 被审提交上的 CI 仍在进行（ — Qwen Code · qwen3.8-max Reviewed at |
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
qwen-code-ci-bot
approved these changes
Aug 28, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed — no blockers. Suggestions are inline.
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/e2e.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
已审查——无阻断问题。 建议见行内评论。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/e2e.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— qwen3.8-max via Qwen Code /review (v0.22.2)
| # sibling runners of the same run, and the re-run passed on another | ||
| # runner. A retry reuses the docker build cache and almost always | ||
| # passes; a genuine failure fails this step and with it the job. | ||
| - name: 'Build the sandbox image (retry)' |
Collaborator
There was a problem hiding this comment.
[Suggestion] The bounded retry is hand-rolled per workflow step instead of living at the depth where the failure lives: .github/workflows/build-and-publish-image.yml already carries this exact shape (landed for issue #9898), this PR writes copy 2 here in e2e.yml, and release.yml's integration_docker job runs the identical npm run build:sandbox -- -s (release.yml:309) with no retry at all while publish gates on needs.integration_docker.result == 'success'. The next transient docker-build flake in that job turns a release red and invites either a third hand-rolled copy of the same block or use of the force_skip_tests dispatch input (release.yml:36) to bypass docker validation entirely; every copy also carries its own drift hazard — the very reason this PR adds six tests asserting the two e2e.yml steps stay in sync. Consider lifting the retry once — either a composite action under .github/actions/ used by e2e.yml and release.yml (and replaceable into build-and-publish-image.yml), or, minimally for the two run:-shaped call sites, a shell-level retry in a single step (npm run build:sandbox -- -s || { echo '::warning::first attempt failed; retrying'; npm run build:sandbox -- -s; }). At minimum, give release.yml's integration_docker the same protection this PR gives e2e.
中文说明
[建议] 这个有界重试是在每个工作流步骤里手工复制的，而不是放在故障实际发生的层级：.github/workflows/build-and-publish-image.yml 已经带有完全相同的结构（为 issue #9898 落地），本 PR 在 e2e.yml 写了第 2 份拷贝，而 release.yml 的 integration_docker 任务运行着完全相同的 npm run build:sandbox -- -s（release.yml:309）却没有任何重试，同时 publish 又以 needs.integration_docker.result == 'success' 作为门控。该任务下一次发生瞬时 docker 构建故障时，发布流程会变红，并诱发第三种选择：要么再手抄一份相同的代码块，要么使用 force_skip_tests 派发输入（release.yml:36）完全绕过 docker 校验；每一份拷贝还自带漂移风险 —— 这正是本 PR 要新增六个测试来保证 e2e.yml 两个步骤保持同步的原因。建议把重试一次性提升到对应层级 —— 要么是 .github/actions/ 下的复合动作，供 e2e.yml 与 release.yml 使用（也可替换进 build-and-publish-image.yml）；要么，对这两个 run: 形态的调用点，最小化地在单个步骤内做 shell 级重试。至少应让 release.yml 的 integration_docker 获得与本 PR 给 e2e.yml 相同的保护。
— qwen3.8-max via Qwen Code /review (v0.22.2)
| expect(buildStep['continue-on-error']).toBe(true); | ||
| }); | ||
|
|
||
| it('pins the first build step id the retry gate references', () => { |
Collaborator
There was a problem hiding this comment.
[Suggestion] The six new tests pin every property of the retry except the relative order of the two steps; steps.find(...) is position-independent, so if a future edit moves the 'Build the sandbox image (retry)' block above 'Build the sandbox image' (e.g., regrouping the docker setup steps), all six tests stay green while the retry is silently disabled — on any docker leg where the first attempt hits a transient failure (exactly the issue #10355 trigger), the retry's if evaluates steps.build-sandbox.outcome against a step that has not yet run, the context entry is empty, '' == 'failure' is false, and the retry is skipped with no error: the fix stops working silently. This is the same silent-invalidation class this test's own comment warns about for id renames. Verified by mutation in a scratch tree: reordering the two steps keeps all 8 committed tests green. Pin the order too, e.g. inside the sandbox image build retry describe block:
expect(steps.indexOf(retryStep)).toBeGreaterThan(steps.indexOf(buildStep));If you add it, prove it by mutation: move the retry step above the build step in e2e.yml and confirm the new test goes red.
中文说明
[建议] 新增的六个测试钉住了重试的每一项属性，唯独没有钉住两个步骤的相对顺序；steps.find(...) 与位置无关，因此如果未来的编辑把 'Build the sandbox image (retry)' 块移到 'Build the sandbox image' 之上（例如重新分组 docker 准备步骤），所有六个测试仍是绿色，而重试已被悄悄禁用 —— 在任何首次构建遇到瞬时故障的 docker 分支上（正是 issue #10355 的触发场景），重试的 if 会对一个尚未运行的步骤求值 steps.build-sandbox.outcome，上下文条目为空，'' == 'failure' 为假，重试被无声跳过：修复静默失效。这与本测试自身注释就 id 重命名所警告的静默失效属于同一类。已在临时树中通过变异验证：把两个步骤调换顺序后，全部 8 个已提交的测试仍为绿色。建议把顺序也钉住，例如在 sandbox image build retry describe 块内：
expect(steps.indexOf(retryStep)).toBeGreaterThan(steps.indexOf(buildStep));如果采纳，请用变异证明：在 e2e.yml 中把重试步骤移到构建步骤之上，确认新测试变红。
— qwen3.8-max via Qwen Code /review (v0.22.2)
| expect(retryStep['continue-on-error']).toBeUndefined(); | ||
| }); | ||
|
|
||
| it('rebuilds with the same script and the same skip flag', () => { |
Collaborator
There was a problem hiding this comment.
[Suggestion] The retry is by design an identical second attempt, but these tests pin only today's two env keys and a toContain on the run command — not equivalence between the two steps — so any future edit to the first step's env: or run: silently diverges the retry while all tests stay green. Concretely: a maintainer adds an env knob the build script already supports (e.g. QWEN_SANDBOX_IMAGE_TAG or BUILD_SANDBOX_FLAGS) to the first step's env: block to fix an unrelated problem and doesn't touch the retry step a dozen lines below; on the next transient docker failure, attempt 1 fails and the retry builds a differently-tagged/configured image, so the shard's resulting image depends on which attempt happened to succeed (E2E tests then run against the wrong or missing image tag). Verified by mutation in a scratch tree: both an env divergence and a run: divergence survive the committed suite. Pin the actual invariant — "the retry is an identical attempt" — with equality instead of containment:
it('retries an identical attempt', () => {
expect(retryStep.run).toBe(buildStep.run);
expect(retryStep.env).toEqual(buildStep.env);
});(The existing per-key assertions can stay for their explanatory comments.) If you add it, prove it by mutation: add a key to the first build step's env: without mirroring it on the retry step and confirm the new test goes red — today no test catches that mutation.
中文说明
[建议] 重试在设计上是完全相同的第二次尝试，但这些测试只钉住了当前的两个 env 键和 run 命令的 toContain —— 而非两个步骤之间的等价性 —— 因此未来对第一个步骤 env: 或 run: 的任何修改都会让重试悄悄分叉，而所有测试仍是绿色。具体场景：维护者为修复一个无关问题，在第一个步骤的 env: 块中加入构建脚本已支持的 env 旋钮（例如 QWEN_SANDBOX_IMAGE_TAG 或 BUILD_SANDBOX_FLAGS），而没有改动十几行下面的重试步骤；下一次瞬时 docker 故障时，第一次尝试失败，重试构建出的镜像带有不同的标签/配置，分片最终使用哪个镜像取决于哪次尝试碰巧成功（E2E 测试会运行在错误或缺失的镜像标签上）。已在临时树中通过变异验证：env 分叉与 run: 分叉两种变异都能通过已提交的测试套件。建议用等价而非包含来钉住真正的不变量 ——「重试是完全相同的一次尝试」：
it('retries an identical attempt', () => {
expect(retryStep.run).toBe(buildStep.run);
expect(retryStep.env).toEqual(buildStep.env);
});（原有的逐键断言可保留其说明性注释。）如果采纳，请用变异证明：在第一个构建步骤的 env: 中新增任意键而不同步到重试步骤，确认新测试变红 —— 目前没有任何测试能捕获该变异。
— qwen3.8-max via Qwen Code /review (v0.22.2)
Collaborator
|
Released in v0.22.3. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Adds one bounded retry around the sandbox image build in the E2E workflow's Linux docker leg. The first build attempt runs with
continue-on-errorand a stable step id; a second step rebuilds only when the first attempt's outcome isfailure, with the same build command and the same docker-leg environment. A genuine build failure still fails the retry and the shard — only single-attempt transient failures are absorbed. The new shape is pinned by six cases in the existing e2e-workflow test.Why it's needed
The E2E run for commit
bc6f1a015c(run 33139344576, tracked in the linked issue) failed before any test ran: the docker shard's sandbox image build died after ~85 seconds on one pool runner with a bare exit-code-1 annotation. The identical build of the same commit passed on two sibling pool runners in the same run, every other job was green, and the re-run of the failed job on another runner passed — a transient pool-runner environment failure, not a code defect (the commit in question only touches integration-test global setup). The same step died once before on a different runner and commit (run 33066706127), so the class recurs. Each occurrence turns main red and files an autofix issue even though nothing in the tree is broken. This mirrors the bounded retry already landed for the release image build after incident #9898.Reviewer Test Plan
How to verify
continue-on-error: true(a successful retry would otherwise leave the shard red) and the retry must have none (a second failure must stay fatal).npx vitest run --config ./scripts/tests/vitest.config.ts e2e-workflow— 8 tests pass. To see the pins bite, mutate any one guard (dropcontinue-on-error, rename the step id, gate onfailure(), addcontinue-on-errorto the retry, drop-sorQWEN_SANDBOX) and re-run; each mutation fails the suite (all six were probed).npm run test:scripts— green, including the workflow size ratchet (e2e.yml 17,686 bytes, within baseline + allowance).Build the sandbox imagestep followed by a green(retry)step and a green shard, instead of a red shard.Evidence (Before & After)
N/A (CI workflow change, no user-visible surface). Before: one transient docker-build failure on a pool runner exits the shard red → whole E2E run red → autofix issue filed. After: the retry rebuilds (reusing the docker cache) and the shard only goes red when both attempts fail.
Tested on
Environment (optional)
Linux runner; workflow YAML verified by parsing and by the scripts test suite. The docker build path itself cannot run in this environment (no docker daemon); the PR's own docker E2E legs are the live check.
Risk & Scope
steps.build-sandbox.outcomegate evaluates toskippedon the non-docker legs, so the retry never runs there.release.yml's sandbox image build shows no such failure and is untouched; the yamllint CI lane could not run locally (yamllint not installed on the runner) but the YAML follows the file's existing style; the docker build itself cannot be exercised here.Linked Issues
Fixes #10355
中文说明
本 PR 做了什么
为 E2E 工作流 Linux docker 分支中的沙箱镜像构建增加一次有界重试。第一次构建尝试带
continue-on-error与稳定的步骤 id；第二个步骤仅在第一次尝试的 outcome 为failure时、以相同的构建命令与相同的 docker 分支环境变量重新构建。真实的构建失败仍会让重试与分片失败 —— 只有单次尝试的瞬时故障会被吸收。新结构由现有 e2e-workflow 测试中的六个用例钉住。为什么需要
提交
bc6f1a015c的 E2E 运行（run 33139344576，即关联 issue 所追踪的运行）在任何测试运行之前失败：docker 分片的沙箱镜像构建在一台池内运行器上约 85 秒后以一条光秃秃的退出码 1 注解结束。同一提交、同一次运行中，相同的构建在另外两台池内运行器上通过，其他所有任务均为绿色，且失败任务换到另一台运行器重跑后通过 —— 这是池内运行器环境的瞬时故障，而非代码缺陷（该提交只改动了集成测试的 global setup）。同一步骤此前在另一台运行器、另一个提交上也失败过一次（run 33066706127），说明该类别会复发。每一次发生都会让 main 变红并创建 autofix issue，即使代码树里没有任何东西坏了。这与 #9898 事件后为发布镜像构建落地的有界重试模式一致。评审者测试计划
如何验证
continue-on-error: true（否则重试成功也会让分片保持红色），重试步骤必须没有它（第二次失败必须保持致命）。npx vitest run --config ./scripts/tests/vitest.config.ts e2e-workflow—— 8 个测试通过。想看钉住效果，可变异任一守卫（去掉continue-on-error、重命名步骤 id、以failure()作为门控、给重试加continue-on-error、去掉-s或QWEN_SANDBOX）后重跑；每种变异都会使套件失败（六项均已探针验证）。npm run test:scripts—— 绿色，包含工作流大小棘轮（e2e.yml 17,686 字节，在基线 + 余量之内）。Build the sandbox image步骤 + 绿色的(retry)步骤 + 绿色分片，而不是红色分片。证据（前后对比）
N/A（CI 工作流变更，无用户可见界面）。变更前：池内运行器上一次瞬时的 docker 构建失败使分片变红 → 整个 E2E 运行变红 → 创建 autofix issue。变更后：重试（复用 docker 缓存）重新构建，只有两次尝试都失败时分片才变红。
测试环境
环境（可选）
Linux 运行器；工作流 YAML 通过解析与 scripts 测试套件验证。docker 构建路径本身无法在本环境演练（无 docker 守护进程）；PR 自身的 docker E2E 分支是实际检验。
风险与范围
steps.build-sandbox.outcome门控在非 docker 分支上求值为skipped，因此重试永远不会在那里运行。release.yml的沙箱镜像构建未出现此失败，保持不动；yamllint CI 通道无法在本地运行（运行器未安装 yamllint），但 YAML 遵循文件既有风格；docker 构建本身无法在此演练。关联 Issue
Fixes #10355