Skip to content

fix(ci): avoid apt on self-hosted Playwright smoke - #6865

Merged
yiliang114 merged 6 commits into
QwenLM:mainfrom
yiliang114:cx/fix-6856-playwright-apt-lock
Jul 14, 2026
Merged

fix(ci): avoid apt on self-hosted Playwright smoke#6865
yiliang114 merged 6 commits into
QwenLM:mainfrom
yiliang114:cx/fix-6856-playwright-apt-lock

Conversation

@yiliang114

@yiliang114 yiliang114 commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

What this PR does

Keeps the web-shell browser smoke on the selected Linux runner. GitHub-hosted runners continue installing Chromium with its Linux packages, while self-hosted ECS runners install only the browser and do not invoke apt-get during the job.

Why it's needed

Playwright's --with-deps option invokes apt-get. CI run 29302774843 failed before the browser test started because another process held the shared ECS runner's apt lock, then passed unchanged on another runner. Linux packages belong in the ECS runner image; the CI job should not mutate the host package database on every run.

Reviewer Test Plan

How to verify

Confirm the smoke job still follows the existing selected-runner routing. On GitHub-hosted runners it should run npx playwright install --with-deps chromium; on self-hosted runners it should run npx playwright install chromium and never invoke apt. The ECS image must already provide Playwright's Linux dependencies.

Evidence (Before & After)

Before: CI attempt 1 failed in the Playwright install step with /var/lib/apt/lists/lock held by another process. After: the focused workflow test passes 5/5, and actionlint and yamllint pass.

Tested on

OS Status
🍏 macOS ✅ tested
🪟 Windows N/A
🐧 Linux ⚠️ not tested

Environment (optional)

Local workflow validation on macOS. This fork PR routes to GitHub-hosted CI, so the ECS branch still requires validation after the runner image rollout.

Risk & Scope

  • Main risk or tradeoff: Self-hosted browser execution now relies on Playwright's Linux dependencies being present in the ECS runner image.
  • Not validated / out of scope: The ECS AMI/image and launch template live outside this repository; this PR does not provision them.
  • Breaking changes / migration notes: Roll out the runner image before merging this workflow change.

Linked Issues

Related to #6856

Does not close #6856 until the self-hosted runner image rollout is complete and verified.

中文说明

这个 PR 做了什么

web-shell 浏览器冒烟测试继续使用现有的 selected Linux runner 路由。GitHub-hosted runner 仍安装 Chromium 和 Linux 系统依赖;self-hosted ECS runner 只安装浏览器,不再在任务执行期间调用 apt-get

为什么需要

Playwright 的 --with-deps 会调用 apt-get。CI run 29302774843 在浏览器测试开始前失败,因为共享 ECS runner 上另一个进程持有 apt 锁;代码不变换 runner 重跑后通过。Linux 系统依赖应固化在 ECS runner 镜像中,CI Job 不应每次运行都修改宿主机包数据库。

Reviewer 测试计划

如何验证

确认 smoke Job 仍跟随现有 selected-runner 路由。GitHub-hosted runner 应执行 npx playwright install --with-deps chromium;self-hosted runner 应执行 npx playwright install chromium,且不调用 apt。ECS 镜像必须已经提供 Playwright 所需的 Linux 系统依赖。

证据(Before & After)

Before:CI attempt 1 在 Playwright 安装阶段因另一个进程持有 /var/lib/apt/lists/lock 而失败。After:聚焦 workflow 测试 5/5 通过,actionlint 和 yamllint 通过。

测试平台

OS 状态
🍏 macOS ✅ 已测试
🪟 Windows N/A
🐧 Linux ⚠️ 未测试

环境(可选)

在 macOS 上完成本地 workflow 验证。fork PR 会路由到 GitHub-hosted CI,因此 ECS 分支仍需在 runner 镜像发布后验证。

风险与范围

  • 主要风险或取舍:self-hosted 浏览器测试现在依赖 ECS runner 镜像已经包含 Playwright 所需的 Linux 系统依赖。
  • 未验证 / 范围外:ECS AMI/镜像及 launch template 位于本仓库之外,本 PR 不负责发布镜像。
  • 破坏性变更 / 迁移说明:应先发布 runner 镜像,再合并此 workflow 修改。

关联 Issue

关联 #6856

在 self-hosted runner 镜像完成发布并验证前,这个 PR 不应关闭 #6856

@qwen-code-ci-bot

qwen-code-ci-bot commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the PR! (Re-run — the previous review was against 1a879479; the approach has since been reworked.)

Template looks good ✓

Problem: Observed bug with evidence. CI run 29302774843 failed with apt lock contention (/var/lib/apt/lists/lock) on a shared self-hosted ECS runner during npx playwright install --with-deps chromium. Same job passed on rerun with a different runner. Root cause is clear: --with-deps triggers apt-get, which races with other processes on shared self-hosted runners.

Direction: Aligned. This is a CI stability fix for a real, observed failure. Linux packages belong in the runner image, not in a per-job apt-get that can race. The ECS runner image should already have Playwright's system deps baked in.

Size: Not applicable — CI-only change (.github/workflows/ci.yml + one test file), no core paths touched. 30 production lines changed.

Approach: The approach has been reworked since the first pass. Previously the fix pinned web_shell_e2e_smoke to ubuntu-latest; now it keeps the dynamic runner routing and splits the Playwright install into two conditional steps — --with-deps on hosted, plain install on self-hosted. This is more nuanced: it fixes the apt race without abandoning self-hosted routing for this job.

One addition beyond the minimal fix: a linux_runner workflow_dispatch input is added so maintainers can manually trigger CI on a specific runner type. This is useful for validating runner image changes but is scope beyond the apt fix itself. Not a concern — just noting it.

Moving on to code review. 🔍

中文说明

感谢贡献!(Re-run — 上次审查基于 1a879479,此后方案已大幅调整。)

模板完整 ✓

问题:已观测到的 bug,有证据。 CI run 29302774843 在共享 self-hosted ECS runner 上因 npx playwright install --with-deps chromium 触发 apt 锁竞态(/var/lib/apt/lists/lock)而失败。同一 Job 换 runner 重跑后通过。根因明确:--with-deps 会调用 apt-get,在共享 self-hosted runner 上与其他进程竞态。

方向:对齐。这是针对已观测到的 CI 失败的稳定性修复。Linux 系统依赖应固化在 runner 镜像中,不应每次 Job 都跑 apt-get

规模:不适用——仅 CI 改动(.github/workflows/ci.yml + 一个测试文件),未触及核心路径。30 行生产代码变更。

方案:方案相比上次审查已大幅调整。之前是将 web_shell_e2e_smoke 固定到 ubuntu-latest;现在保留动态 runner 路由,将 Playwright 安装拆为两个条件步骤——hosted 用 --with-deps,self-hosted 不用。更细致的方案:修复 apt 竞态的同时不放弃 self-hosted 路由。

在最小修复之外增加了一个 linux_runner workflow_dispatch 输入,允许维护者手动在特定 runner 类型上触发 CI。这对验证 runner 镜像变更有用,但超出了 apt 修复本身的范围。不是问题——仅记录。

进入代码审查 🔍

Qwen Code · qwen3.7-max

Reviewed at 7f84860226f44bfd4c426b7b7176194402777eb7 · re-run with @qwen-code /triage

@qwen-code-ci-bot

qwen-code-ci-bot commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Code Review

Independent proposal before reading the diff: to fix the apt lock race on self-hosted runners without abandoning the dynamic runner routing, split the Playwright install into two conditional steps — --with-deps for hosted (fresh VM, no system deps), plain install for self-hosted (deps baked into the runner image). Optionally add a workflow_dispatch knob for manual validation. Keep the checkout/verify logic aware of the new event type.

The PR matches this closely. The key changes:

.github/workflows/ci.yml — The Playwright install is now split into two conditional steps gated on runner.environment: hosted gets npx playwright install --with-deps chromium, self-hosted gets npx playwright install chromium (no apt). The smoke job now also runs on workflow_dispatch, with a linux_runner choice input for manual validation. The classify_pr gate and pick_runner logic correctly handle the new event type. The checkout ref expression falls through cleanly for dispatch events (branch_ref input → github.ref). The head-commit verification step is correctly gated to pull_request only.

scripts/tests/no-ak-integration-ci.test.js — New test asserts the smoke job has both ubuntu_router references and both Playwright install variants. Simple and effective.

No correctness bugs, no security concerns, no regressions. The conditional expressions are clean and mutually exclusive (runner.environment is either github-hosted or self-hosted). Existing self-hosted setup steps (Node fallback, npm cache) remain unchanged and are still unreachable for fork PRs routed to hosted runners.

One minor observation: the test name "does not install Linux packages on self-hosted Playwright runners" is slightly misleading — the test asserts the existence of both install variants, not that self-hosted runners don't get --with-deps. The YAML conditionals enforce the separation, not the test. Not blocking.

Testing

This is a CI workflow change — the meaningful verification is the CI run itself, not a tmux CLI session. All checks on the PR pass:

All checks were successful
0 cancelled, 0 failing, 10 successful, 17 skipped, and 0 pending checks

✓  Qwen Code CI/Classify PR (pull_request)                               3s
✓  Qwen Code CI/Test (ubuntu-latest, Node 22.x) (pull_request)     30m6s
✓  🧐 Qwen Pull Request Review/precheck-pr / precheck              5s
✓  Qwen Code CI/web-shell E2E Smoke (ubuntu-latest, Node 22.x)    4m53s

Key results:

  • Test job (30m6s) — includes the updated no-ak-integration-ci.test.js suite, all pass
  • Web-shell E2E Smoke (4m53s) — the actual smoke test ran successfully (on hosted runner, since this is a fork PR)
  • precheck-pr (actionlint + yamllint) — YAML syntax valid
$ python3 -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml')); print('YAML valid')"
YAML valid

Note: This fork PR routes to GitHub-hosted runners, so the --with-deps (hosted) path was exercised. The self-hosted path (plain npx playwright install chromium) will need validation when the ECS runner image includes Playwright's system deps — the PR author acknowledges this in the Risk section.

中文说明

代码审查

读 diff 前的独立方案:在不放弃动态 runner 路由的前提下修复 apt 锁竞态,将 Playwright 安装拆为两个条件步骤——hosted 用 --with-deps(全新 VM 无系统依赖),self-hosted 不用(依赖固化在镜像中)。可选地加一个 workflow_dispatch 旋钮供手动验证。checkout/verify 逻辑需感知新事件类型。

PR 与此高度一致。关键改动:

.github/workflows/ci.yml — Playwright 安装按 runner.environment 拆为两个条件步骤:hosted 执行 npx playwright install --with-deps chromium,self-hosted 执行 npx playwright install chromium(不调 apt)。smoke Job 新增 workflow_dispatch 触发支持,附带 linux_runner 选择输入。classify_pr gate 和 pick_runner 逻辑正确处理新事件类型。checkout ref 表达式对 dispatch 事件有合理的 fallback。head commit 验证步骤正确地仅在 pull_request 时执行。

scripts/tests/no-ak-integration-ci.test.js — 新测试断言 smoke Job 同时包含 ubuntu_router 引用和两种 Playwright 安装变体。简洁有效。

无正确性 bug,无安全隐患,无回归。条件表达式干净且互斥。已有的 self-hosted 设置步骤(Node 回退、npm 缓存)保持不变。

一个小观察:测试名 "does not install Linux packages on self-hosted Playwright runners" 略有误导——测试断言的是两种安装变体的存在,而不是 self-hosted runner 使用 --with-deps。是 YAML 条件语句保证了分离,不是测试。不阻塞。

测试

这是 CI workflow 改动——有效验证来自 CI 运行本身。PR 上的所有检查通过:Test Job(30m6s)包含更新的测试套件全部通过;Web-shell E2E Smoke(4m53s)在 hosted runner 上成功运行;precheck-pr(actionlint + yamllint)YAML 语法有效。

此 fork PR 路由到 GitHub-hosted runner,因此验证了 --with-deps(hosted)路径。self-hosted 路径(纯 npx playwright install chromium)需在 ECS runner 镜像包含 Playwright 系统依赖后验证——作者在 Risk 部分已说明。

Qwen Code · qwen3.7-max

Reviewed at 7f84860226f44bfd4c426b7b7176194402777eb7 · re-run with @qwen-code /triage

@qwen-code-ci-bot

qwen-code-ci-bot commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Confidence: 5/5 — Clean, targeted fix for an observed CI failure; every conditional is correct and well-tested.

The approach has been reworked since the first review (which pinned to ubuntu-latest), and the new approach is arguably better: instead of abandoning self-hosted routing for the smoke test entirely, it keeps the dynamic runner selection and only changes what happens during the Playwright install step. Self-hosted runners skip --with-deps (no apt), hosted runners keep it. The fix is surgical — two conditional steps replacing one unconditional one.

The workflow_dispatch additions (runner choice input, classify_pr support, checkout ref fallback) are well-implemented and useful — they let maintainers manually validate runner image changes without pushing test commits. Not strictly necessary for the apt fix, but a reasonable companion change in the same PR.

All CI checks pass. The test correctly asserts the existence of both install variants. The only caveat (acknowledged by the author) is that the self-hosted path needs the ECS runner image to already include Playwright's system deps — that's an out-of-repo dependency, not a code concern.

Approving. ✅

中文说明

信心:5/5 — 针对已观测到的 CI 失败的干净、精准修复;每个条件表达式正确且经过测试。

方案自上次审查后已调整(之前固定到 ubuntu-latest),新方案更好:不放弃 smoke test 的 self-hosted 路由,而是只改变 Playwright 安装阶段的行为。Self-hosted runner 跳过 --with-deps(不调 apt),hosted runner 保持原样。修复精确——两个条件步骤替换一个无条件步骤。

workflow_dispatch 新增内容(runner 选择输入、classify_pr 支持、checkout ref fallback)实现良好且实用——允许维护者手动验证 runner 镜像变更,无需推送测试 commit。对 apt 修复本身不是必须的,但作为同一 PR 的伴随改动合理。

所有 CI 检查通过。测试正确断言两种安装变体的存在。唯一的注意事项(作者已说明)是 self-hosted 路径需要 ECS runner 镜像已包含 Playwright 系统依赖——这是仓库外的依赖,不是代码问题。

已批准。✅

Qwen Code · qwen3.7-max

Reviewed at 7f84860226f44bfd4c426b7b7176194402777eb7 · re-run with @qwen-code /triage

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, looks ready to ship. ✅

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. Suggestions are inline. Not reviewed: chunk 1 — no agent reported covering these; nobody read them.

— qwen3.7-max via Qwen Code /review

Comment thread scripts/tests/no-ak-integration-ci.test.js Outdated
Comment thread .github/workflows/ci.yml Outdated
@yiliang114 yiliang114 changed the title fix(ci): keep web-shell Playwright smoke on hosted runner fix(ci): avoid apt on self-hosted Playwright smoke Jul 14, 2026

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. Suggestions are inline. Not reviewed: chunk 1 — no agent reported covering these; nobody read them.

— qwen3.7-max via Qwen Code /review

Comment thread .github/workflows/ci.yml
@yiliang114

Copy link
Copy Markdown
Collaborator Author

Added one more pre-merge validation pass for the updated workflow on a real self-hosted ECS runner.

Manual dispatch run: https://github.com/QwenLM/qwen-code/actions/runs/29322239672

What this validated:

  • workflow_dispatch can now target the self-hosted Linux path explicitly
  • Classify PR selected the ECS runner successfully
  • Test (ubuntu-latest, Node 22.x) completed successfully on the self-hosted runner
  • web-shell E2E Smoke (ubuntu-latest, Node 22.x) also completed successfully on the self-hosted runner
  • the self-hosted-only branches were exercised as intended (Use pre-installed Node.js, persistent npm cache, and Install Playwright Chromium (self-hosted))

This run used the temporary branch cx/validate-6856-manual-dispatch only to execute the modified workflow before merge. The actual workflow changes are already included in this PR branch.

@yiliang114 yiliang114 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review — clean and correct. Verified the routing chain: workflow_dispatch makes test run (its if: github.event_name != 'push' holds) and ci_profile defaults to full (the classifier only downgrades on same-repo PRs), so web_shell_e2e_smoke's needs.test.outputs.ci_profile == 'full' is satisfied and the smoke job actually runs under dispatch. The Playwright split is the right fix for the apt-lock flake — --with-deps on a shared self-hosted host racing apt-get is exactly the failure in run 29302774843.

One minor nit (non-blocking): the self-hosted install step keys on runner.environment == 'self-hosted'. If an enterprise runner ever appears, it matches neither branch and Chromium isn't installed (smoke fails with no browser). runner.environment != 'github-hosted' for the no-deps step would be more future-proof, but QwenLM on github.com won't hit enterprise runners, so it's purely defensive.

LGTM.

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not reviewed: chunk 1 — no agent reported covering these; nobody read them.

— qwen3.7-max via Qwen Code /review

Comment thread scripts/tests/no-ak-integration-ci.test.js Outdated
Comment thread .github/workflows/ci.yml
@yiliang114

Copy link
Copy Markdown
Collaborator Author

Automated follow-up summary:

  • Fixed the CI-breaking Playwright assertion in 7f84860226 by checking the self-hosted command independently and retaining coverage that hosted runners use --with-deps.
  • Did not add workflow-dispatch input logging; that is an observability enhancement outside this PR's apt-lock fix.

Both threads are being resolved.

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. Suggestions are inline. Not reviewed: chunk 1 — no agent reported covering these; nobody read them.

— qwen3.7-max via Qwen Code /review

Comment thread scripts/tests/no-ak-integration-ci.test.js
Comment thread .github/workflows/ci.yml
@yiliang114

Copy link
Copy Markdown
Collaborator Author

Automated follow-up summary for the latest review:\n\n- No code change for stronger YAML-structure assertions around the hosted/self-hosted Playwright steps. The current workflow behavior is not reported broken; this is future regression-hardening for the test.\n- No code change for adding workflow-dispatch runner-selection coverage. That branch is not reported broken, and covering the full dispatch input/fallback matrix would expand the test surface beyond this bounded apt-lock fix.\n\nBoth items are Suggestions rather than current behavior or CI defects. The two threads are being resolved under the minimal-scope policy.

@yiliang114

Copy link
Copy Markdown
Collaborator Author

@qwen-code /triage

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, looks ready to ship. ✅

@yiliang114
yiliang114 added this pull request to the merge queue Jul 14, 2026
Merged via the queue into QwenLM:main with commit 344c006 Jul 14, 2026
66 of 67 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: avoid Playwright apt lock contention in web-shell smoke job

3 participants