ci: quarantine cron-interactive from push E2E to nightly-only - #6986
Merged
Conversation
cron-interactive.test.ts is inherently timing-flaky: it relies on wall-clock cron fire (*/1 at minute boundary) + real model latency, with a 90s waitForScreen window. On slower macOS runners this occasionally exceeds the timeout, turning push CI red and triggering wasteful autofix attempts (~50min). Changes: - Exclude cron-interactive from push-triggered Linux and macOS jobs via vitest --exclude - Add new cron-interactive-nightly job: runs only on schedule/ workflow_dispatch, with continue-on-error so flakes are visible but do not fail the workflow - Mirrors the existing web-shell-browser-regression pattern Coverage is preserved: cron regressions are still caught by nightly runs within ~24h. The root fix (injectable clock seam in CronScheduler) is tracked separately in #6487/#6982. Closes #6982
qwen-code-ci-bot
requested changes
Jul 15, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Hey @yiliang114 — the approach here (quarantining the flaky cron-interactive test to nightly) looks right and the linked issue #6982 has great context. But the PR body doesn't follow the PR template, which is required for all PRs.
What's missing:
## What this PR does— PR uses## Summaryinstead## Why it's needed— PR uses## Changes/## Why this is safeinstead## Reviewer Test Plan— entirely missing. Even for a CI-only change, this section helps reviewers know what to verify. For CI config, you can describe the verification approach (e.g. "merge and observe the next push run should skip cron-interactive; trigger workflow_dispatch to confirm the nightly job runs it") and mark Before/After as N/A## Risk & Scope— entirely missing. The "Why this is safe" and "Not in this PR" sections cover some of this, but the template section is the standard place for risk/tradeoff/out-of-scope notes## Linked Issues— PR uses## Relatedinstead
Could you restructure the PR body to use the template headings? The content is already solid — it just needs to fit the expected format so reviewers (and future git log readers) can find what they need.
中文说明
嗨 @yiliang114 — 方案本身没问题（把 flaky 的 cron-interactive 测试移到 nightly），关联的 issue #6982 也写得很清楚。但 PR 正文没有按照 PR 模板 的格式来写，这是所有 PR 都需要遵守的。
缺少的部分：
## What this PR does— PR 用了## Summary## Why it's needed— PR 用了## Changes/## Why this is safe## Reviewer Test Plan— 完全没有。即使是纯 CI 改动，这个部分也能帮助 reviewer 知道如何验证。对于 CI 配置，可以写验证方案（例如"合入后观察下次 push 运行应跳过 cron-interactive；触发 workflow_dispatch 确认 nightly job 能运行该测试"），Before/After 标 N/A 即可## Risk & Scope— 完全没有。"Why this is safe" 和 "Not in this PR" 涵盖了部分内容，但模板的 Risk & Scope 是标准的风险/权衡/范围说明位置## Linked Issues— PR 用了## Related
能否按模板的标题重新组织一下 PR 正文？内容本身已经很好了，只是需要调整格式，让 reviewer（和未来 git log 的读者）能方便地找到需要的信息。
— Qwen Code · qwen3.7-max
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed. Suggestions are inline.
— qwen3.7-max via Qwen Code /review
| OPENAI_BASE_URL: '${{ secrets.OPENAI_BASE_URL }}' | ||
| OPENAI_MODEL: '${{ secrets.OPENAI_MODEL }}' | ||
| run: 'npm run test:e2e' | ||
| run: 'npx cross-env VERBOSE=true KEEP_OUTPUT=true QWEN_SANDBOX=false vitest run --root ./integration-tests --exclude "**/interactive/cron-interactive.test.ts"' |
Collaborator
There was a problem hiding this comment.
[Suggestion] The macOS job inlines the full vitest command instead of delegating to the test:e2e npm script like every other job. This means future changes to test:e2e or test:integration:sandbox:none in package.json (new flags, reporters, pool options) will silently not apply to the macOS job.
Concrete cost: the Linux job uses npm run test:integration:sandbox:none -- --exclude '...' and automatically picks up script changes. The macOS job should follow the same pattern through test:e2e:
Suggested change
| run: 'npx cross-env VERBOSE=true KEEP_OUTPUT=true QWEN_SANDBOX=false vitest run --root ./integration-tests --exclude "**/interactive/cron-interactive.test.ts"' | |
| run: 'npm run test:e2e -- --exclude "**/interactive/cron-interactive.test.ts"' |
This expands through test:e2e → test:integration:sandbox:none → vitest run --root ./integration-tests --exclude "**/interactive/cron-interactive.test.ts", preserving VERBOSE, KEEP_OUTPUT, and QWEN_SANDBOX from the script chain while picking up any future script modifications.
— qwen3.7-max via Qwen Code /review
wenshao
approved these changes
Jul 15, 2026
Collaborator
|
@qwen-code /triage |
Collaborator
|
Thanks for the PR! Template looks good now ✓ — the previous round flagged missing headings and you've restructured to match the template. All required sections present. Problem: observed and well-documented. Direction: aligned. Quarantining a timing-flaky test to nightly-only runs is a pragmatic CI hygiene move. The test never caught a real regression during its 3.5-month flaky period, so the immediate-push coverage it provides is noise, not signal. Catches regressions within ~24h via nightly instead — reasonable tradeoff. Size: not applicable — Approach: minimal and focused. One file changed, does exactly what it says: Moving on to code review. 🔍 中文说明感谢贡献！ 模板现在完整 ✓ — 上一轮指出了缺少的标题，你已经按模板重新组织了，所有必要部分都在。 问题：已观测且有充分记录。 方向：对齐。将 timing-flaky 测试移到 nightly 是务实的 CI 维护。该测试在 3.5 个月的 flaky 期间从未捕获过真实回归，push 时立即运行提供的是噪音而非信号。通过 nightly 在 ~24h 内捕获回归 — 合理的折衷。 规模：不适用 — 仅改动 方案：最小且聚焦。改一个文件，做了它说要做的事：push job 加 进入代码审查 🔍 — Qwen Code · qwen3.7-max Reviewed at |
Collaborator
|
Code review: The diff is clean — one file, focused changes, no drive-by edits. Verified each piece:
No critical blockers, no security concerns, no AGENTS.md violations. Verification (YAML inspection — CI-only change, tmux not applicable):
One observation (non-blocking): the nightly job runs on 中文说明代码审查： Diff 很干净 — 一个文件，聚焦的改动，没有夹带其他变更。逐项验证：
无关键阻断、无安全问题、无 AGENTS.md 违规。 验证（YAML 结构检查 — 纯 CI 改动，tmux 不适用）：
一个观察（非阻断）：nightly job 运行在 — Qwen Code · qwen3.7-max Reviewed at |
Collaborator
|
Confidence: 5/5 — clean across every stage, no reservations. This is exactly the kind of PR you want to see: a well-scoped fix to a well-documented problem. The flaky test has been poisoning push CI for 3.5 months with no real regression caught in that time. Moving it to nightly is the right call — it preserves coverage within ~24h while stopping the noise on every push. The implementation is minimal (one file, 52 additions), follows the existing LGTM — approving. ✅ 中文说明信心: 5/5 — 所有阶段都干净，没有顾虑。 这正是你想看到的 PR：对充分记录的问题进行范围合理的修复。flaky 测试已经在 3.5 个月内持续干扰 push CI，且期间未捕获过任何真实回归。移到 nightly 是正确选择 — 在 ~24h 内保留覆盖率，同时停止每次 push 的噪音。 实现最小化（一个文件，52 行新增），精确遵循现有的 LGTM — 批准。✅ — Qwen Code · qwen3.7-max Reviewed at |
qwen-code-ci-bot
approved these changes
Jul 15, 2026
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
doudouOUC
reviewed
Jul 15, 2026
doudouOUC
left a comment
doudouOUC
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed — no blockers. Suggestions are inline.
— qwen3.7-max via Qwen Code /review
Comment on lines
+148
to
+149
| cron-interactive-nightly: | ||
| name: 'cron-interactive E2E (nightly)' |
Collaborator
There was a problem hiding this comment.
[Suggestion] continue-on-error: true combined with the existing failure-notification workflow (main-ci-failure-issue.yml) creates a silent-coverage gap. That workflow filters on conclusion == 'failure' (which continue-on-error prevents) AND event == 'push' (but this job fires on schedule). Net result: if a real cron regression lands, the nightly job fails silently — the workflow badge stays green, no issue is created, and nobody is alerted. The PR acknowledges the ~24h detection delay as an acceptable tradeoff, but there is currently no detection path, not even the delayed one.
Consider adding a conditional notification step:
- name: 'Report nightly failure'
if: failure()
env:
GH_TOKEN: '${{ secrets.GITHUB_TOKEN }}'
run: |
gh issue create --title "Nightly cron-interactive E2E failure ($(date +%F))" \
--label "ci/nightly-failure" \
--body "See: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"This preserves continue-on-error (flakes don't block CI) while ensuring genuine regressions surface within ~24h.
— qwen3.7-max via Qwen Code /review
Comment on lines
+150
to
+152
| runs-on: 'ubuntu-latest' | ||
| if: |- | ||
| ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }} |
Collaborator
There was a problem hiding this comment.
[Suggestion] The nightly job runs only on ubuntu-latest, but the PR description and linked issue evidence show the flakes were worst on macOS runners ("failed only on macOS"). A real cron regression that manifests only on macOS (e.g., a platform-specific timing issue in CronScheduler) would go undetected by the nightly job.
Consider adding macOS to the nightly matrix:
strategy:
matrix:
os: [ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}Or at minimum, document that macOS coverage is intentionally dropped for nightly.
— qwen3.7-max via Qwen Code /review
| # This test is inherently timing-flaky (wall-clock cron fire + real model | ||
| # latency). Run it nightly only so flakes do not turn push CI red. | ||
| # continue-on-error prevents this job from marking the workflow as failed. | ||
| continue-on-error: true |
Collaborator
There was a problem hiding this comment.
[Suggestion] No timeout-minutes on this job. The cron-interactive test depends on wall-clock scheduling (*/1 at minute boundary) + real model latency with a 90s waitForScreen window. If the test hangs (e.g., the cron event never fires), the job runs until GitHub's default 360-minute runner timeout. No other job in this workflow sets timeout-minutes either, but this nightly job is the highest-risk candidate for hangs.
timeout-minutes: 30— qwen3.7-max via Qwen Code /review
yiliang114
added a commit
that referenced
this pull request
Aug 27, 2026
Run 33069559004 shows the stall class is broader than mem0: on macOS and ecs-qwen pool, external-context-auto-recall, context-compress-interactive and qwen-serve-channel-workers also fail while ubuntu-hosted stays green (#10198's own ubuntu CI passed at merge). Quarantine them with mem0 per the #6986 precedent; nightly keeps them as a canary.
doudouOUC
pushed a commit
to doudouOUC/qwen-code
that referenced
this pull request
Aug 27, 2026
…efactor (QwenLM#10290) * ci: quarantine external-context mem0 E2E from push lanes (QwenLM#10272) The interactive external-context-mem0-write suite hangs at CLI startup ('Connecting to MCP servers...') on macOS and ecs-qwen pool runners — bisected to QwenLM#10128, tracked in QwenLM#10272, ubuntu-hosted unaffected. Every push E2E run currently fails on it, masking all other signal. Follow the cron-interactive precedent (QwenLM#6986): exclude it from the push lanes (linux both sandbox legs, macOS) and keep it in the nightly isolated matrix so the regression stays visible and the fix is verified when it lands. * ci: extend quarantine to the platform-stalled interactive/serve set Run 33069559004 shows the stall class is broader than mem0: on macOS and ecs-qwen pool, external-context-auto-recall, context-compress-interactive and qwen-serve-channel-workers also fail while ubuntu-hosted stays green (QwenLM#10198's own ubuntu CI passed at merge). Quarantine them with mem0 per the QwenLM#6986 precedent; nightly keeps them as a canary. * fix(cli): handle synchronous goal runtime unavailability
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Quarantine the timing-flaky
cron-interactive.test.tsfrom push-triggered E2E runs into a nightly-only isolated job. The test is excluded from both Linux and macOS push jobs via vitest--exclude, and a newcron-interactive-nightlyjob runs it on schedule/manual dispatch withcontinue-on-error: true.Why it's needed
cron-interactive.test.tsis inherently timing-flaky: it relies on wall-clock cron fire (*/1at minute boundary) + real model latency, with a 90swaitForScreenwindow. On slower macOS runners this occasionally exceeds the timeout, turning push CI red onmainand triggering wasteful autofix attempts (~50min). This has been intermittently flaky for ~3.5 months (#2731 → #3402 → #3992 → #6016). Coverage is preserved: cron regressions are still caught by nightly runs within ~24h.Reviewer Test Plan
How to verify
cron-interactive.test.ts— inspect theRun E2E testsstep in the Linux and macOS jobs for the--excludeflag.cron-interactive-nightlyjob only runs onscheduleorworkflow_dispatch(same pattern asweb-shell-browser-regression).continue-on-error: trueis set so flakes don't fail the workflow.workflow_dispatchmanually to verify the nightly job runs the cron-interactive tests correctly.Evidence (Before & After)
Before: push E2E runs all tests including cron-interactive → occasional timeout failures on macOS (e.g. run 29424611412).
After: push E2E skips cron-interactive; nightly job runs it in isolation with
continue-on-error.Tested on
CI-only change — verified by inspecting workflow YAML structure. Actual CI run will confirm on merge.
Environment (optional)
N/A — pure CI configuration change.
Risk & Scope
Linked Issues
Closes #6982
Refs: #2731, #3402, #3992, #6016, #6987
中文说明
这个 PR 做了什么
将 timing-flaky 的
cron-interactive.test.ts从 push 触发的 E2E 运行中隔离出来，移到仅 nightly 运行的独立 job。通过 vitest--exclude从 Linux 和 macOS push job 中排除该测试，新增cron-interactive-nightlyjob 在 schedule/manual dispatch 时运行，设置continue-on-error: true。为什么需要
cron-interactive.test.ts存在固有的 timing-flakiness：依赖墙钟整分钟触发的 cron（*/1）+ 真实模型延迟，waitForScreen窗口 90 秒。在较慢的 macOS runner 上偶尔超时，导致 main CI 变红并触发浪费的 autofix 尝试（~50 分钟）。这个问题已经间歇性存在约 3.5 个月。覆盖率不受影响：cron 回归仍然在 nightly 运行中被捕获（~24h 内）。风险与范围