test(review): stamp the ledger entries when the test runs, not when it loads - #10878
Conversation
…t loads The three entries were stamped from a `Date.now()` evaluated in the describe body — at collection — while `beforeEach` writes the plan file at execution. Entries are fenced against that file's mtime with RUN_EPOCH_SLACK_MS (2s) of slack, so the stamps go stale by however long collection took and the fence drops them as a previous run's. Reproduced against the real module: at a 0ms and 1.5s gap the count reads 2, at 3s it reads 1, past a minute it reads 0 — the same 1/0/0 the three attempts of release run 33713579913 read after collecting for 2260s. Claude-Session: https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8
That change is a defect — a stamp taken at collection, fenced against a file written at execution — while this PR is a tradeoff about where wall-clock budgets are worth asserting. They review differently, so they ship separately. See #10878. Claude-Session: https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8
|
Thanks for the PR! Template looks good ✓ Problem: observed, not theoretical. Release run 33713579913 collected for ~2260s on a contended host, and the Direction: aligned — a latent flake in the review-command ledger tests that a slow CI host turned into a real failure; fixing the stamp keeps the suite trustworthy. No product surface involved. Size: not applicable — one test file, +9/−3, no core production paths. Approach: the scope is exactly right. One statement moves into the helper that consumes it, the single other consumer inlines its own stamp, offsets and assertions are unchanged. I checked the sibling PR #10870 — it touches the latency-budget files, not this one, so the split is clean. Risk: no elevated risk signals (test-only change, no high-risk paths matched). Moving on to code review. 🔍 中文说明感谢贡献! 模板完整 ✓ 问题: 已观测到,不是理论问题。Release run 33713579913 在一台被争抢的宿主上收集了约 2260 秒, 方向: 对齐——这是 review 命令 ledger 测试中一个潜伏的 flake,被慢速 CI 宿主变成了真实失败;修正打戳时机可保持套件可信。不涉及产品面。 规模: 不适用——单个测试文件,+9/−3,不涉及核心生产路径。 方案: 范围恰好。一条语句移入使用它的辅助函数,另一个使用处内联取戳,偏移与断言均未改变。已核对姊妹 PR #10870——它改的是 latency-budget 相关文件,与本文件无交集,拆分干净。 风险: 无升级风险信号(纯测试改动,未命中高风险路径)。 进入代码审查 🔍 — Qwen Code · qwen3.8-max Reviewed at |
Code reviewBefore reading the diff, my independent proposal from the description alone: the stamp must move from collection scope into execution scope — either a What I verified against the code at the reviewed commit:
No blockers; nothing to change. Test evidence — this run's CI, read via the APII did not build or run the PR's code — this lane never does. Below is the PR's own CI on the reviewed commit, fetched once (not polled); the table updates in place once CI settles. The fix is substantiated by construction rather than by a rerun: the stamp is now taken inside the test body, after CI at fetch time — unit suite, lint, and integration still running; nothing failing:
Not verified: the suite under artificially slow collection — that trigger cannot arise after this change by construction, so there is nothing to rerun. 中文说明代码审查读 diff 之前,我仅凭描述给出的独立方案:时间戳必须从收集作用域移入执行作用域——要么在 在受审提交上对照代码核实:
无阻塞问题;无需修改。 测试证据——本次运行 CI,经 API 读取本流程不构建、不运行 PR 代码。以上是 PR 自身 CI 在受审提交上的状态,一次性读取(不轮询);CI 结束后表格会就地更新。 该修复由构造支撑,而非靠重跑:时间戳现在在测试体内、 未验证:人为放慢收集下的套件——该触发条件在本次改动后按构造不可能再出现,无需重跑。 — Qwen Code · qwen3.8-max Reviewed at |
|
Confidence: 5/5 — an observed flake whose mechanism matches the fence code exactly, fixed by moving one statement; verified by construction. Stepping back: the diagnosis is the strong part of this PR. The 2 → 1 → 0 progression across retries is exactly what a stamp going stale against a 2s epoch slack produces, and I confirmed each load-bearing piece in Approval is deferred only because the unit suite, lint, and integration jobs are still running on this commit — approval lands automatically once every check on 中文说明置信度:5/5 —— 已观测到的 flake,机制与围栏代码完全吻合,仅移动一条语句即修复;构造层面已验证。 回头看:诊断是这个 PR 的亮点。重试间 2 → 1 → 0 的递减正是时间戳相对 2 秒 epoch 容差变陈旧的结果;我逐一在 暂缓批准仅因该提交上的单元套件、lint 与集成任务仍在运行——待 — Qwen Code · qwen3.8-max Reviewed at |
chiga0
left a comment
There was a problem hiding this comment.
Scope: test file only — packages/cli/src/commands/review/lib/run-ledger.test.ts (9 additions / 3 deletions). Cross-file context read: run-ledger.ts (production source — appendRunSession, parseSessions, epoch fence) and prompt-record.ts (RUN_EPOCH_SLACK_MS = 2000). No execution rung run; no environmental-behavior change in production code.
No blocking findings.
Approval blockers: none.
What was checked:
Contract / epoch-fence correctness. The fencing condition in parseSessions keeps entries where atMs >= runEpochMs(planPath) and runEpochMs = planMtimeMs − RUN_EPOCH_SLACK_MS (i.e. plan-file mtime − 2 s). beforeEach writes the plan file at test execution time. A Date.now() stamp taken at Vitest collection time — before any beforeEach has run — can therefore be arbitrarily earlier than the plan file's mtime. On the release run cited in the comment (run 33713579913, 2260 s collection latency), the stamp was 2258 s before the epoch, so every entry was fenced out and the three affected tests read 1 / 0 / 0 against an expected 2. Moving the stamp inside threeSessions() means it is evaluated at call time, after beforeEach has written the plan, putting the stamp within the slack window. Verified: appendRunSession takes nowMs as its third parameter and stores it as atMs; the fix feeds a fresh Date.now() through threeSessions() — correct.
The lone standalone test. it('is zero on a fresh ledger, whatever excludes') previously used the outer now variable; after the fix now is no longer in scope there. The replacement Date.now() is called inline at execution time (after beforeEach), so the same epoch-fence concern does not apply. The test's assertion (one entry, ledgerResumeCount = 0) does not depend on relationships between multiple session timestamps, so a fresh inline stamp is correct here.
No production-code change. The diff is test-only; the epoch fence, appendRunSession, and ledgerResumeCount implementations are unchanged.
Existing reviews / cross-check. No prior non-bot reviews on this PR; no conflicting findings to reconcile.
Not reviewed: no terminal-dependent or environment-dependent production change, so execution rungs not run.
Reviewed with AI assistance.
qwen-code-dev-bot
left a comment
There was a problem hiding this comment.
Reviewed at head 1b9af173.
- The mechanism checks out against the module at this head:
runEpochMs=statSync(plan).mtimeMs - RUN_EPOCH_SLACK_MS(2000),beforeEachrewrites the plan at execution, so a stamp taken at collection ages past the fence exactly as described — and the observed 2→1→0 retry gradient is the oldest-entry-first drop (now,now+1000,now+2000) the table predicts. Stamping insidethreeSessions()makes the stamp↔mtime gap milliseconds by construction. TheatMs <= runCeilingMs()side stays safe:S2at stamp+2000 is read within the sameit(), so ceiling (read-time + 2000) still admits it. - No dangling references to the removed describe-scoped
now(the only bare uses are now function-local orDate.now()); the other twoconst nowdeclarations are indeed already insideit()bodies. - Ran locally against the PR-head files (
run-ledger.tsandprompt-record.tsare byte-identical to current main):run-ledger.test.ts72/72 passed. - No prior reviews or threads to re-verify; no CI failures on this head (Test/Lint lanes still running — per the channel convention the call is on the review itself). No new Criticals found in the one-file test change.
qqqys
left a comment
There was a problem hiding this comment.
Reviewed at head 1b9af173. First round on this PR (no prior reviews or threads), and my Critical-only pass finds no blocking defects.
The change, verified at this head: the ledger entries in the ledgerResumeCount block are now stamped at call time instead of collection time — const now = Date.now() moves inside threeSessions(), and the fresh-ledger case takes its own Date.now(). This is the right fix for the observed flake: readSessions fences entry timestamps against the plan file's mtime with only RUN_EPOCH_SLACK_MS (2 s) of slack, and the describe's beforeEach rewrites that file, so any stamp older than collection time plus 2 s is dropped as a previous run's — exactly the shape in release run 33713579913, where a 2260 s collection on a contended host made the block read 1/0/0 against an expected 2.
Semantic preservation checked: every test in the block calls threeSessions() exactly once (or stamps its own fresh timestamp), so the per-call stamp changes no assertion — the expected resume counts (2/2/1/0/0) are untouched; no other reference to the moved binding remains in the block, and the similarly named local in the sessionEntryCount describe is a separate declaration. Test-only change; no production code, no runtime surface.
CI at this head: no failing or cancelled checks at review time; Test (ubuntu-latest, Node 22.x), Lint & Static, the no-AK integration lane, review-pr and triage are still pending, which does not gate this review per policy.
…#10870) * test: stop millisecond budgets from measuring the shared pool A budget written on a developer machine measures the code. On the shared ECS pool it measures the neighbours: the same third of the same suite runs in 6.7 or 36 minutes there depending only on which host it lands on, and at that spread every one of these bounds is a coin flip. Route the eleven wall-clock assertions through a helper that skips them when the workflow marks the run as pool-hosted. They still run locally and on the GitHub-hosted lanes, which is where the number means something. Claude-Session: https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8 * test: cover the inline wall-clock budgets too The first pass matched only `expect(elapsed).toBeLessThan(N)`, which is under a third of the family. Twenty-three more write the measurement inline — `expect(Date.now() - started).toBeLessThan(N)` — including two in acp-http/transport.test.ts, the file that flaked on QwenLM#10842's CI run. Two sites stay as they are: NativeLspService's bound runs under fake timers, and hook-runner.process.test.ts belongs to QwenLM#10842. Claude-Session: https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8 * ci: move the latency gate to the end of the env block Three CI PRs were extending the same anchor after NO_COLOR, so whichever landed second had to resolve a conflict that carried no disagreement. Claude-Session: https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8 * test: keep a relaxed bound where the duration is the property Five of the quarantined cases have no other expect(): a complexity bound or a no-zombie guarantee is the whole test, so skipping on the pool would leave them running and checking nothing — greener than before and worth less. They now keep a bound 20x wide there: far beyond the ~5x contention this fleet shows, far under what a quadratic regression costs. Claude-Session: https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8 * test(review): stamp the ledger entries when the test runs, not when it loads The three entries were stamped from a `Date.now()` evaluated in the describe body — at collection — while `beforeEach` writes the plan file at execution. Entries are fenced against that file's mtime with 2s of slack, so the stamps go stale by however long collection took and the fence drops them as a previous run's. Reproduced against the real module: at a 0ms and 1.5s gap the count reads 2, at 3s it reads 1, past a minute it reads 0 — the same 1/0/0 the three attempts of release run 33713579913 read after collecting for 2260s. Claude-Session: https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8 * ci: restore the latency gate dropped by the previous commit The previous commit swept in a stale copy of ci.yml left in the worktree and deleted this branch's own env key. Nothing about the gate changed. Claude-Session: https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8 * test: move the ledger stamp fix to its own PR That change is a defect — a stamp taken at collection, fenced against a file written at execution — while this PR is a tradeoff about where wall-clock budgets are worth asserting. They review differently, so they ship separately. See QwenLM#10878. Claude-Session: https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8 * test: make the pool-lane bounds able to fail again The relaxed poolMultiplier bounds could never fire: the transport zombie tests' own 3s safety abort capped elapsed far below 1500x20, and the serve close bound (5500x20) sat above both the 5s force-close timer and the pool's 60s testTimeout, so vitest decided the outcome first. Assert what the lanes can actually observe: the zombie tests record whether the safety abort fired (a zombie ends only via that abort) and drop the multiplier; the close bound uses a 10x multiple that stays under testTimeout while a real stall still fails it. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> Patrol-Run: qwen-pr-closeout/jmtl54lfvf2 * test: parse QWEN_SKIP_LATENCY_BUDGETS as a boolean '0' and 'false' are truthy strings, so the raw truthiness gate skipped every budget for them — the opposite of what exporting =0 to re-enable budgets intends. Parse '1'/'true'/'yes' as skip; anything else enforces. Both byte-identical copies updated in lockstep; helper suites pin the gate both ways, including that poolMultiplier must not relax the bound when the switch is unset. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> Patrol-Run: qwen-pr-closeout/jmtl54lfvf2 * test: pin that poolMultiplier never relaxes the bound off the pool Add a case to both latency-budget test copies proving that with QWEN_SKIP_LATENCY_BUDGETS unset, poolMultiplier must not move the bound: the probe (1500 vs a 100 ms budget, x20) throws off the pool while the same call keeps the relaxed bound on it. The unconditional-multiplier mutant now fails this case in both packages. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> Patrol-Run: qwen-pr-closeout/jmtl79r9if7 * test: pin the QWEN_SKIP_LATENCY_BUDGETS wiring in ci.yml Extend the ECS-predicate pin case so the new skip line is nailed verbatim beside its four VITEST_* siblings. Without the pin, an edit that alters the predicate or renders '1' on every lane silently disables every millisecond budget while the helper tests stay green; the pin goes red on that mutant. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> Patrol-Run: qwen-pr-closeout/jmtl79r9if7 * test: keep the overall-cap bound asserting on the pool lane Give the wall-clock cap case poolMultiplier 5 so the pool lane keeps an upper bound at 4000ms: still under the ~5000ms drain a broken cap drains sequentially, but above a ~100ms firing under the fleet's ~5x contention. Without it the pool lane kept only the trivially-true lower bound, so the cap regression this test exists to catch shipped green there; the default x20 multiple would clear the drain too. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> Patrol-Run: qwen-pr-closeout/jmtl79r9if7 * test(ci): raise the scripts suite timeout where the suite actually reads it 30s was the quiet-host figure. Release run 33725742855 lost its Quality Checks (Scripts) job to two files at once — qwen-autofix-workflow, whose heaviest case measures ~14s idle, and acp-serve-boundary-guard, which nothing here touches — neither of them slow, both past 30s on a contended host. An earlier attempt put `vi.setConfig({ testTimeout: 90_000 })` at the top of one of those files. It is still on main and it does not work: these cases register their timeout at collection, before a runtime call can move it, and the same file timed out at 30000ms with that line in place. Remove it and set the value where the suite reads it, behind an env knob so the pool can be retuned without a PR. Claude-Session: https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8 * test(agent-view): give the ready wait the product's own budget Four cases capped the worker-ready wait at 1s while the product allows 15s (DEFAULT_WORKER_READY_TIMEOUT_MS). None of them asserts anything about that deadline — it is scaffolding for the state they do assert. On the shared pool a worker takes longer than a second to report ready, so they failed three times each, with --retry=2 already on, in release run 33713579913. The two cases that DO assert the timeout keep their 1ms. Claude-Session: https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8 * test(serve): give the archive waits the room the shared host needs `vi.waitFor` defaults to a 1000ms deadline — a developer-machine figure, hardcoded in Vitest with no global knob. On the shared pool the archive install and its settle take longer than that, and release run 33713579913 lost both of these cases with --retry=2 already on. Only the five waits inside those two cases move. The deadline is scaffolding for the state they assert, never the thing under test. Claude-Session: https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8 * test: keep the complexity bounds asserting on the pool lane Fourteen sites are ReDoS and complexity guards — 'stays linear on pathological inputs', 'collapses an exponentially branching $ref type tree', 'does not catastrophically backtrack', 'runs in bounded time'. Skipping them outright leaves assertions that a quadratic regression satisfies, on the one lane every PR runs: ci-bot's witness is a 1.5s busy-spin in budgetGapDisclosures, green on the pool and red under the strict bound. The earlier pass only caught the sites with no other expect() at all. That judgement was too narrow — what matters is whether the remaining assertions survive the regression the bound exists to catch. Multipliers land the relaxed bound near 20s, a third of the pool's 60s testTimeout, so Vitest never decides first. review-footer's refusing-run case carries its own 20s timeout, so it takes x5. Claude-Session: https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8 * ci: quarantine latency budgets in autofix gates Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * test(ci): let autofix upsert use the suite timeout Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> --------- Co-authored-by: yiliang114 <jinjing.zzj@gmail.com> Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
|
Released in v0.23.0. |
What this PR does
This PR moves one
Date.now()inrun-ledger.test.tsout of thedescribebody and into the helper that uses it, so the ledger entries are stamped when the test runs rather than when the file is collected.Why it's needed
ledgerResumeCountreads entries through a fence:The block stamped its three entries from a
Date.now()evaluated in thedescribebody — at collection — whilebeforeEachwrites the plan file at execution. The entries are therefore older than the file they are fenced against by however long collection took, and once that exceeds the 2-second slack the fence drops them as a previous run's.Collection is normally a few hundred milliseconds, so the gap stayed inside the slack and the bug stayed latent. Release run 33713579913 collected for 2260 seconds on a contended host, and the block read 1, then 0, then 0 across its three
--retry=2attempts against an expected 2 — three different wrong answers, because the gap kept growing between attempts.Reproduced against the real module, bundled standalone so it runs without the workspace build:
ledgerResumeCountThe boundary sits exactly at
RUN_EPOCH_SLACK_MS, and the 2 → 1 → 0 progression is the one CI reported.The fence itself is correct — an entry older than the plan file it claims to have seen is from a previous run. The test was asserting against a stamp it took too early.
Reviewer Test Plan
How to verify
cd packages/cli && npx vitest run src/commands/review/lib/run-ledger.test.ts— theledgerResumeCount — entries past the originalblock passes, as it did before on a fast machine.it(), so their gap is milliseconds regardless of how long collection took.const now = Date.now()in this file (lines 108 and 818) are already insideit()bodies and are left alone — only thedescribe-level one was evaluated at collection.Evidence (Before & After)
Before:
expected 1 to be 2, thenexpected +0 to be 2, thenexpected +0 to be 1atrun-ledger.test.ts:149, inWorkspace Tests (3/3)of release run 33713579913, alongside 10,611 passing tests.After: the stamp is taken at call time; the table above shows the count holds at 2 for any gap when the stamp is current.
Tested on
Environment (optional)
The suite itself was not run here — this worktree has no built workspace
dist/, and building it is what this machine cannot do. The mechanism was verified instead by bundlingrun-ledger.tsstandalone with esbuild (its only cross-package import stubbed) and drivingappendRunSession/ledgerResumeCountdirectly at the gaps in the table. The file parses under esbuild and Prettier.Risk & Scope
now,now + 1000,now + 2000) and every assertion are unchanged.[vitest-worker]: Timeout calling "onTaskUpdate"unhandled error, which--retrydoes not cover, and shard 2/3 exhausting the job's 45-minute cap. Both are being handled separately.Linked Issues
Split out of #10870, which covers the wall-clock budget assertions. This one is a defect rather than a tradeoff, so it stands alone.
中文说明
本 PR 做了什么
把
run-ledger.test.ts中的一处Date.now()从describe体移入使用它的辅助函数,使 ledger 条目在测试执行时打时间戳,而不是在文件被收集时。为什么需要
ledgerResumeCount通过一道围栏读取条目:该代码块用一个在
describe体中求值的Date.now()给三个条目打戳 —— 也就是在收集阶段 —— 而beforeEach是在执行阶段写 plan 文件。因此条目比它们所对照的那个文件旧了「收集耗时」那么多,一旦超过 2 秒的容差,围栏就会把它们当作上一次 run 的数据丢弃。收集通常只有几百毫秒,差距落在容差内,所以这个缺陷一直潜伏。release run 33713579913 在一台被争抢的宿主上收集了 2260 秒,该代码块在
--retry=2的三次尝试中分别读到 1、0、0,而期望值是 2 —— 三个不同的错误答案,因为尝试之间差距还在扩大。针对真实模块复现(单独打包,无需 workspace 构建):
ledgerResumeCount拐点正好落在
RUN_EPOCH_SLACK_MS上,而 2 → 1 → 0 的递减正是 CI 报告的那一组。围栏本身是正确的 —— 一个比它声称见过的 plan 文件还旧的条目,确实来自上一次 run。是测试拿了一个取得过早的时间戳去断言。
Reviewer Test Plan
如何验证
cd packages/cli && npx vitest run src/commands/review/lib/run-ledger.test.ts——ledgerResumeCount — entries past the original代码块通过,与它此前在快机器上的表现一致。it()内取得,无论收集耗时多久,两者差距都是毫秒级。const now = Date.now()(第 108、818 行)本就在it()体内,未做改动 —— 只有describe级那一处是在收集阶段求值的。Evidence(修复前后)
修复前:release run 33713579913 的
Workspace Tests (3/3)中,run-ledger.test.ts:149报expected 1 to be 2、expected +0 to be 2、expected +0 to be 1,同时另有 10611 个测试通过。修复后:时间戳在调用时取得;上表显示只要戳是当前的,计数恒为 2。
测试环境
Environment(可选)
本地没有跑该测试套件 —— 这个 worktree 没有构建产物
dist/,而构建正是这台机器做不到的事。改为用 esbuild 将run-ledger.ts单独打包(其唯一的跨包导入用 stub 替代),直接驱动appendRunSession/ledgerResumeCount在上表各个间隔下验证机制。该文件通过 esbuild 解析与 Prettier 检查。风险与范围
now、now + 1000、now + 2000)与所有断言均未改变。[vitest-worker]: Timeout calling "onTaskUpdate"的 unhandled error(--retry覆盖不到),以及 shard 2/3 耗尽 job 的 45 分钟上限。两者另行处理。关联 Issue
从 #10870 拆分而出,那个 PR 处理的是挂钟预算断言。本 PR 是一个缺陷而非取舍,因此单独成 PR。
https://claude.ai/code/session_01MCE9CXnMVrX4fUxHpQoUr8