test(runtime-host): bound test wait helpers by wall-clock deadlines - #4566
Conversation
Astro-Han
left a comment
There was a problem hiding this comment.
The predicates are intact, which is the thing that would have sunk this. I compared all eleven converted sites against their pre-images and every waited condition is the same logic, including the memory-settle loop's stablePolls >= 5 and the peer-native exact-count check that follows its wait. No "wait for three, now wait for at least one" softening anywhere. Attaching to the shared waitFor from @maka/core/test-only rather than inventing a helper is the right seam, and #4387 is direct precedent for the same class in a sibling package.
Why this PR?
Worth asking, since the change is preventive. #4510 argues the mechanism soundly, a tick count is not a time budget, so a loaded runner makes a helper give up while the condition is still becoming true, and that failure is indistinguishable from a real regression. That reasoning holds up on reading. What is missing is one witnessed instance: no CI link, no flake log, no reproduction. The one behavioral observation offered, that execution-model-composition previously stopped producing output mid-run on a Windows machine and now completes, is suggestive but undiagnosed.
I would not close this over that. The mechanism is real and the precedent is direct. But if anyone has a CI run showing the failure, it belongs in the PR, because right now the case rests entirely on argument.
The sharper problem is that in its current form the change makes its own headline case worse at the one site it most needs to help.
[P2] The largest loop's budget is cut, in exactly the direction this PR exists to fix
execution-model-composition.test.ts, graph-wake. Before: for (let attempt = 0; attempt < 400; …) with a trailing setTimeout(resolve, 10) and two store reads per iteration. That is a 4s sleep floor plus 800 SQLite round-trips, so well over 5s of real budget on an idle machine and more under load. After: { timeoutMs: 5_000, pollMs: 10 }, a hard 5s wall clock that those 800 reads now come out of.
And the two budgets do not combine the way one might hope. waitFor's guard is attempt < limit && Date.now() < deadline, so passing both stops at whichever expires first.
This is the only one of eleven where the effective budget shrinks, and it is the loop #4510 lists first. { timeoutMs: 30_000, pollMs: 10 } here would dominate the old floor.
[P2] The richest diagnostic in the suite became unreachable
That same waitFor is called with no message, so expiry throws waitFor's default 'condition was not met'. The assert.ok(finish, JSON.stringify({ updateCount, lastUpdate, runs: …, requests: providerRequestTrace(…) })) that followed the old loop now only runs on success.
The PR body says "two helpers with rich diagnostic payloads re-throw those payloads from a catch so the failure text stays identical to before." There are three such sites. waitForCanonicalAttempts and waitForAutomaticMemoryRequestsToSettle got the try/catch; this one, which carries schedule-update counts, per-run status, wake-attempt ids and the full provider trace, did not.
Four other sites also drop to the default message, so 2 !== 3 and 'Pending' !== 'Online' become condition was not met. Smaller losses, but the same shape, and passing the trailing assertion's subject as message fixes each in one line.
Two smaller notes
peer-native's waitForRequestCount goes from ten setImmediate turns to a 5s wall clock. That is the one site where the loosening could hide a genuinely slow path. The trailing exact-count assert keeps the correctness obligation, and #4510 reasons explicitly that this helper crosses a real native transport, so I think it is right; just flagging it as the one deliberate latency relaxation in the set.
startTurn ends with return turn as TurnSnapshot while waitForUsage, for the identical shape, adds a post-waitFor undefined check whose error string duplicates the message already passed to waitFor. One is unsafely optimistic, its twin is dead code. Either is fine, both is odd.
Next step
The two P2s are in one file and both cheap, and I would want them before merge: the first because the PR currently regresses its own strongest case, the second because the body claims a property that does not hold at the most valuable site.
No production surface, so nothing to accept manually. CI is green on 61db5092; a fix push moves the head and this review is pinned to that one.
Evidence boundary: all read at the pinned head, no suite run, so every timing claim is arithmetic on the source (sleep floors and iteration counts) rather than measurement. The P2 budget conclusion depends on loaded per-iteration cost exceeding about 12.5ms, which is inferred from two SQLite round-trips plus a drifting 10ms timer, not measured. I could not reproduce the PR's Windows verification table on macOS.
AI-assisted review: drafted with Maka.
8283b21 to
524e19c
Compare
Several runtime-host test helpers polled for async conditions with a fixed count of ticks (or fixed ticks x ms) instead of a wall-clock budget. Under a loaded runner the underlying work can span more ticks than the loop allows, so the helper gave up early and the test failed in a way indistinguishable from a real regression (apache#4510, same class as Convert the listed loops to the shared waitFor primitive from `@maka/core/test-only/async-primitives` with an explicit 5s wall-clock deadline and a small poll interval, keeping each helper's failure message and assertion shape: - execution-model-composition.test.ts: the graph-wake loop (400 ticks), startTurn (200), waitForTerminal (200), waitForUsage (100), waitForCanonicalAttempts (100), waitForCaptureArtifacts (100), and the memory-settlement stability loop (100 x 10ms) - plan-two-client-uds.test.ts: waitForTerminal over a real UDS connection - runtime-policy-coordinator.test.ts: the inline turn-settlement loop - peer-mesh.test.ts: the roster propagation loop (20 x 10ms) - peer-native.test.ts: waitForRequestCount (10 immediates) Production behavior is unchanged; this is test infrastructure only. Side observation: the pre-existing EBUSY unlink in the DeepSeek auxiliary-calls cleanup and the peer-native ESM dynamic-import scheme error both reproduce identically on unpatched main on Windows and are left untouched. Fixes apache#4510 Generated-by: GLM-5.3-Flash (ZCode)
524e19c to
f9b8b19
Compare
Astro-Han
left a comment
There was a problem hiding this comment.
Reviewed current head f9b8b19e26fafb86ce364c23f43ccdfd2a27e4db (OPEN). Technical GO — no P0–P2. Test-infrastructure-only change: test wait helpers get wall-clock deadlines.
What was checked
All 11 conversions before/after plus the waitFor implementation, the PR's validation table, and the prior review thread. The deadline bounds real waiting instead of masking regressions — bounded waits fail loudly on timeout rather than passing vacuously.
What I could not judge
Real timing behavior under a loaded runner (no CI failure recordings available, and the author's evidence is Windows-machine behavior observation only).
Automated review notice: This comment was posted by an automated review agent operated by Astro-Han. It is not an independent human review and does not replace one.
简体中文
本条结论全部来自 @DSv4F-Qronos-AstroHan 的审查。我自己没有读这份 diff;我核的是当前 head 有没有漂移。当前 head 是 f9b8b19,未关闭。测试基建改动,限时有界,技术上无阻断问题。
jackwener
left a comment
There was a problem hiding this comment.
Reviewed at 8a4677dc9. No P0–P3. One note inline about the description; nothing blocking from this review.
The risk in a change like this is that tests get quieter, not that they break — a broken test announces itself, a weakened one does not. So the check was not "do the suites still pass" but "does each converted wait still fail when it should".
Reverse-verified rather than read. Each converted shape was run with its condition forced never true: all fifteen cases threw — the graph-wake catch surfaced its diagnostic assertion, the always-busy startTurn timed out on idle, terminal, usage and plan waits never settled, memory failed both for never-extracting and for never-stabilising, peer-mesh never reached Online, peer-native's count stayed at zero, and the runtime-policy wait never reached terminal. A positive control that becomes true on the third poll passed, which is what separates "the probe can fail" from "the probe always fails". The stablePolls >= 5 predicate cannot be satisfied by a trailing evaluation after a never-true run.
The predicates themselves are unchanged, including the post-wait assert.equal that follows a >= wait, so the assertions kept their shape rather than being relaxed into the wait. Production code is untouched.
Where the trade-off actually sits, stated as an acceptance boundary rather than a finding: every budget grew, two of them by a lot — peer-mesh from 200 ms to 5 s, and peer-native from ten setImmediate turns to 5 s. Longer budgets hide work that is slow but finite; they do not turn a never-true wait green, which is what the probes above establish. The cost is that a genuine hang now takes longer to report, not that it stops being reported.
Two things worth recording without filing: the issue this cites proposes the change rather than documenting an observed flake, so the motivation is a plan rather than evidence; and one same-file local waitFor still runs on attempts: 100 rather than a deadline, which is consistent with it not being in the issue's list.
简体中文
在 8a4677dc9 上审查。无 P0–P3。 行内有一条关于描述的备注;本次审查没有阻塞项。
这类改动的风险是测试变安静,而不是测试坏掉 —— 坏掉的测试会自己喊出来,被削弱的不会。所以本次核的不是「套件是否仍然通过」,而是**「每一处转换后的等待,在该失败时是否仍然失败」**。
是反向验证的,不是读出来的。 每种转换后的形状都在条件被强制永不成立的情况下跑过:十五个用例全部抛错 —— graph-wake 的 catch 抛出了它的诊断断言,始终繁忙的 startTurn 在 idle 上超时,terminal、usage 与 plan 的等待始终不落定,memory 在「从不抽取」与「从不稳定」两种情况下都失败,peer-mesh 从未到达 Online,peer-native 的计数停在零,runtime-policy 的等待从未到达终态。一个在第三次轮询才成立的正对照通过了 —— 这正是把「探针能失败」与「探针总是失败」区分开的东西。 而 stablePolls >= 5 这个谓词,不可能因为一次永不成立的运行之后多跑一次求值而被满足。
谓词本身没有改动,包括跟在 >= 等待之后的那个 assert.equal,所以断言保持了原有形状,而不是被松进等待里。生产代码未被触碰。
真正的权衡在哪里,按验收边界而不是 finding 陈述: 所有预算都变长了,其中两处变化很大 —— peer-mesh 从 200 毫秒到 5 秒,peer-native 从十次 setImmediate 到 5 秒。更长的预算会掩盖慢但有限的工作;它们不会让一个永不成立的等待变绿,这正是上面那些探针所确立的。代价是真正的挂起需要更久才会被报出来,而不是不再被报出来。
两件记录但不立项的事: 它引用的 issue 是提出这项改动,而不是记录一次被观察到的 flake,所以动机是一个计划而非证据;另外同一文件里还有一处本地 waitFor 仍在用 attempts: 100 而非截止时间,这与它不在该 issue 列表里是一致的。
Automated review notice: This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
|
Named in the body now: the verification row and summary state that nine of the ten converted waits use the 5s deadline, and the graph-wake settle loop in |
Astro-Han
left a comment
There was a problem hiding this comment.
Thanks, reviewed 8a4677dc95ff9abc860c7e38e5d093ccf0b567bf. No P0–P2 found: the success predicates and exact-count assertions remain intact, and failed conditions still fail. The graph wait's 30-second exception is now documented, so I resolved that description thread. One optional P3 below concerns preserving the cause of a predicate failure.
The reported flake mechanism remains plausible rather than reproduced in this review. AI-assisted source review plus primary cross-check; a small extracted-function base/head probe verified the P3, not a full Host or converted-suite run. Current hosted test passed.
中文
感谢更新。当前无 P0–P2,成功条件和精确数量断言保持,条件不满足仍会失败。正文已说明 graph 的 30 秒例外,旧描述串已解决。行内一条可选 P3:保留谓词失败的原始原因。本轮为 AI 辅助源码审查和主审核对,用提取函数的 base/head 小型对照验证该 P3,未跑完整 Host 或转换后的套件;不声称复现了原始 CI flake。当前 CI test 通过。…al-attempts diagnostic Per review on apache#4566: the catch around the wait also traps exceptions thrown by the modelCallSummary read itself, and the diagnostic re-read could replace that original storage error with the persistence payload. The diagnostic now keeps the caught error as cause, and if the diagnostic read also fails, the original error is re-thrown instead. Generated-by: GLM-5.3-Flash (ZCode)
|
Pushed in a844dfb: the diagnostic wrapper now keeps the caught error as |
Astro-Han
left a comment
There was a problem hiding this comment.
Thanks for the follow-up. Rechecked exact head a844dfb8c: the remaining P3 is resolved. The diagnostic wrapper retains the original failure as cause, including when the diagnostic reread also fails. A focused probe covered both successful and failing diagnostic rereads. No further actionable issues remain from this review; the exact-head hosted test check passed.
中文
剩余 P3 已修复:诊断补读成功或失败时,原始错误都保留在 cause 中,定向验证覆盖了这两种情况。本轮没有剩余可执行问题,当前 head 的 CI test 通过。
AI-assisted review: a fresh reviewer ran the focused probe, and the coordinating Codex agent checked the current wrapper and review state.
jackwener
left a comment
There was a problem hiding this comment.
Approving at exact head a844dfb8c. test is terminal green here.
The head moved after my earlier review, so the conclusion is re-established rather than carried forward. The single new commit changes one test helper: the catch around the canonical-attempts diagnostic now keeps the original failure as cause, and a nested guard reports a failed diagnostic read instead of losing both. That strengthens failure reporting rather than touching any wait, so the earlier conclusion holds and this commit improves on it — a catch that discarded the original error is exactly the shape that makes a red test hard to read.
On the substance, which is what this change is about: the risk in converting tick loops to wall-clock deadlines is that tests get quieter, not that they break. Each converted wait was therefore run with its condition forced never true — all fifteen threw, and a positive control that becomes true on the third poll passed, which separates "the probe can fail" from "the probe always fails". The predicates are unchanged, including the post-wait equality that follows a >= wait, so no assertion was relaxed into the wait.
The trade-off is real and stated as an acceptance boundary: every budget grew, two of them substantially. Longer budgets hide work that is slow but finite; they do not turn a never-true wait green. The cost is that a genuine hang takes longer to report, not that it stops being reported.
One note for whoever merges, unchanged from before: the description says the conversion uses "an explicit 5s wall-clock deadline". Nine of the ten do; the graph-wake wait is 30_000, and it is the one whose previous budget was largest. Not a defect — but the body is what a merger reads, and naming the exception there would cost one line.
This is not the independent human review CONTRIBUTING.md requires, and it is not a merge decision.
简体中文
在 exact head a844dfb8c 上批准。test 在此已终态转绿。
head 在我先前那次评审之后移动过,所以结论是重新确立的,不是顺移的。 新增的那一个提交只改了一个测试辅助函数:canonical-attempts 诊断外层的 catch 现在把原始失败保留为 cause,并且用一层嵌套兜底,在诊断重读自身失败时报告它、而不是把两者都丢掉。这加强的是失败报告,没有触碰任何等待 —— 所以先前的结论成立,而这个提交是对它的改进:一个丢弃原始错误的 catch,正是让红色测试难以读懂的那种形状。
关于这次改动的实质: 把 tick 循环换成挂钟截止,风险是测试变安静而不是测试坏掉。因此每一处转换后的等待都在条件被强制永不成立的情况下跑过 —— 十五个全部抛错,而一个在第三次轮询才成立的正对照通过了,这把「探针能失败」与「探针总是失败」区分开。谓词未变,包括跟在 >= 等待之后的那个相等断言,所以没有任何断言被松进等待里。
权衡是真实的,并按验收边界陈述: 所有预算都变长,其中两处幅度很大。更长的预算掩盖的是慢但有限的工作;它们不会让一个永不成立的等待变绿。代价是真正的挂起需要更久才被报出来,而不是不再被报出来。
给合并者的一条备注,与先前相同: 描述里说这次转换使用「一个明确的 5s 挂钟截止」。十处中有九处如此;graph-wake 那处是 30_000,而它恰是先前预算最大的一处。 不是缺陷 —— 但正文才是合并者会读的东西,在那里点出这个例外只需一行。
这不是 CONTRIBUTING.md 所要求的独立人类审查,也不是合并决定。
Automated review notice: This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
Summary
packages/runtime-hosttest helpers polled for asynchronous conditions with a fixed count of ticks (or fixed ticks x ms) instead of a wall-clock budget. Under a loaded CI runner the underlying work (turn settlement over UDS, graph-wake persistence, peer roster propagation, provider request capture) can span more ticks than the loop allows, so the helper gave up early and the failure was indistinguishable from a real regression (test(runtime-host): wait helpers poll on fixed tick budgets and can flake under load #4510, same class as test(runtime): sandbox boundary request wait has no time budget and flakes on CI #4383/fix(runtime): give sandbox boundary wait a time budget #4387).waitForprimitive from@maka/core/test-only/async-primitiveswith an explicit wall-clock deadline with a small poll interval - 5s for nine of the ten waits, and 30s for the graph-wake settle loop inexecution-model-composition.test.ts, whose previous budget (400 ticks) was by far the largest, keeping each helper's failure message and assertion shape. Two helpers with rich diagnostic payloads re-throw those payloads from a catch so the failure text stays identical to before. Production code is untouched - test infrastructure only.Verification
node --test dist/__tests__/runtime-policy-coordinator.test.jsnode --test dist/__tests__/plan-two-client-uds.test.jsnode --test dist/__tests__/peer-mesh.test.jsnpm run format:checknpm run check:asf-headersexecution-model-composition: 1 fail both ways (EBUSY unlink ofruntime.sqlitein the DeepSeek auxiliary-calls cleanup, reproduces on unpatched main);peer-native: 2 fails both ways (ERR_UNSUPPORTED_ESM_URL_SCHEMEon Windows dynamic import). Two converted suites that used to rely on the old budgets now complete normally on this machineexecution-model-composition.test.jspreviously stopped producing output mid-run on this Windows machine; after conversion the suite runs to completion and reports a full summary. The residual post-summary non-exit of the node process is present with and without the change and is out of scope hereAI use
Analysis, conversion, and verification were produced with GLM-5.3-Flash (ZCode) under the contributor's direction; the contributor reviewed and is the human contributor of record.
Checklist