test: scale wall-clock timing budgets on loaded runners - #2509
Conversation
The workspace-snapshot perf guard failed at 1.635s against a hard 1.5s median budget while the machine was busy with parallel builds, and the Kiro probe's teardown budgets are load-sensitive the same way. Both are regression guards against algorithmic blowups and hung processes, not latency SLOs. Add `TestTimingBudget` so timing budgets scale 3x under CI, and measure the perf guard as best-of-three instead of the median — the fastest run is the one least interrupted by other work.
clawsweeper
Bot
added
rating: 🐚 platinum hermit
|
Codex review: needs maintainer review before merge. Reviewed August 2, 2026, 9:48 PM ET / August 3, 2026, 01:48 UTC. ClawSweeper reviewWhat this changesThe branch adds a shared test timing-budget helper, scales three wall-clock test limits on CI-like runners, and changes the workspace snapshot performance check from median-of-three to fastest-of-three timing. Merge readinessThis owner-authored PR is still necessary: current Priority: P3 Review scores
Verification
How this fits togetherCodexBar’s Swift test suite validates local usage snapshots and CLI subprocess cleanup. The changed assertions observe elapsed time after those operations and determine whether the suite reports a performance or teardown regression. flowchart LR
A[Test runner environment] --> B[Timing budget helper]
C[Workspace snapshot build] --> D[Three elapsed-time samples]
D --> E[Fastest-run performance guard]
B --> E
F[Kiro CLI subprocess tests] --> G[Timeout and teardown guards]
B --> G
E --> H[Test result]
G --> H
Decision needed
Why: The code is mechanically small and preserves the non-timing correctness assertions, but the preferred balance between flake resistance and sensitivity to sustained performance regressions is an owner-level test-policy choice. Before merge
Agent review detailsSecurityNone. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Merge the focused helper once the owner accepts the deliberate fastest-run trade-off and the current macOS test shard passes, keeping the existing snapshot-content and subprocess-cleanup assertions unchanged. Do we have a high-confidence way to reproduce the issue? Yes, at source level: current main uses fixed wall-clock thresholds for the three identified tests, and the PR body supplies a concrete loaded-runner failure observation. This read-only review did not execute the timing-sensitive tests because repository policy prohibits validation that could trigger local credential prompts without explicit request. Is this the best way to solve the issue? Yes, with a maintainer policy choice: centralizing the three CI-aware budgets is narrower and more maintainable than individual ad hoc relaxations, while the existing functional assertions still detect incorrect snapshots and incomplete subprocess cleanup. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 0c5255c1838e. LabelsLabel justifications:
EvidenceWhat I checked:
Likely related people:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
HistoryReview history (3 earlier review cycles) |
1 participant
Problem
CodexModelsPerformanceTestsasserts a hardmedian < .seconds(1.5)wall-clock budget on the workspace-snapshot build. It failed at 1.635s on a machine that was busy running parallel builds — the code was fine, the machine was loaded.KiroStatusProbeTestshas the same shape of assertion for process-timeout and teardown behaviour (< .seconds(7),< .seconds(2)), which degrades the same way under contention.Both are regression guards: they exist to catch an algorithmic blowup in snapshot building, and a hung child process on teardown. Neither is a latency SLO, so neither should fail because a CI shard was sharing a runner.
Change
TestTimingBudgethelper: budgets scale 3x whenCIorGITHUB_ACTIONSis set.The guard still bites: a real O(n^2) regression in snapshot building would blow past 2s on the fastest run, not just the median.
Proof
Local, on this machine:
swift test --filter CodexModelsPerformanceTests— passes, fastest run 0.475s against the 2s budget.swift test --filter KiroStatusProbe— 55 tests pass in 28s.Scripts/lint.sh— rc=0.