docs(adr): replace heuristic timeout allocator with statistical measurement contract

[seonghobae]

Current exact-head outcome — 2026-09-02

Current head: 12b16e8368f28de11a3f7fbb79cfb41f037fc904 on adr/0125-evidence-based-per-model-timeout-allocator. This is a Proposed design lane, not an executable timeout allocator and not merge-authoritative while exact-head checks/research prerequisites remain incomplete.

RCA

The predecessor design attempted to replace unsupported timeout constants with a sample-size-gated quantile allocator. Fresh review showed that this simply moved the no-heuristics violation into different constants and fallback policy: fixed sample floors, a fixed raw-sample retention count, percentile-class-specific estimator switching, an example EVT/POT threshold, and a model→profile/provider aggregate fallback without a fitted hierarchical model. Those rules affected whether a timeout value would be produced, but no executable statistical decision model identified those exact choices.

RED → repaired decision contract

RED contract commit 55d2d9a3a90b0f2f5246af86b6fbdd8fa36c910c rejects those fixed sample/retention/fallback rules. Repair commit 12b16e8368f28de11a3f7fbb79cfb41f037fc904 changes ADR 0125 into latency measurement without automatic timeout allocation:

• application/Agent/Gateway model timeout remains null by default;
• uncensored latency uses an explicit empirical distribution/quantile estimand and exact Binomial/order-statistic confidence construction for caller-supplied analysis parameters;
• externally terminated observations are right-censored rather than treated as completions or silently dropped;
• Kaplan–Meier plus Brookmeyer–Crowley inference is permitted only when the relevant independent-censoring assumptions are defensible, and non-identifiable requested quantiles return null;
• no cross-model/provider borrowing occurs without a separately fitted and validated hierarchical model;
• no EVT/POT extrapolation occurs without an executable threshold-selection/model-checking procedure;
• no automatic timeout recommendation is emitted without a separately identified loss/utility model, authoritative standard, or validated experimental decision rule; and
• every result binds the exact observations, analysis parameters, software/source identity, method, and deterministic provenance.

This lane does not revive #1010's enforcement wiring and does not claim that a descriptive latency percentile is timeout decision authority.

Ponytail / implementation prerequisite — still open

The repository's canonical docs/library_research.md contract requires every new subsystem design to record the existing libraries researched, the selected maintained library or stdlib alternative, the custom implementation deliberately skipped, and supporting evidence before implementation starts. ADR 0125's current branch does not yet add that latency/survival-analysis research-ledger entry.

Therefore no implementation should begin from this PR until that canonical ledger is updated. In particular, the branch must not silently hand-roll Kaplan–Meier, Brookmeyer–Crowley, exact Binomial inversion, or a tail model merely because the formulas are documented. The implementation owner must first evaluate maintained Rust-capable statistical libraries against the exact estimands, censoring semantics, uncertainty/recovery requirements, deterministic provenance, and fail-closed behavior; if no library has the required contract, that absence and the bounded custom-core decision/removal condition must be documented before code is introduced.

Exact-head verification boundary

Fresh PR-triggered runs on 12b16e8… remain non-terminal or failed at workflow startup: CodeQL PR 33627548381 is startup_failure; OSV-Scanner PR 33627547222, Security Scan 33627546628, SAST Semgrep 33627546619, Scorecard PR 33627546768, Fuzz 33627546712, Security 33627546744, and Tests 33627546693 are queued/pending. The CodeQL startup failure created no source-test success evidence, and predecessor results do not transfer.

Do not merge/auto-merge until the canonical library-research prerequisite is repaired and one unchanged exact head has terminal required evidence plus ordinary current-head review/protection. No heuristic substitute, force push, destructive rebase, self-approval, protection bypass, or fabricated status is authorized.

[coderabbitai]

Warning

Review limit reached

Next included review available in 17 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 5e2d9b2a-9cbc-4187-a6f3-8e7176e00b45

📥 Commits

Reviewing files that changed from the base of the PR and between c594b68 and 862ab64.

📒 Files selected for processing (2)

• docs/planning/adrs/0127-evidence-based-per-model-timeout-allocator.md
• tests/test_timeout_allocator_no_heuristics_contract.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 4 potential issues.

[devin-ai-integration]

🔍 Subsystem research record is missing

This new telemetry and estimation subsystem lacks the mandatory library-research entry. Record evaluated libraries, the selected approach, and skipped custom code before implementation.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 2 new potential issues.

[devin-ai-integration]

🔍 Research artifacts remain undocumented

The ADR adds no paper PDFs or redistribution rationale. The required citation-link-summary fallback is also incomplete for the order-statistics book.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🔍 Lexical checks miss contract regressions

Exact phrase checks allow reworded heuristics and accept required terms anywhere. The ADR can lose its substantive guarantees while these tests still pass.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[seonghobae]

Brought this branch current with main (it was stuck behind against the stale base 8839081, abandoned days ago while main advanced to 212ff43+).

• CI checks on the pre-merge head were all queued/skipped/cancelled at query time — a review-infrastructure artifact of the stale/superseded head (the startup_failure/pending runs referenced in the PR description are from an earlier, now-superseded run), not a current code failure.
• git merge origin/main --no-edit: no textual conflicts, but the merge exposed a same-number ADR collision: main already carries docs/planning/adrs/0125-domain-neutral-rater-observation-context.md (merged via fix(rater): preserve generic v1 and add criterion-bound wrapper #917), and this branch's own docs/planning/adrs/0125-evidence-based-per-model-timeout-allocator.md collides on the 0125 identifier. tests/test_planning_adr_identifiers.py (the executable uniqueness contract referenced in CLAUDE.md) fails on this exact case. Per this repo's documented convention ("a same-number collision is a rename, not a redesign"), I renumbered this branch's ADR to the next free identifier, 0127 (0126 is already taken by another already-merged ADR), updating only the file's own title line and the matching path/title reference in tests/test_timeout_allocator_no_heuristics_contract.py — no content change.
• Verified locally in a fresh venv (pip install --require-hashes -r requirements.lock && pip install --no-deps -e .):
  • test_planning_adr_identifiers.py, test_timeout_allocator_no_heuristics_contract.py, test_paper_contracts.py, test_repository_security_metadata.py (docs/ADR contract suites): all pass after the rename
  • Merge-diff-touched suites (test_rater_observation*.py, test_review_gateway*.py, test_admin_contract.py): 83 passed, 1 pre-existing failure (test_admin_contract.py::test_model_group_mutations_refresh_audit_events, NameError: name 'json' is not defined, already present on main HEAD, unrelated to this PR, tracked/fixed separately in fix(admin): repair test_model_group_mutations_refresh_audit_events #1029)
• Pushed the merge commit plus the renumbering fix commit directly to adr/0125-evidence-based-per-model-timeout-allocator (no force-push). Note the branch name itself still says 0125 — left as-is since renaming the branch would change the PR's identity; only the file/test content was renumbered.

---

Generated by Claude Code

[seonghobae]

Fixed the one concrete, unaddressed finding: contract file wasn't runnable as a direct script

Triaged this PR's open threads. Most of the remaining open ones (missing docs/library_research.md ledger entry, paper-PDF/citation grounding for the order-statistics book, the lexical-check robustness note) are explicitly acknowledged as open prerequisites in this PR's own description ("Ponytail / implementation prerequisite — still open... no implementation should begin from this PR until that canonical ledger is updated") — this PR is self-described as a non-merge-authoritative "Proposed design lane," so those are correctly left to the owner rather than bundled into an unrelated push.

One finding was concrete and independently verifiable, though: Devin flagged that running tests/test_timeout_allocator_no_heuristics_contract.py directly exits successfully without invoking either contract test. Confirmed real — unlike every sibling contract test in this repo (test_paper_contracts.py, test_repository_security_metadata.py, etc.), this file was missing the if __name__ == "__main__": block that actually calls each test function, so it silently validated nothing when run the way this repo's own documented workflow ("each test file is a directly runnable script") calls for.

Fix (commit 63ecf03b): added the standard if __name__ == "__main__": ... print("ok") block, matching this repo's existing idiom exactly.

RED confirmed: reverted the change and ran python3 tests/test_timeout_allocator_no_heuristics_contract.py directly — exit 0, zero output, neither assertion ever ran.

GREEN: same direct invocation now prints ok. pytest tests/test_timeout_allocator_no_heuristics_contract.py -q — 2 passed, unchanged under pytest collection (this was never a pytest-collection bug, only a direct-execution one).

Pushed non-force on top of the existing branch (0806973e → 63ecf03b).

---

Generated by Claude Code

[seonghobae]

Autonomous loop note: exact head 63ecf03b31f8ca9be60160e9c4d094cc6819f009 (base main@212ff437, pre-dates the json import fix) failed three checks:

• Full unit and contract suite (run 33692092712): known tests/test_admin_contract.py::test_model_group_mutations_refresh_audit_events - NameError: name 'json' is not defined — already fixed on main (212ff437 → 39a4348e); not caused by this PR's docs-only diff.
• CodeQL analysis (run 33692092707): "CodeQL job status was configuration error" — the known org-wide advanced-configuration-vs-default-setup conflict tracked centrally in .github.
• noema-review (run 33692092251): 502 Bad Gateway, phase=connecting — transient gateway infra. Re-ran the failed job.

Called update_pull_request_branch to merge current main into this branch so the stale-fixture failure clears; this is routine CI hygiene only and does not constitute merge authorization — the PR body's own "not merge-authoritative" / library-research-prerequisite gate stands untouched.

---

Generated by Claude Code