fix(auto-routing): keep session incumbents inside the benchmark noise band

[iscekic]

Summary

In cost_per_accuracy mode, computeDecision kept a session's incumbent model only when the incumbent cleared the route's minAccuracy bar (0.95). That bar is a point estimate over 10 distinct benchmark cases per route (repetitions re-run the same prompts and add no independent tasks), so pass/fail flips on binomial noise — and every flip ejects every session parked on that model, discarding a warm prompt cache at ~72k median context.

The change replaces the meetsThreshold requirement on the sticky path with a tolerance band: keep a cost-mode incumbent whose route accuracy is within 0.10 of minAccuracy. best_accuracy mode still requires meetsThreshold, and the existing cost escape (fresh pick cheaper by more than switchCostFactor) still fires. switchReason telemetry now derives from the same predicate as the keep decision, so a band-eligible incumbent that is cost-switched is labelled cost, not threshold.

Measured against the live routing table (decider-2026-07-22T10-51-06-305Z) and a pinned 24-hour telemetry window (2026-07-26T12:00Z – 2026-07-27T12:00Z, Axiom cloudflare-logpush):

Measure	Baseline	Predicted after deploy
Real model switches/day	109 (threshold 91, cost 16, capability 2)	threshold ~42, cost ~22, capability 2
Sessions with ≥1 switch/day	54 (23 multi-switch, one with 11)	should fall materially

The band makes 49 of the 91 threshold-forced incumbents eligible to stay; 6 of those still fail the unchanged cost condition and are relabelled threshold → cost rather than prevented — hence the predicted cost rise, which is expected behaviour, not a regression. Net effect: 43 of 91 daily forced switches (47 %) disappear, including 9 of the 14 daily punts onto claude-sonnet-5 in planning_design/technical_planning (its sole passer, at 6.2–14.2× the incumbents' benchmark-case cost). The split is discriminating: the largest remaining ejection (minimax-m3 leaving debugging/bug_fixing at 24/30) is genuinely weak there and still fires.

Correction to the source document. The source doc's diagnosis is confirmed (83 % of switches are threshold-forced; 98 % follow a route change, concentrated in sessions oscillating between semantically adjacent routes). Its magnitude is wrong by ~53×: it modelled 22.6 switches per 100 turns by assuming a 50 % route-change probability per turn; the measured rate is 0.43 per 100 decisions, because the classifier cache hits 95 % of the time within a conversation. The modelled "$300–400/day of forced-switch cache rebuilds" is really $2–25/day. The justification for shipping is therefore not the line item but the 54 sessions/day that get a model swapped mid-conversation — a visible latency and behaviour event — 83 % of them decided by a gate the benchmark cannot resolve.

Honest disclosure — what the band retains. Classifying all 43 net retentions: A1 (19/day) keeps an incumbent that is materially cheaper (0.07–0.86×) than a threshold-clearing fresh pick; A2 (11/day) declines a qualified upgrade for cache continuity alone — the modal case keeps kimi-k2.7-code (27/30) over minimax-m3 (29/30, clears the bar) at a $0.000005 cost difference, i.e. it acts on a difference of 2 graded cases out of 30 that n = 10 cannot resolve (the one-sided Wilson upper bound for 0.9000 is 0.977, above the bar); B (13/day) chooses between two models that both miss the bar on a zero-passer route. A2 is the class a reviewer should challenge: this change forgoes ~11 qualified upgrades/day by design. A dominance guard that would cut the 10/day strictly-dominated retentions was considered and rejected — it acts on avgCostUsd margins (1.16–1.26×) that are not trustworthy at that size, and it barely touches A2 (11 → 10/day) because in the modal case the fresh pick costs fractionally more. Re-examine both if the routing table ever carries a real per-turn cost signal (cache-read price rather than benchmark-case average).

Simpler-shape decision (recorded per workflow). The source doc asked for a Wilson lower bound / publish-time hysteresis on meetsThreshold (Step 2) and an ε band on the sticky condition (Step 3). Implementing both separately means publishing per-candidate case counts or interval bounds through RankedCandidateSchema → the table builder → a new D1 column → rowsToRoutingTable, then computing an interval at serve time. At the live dataset size that machinery is inert: with n = 10 the Wilson boundary sits at an observed accuracy of ≈ 0.837, a 0.10 band keeps everything ≥ 0.85, and the two rules differ only on accuracies in [0.837, 0.85) — of which the live table contains none (every published accuracy is a multiple of 1/30). On the data actually served the two rules select identical models, so this PR ships the constant, calibrated against the measured identical-behaviour plateau [0.0833, 0.1167) ("keep ≥ 26/30, eject ≤ 25/30"), with the upgrade path marked in a code comment.

Rollback: revert-and-redeploy; the change takes effect on the next auto-routing worker deploy. Deliberately no kill-switch knob — adding one would mean the same contract/migration surface this PR avoids.

Deferred, with re-entry criteria (not silently dropped):

• (a) Adding benchmark cases to hot routes (source Step 2's first move). New decider cases need a full benchmark run to validate, and a poorly specified case shifts the very thresholds this change makes robust. Re-entry: its own change, acceptance = a published benchmark run, once the per-route case results in the benchmark D1 are readable.
• (b) Route-flip debounce (source Step 4). The phenomenon is real and dominant (100 % of switches follow a route change; multi-switch sessions visibly alternate adjacent routes), but the band breaks those cycles at the source, and whatever it does not catch is where keeping the model costs the most accuracy — where a debounce is least defensible. Debounce also needs new Durable Object state and one turn of delayed adaptation. Re-entry: on ≥ 3 days of post-deploy data, if switch volume has not fallen materially and multi-switch sessions still alternate, build it.
• (c) "Within ε of the best candidate" arm / stickiness on empty passing sets (source Step 3's second arm). Deferred as a genuinely different policy with a worse failure mode: relative-to-best keeps an incumbent where nothing is good (on codebase_understanding the field spans 16/30–28/30; "within ε of best" could retain a 0.53 model under a 0.60 best). Note the shipped absolute band already fires on zero-passer routes — investigation/codebase_understanding has 0 passers but 9 of 14 candidates in band — so the arm is not needed to reach the degenerate routes. Re-entry: a route with meaningful traffic that keeps switching and has no in-band candidates at all gets its own change and review.
• (d) Cost-relative override (keep a below-band incumbent when the only passer is dramatically more expensive). Deferred as a second, uncalibrated policy; the band plus the existing cost condition covers the common case.

Post-deploy validation (not a gate on this PR): re-run the pinned-window Axiom query on ≥ 3 days of data. Primary signal: threshold switches ~91/day → ~42/day (falsifiable prediction; a materially different outcome means the mechanism model is wrong), cost 16 → ~22 (relabels), capability unchanged at 2. A cost count materially above ~22 is the real warning sign — it would mean in-band incumbents are kept and then cost-switched a turn later, churn the band was supposed to remove rather than relocate. Guardrail: watch per-route decidedModel mix for drift toward the expensive end, since an in-band incumbent can now sit at up to 3× the cheapest eligible model's cost on every subsequent turn — the main product risk against the one-time rebuild saving. The prediction is void if a new routing table publishes before measurement.

Verification

No manual device or UI testing: the change alters which model a server-side routing decision returns inside a worker with no UI, no new failure mode, and no user-facing surface — the feature-state matrix is structurally not applicable, and no mobile/web client renders anything derived from sticky or switchReason.

• pnpm vitest run src/decision-engine.test.ts — 38/38, including 5 new fixtures taken verbatim from the live routing table (zero-passer keep, sole-expensive-passer keep, qualified-upgrade-declined keep, in-band cost relabel, best_accuracy still ejects)
• pnpm test in services/auto-routing (143/143) and packages/auto-routing-contracts (84/84) — every pre-existing test passes unedited (no existing test encodes "below threshold ⇒ always switch"; the one candidate, weak/chat at 0.5 vs a 0.6 band floor, was pre-verified)
• Root pnpm typecheck, pnpm lint, git diff --check clean
• Independent reviewer pass: No findings (residual risks noted below)

Visual Changes

N/A

Reviewer Notes

• Everything hangs on one predicate, incumbentStickyEligible, which is shared by the keep decision and the switchReason derivation so the two cannot disagree — the one way this change could silently corrupt its own before/after measurement was keying switchReason to meetsThreshold while keeping on the band. Test 5d (planning_design/architecture_design, inkling $0.00660620 vs incumbent claude-sonnet-5 $0.03943792, both below bar but in band → must read cost) exists specifically to catch that; test 5e (same route as 5b, mode flipped to best_accuracy, accuracy gap 0.0334 < the 0.05 switch threshold) exists to catch the band leaking into best_accuracy.
• Fixture arrays list the named fresh pick first because pickFreshCandidate returns candidates[0] in cost mode — a fixture listing the incumbent first would silently test a different scenario.
• Residual testing risks from the independent review: no unit case pins the exact band floor (live fixtures sit strictly above it); 5d/5e discrimination was verified by fixture arithmetic and control-flow reading rather than by breaking the predicate; the post-deploy Axiom checks above remain the real production gate.
• The second commit (docs(mobile): record planner workflow learnings) is a required workflow deliverable, unrelated to the behavioural change.

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

Since the last review, only apps/mobile/.kilo/WORKFLOW_LEARNINGS.md changed (a 14-line docs-only addition of orchestrator workflow notes); no code in services/auto-routing changed, so the previously reviewed incumbentStickyEligible sticky-band logic and its fixtures stand as-is with no new risk introduced.

Files Reviewed (1 file)

• apps/mobile/.kilo/WORKFLOW_LEARNINGS.md

Previous Review Summary (commit b86d01f)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit b86d01f)

Status: No Issues Found | Recommendation: Merge

Executive Summary

Reviewed the sticky-band predicate change in computeDecision (services/auto-routing/src/decision-engine.ts) and its five new fixture tests against the live routing table; the shared incumbentStickyEligible predicate is applied consistently to both the keep decision and the switchReason telemetry, and every new test's expected outcome checks out against the implementation's arithmetic (band floor, cost-escape, and best_accuracy threshold branches). No security, correctness, or runtime issues found in the changed lines; the docs-only WORKFLOW_LEARNINGS.md addition carries no functional risk.

Files Reviewed (3 files)

• services/auto-routing/src/decision-engine.ts
• services/auto-routing/src/decision-engine.test.ts
• apps/mobile/.kilo/WORKFLOW_LEARNINGS.md

---

_{Reviewed by claude-sonnet-5 · Input: 22 · Output: 3.6K · Cached: 599.7K}

_{Review guidance: REVIEW.md from base branch main}