feat: replace routing heuristics with measured evidence ledgers - #847
Conversation
Remove DOMAIN_HINTS/COMPLEX_HINTS keyword tables; route via eligibility contracts, declaration priority/capability fit/cosine affinity over operator-declared metadata, and measured intra-group quality then EWMA tokens-per-second. Add structured fail-closed triage gas with content-hash verdict caching and real-time fast-mlsirm judging on direct routes that feeds a Beta-Bernoulli quality ledger with in-budget failover. ADR 0027 + doctoring APA 7 references (Jacobson 1988; Gelman et al. 2013; Karpukhin et al. 2020; Ong et al. 2024; Chen et al. 2023; Zheng et al. 2023; Jeon et al. 2021). Gap baseline added at docs/product-technical-gap-baseline.md.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…overy' into codex/pr847-review
…overy' into codex/pr847-review
|
Review remediation on exact head
Evidence: 74 focused tests passed before base refresh; 68 integration/focused tests passed after merging current #834. The pre-refresh full suite completed 1923 passed with two free-model error-precedence regressions; both regressions were fixed and their focused tests passed. Hosted exact-head checks/review remain authoritative. |
93b8016
into
feat/model-group-cost-aware-discovery
| **self.policy.as_dict(), | ||
| "roles": list(self.ROLE_TAGS), |
There was a problem hiding this comment.
🔍 Admin console load crashes on removed hints field
admin_state no longer emits complex_hints, but the console's load still reads state.policy.complex_hints.length. The field is now undefined, so reading .length throws and aborts load before agents, secondary views, and the simulator render, leaving the console blank.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if not self.policy.realtime_judge: | ||
| return { | ||
| "accepted": True, | ||
| "reason": "single route path", | ||
| "verifier_output": answer, | ||
| "judge": "model", | ||
| } | ||
| fallback_report = {"verifier_output": answer} | ||
| base = self._model_judge_verification( | ||
| text, fallback_report, free_only=free_only | ||
| ) | ||
| accepted = bool(base.get("accepted")) | ||
| _record(accepted) | ||
| return base |
There was a problem hiding this comment.
🔍 Default real-time judge degrades routing without an installed judge
realtime_judge defaults on. When the fast-mlsirm judge cannot be resolved, _model_judge_verification returns accepted=False, so route_once rejects every direct answer, spends an extra failover call, returns the answer marked unaccepted, and records a quality-ledger failure for each member. Uniform failures then drive _measured_member_order, flattening intra-group ranking. Deployments without a judge must set --no-realtime-judge.
Was this helpful? React with 👍 or 👎 to provide feedback.
| def _needs_workflow(self, text: str) -> bool: | ||
| lowered = text.lower() | ||
| hits = sum(1 for hint in self.COMPLEX_HINTS if hint in lowered) | ||
| return hits >= self.policy.conduct_hint_threshold or len(text) > self.policy.route_text_length_threshold | ||
| """Route-vs-conduct decision from a strict structured triage verdict. | ||
|
|
||
| Keyword hint tables are intentionally absent: keyword matching cannot | ||
| handle negation, mixed language, or tasks that quote trigger words, and | ||
| hand-tuned thresholds are not evidence. The verdict comes from one | ||
| exact-schema model call (cached by content hash) and fails closed to | ||
| the orchestrated path on any uncertainty. | ||
| """ | ||
| return bool(self._triage_fn(text)) |
There was a problem hiding this comment.
📝 Info: Auto-mode triage now issues a blocking model call
_needs_workflow now delegates to a structured triage model call (content-hash cached), so would_route and _dispatch issue a synchronous provider request on cache miss where none existed before. Mock transports cannot emit the strict JSON, so they fail closed and route every auto-mode request to conduct unless _triage_fn is overridden.
Was this helpful? React with 👍 or 👎 to provide feedback.
| verification = self._realtime_route_judge( | ||
| text=text, | ||
| answer=answer, | ||
| served_id=served_id, | ||
| latency_seconds=latency_seconds, | ||
| usage=attempt_usage, | ||
| free_only=free_only, | ||
| ) |
There was a problem hiding this comment.
🔍 Judge and triage provider calls are not billed
The real-time route judge and triage each issue provider requests that never enter the run trace, so spend_analytics (which sums trace-row usage) under-reports the token cost of auto/route traffic while realtime_judge is on.
Was this helpful? React with 👍 or 👎 to provide feedback.
* feat(routing): add operator-managed model groups * fix(api): enforce model group create semantics * fix(routing): preserve eligibility and REST contracts * docs(groups): remove ephemeral model example * fix(groups): route advertised aliases end to end * ci: hourly OpenCode maintenance agent routed through this gateway (#835) * ci: add hourly OpenCode maintenance agent routed through this gateway The scheduled job boots the contextual-orchestrator gateway with the five org provider secrets seeded into the process-local KV registry (bootstrap transport only), auto-discovers chat-capable models, assigns the ox-alpha measured-routing group across OpenRouter + OpenCode Zen aliases, then runs the pinned OpenCode CLI pointed at http://127.0.0.1:8000/v1 with model 'ox-alpha' so the agent's own traffic exercises group routing. The agent works the PR queue (review -> fix -> recheck -> merge), root-causes failing checks, and advances docs/product-technical-gap-baseline.md when the PR queue is empty. COPILOT_GITHUB_TOKEN is not used; the existing review-agent key scheme is untouched. * fix(ci): remove ephemeral model binding * fix(ci): install gateway and grant branch writes * fix(groups): preserve internal dispatch default * fix: isolate model group routing evidence * fix: preserve conduct semantics for model groups * ci: minimize hourly loop permissions and installs * fix: measure streamed group routing * feat: route model groups across modalities (#837) * feat: normalize and edit model groups (#838) * fix: preserve capability routing contracts * release: v0.2.0 — model groups, cost-aware discovery, changelog baseline Bump 0.1.0 -> 0.2.0 and add the canonical Keep-a-Changelog file with the 0.1.0 baseline and the 0.2.0 additions (operator-managed model groups, measured group routing, OpenCode Zen discovery + free-tier classification, Strix B105 root-cause remediation). * fix(discovery): tolerate null modality arrays * docs: specify model group product and technical contracts * release: align v0.2.0 changelog and lock metadata * feat: stream orchestrated reasoning summaries * fix: keep free reasoning streams fail closed * fix: preserve free routing evidence and analytics * fix: distinguish failed Responses streams in analytics * fix: pin structured free judge to selected agent * fix: lock container dependencies and virtual capabilities * fix: keep free passthrough on zero-cost models * fix: prune removed routing measurements * refactor: remove unreachable responses passthrough branch * fix: route virtual models across media capabilities * fix: retain reset candidate routing rows * fix: preserve Responses instructions in workflows * test: align model-group missing-member code with canonical agent_not_found #831 unified worker-agent not-found errors on agent_not_found; the model-group CRUD contract now asserts that same canonical code. * fix(api): model-group creation returns canonical agent_not_found for unknown members * ci: route hourly OpenCode loop through auto * fix: reject unsupported orchestrated structured output * docs: correct Responses stream options error * fix: retrieve URL-encoded model identifiers * fix: measure free capability and failover routing * fix: resolve model group review findings * fix: close model group integration gaps * fix: align provider inventory and session cache scope * fix: harden compose secrets and repeated reasoning summaries * fix: contain binary response disconnects * fix: ground Zen free discovery in structured costs * fix: keep group judge within allowed members * fix: preserve free catalog evidence across reloads * docs: assign unique model-group ADR number * fix: preserve catalog capability evidence * fix: preserve declared Bytez endpoint capabilities * fix: validate batch model identity at ingress * fix(api): normalize missing model group errors * docs(prd): align product bets with model groups * fix(discovery): filter chat-only Bytez transports * docs(adr): reserve model-group decision identifier * feat: replace routing heuristics with measured evidence ledgers (#847) * feat: replace routing heuristics with measured evidence ledgers Remove DOMAIN_HINTS/COMPLEX_HINTS keyword tables; route via eligibility contracts, declaration priority/capability fit/cosine affinity over operator-declared metadata, and measured intra-group quality then EWMA tokens-per-second. Add structured fail-closed triage gas with content-hash verdict caching and real-time fast-mlsirm judging on direct routes that feeds a Beta-Bernoulli quality ledger with in-budget failover. ADR 0027 + doctoring APA 7 references (Jacobson 1988; Gelman et al. 2013; Karpukhin et al. 2020; Ong et al. 2024; Chen et al. 2023; Zheng et al. 2023; Jeon et al. 2021). Gap baseline added at docs/product-technical-gap-baseline.md. * fix: keep routing evidence units and capability boundaries honest * fix(admin): tolerate unavailable model-group state * docs(loop): require PRD and measured web capacity * fix(admin): remove retired policy hint metric * fix(routing): validate evidence before mutation * fix(discovery): remove model-name free inference guidance * fix: complete #834 model-group persistence on the normalized agent-pool schema - model_group/model_group_member relations compose with main's normalized agent_pool (no JSON shadow); save() maintains membership, load_all() restores group_name via join. - Legacy payload promotion reads agent_pool_legacy_payloads during the migration window and drops it after promoting group names. - DB-naming gate now extracts SQL from AST string constants so prose in comments can no longer produce false identifier violations. - Batch runner signature aligned (messages, mode, model) with the merged LocalBatchBackend; stream-route fake accepts the merged caller kwargs. * test: case-sensitive DDL pattern so prose cannot fake identifiers * docs: refresh protected-main gap evidence * docs: record exact provider regression suite * docs: correct exact provider head * fix(admin): make guidance customer-actionable * fix(admin): remove internal terms from customer copy * docs(ui): record customer-copy screenshot audit * test(ci): lock hourly orchestrator loop contract * fix: replace synthetic admin status with truthful empty states * docs: refresh exact-head remediation baseline * fix(stream): cancel orchestration after Responses disconnect * test: align boundary contracts with current routing * docs: complete public boundary docstrings * docs: refresh exact-head product gap queue * test: enforce complete public docstrings * ci: enforce protected exact-head merge loop
1 participant
Summary
Stacked on #834 (ADR 0026). Removes every task-keyword heuristic from the
routing path (ADR 0027):
DOMAIN_HINTS/COMPLEX_HINTStables and the conduct-hintthreshold from route selection.
(
-role_fit, -priority, has_affinity, -cosine, id) with cosine affinitycomputed over operator-declared metadata documents via the pool's own
embedding member (LRU-cached per text hash) -> measured intra-group
ordering (judged quality first, then
stability x EWMA tokens/second).{"workflow_required": bool}or fail closed to conducted orchestration;verdicts memoized by content hash.
Beta-Bernoulli quality ledger; rejects record failures and fail over
within the retry budget.
--no-realtime-judgekeeps legacy shape._select_agentas anendpoint-compatibility contract (not a heuristic).
Docs
docs/planning/adrs/0027-anti-heuristic-routing-evidence.mddocs/doctoring/measured-routing-evidence.md(Jacobson 1988; Gelman et al. 2013; Karpukhin et al. 2020; Ong et al.
2024; Chen et al. 2023; Zheng et al. 2023; Jeon et al. 2021)
docs/product-technical-gap-baseline.mdTest plan
tests/test_measured_routing_evidence.py: 29 tests — exact JacobsonTPS arithmetic, Laplace-prior stability products, deterministic mock
embeddings + cosine guards, strict triage parser (7 malformed classes),
verdict caching, judge accept/reject ledger updates, in-budget failover,
dual-ledger admin state.
counts stay exact measures of worker executions.