fix(routing): remove heuristic batch and embedding decisions

[seonghobae]

Current exact-head fuzz integration repair — 2026-09-07

• Exact head: 37cf3f6bb0810ff3bcf0132ac08f093523809667
• Lifecycle: Draft / Proposed
• Security and Quality run 34066307901 on predecessor 3f4a895… reached the full suite and failed 346 tests; its fuzz shard isolated one independent fixture regression. test_orchestration_on_arbitrary_prompt constructed three eligible mock agents and therefore hit the intended ambiguous-selection fail-closed contract before exercising prompt/trace/SSE invariants.
• The current commit replaces only that offline fuzz fixture with one explicit eligible agent. It restores end-to-end fuzz reachability without ranking, priority, tie-breaking, exception swallowing, or any production selector change. Dedicated no-heuristics tests continue to require ambiguous multi-agent selection to fail closed.
• This repair does not claim the other full-suite failures are resolved and does not transfer predecessor GREEN evidence. Fresh exact-head hosted checks are queued/non-terminal.

Root cause

Several production routing, benchmark, and compute-allocation seams made substantive decisions without an identified model or authoritative measurement contract:

• RoutingPolicy inferred sync-vs-batch from latency_tolerant, priority, and batch_min_tokens;
• the local embedding backend fabricated a SHA-256-derived pseudo-vector;
• cheapest_upstream assumed a representative 1000-prompt/1000-completion-token request shape;
• unspecified embedding requests could select by price and then static/input order;
• ModelGroupRouter synthesized P(success)/EWMA latency as route authority;
• psychometric routing transferred scores to unseen prompts by nearest-vector cosine and broke fitted-probability ties by identifier;
• the NIM benchmark estimated tokens from character length and chose a “cheapest” worker by an implicit 1:1 input/output price weight with model-id tie-breaking;
• the NIM evidence gate still used hand-selected paired-task/completion floors as production sufficiency authority;
• the reasoning-effort profile still contains repository-authored role→effort defaults, synthetic shrinkage/ability arithmetic and a fixed production-unlock threshold.

Under the no-heuristics contract, missing routing/accounting/evaluation evidence must fail closed rather than be replaced by a hand-authored selector.

Implemented production boundary

The current branch already changes the core routing seams so that:

• sync versus batch follows explicit channel=sync|batch, with compatibility latency/priority/token hints non-authoritative;
• SHA pseudo-embeddings are fail-closed tombstones and local embedding requires an explicit semantic embedder;
• table-driven cost comparison requires exact prompt/completion token quantities and a unique minimum; ties remain unresolved;
• ambiguous embedding pools require explicit eligible identity until an independently evaluated router exists;
• package initialization patches TaskOrchestrator to the evidence-only model-selection bridge: multiple general model candidates require complete, unique, exact-context fast-mlsirm psychometric evidence or explicit eligible model/agent selection;
• unseen prompts cannot borrow nearest-context psychometric scores and equal fitted probabilities remain unresolved;
• model-group transport/quality measurements remain diagnostics; composite score/rank seams fail closed;
• NIM benchmark accounting accepts complete provider-reported nonnegative integer token usage only; missing/malformed usage fails closed;
• when NIM request mix is unknown, automatic price selection is allowed only for a uniquely component-wise dominant complete input/output price vector; equal, crossing, or incomplete vectors remain unresolved;
• unsupported dry-run cells remain failures, so no Pareto frontier or paired comparison is fabricated from unresolved routing evidence.

The earlier NIM token/accounting repair has executable RED→GREEN evidence recorded on its source-fix lineage (tests/test_nim_benchmark_no_heuristic_tokens.py plus tests/test_nim_benchmark.py: 102 passed on the repaired worktree before that one-shot was removed). That predecessor evidence is not transferred as hosted evidence for the current head.

Active exact-head follow-up repairs

Exact current head, freshly re-fetched on 2026-09-02: 512ee46b42d71300e5bdc9170027915d6ddb1c2a.

The changed-file set still contains temporary source-fix machinery for optimizer provenance, optimizer selection, and reasoning effort. Their presence is not completion evidence; do not overlap those source writers with a competing mutation.

A fresh exact-head source read also proves that contextual_orchestrator/nim_benchmark.py still constructs OrchestrationPolicy with route_p95_seconds=2.5 and workflow_planning="template", with max_workflow_steps=MAX_WORKFLOW_DEPTH. Protected main contains the same route_p95_seconds=2.5 policy. These values remain open no-heuristics findings until the branch either (a) binds each decision to an executable research/experimental/model authority or explicit caller-supplied governed evaluation design, or (b) fails closed instead of selecting a route/plan/compute allocation. Do not replace them with different repository-authored constants. MAX_WORKFLOW_DEPTH=5 must likewise be treated as decision authority only if its cited issue/experiment is verified to govern this exact evaluation; a Conductor/TRINITY name in a comment is not by itself evidence that those papers authorize five steps.

Current PR-triggered exact-head Tests, Security, Fuzz, SAST Semgrep, and Security Scan runs are queued. Queued state is non-passing and does not transfer predecessor evidence.

Research conformance refresh

docs/doctoring/routing-literature-refresh-2026-09.md refreshes the research boundary against TRINITY, the Conductor, the 2026 Sakana Fugu technical report, Select-then-Solve, and TwinRouterBench. The refreshed conclusion is deliberately narrower than those learned systems: their results support learned/validated per-task coordination and execution-grounded evaluation, but do not authorize hand-authored thresholds, weights, provider preferences, static workflow choices, or surrogate scores in this repository. Until a learned router or other explicit decision model has deployment-valid training/evaluation evidence, ambiguous selection remains fail-closed or requires exact-context fast-mlsirm evidence/explicit unique selection.

Current merge boundary

The PR is mechanically mergeable but remains Draft and is not merge-ready. Temporary one-shot repair artifacts are still in the changed-file set and current exact-head required evidence is non-terminal. Landing requires the one-shot artifacts to be absent from one unchanged exact head, their intended production source changes to be present and verified, the newly re-confirmed route_p95_seconds/workflow-planning/compute-allocation findings above to be resolved without another heuristic, fresh hosted Tests/Security/Fuzz/Semgrep/Security Scan and required central review evidence to be terminal-success, all valid exact-head review findings to be resolved, and ordinary repository protection/independent review to be satisfied. No force push, self-approval, administrative bypass, gate weakening, or fabricated/predecessor evidence is authorized.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

2026-09-02 no-heuristics continuation — exact-head RCA found two additional NIM decision defects on the canonical branch: (1) build_worker_agents/request planning admitted only the first seven chat-eligible models in model-id order, so an arbitrary cardinality and serialization identity controlled evaluation admission; (2) best_single_worker_hindsight broke equal measured quality with policy_name, making model identity an undocumented tie-break. RED regressions are now committed in tests/test_nim_benchmark_no_heuristic_candidates.py. The existing NIM one-shot was extended to retain every capability-proven chat-eligible candidate, reserve the all-discovered pre-probe upper bound, and return no hindsight winner when measured maxima tie. The exact-text candidate repair was also reconciled with the preceding output-token repair so it consumes the post-repair max_output_tokens: int | None signature rather than aborting on its stale pre-repair form. Current source-fix run 33574359373 is queued; queued evidence is non-passing and non-blocking. No force push, self-approval, gate weakening, or completion claim.

[seonghobae]

@jules Fresh exact-head inventory shows this Draft has advanced to 512ee46b42d71300e5bdc9170027915d6ddb1c2a, and the live tree still contains multiple purpose-complete source-mutating repair lanes, including .github/workflows/source-fix-1000-optimizer-provenance.yml, .github/workflows/source-fix-1000-optimizer-selection.yml, .github/workflows/source-fix-1000-reasoning-effort.yml plus their .github/source-fix-1000-*.trigger files and repair drivers. The optimizer workflows run RED, execute a script that rewrites source, delete themselves, commit, reconcile the remote branch, and push with contents: write. This is explicitly outside the fleet's acceptable development boundary even if exact-head guards/non-force merge logic are present.

Repair the current branch directly through ordinary source/test commits; do not wait for these workflows to self-publish and do not add another source-fix successor.

Acceptance for the next exact head:

• apply each still-valid optimizer/reasoning/NIM causal repair directly, with its RED regression preserved as ordinary test history;
• delete every branch-specific source-fix-1000* workflow, trigger, and repair script from the tree;
• preserve the intended no-heuristics invariants: no fabricated pseudo-embedding, no guessed batch-vs-sync route, no static/tie-break price winner, no nearest-context psychometric transfer, no repository-authored reasoning-effort/production-unlock threshold, and no ad-hoc optimizer selection authority without executable provenance;
• keep observed benchmark/evaluation counts as evidence rather than turning them into hand-selected admission thresholds;
• run the focused optimizer/reasoning/NIM/psychometric suites plus the repository's full exact-head Tests/Fuzz/Security/SAST/Security Scan when hosted capacity admits them;
• keep this PR Draft until the final unchanged head contains no self-modifying repair machinery and all current valid review findings are repaired/obsolete.

Do not force-push, weaken gates, transfer predecessor GREEN, or treat a queued repair workflow as production evidence.

[seonghobae]

Fresh exact-head/source revalidation found additional production policy defaults in contextual_orchestrator/orchestrator.py that remain unchanged on both protected main and #1000 head 235dfe5e7e6f34369db6263e913cc79b153afee8: OrchestrationPolicy.route_p95_seconds = 2.5, workflow_planning = "template" with generated-plan failure falling back to that fixed template, and max_workflow_steps = 6. These are decision-/test-time-compute-affecting repository defaults and are not justified merely by the class docstring's evidence-based triage claim. They should be included in this canonical no-heuristics owner lane unless a narrower existing owner with executable research/standard provenance already governs them. Do not replace them with different constants: add RED contracts first where practical, then require explicit caller/experiment/learned-policy authority or fail closed. I did not move this branch because its two exact-head self-fix writers are still active/queued; concurrent single-writer intent is preserved.

[seonghobae]

Adjudication evidence (host 1 session, 2026-09-06 KST; full report with commands in #1080). Nothing here closes, flips, or retargets anything — the decision is the opener's.

Stray CI machinery in the tree: three branch-scoped .github/workflows/source-fix-1000-*.yml, three .github/*.trigger files, and four scripts/ci/repair_pr1000_*.py with no tests. Production behaviour is changed by import-time monkeypatching in contextual_orchestrator/__init__.py (TaskOrchestrator._ranked_agents/_measured_member_order/get_model_group, CostRoutingCoordinator._resolve_embedding_target, batch_routing.RoutingPolicy/cheapest_upstream), which no file-level review sees and which silently overrides #1034's sample= Thompson path, #911's refresh= return, and #983's _request_candidate_allowed filter with zero textual overlap. This PR has to go last across every cluster and needs a full-suite run on the merged tree, not file-level cleanliness; it also deletes existing routing tests (latency_tolerant/priority/batch_min_tokens) and rewrites ADR 0002/0003/0034, nim_benchmark.py and psychometric_routing.py (re-derive on top of #1074's estimand change). Draft hold respected.

[seonghobae]

Exact-log RCA and non-force source-fix repair — 2026-09-07

The stalled reasoning-effort source-fix run 33598019114, job 100145365215, proved the intended no-heuristics regression RED and applied the production transformation, then failed focused verification during collection. The exact failure was:

tests/test_generated_workflow.py:37
def _explicit_role_effort_catalog():
SyntaxError: invalid syntax

Root cause: scripts/ci/repair_pr1000_reasoning_effort.py appended the test-only helper after the text PLAN = {\n, placing a function definition inside the dictionary literal.

Commit 14fe2e5eb28e0a6d0376d35c9f6e771e332193e6 changes only the repair driver's insertion anchor and its trigger. The helper is now inserted after the complete contextual_orchestrator.orchestrator import block and before PLAN. An in-memory application of the exact transformation passed ast.parse; the remote branch was then advanced by fast-forward from 512ee46… with no force update.

Fresh source-fix run 34066033532 and all other exact-head workflows are queued for 14fe2e5…. This is not GREEN or merge evidence yet. If the one-shot succeeds, it must publish the production/test/docs repair and retire its own workflow, trigger, and driver before the resulting head is reviewed.

[seonghobae]

Follow-up exact-log RCA — run 34066033532

The anchor correction worked: RED-before-change and the production transformation both completed, and tests/test_generated_workflow.py collected successfully. Focused verification then reported 57 failures / 48 passes.

Two causal classes remain:

1. Five tests/test_reasoning_effort_profile.py failures were a source-fix fixture defect: _explicit_role_effort_catalog() referenced ReasoningEffortProfile without importing it. Commit 3f4a8959121a6f8c8b76e068062f5d09d1c1b7b9 repairs the driver import transformation. It intentionally does not touch the trigger, so it does not start another known-red publication attempt.
2. The remaining integration failures are not a formatting/test-runner issue. The branch's evidence-only model selector now fails closed for ambiguous multi-candidate pools, while CLI, mixed-pool, generated-workflow, and 49 virtual-provider/passthrough cases still assume an ordered provider fallback without supplying exact-context psychometric evidence or explicit selection. Representative actual error: multiple eligible agents require complete exact-context psychometric routing evidence or explicit model/agent selection.

Those integration failures must be repaired without reinstating priority/discovery-order/name fallback and without merely weakening the tests. The correct owner work is to define explicit eligibility/selection evidence in the fixtures and production virtual-provider contract, or fail closed where no such authority exists. Until that integrated RED→GREEN repair lands, the reasoning-effort one-shot must remain unexecuted and #1000 remains Draft/non-merge-ready.

[seonghobae]

Exact-head RCA and bounded repair for Security and Quality run 34066307901:

• job 101575589554 reached the full suite and reported 346 failures after the evidence-only selector correctly refused to choose among multiple eligible agents without complete exact-context psychometric evidence or explicit selection;
• job 101575589694 isolated the smallest independent regression: tests/fuzz/test_fuzz_properties.py::test_orchestration_on_arbitrary_prompt always constructed three eligible mock agents, so the newly documented fail-closed outcome was misclassified as a fuzz crash (minimal Hypothesis example: empty prompt, mode="auto");
• commit 37cf3f6 repairs only that fixture. The end-to-end fuzz target now uses one explicit offline fixture agent, so there is no routing decision or priority/tie-break allocation, while prompt processing, route/conduct, trace serialization, and SSE framing remain exercised for arbitrary input. The dedicated no-heuristics selector contract continues to test that ambiguous multi-agent selection fails closed.

This does not claim that the other 345 full-suite failures are fixed, does not weaken the production selector, and does not transfer the predecessor run as evidence. The branch remains Draft pending fresh exact-head hosted results and integration repair.

[seonghobae]

CO #1067/#1074 integration notice: read the full existing 715f24a delta and freshly fetched this branch at 37cf3f6. The measurement_evidence_only/null-threshold classification repair already exists here; no duplicate estimator or replacement cutoff is needed. We intend to preserve that decision-authority slice on the psychometric parent, then normally integrate the child locked-cohort and failure-inclusive pairing changes. The separate explicit token-allocation slice remains owned here; this is not a claim of complete 715f24a or PR1000 supersession.

Counterexample reproduced on child54b2b809: the same 30 successful route_once/conduct_bounded task pairs receive production_candidate_review with completion1.0 for declared locked_task_count30 and30000 alike. This is a constructed helper-contract case, not evidence of live contamination. The current owner helper still lacks the child locked-cell restriction, so integration must retain both. We will not move this PR branch, rerun its failed suite blindly, or treat its current integration failures as resolved.