Combo + quota-shared deep audit: 5 fixes + strategy validation + E2E + complexity routing (2026) by diegosouzapw · Pull Request #3779 · cheaper-inference/OmniRoute

diegosouzapw · 2026-06-13T16:36:49Z

Deep combo + quota-shared audit → 4 waves (TDD)

Deep audit of the combo system (all strategies + rules + proxy) and quota-shared,
then fixes, strategy validation, E2E coverage, and one new 2026 routing strategy.
Every change is TDD (red→green); proxy/credential isolation was verified clean
(each target uses its own credentials+proxy via AsyncLocalStorage — no leak).

Wave 1 — repair 5 dead/broken rules (`fix`)

Streaming USD (HIGH): the streaming quota-share hook recorded usd:0
hardcoded, so USD-unit shared pools (e.g. DeepSeek usd/monthly) never
enforced on streaming traffic. Extracted a testable recordStreamingConsumption
- shared buildConsumptionCost; streaming now accrues the real cost.
Pool-usage route blank: resolvePlan(connId, "") degraded catalog-only
pools to an empty (dimension-less) snapshot. Resolves the provider from the
connection (resolveConnectionProvider, awaited) — also fixes the same
missing-await bug in the plans-route helper.
Provider-diversity dead: recordProviderUsage had zero callers, so
/api/analytics/diversity always reported 1.0/empty. Wired into
recordComboRequest (the single chokepoint for every strategy).
config.maxComboDepth dead: depth was always the hardcoded 3. Now flows
through the nesting functions (clampComboDepth, 1..10) at dispatch + DAG
validation.
Scoring hardening: clamp every factor and the final score to [0,1]
(clamp01 maps non-finite → 0) — a NaN/negative input can no longer produce a
NaN score (sorts nondeterministically) or a >1 score.

Wave 2 — validate every strategy (`test`)

Direct coverage for the auto-router strategies that had none (cost / latency /
sla-aware / lkgp / selectWithStrategy + silent fallback + aliases). The
combo-hedging placeholder (two assert.ok(true)) now exercises the
predictive-TTFT decision (shouldSkipForPredictedTtft, extracted) and the
hedging/zero-latency config resolution.

Wave 3 — E2E coverage (`test`)

3-target priority failover (500 → 503 → success), per-target timeout failover
(targetTimeoutMs → 524 → next target), and a real strategy:auto dispatch
end-to-end. The suite previously topped out at a 2-target single-hop failover with
no timeout-driven path and no auto dispatch.

Wave 4 — complexity-aware routing (2026) (`feat`)

Routes by the request's intrinsic difficulty: classifyRequestComplexity over the
existing specificity detector → score / level / recommended tier, with a 2026
tool-use escalation (tool/function schemas floor the tier at "cheap"). Fed into
scoreAutoTargets via the existing-but-never-passed manifest hint. Opt-in
behind config.complexityAwareRouting (default OFF — null hint is byte-for-byte
identical to before).

Root-cause for audit Bug feat(versioning): set initial version to 0.0.1 and restore dynamic model import #4: calculateTierAffinity/calculateSpecificityMatch
used a lazy require("../tierResolver") that silently throws under ESM → the two
factors (0.10 combined weight) returned a constant 0.5 even when a hint was
passed. Converted to a static import → genuinely live when complexity routing is on.

Validation

7 new test files, all green deterministically; lint clean; full-tsc clean on
touched files (only pre-existing app-tsc noise remains).
No regressions: combo-routing-engine 81, autoCombo vitest 52, quota-plans-crud 10,
broad quota unit 71, combo-config/metrics, providerDiversity 10.
Pre-existing flaky (NOT from this branch): combo-stream-readiness-fallback
"all models 504" varies 0/4–3/4 across runs and fails on the base combo.ts too
(timing-sensitive infra flake in a file this branch does not touch).

Deferred (honest)

E2E for combo+shared-quota pool-exhaustion re-route and soft-penalty re-score:
these are the combo↔quota integration paths the audit flagged as
architecturally weak (~60% — soft penalty is ephemeral / post-selection). They
should be hardened (feed quota into pre-dispatch scoring) before an E2E test, not
pinned in their current form.

Deep-audit findings (combo + quota-shared), each fixed red→green: - quota: streaming requests now accrue the real USD cost (was hardcoded usd:0), so USD-unit shared pools (e.g. DeepSeek usd/monthly) finally enforce on streaming traffic — the dominant request shape. Extracted a testable recordStreamingConsumption + shared buildConsumptionCost. - quota: pool-usage route resolves the provider from the connection (resolveConnectionProvider, awaited) so catalog-only pools surface their plan dimensions; was resolvePlan(connId, "") -> empty -> blank dashboard. Also fixes the same missing-await bug in the plans route helper. - autoCombo: wire recordProviderUsage into recordComboRequest (the single chokepoint for every strategy) so /api/analytics/diversity reflects real provider usage; recordProviderUsage previously had zero callers. - combo: config.maxComboDepth now flows through the nesting-depth functions (clampComboDepth, 1..10 hard cap) at dispatch + DAG validation; was dead config with depth always hardcoded to 3. - autoCombo scoring: clamp every factor and the final score to [0,1] (clamp01 maps non-finite -> 0), preventing a NaN/negative input from producing a NaN score (sorts nondeterministically) or a >1 score. Validation: 5 new test files (TDD); lint clean; full-tsc clean on touched files (only pre-existing app-tsc noise remains). No regressions: combo-routing-engine 81/81, autoCombo vitest 63/63, quota-plans-crud 10/10, combo-metrics 25/25.

Closes the strategy-validation gaps surfaced by the audit: - router-strategies.test.ts: direct coverage for the pluggable auto-router strategies that had none — cost (cheapest, skips OPEN, eco alias), latency (lowest p95 + error penalty, fast alias), sla-aware (soft + hardConstraints, sla alias), lkgp (hit / OPEN-fallback / disabled-fallback), and the selectWithStrategy entry point incl. the documented silent fallback to rules for unknown names + listStrategies registry. - combo-hedging.test.ts: was a placeholder (two assert.ok(true)); now exercises the predictive-TTFT skip decision and the hedging/zero-latency config resolution (defaults off, per-combo override). Full hedging dispatch stays covered in combo-routing-engine.test.ts. - combo.ts: extracted shouldSkipForPredictedTtft as a pure, testable predicate (>= 5 samples AND avg latency > ceiling); behavior-preserving — the inline predictive-TTFT check now delegates to it. router-strategies 16/16, combo-hedging 7/7, combo-routing-engine 81/81 (no regression), lint + filtered full-tsc clean.

… and auto dispatch The integration suite topped out at a 2-target (single-hop) failover and had no timeout-driven failover nor a real strategy:auto dispatch test. Adds: - a 3-target priority chain (500 -> 503 -> success); - a per-target timeout (targetTimeoutMs) failover to a healthy target; - a strategy:auto combo dispatched end-to-end (scored selection -> upstream 200). All 3 green; lint clean.

…factors Implements the top 2026 routing strategy from the audit research: route by the intrinsic difficulty of the request, not just provider stats. - complexityRouter.ts: classifyRequestComplexity() over the existing specificity detector (code/math/reasoning/context/tool/domain signals) → score, level (trivial..expert) and a recommendedTier (free/cheap/premium). Adds the 2026 tool-use escalation the research highlighted: a request carrying tool/function schemas (or agentic tool signals) is floored at the "cheap" tier so scoring favors function-calling-reliable models, even when the prose looks trivial. - combo.ts: the auto router now builds a tier hint from the request and feeds it into scoreAutoTargets (newly exported + manifestHint param). Opt-in via the new config.complexityAwareRouting flag (default OFF — zero behavior change unless enabled); a null hint is byte-for-byte identical to before. - scoring.ts: convert calculateTierAffinity/calculateSpecificityMatch from a lazy `require("../tierResolver")` (which silently throws → 0.5 under ESM) to a static import. This was the deeper root cause of audit Bug #4: the tierAffinity + specificityMatch factors (0.10 combined weight) returned a constant 0.5 even when a hint WAS passed. Now they are genuinely live when complexity routing is on. TDD: complexity-router 4/4, complexity-aware-scoring-wiring 2/2 (hint changes score; null hint unchanged). No regressions: combo-routing-engine 81, combo-hedging 7, combo-config, autoCombo vitest 52. lint clean.

chatgpt-codex-connector · 2026-06-13T16:36:54Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

gemini-code-assist

Code Review

This pull request introduces complexity-aware routing to classify request difficulty, hardens auto-combo scoring with clamping to prevent NaN propagation, and enforces a safety cap on nested combo depth. It also resolves regressions in streaming quota consumption by recording real estimated costs and fixes provider resolution for catalog-only pools. The review feedback identifies several critical safety improvements, specifically recommending optional chaining guards to prevent potential TypeError exceptions when accessing result.breakdown, hint.specificity, and autoManifestHint, as well as clamping connectionDensity to ensure consistency.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-06-13T16:38:50Z

+  const level = getSpecificityLevel(result.score);
+
+  const explicitTools = Array.isArray(input.tools) && input.tools.length > 0;
+  const hasToolUse = explicitTools || result.breakdown.toolCalling > 0;


To prevent potential runtime TypeError exceptions if result.breakdown is undefined, use optional chaining and nullish coalescing to safely access toolCalling.

Suggested change

const hasToolUse = explicitTools || result.breakdown.toolCalling > 0;

const hasToolUse = explicitTools || (result.breakdown?.toolCalling ?? 0) > 0;

gemini-code-assist · 2026-06-13T16:38:50Z

  try {
-    const { classifyTier } = require("../tierResolver");
    const assignment = classifyTier(candidate.provider, candidate.model);
    const specificityScore = hint.specificity.score;


Use optional chaining to safely access hint.specificity.score to prevent a potential TypeError if hint.specificity is undefined.

Suggested change

const specificityScore = hint.specificity.score;

const specificityScore = hint.specificity?.score;

if (specificityScore === undefined) return 0.5;

gemini-code-assist · 2026-06-13T16:38:50Z

-    resetWindowAffinity: candidate.resetWindowAffinity ?? 0.5,
+    contextAffinity: clamp01(candidate.contextAffinity ?? 0.5),
+    resetWindowAffinity: clamp01(candidate.resetWindowAffinity ?? 0.5),
    connectionDensity: Math.min(1, Math.max(0, ((candidate.connectionPoolSize ?? 1) - 1) / 10)),


For consistency with the other scoring factors in this object, use the imported clamp01 helper to clamp connectionDensity to the [0, 1] range.

Suggested change

connectionDensity: Math.min(1, Math.max(0, ((candidate.connectionPoolSize ?? 1) - 1) / 10)),

connectionDensity: clamp01(((candidate.connectionPoolSize ?? 1) - 1) / 10),

gemini-code-assist · 2026-06-13T16:38:50Z

+          autoManifestHint.recommendedMinTier = escalateTier(
+            autoManifestHint.recommendedMinTier as ComplexityTier,
+            classification.recommendedTier
+          ) as typeof autoManifestHint.recommendedMinTier;
+          log.info(
+            "COMBO",
+            `Complexity-aware routing: level=${classification.level} score=${classification.score} minTier=${autoManifestHint.recommendedMinTier} tools=${classification.hasToolUse}`
+          );


Guard the property access and escalation logic with an if (autoManifestHint) check to prevent potential TypeError exceptions if generateRoutingHints returns null or undefined.

if (autoManifestHint) { autoManifestHint.recommendedMinTier = escalateTier( autoManifestHint.recommendedMinTier as ComplexityTier, classification.recommendedTier ) as typeof autoManifestHint.recommendedMinTier; log.info( "COMBO", "Complexity-aware routing: level=" + classification.level + " score=" + classification.score + " minTier=" + autoManifestHint.recommendedMinTier + " tools=" + classification.hasToolUse ); }

…ize gate The W1+W4 audit changes grew open-sse/services/combo.ts past its frozen file-size baseline (5054), failing the Fast Quality Gates. The cleanly extractable W4 block — the inline complexity-hint construction in the auto-router branch — is moved into autoCombo/complexityRouter.ts as buildComplexityRoutingHint(), where the complexity-routing logic already lives. combo.ts now calls it behind the same opt-in flag; behavior is byte-for-byte identical (the ruleInput construction, tool-use escalation and fail-open catch are preserved verbatim). The remaining +77 LOC over baseline is the irreducible residual (maxDepth threading across 6 signatures + dispatch/DAG sites, scoreAutoTargets export + manifestHint param) — signature/threading edits, not a movable block — so it is consciously re-baselined (combo.ts 5054 -> 5131) with a justification key in file-size-baseline.json; structural shrink stays owned by #3501. TDD: 2 new direct tests for buildComplexityRoutingHint (tool-use tier floor; null-body fail-open). No regressions: complexity-router 6, complexity-aware- scoring-wiring 2, combo-hedging 7, combo-routing-engine 81, autoCombo vitest 62. lint + typecheck (touched files) clean; check:file-size green.

…ent) connectionDensity used an inline Math.min/max that bounded [0,1] but was NOT NaN-safe and was inconsistent with the other W1-clamped factors (and the calculateFactors comment claiming clamp01 guards NaN for every factor). A non-finite connectionPoolSize propagated NaN into the weighted sum, and the final clamp01 then zeroed the whole score -- unfairly sinking the candidate. clamp01 now maps it to 0 in isolation. Resolves a gemini-code-assist medium suggestion. TDD: +1 test (large pool stays <=1; NaN pool -> finite). No regressions: scoring-clamp 7, autoCombo vitest 62, tsc/eslint clean.

oyi77 · 2026-06-13T18:04:33Z

Addressed the review comments from gemini-code-assist:

Fixed (commit 059cc4c93):

complexityRouter.ts:56 — result.breakdown?.toolCalling ?? 0 optional chaining guard
scoring.ts:159 — hint.specificity?.score + undefined guard to prevent TypeError
scoring.ts:203 — connectionDensity was already using clamp01 in the latest revision ✅

Cherry-pick: git cherry-pick 059cc4c93 from oyi77/audit/combo-quota-validation

Preview of 059cc4c93:

- const hasToolUse = explicitTools || result.breakdown.toolCalling > 0;
+ const hasToolUse = explicitTools || (result.breakdown?.toolCalling ?? 0) > 0;

- const specificityScore = hint.specificity.score;
+ const specificityScore = hint.specificity?.score;
+ if (specificityScore === undefined) return 0.5;

Resolve file-size-baseline.json conflict in the _rebaseline keys block: keep this PR's combo_quota_audit key + release's v3824_3776 key (#3776); drop the now-redundant drift_3780 key (release's #3776 key already documents the base.ts 1205->1218 carry-over from #3780). frozen values auto-merged (combo.ts 5131 mine, base.ts 1218 both, chatCore.ts 5808 theirs). Verified on the merged tree: all 14 Fast Quality Gates pass (incl. file-size: chatCore 5736<=5808, combo 5131<=5131), W1 streaming hook intact, quota-streaming 8 / complexity-router 6 / scoring-clamp 7 / combo-routing-engine 81 green. Pre-existing release app-tsc noise in chatCore (3026-4380, from #3775) is untouched by this PR.

kilo-code-bot · 2026-06-13T18:14:25Z

Code Review Summary

Status: 2 Issues Found | Recommendation: Address before merge

Overview

Severity	Count
CRITICAL	0
WARNING	2
SUGGESTION	0

Issue Details (click to expand)

WARNING

File	Line	Issue
`open-sse/services/autoCombo/complexityRouter.ts`	56	Existing comment: Optional chaining recommended for `result.breakdown.toolCalling` access to prevent potential TypeError if breakdown is undefined (defensive coding).
`open-sse/services/autoCombo/scoring.ts`	159	Existing comment: Optional chaining recommended for `hint.specificity.score` access to prevent potential TypeError (defensive coding).

Other Observations (not in diff)

Positive Changes

The 4 waves of changes are well-structured and solve real issues:

Streaming USD fix (spendRecorder.ts): Extracts testable recordStreamingConsumption and buildConsumptionCost to properly record USD costs for streaming responses (previously hardcoded usd: 0).
Pool usage provider resolution (connectionProvider.ts, route.ts): Fixes the resolvePlan(connId, "") bug where empty provider string caused catalog-only pools to return dimension-less snapshots.
Provider diversity wiring (comboMetrics.ts): Wires recordProviderUsage into the combo metrics pipeline so /api/analytics/diversity now reports actual usage.
Scoring hardening (scoring.ts): Properly clamps all factors and final score to [0,1] using clamp01, preventing NaN propagation and out-of-range scores.
Max combo depth config (combo.ts): Wires config.maxComboDepth through clampComboDepth() and all depth-related functions (was hardcoded to 3).
Complexity-aware routing (complexityRouter.ts): New opt-in 2026 feature that classifies request difficulty and feeds tier hints to auto-router scoring.

Test Coverage

7 new test files covering: 3-way failover, timeout failover, auto combo E2E, max depth config, provider diversity wiring, complexity scoring wiring, scoring clamp/NaN handling, streaming consumption.

Files Reviewed (10 files)

src/lib/quota/spendRecorder.ts - +77 lines (streaming consumption fix)
src/lib/quota/connectionProvider.ts - New file (provider resolution)
src/app/api/quota/pools/[id]/usage/route.ts - 6 lines modified
open-sse/handlers/chatCore.ts - Uses buildConsumptionCost/recordStreamingConsumption
open-sse/services/autoCombo/scoring.ts - Scoring factor hardening + static import fix
open-sse/services/autoCombo/complexityRouter.ts - New file (complexity routing)
open-sse/services/combo.ts - maxDepth threading, scoreAutoTargets export
open-sse/services/comboConfig.ts - Added complexityAwareRouting config
open-sse/services/comboMetrics.ts - Provider diversity recording
file-size-baseline.json - Rebaseline for changed file sizes

_{Reviewed by laguna-m.1-20260312:free · 4,072,398 tokens}

…n stale codex/plugins tests - Date the [3.8.24] section (2026-06-13) and add the missing CHANGELOG bullet for the combo + quota-shared deep audit (#3779). - Align two unit tests left stale by intentional behavior changes this cycle: executor-codex now asserts gpt-5.4-mini xhigh passthrough (#3756 default), and plugins-route-error-sanitization now covers the /api/plugins/marketplace route added by #3656 (verified Hard Rule #12 compliant). No production change.

…+ complexity routing (cheaper-inference#3779) Deep audit of the combo + quota-shared system, delivered as 4 TDD waves. - Wave 1: repair 5 dead/broken rules — streaming USD recording, pool-usage provider resolution, provider-diversity wiring, maxComboDepth threading, scoring clamp/NaN-safety (incl. connectionDensity). - Wave 2: validate every auto-router strategy (cost / latency / sla-aware / lkgp / selectWithStrategy + aliases) and the predictive-TTFT decision. - Wave 3: E2E coverage — 3-hop priority failover, per-target timeout failover, real strategy:auto dispatch. - Wave 4: complexity-aware routing (2026, opt-in) over the existing specificity detector, plus revival of the dead tierAffinity / specificityMatch scoring factors (require-in-ESM root cause -> static import). Proxy/credential isolation verified clean (each target uses its own credentials+proxy via AsyncLocalStorage). file-size reconciled (combo.ts re-baseline after extracting buildComplexityRoutingHint to complexityRouter.ts; base.ts release-drift from cheaper-inference#3780). Fast Quality Gates + semgrep green.

…elog bullet, align stale codex/plugins tests - Date the [3.8.24] section (2026-06-13) and add the missing CHANGELOG bullet for the combo + quota-shared deep audit (cheaper-inference#3779). - Align two unit tests left stale by intentional behavior changes this cycle: executor-codex now asserts gpt-5.4-mini xhigh passthrough (cheaper-inference#3756 default), and plugins-route-error-sanitization now covers the /api/plugins/marketplace route added by cheaper-inference#3656 (verified Hard Rule cheaper-inference#12 compliant). No production change.

diegosouzapw added 4 commits June 13, 2026 12:49

gemini-code-assist Bot reviewed Jun 13, 2026

View reviewed changes

diegosouzapw added 3 commits June 13, 2026 14:12

chore(file-size): reconcile base.ts baseline drift from #3780

94b47d2

diegosouzapw merged commit 45b0672 into release/v3.8.24 Jun 13, 2026
2 checks passed

diegosouzapw deleted the audit/combo-quota-validation branch June 13, 2026 19:12

diegosouzapw mentioned this pull request Jun 13, 2026

Release v3.8.24 #3747

Merged

Navigation Menu

Repository navigation

Combo + quota-shared deep audit: 5 fixes + strategy validation + E2E + complexity routing (2026) - #3779

Combo + quota-shared deep audit: 5 fixes + strategy validation + E2E + complexity routing (2026)#3779
diegosouzapw merged 8 commits into
release/v3.8.24from
audit/combo-quota-validation

diegosouzapw commented Jun 13, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jun 13, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Jun 13, 2026

Uh oh!

gemini-code-assist Bot Jun 13, 2026

Uh oh!

gemini-code-assist Bot Jun 13, 2026

Uh oh!

gemini-code-assist Bot Jun 13, 2026

Uh oh!

oyi77 commented Jun 13, 2026

Uh oh!

kilo-code-bot Bot commented Jun 13, 2026 •

edited

Loading

WARNING

Positive Changes

Test Coverage

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	const hasToolUse = explicitTools \|\| result.breakdown.toolCalling > 0;
	const hasToolUse = explicitTools \|\| (result.breakdown?.toolCalling ?? 0) > 0;

	const specificityScore = hint.specificity.score;
	const specificityScore = hint.specificity?.score;
	if (specificityScore === undefined) return 0.5;

	connectionDensity: Math.min(1, Math.max(0, ((candidate.connectionPoolSize ?? 1) - 1) / 10)),
	connectionDensity: clamp01(((candidate.connectionPoolSize ?? 1) - 1) / 10),

Conversation

diegosouzapw commented Jun 13, 2026

Deep combo + quota-shared audit → 4 waves (TDD)

Wave 1 — repair 5 dead/broken rules (fix)

Wave 2 — validate every strategy (test)

Wave 3 — E2E coverage (test)

Wave 4 — complexity-aware routing (2026) (feat)

Validation

Deferred (honest)

Uh oh!

chatgpt-codex-connector Bot commented Jun 13, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jun 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 13, 2026

Choose a reason for hiding this comment

Uh oh!

oyi77 commented Jun 13, 2026

Uh oh!

kilo-code-bot Bot commented Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Summary

Overview

WARNING

Positive Changes

Test Coverage

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Wave 1 — repair 5 dead/broken rules (`fix`)

Wave 2 — validate every strategy (`test`)

Wave 3 — E2E coverage (`test`)

Wave 4 — complexity-aware routing (2026) (`feat`)

kilo-code-bot Bot commented Jun 13, 2026 •

edited

Loading