[defer] feat(combo): universal cooldown-aware retry & auto-strategy combo-ref guard

config/quality/file-size-baseline.json

@@ -1,4 +1,5 @@
 {
+  "_rebaseline_2026_07_21_7301_universal_cooldown_retry": "PR #7301 (ViFigueiredo, feat/universal-cooldown-retry) own growth, surfaced during rebase-onto-tip reconciliation (fast-gates PR->release do not run check:file-size): open-sse/services/combo.ts 3388->3479 (+91) generalizes the existing quota-share-only cooldown-aware retry (dispatchWithCooldownRetry) to ALL combo strategies (priority/weighted/round-robin/etc), gates it on the model lockout's REAL reason (not a hardcoded \"rate_limit\") via the existing getModelLockoutInfo/resolveComboCooldownWaitDecision chokepoint, and adds a global comboTimeoutMs guard + aggregated per-target error diagnostics on exhaustion. Companion leaves open-sse/services/combo/comboCooldownRetry.ts (+29), combo/autoStrategy.ts (+9, auto-strategy combo-ref guard so a combo cannot recursively reference itself as a candidate), combo/comboSetup.ts (+3), comboConfig.ts (+6) all stay under cap. Irreducible orchestration wiring at the existing dispatch chokepoint (mirrors the quota-share-only precedent this PR generalizes); not extractable without hiding the retry loop. Covered by tests/unit/combo-auto-candidate-expansion.test.ts (+61, combo-ref guard), tests/unit/combo-routing-engine.test.ts (+68, universal retry across strategies + comboTimeoutMs, no-explicit-any clean), tests/unit/serial/combo-quota-share-cooldown-wait-timing.test.ts (+136, quota_exhausted vs rate_limit reason gating, disabled-flag passthrough). Structural shrink of combo.ts tracked in #3501.",
   "_rebaseline_2026_07_21_7935_vi_locale_residual_ui": "PR #7935 (nguyenha935, fix/vietnamese-locale-residual) own growth: 9 dashboard components gained `useTranslations()` wiring (import + hook call + a handful of `t(\"key\")` call-sites replacing hardcoded English strings) as part of restoring i18n coverage — ComboHealthTab.tsx 1028->1031 (+3), cloud-agents/page.tsx 922->931 (+9), PoolWizard.tsx 1007->1022 (+15), EndpointPageClient.tsx 2612->2615 (+3), health/page.tsx 1091->1095 (+4), ProviderOnboardingWizard.tsx 912->948 (+36, largest — several previously-hardcoded wizard step labels/descriptions), PricingTab.tsx 1012->1017 (+5), ProxyRegistryManager.tsx 1461->1464 (+3), BudgetTab.tsx 1016->1028 (+12). All additions are literal `t(...)`/`tc(...)` call-site swaps for existing UI text, verified byte-identical in intent against the corresponding new `src/i18n/messages/{en,vi}.json` keys (see tests/unit/dashboard-localization-contract.test.ts, tests/unit/i18n-vi-completeness.test.ts, tests/unit/gamification-display-contract.test.ts, tests/unit/cli-catalog-display-contract.test.ts added by the same PR). Fast-gates PR->release do not run check:file-size, so this surfaced only during rebase-onto-tip reconciliation.",
   "_rebaseline_2026_07_21_7908_chathelpers_abort_guard": "PR #7908 (insoln, don't cool down accounts or trip the breaker on client-side stream aborts, #7907) own growth: src/sse/handlers/chatHelpers.ts 876->877 (+1 = the single `isLocalStreamLifecycleError(failure?.message ?? failure)` clause added to executeChatWithBreaker's onStreamFailure connection-disable check, verified working by the existing #4602 test + the PR's own circuit-breaker-client-abort.test.ts, no regressions). Irreducible call-site wiring at the existing failure-classification chokepoint. Fast-gates PR->release do not run check:file-size, so this surfaced only during the /green-prs pre-merge pass.",
   "_rebaseline_2026_07_21_7908_combo_breaker_abort_guard": "PR #7908 pre-green fix (green-prs pipeline): shouldRecordProviderBreakerFailure() (open-sse/services/combo/comboPredicates.ts, not frozen) gained an `error` field so a client-side stream abort no longer trips the whole-provider circuit breaker in the combo path (mirrors the connection-cooldown fix shouldSkipConnDisable() already applies for the same #4602/#7907 policy). Own growth: open-sse/services/combo.ts 3387->3388 (+1, irreducible call-site wiring — the single new `error: errorText,` field passed at the existing shouldRecordProviderBreakerFailure() call site inside handleComboChat's executeTarget). Covered by tests/unit/circuit-breaker-abort-provider-trip-7907.test.ts.",
@@ -189,7 +190,7 @@
     "_rebaseline_2026_06_24_headroom_strategy": "Headroom-aware connection selection (dario technique): combo.ts 3168->3180 (+12 = a new `else if (strategy === \"headroom\")` dispatch branch in handleComboChat that delegates to orderTargetsByHeadroom + its log line, plus the import). The actual logic lives OUT of the god-file: the pure ranker rankByHeadroom/computeHeadroom is the new leaf open-sse/services/combo/headroomRanking.ts (91 LOC, <cap) and the async orderer orderTargetsByHeadroom is appended to the existing open-sse/services/combo/quotaStrategies.ts (<cap) next to its sibling reset-aware/reset-window orderers (reuses their connection-expansion machinery). headroom = 1 - max(util_5h, util_7d) from getSaturation (src/lib/quota/saturationSignals.ts), prefers the connection with the most free capacity. Only the dispatch wiring is irreducible at the existing combo strategy chokepoint (mirrors the reset-aware/reset-window/context-optimized branches); not extractable without hiding the call site. fill-first stays default; all existing strategies untouched. Covered by tests/unit/combo-headroom-ranking.test.ts (pure helper) + tests/unit/combo-headroom-strategy.test.ts (orderer, saturation injected). Structural shrink of combo.ts tracked in #3501.",
     "_rebaseline_2026_06_24_quota_share_strategy": "Dedicated quota-share strategy (Phase 3 #9): combo.ts 3180->3190 (+10 = one new `else if (strategy === \"quota-share\")` dispatch branch in handleComboChat that delegates 100% to selectQuotaShareTarget + its log line, plus the import). All the new logic lives OUT of the god-file in two new leaves under open-sse/services/combo/: quotaShareInflight.ts (in-flight counter with TTL/lease, ~150 LOC <cap) and quotaShareStrategy.ts (per-model bucket gating via isBucketSaturated + DRR proportional to weight + P2C over in-flight, ~240 LOC <cap). Only the dispatch wiring is irreducible at the existing combo strategy chokepoint (mirrors the headroom/reset-aware/reset-window/context-optimized branches); not extractable without hiding the call site. ZERO existing strategy cases were modified — only this branch was added, and the qtSd/ combos switched from fill-first to quota-share in src/lib/quota/quotaCombos.ts. Covered by tests/unit/quota-share-strategy.test.ts (gating, DRR fairness, P2C in-flight, fail-open, activation). Structural shrink of combo.ts tracked in #3501.",
     "_rebaseline_2026_06_24_task_aware_routing": "Task-aware routing strategy (port PR #2045, OmniRoute #4945): combo.ts 3190->3225 (+35) = one new `else if (strategy === \"task-aware\")` dispatch branch delegating 100% to selectTaskAwareTarget + its imports/log lines. All scoring/classification logic lives OUT of the god-file in the new leaf open-sse/services/taskAwareRouting.ts (553 LOC <cap). Only the dispatch wiring is irreducible at the existing combo strategy chokepoint (mirrors quota-share/headroom/reset-aware branches). ZERO existing strategy cases modified. Covered by tests/unit/combo-task-aware.test.ts (35 tests). Structural shrink of combo.ts tracked in #3501.",
-    "open-sse/services/combo.ts": 3388,
+    "open-sse/services/combo.ts": 3479,
     "_rebaseline_2026_06_26_fidelity_gate_extraction": "Milestone-B fidelity-gate wiring residual: bodyToText+gateAdvance extracted to fidelityGateStep.ts (889->854, -35), but the StackOptions.fidelityGate field, the `const fidelityGate` reads at the two stacked-loop dispatch chokepoints, and the import of FidelityGateConfig are irreducible wiring that cannot leave strategySelector without an architectural refactor of the pre-existing stacked pipeline. Net: 889->854 (+6 vs the pre-Milestone-B frozen 848). Covered by tests/unit/compression/*.test.ts (940 pass).",
     "_rebaseline_2026_06_28_5243_risk_gate_prepass": "PR #5243 (compression risk-gate pre-pass) own growth: open-sse/services/compression/strategySelector.ts 854->899 (+45). The three exported entry points (applyCompression/applyStackedCompression/applyStackedCompressionAsync) become thin wrappers over pure-extracted private bodies (runCompression/runStackedCompression/runStackedCompressionAsync) so the risk-gate mask->run->restore wrapper sits strictly OUTSIDE the per-step loop — a single universal integration point. The wrapper logic itself (resolveRiskGate/withRiskGate) lives in the new riskGate/strategyWrap.ts (<cap); the residual growth is the duplicated thin-wrapper signatures + the extracted bodies' dispatch boundary, guarded by a byte-identical parity test (riskGateIntegration). Default off (DEFAULT_COMPRESSION_CONFIG unchanged). Not extractable without hiding the dispatch boundary, mirroring prior compression rebaselines. Structural shrink tracked in #3501.",
     "_rebaseline_2026_06_29_5286_memoization": "PR #5286 own growth: strategySelector.ts 899->960 (+61 = the opt-in result-memoization branches in applyCompression/applyCompressionAsync — principal+determinism gate, makeMemoKey lookup/store with model+supportsVision folded into the key, recompute-with-memo-off). Default off (memoizeCompressionResults), so zero behavior change. The memo helpers live in the leaf resultMemo.ts (<cap); the chokepoint wiring here is not extractable. Structural shrink of this hot-path file tracked in #3501.",
@@ -321,7 +322,7 @@
     "tests/unit/chatcore-translation-paths.test.ts": 2810,
     "tests/unit/chatgpt-web.test.ts": 3170,
     "tests/unit/combo-config.test.ts": 881,
-    "tests/unit/combo-routing-engine.test.ts": 3243,
+    "tests/unit/combo-routing-engine.test.ts": 3311,
     "tests/unit/combo-strategy-fallbacks.test.ts": 880,
     "tests/unit/db-core-init.test.ts": 877,
     "tests/unit/db-migration-runner.test.ts": 1499,

open-sse/services/combo.ts

@@ -130,7 +130,10 @@ import {
   releaseRejectedQualityResponse,
   toRetryAfterDisplayValue,
 } from "./combo/validateQuality.ts";
-import { resolveComboCooldownWaitDecision } from "./combo/comboCooldownRetry.ts";
+import {
+  resolveComboCooldownWaitDecision,
+  ResolveComboCooldownDecisionResult,
+} from "./combo/comboCooldownRetry.ts";
 import {
   computeClosestRetryAfter,
   waitForCooldownAwareRetry,
@@ -1418,11 +1421,21 @@ export async function handleComboChat({
   // re-runs ONLY the set loop (selection / shadow routing / setup above stay
   // untouched), preserving the pre-existing `continue`-to-top-of-set-loop
   // semantics exactly.
-  const comboCooldownWaitEnabled =
-    strategy === "quota-share" && resilienceSettings.comboCooldownWait.enabled;
+  const comboCooldownWaitEnabled = resilienceSettings.comboCooldownWait.enabled;
   let comboCooldownAttempt = 0;
   let comboCooldownBudgetLeftMs = resilienceSettings.comboCooldownWait.budgetMs;
 
+  // Global combo timeout: when set (>0), limits total wall-clock time the combo
+  // spends iterating through targets. After each target completes, if elapsed time
+  // exceeds comboTimeoutMs, remaining targets are skipped and a 504 with aggregated
+  // error diagnostics is returned. 0 = disabled (backward-compatible, unlimited).
+  const comboTimeoutMs = config.comboTimeoutMs || 0;
+  const comboStartTime = Date.now();
+  let comboExpired = false;
+  // Accumulator for per-model error details across targets in the current set try.
+  // Reset at the start of each set retry (same lifecycle as lastError/recordedAttempts).
+  let comboErrors: Array<{ model: string; status: number; error: string }> = [];
+
   // FASE 2.1: per-connection concurrency limit for quota-share. The gating in
   // selectQuotaShareTarget is fail-open and cannot hard-limit a single-connection
   // pool, so we serialize concurrent requests to the selected account through a
@@ -1463,6 +1476,7 @@ export async function handleComboChat({
       const startTime = Date.now();
       let fallbackCount = 0;
       let recordedAttempts = 0;
+      comboErrors = [];
 
       // QA P0: assemble a sanitized diagnostic trace from the state already in scope
       // (pool size + this set-try's exhausted providers/connections + attempt order +
@@ -2237,6 +2251,7 @@ export async function handleComboChat({
             });
             recordedAttempts++;
             lastError = errorText || String(result.status);
+            comboErrors.push({ model: modelStr, status: result.status, error: errorText || String(result.status) });
             if (!lastStatus) lastStatus = result.status;
             if (i > 0) fallbackCount++;
             log.warn("COMBO", `Model ${modelStr} failed with body-specific error, stopping combo`);
@@ -2330,6 +2345,7 @@ export async function handleComboChat({
           });
           recordedAttempts++;
           lastError = errorText || String(result.status);
+          comboErrors.push({ model: modelStr, status: result.status, error: errorText || String(result.status) });
           if (!lastStatus) lastStatus = result.status;
           if (i > 0) fallbackCount++;
           // Wire combo failures into the resilience dashboard (model-level lockout)
@@ -2402,7 +2418,7 @@ export async function handleComboChat({
       };
 
       for (let i = 0; i < orderedTargets.length; i++) {
-        if (anySuccess) break;
+        if (anySuccess || comboExpired) break;
 
         const abortController = new AbortController();
         abortControllers.set(i, abortController);
@@ -2447,6 +2463,17 @@ export async function handleComboChat({
         } else {
           await Promise.race([task, globalPromise]);
         }
+
+        // Global combo timeout check: after each target completes, stop trying
+        // further targets if the total elapsed time exceeds comboTimeoutMs.
+        if (!anySuccess && comboTimeoutMs > 0 && Date.now() - comboStartTime >= comboTimeoutMs) {
+          comboExpired = true;
+          log.info(
+            "COMBO",
+            `Combo global timeout (${comboTimeoutMs}ms) reached after ` +
+              `${i + 1}/${orderedTargets.length} targets (${recordedAttempts} attempted) — stopping`
+          );
+        }
       }
 
       if (!anySuccess && runningTasks.size > 0) {
@@ -2457,6 +2484,40 @@ export async function handleComboChat({
         return await globalPromise;
       }
 
+      // Global combo timeout: return aggregated error immediately, skipping set retries.
+      if (comboExpired) {
+        const summary = comboErrors
+          .slice(0, 5)
+          .map((e) => `${e.model} (${e.status})`)
+          .join(", ");
+        const msg =
+          `Combo global timeout (${comboTimeoutMs}ms) after ${recordedAttempts}/${orderedTargets.length} targets` +
+          (comboErrors.length > 0
+            ? ` | tried: ${summary}${comboErrors.length > 5 ? `... (+${comboErrors.length - 5})` : ""}`
+            : "");
+        const latencyMs = Date.now() - startTime;
+        if (recordedAttempts === 0) {
+          recordComboRequest(combo.name, null, {
+            success: false,
+            latencyMs,
+            fallbackCount,
+            strategy,
+          });
+        }
+        notifyWebhookEvent("request.failed", {
+          combo: combo.name,
+          reason: "COMBO_TIMEOUT",
+          latencyMs,
+          fallbackCount,
+        });
+        return errorResponseWithComboDiagnostics(
+          504,
+          msg,
+          buildComboDiag("combo_timeout"),
+          { code: "COMBO_TIMEOUT", type: "server_error" }
+        );
+      }
+
       // All models failed in this set try
       const latencyMs = Date.now() - startTime;
       if (recordedAttempts === 0) {
@@ -2492,39 +2553,69 @@ export async function handleComboChat({
       }
 
       const status = lastStatus;
-      const msg = lastError || "All combo models unavailable";
+      // Build aggregated error message with per-model failure details for diagnostics.
+      const comboErrorSummary =
+        comboErrors.length > 0
+          ? " [" +
+            comboErrors
+              .slice(0, 5)
+              .map((e) => `${e.model} (${e.status})`)
+              .join(", ") +
+            (comboErrors.length > 5 ? `... (+${comboErrors.length - 5})` : "") +
+            "]"
+          : "";
+      const msg = (lastError || "All combo models unavailable") + comboErrorSummary;
 
       if (earliestRetryAfter) {
-        // Quota-share cooldown-aware retry: instead of crystallizing the 429,
-        // wait out a SHORT transient cooldown and re-run the whole set loop.
-        // Guarded by the helper (quota_exhausted/auth/not-found excluded,
-        // ceiling, attempts, budget). MAX_GLOBAL_ATTEMPTS still bounds total
-        // dispatches.
+        // Cooldown-aware retry: instead of crystallizing the 429/503, wait out
+        // a SHORT transient cooldown and re-run the whole set loop. Guarded by
+        // the helper (quota_exhausted/auth/not-found excluded, ceiling,
+        // attempts, budget). MAX_GLOBAL_ATTEMPTS still bounds total dispatches.
+        // Available to ALL combo strategies (not just quota-share).
         if (comboCooldownWaitEnabled && status === 429) {
-          const decision = resolveComboCooldownWaitDecision({
+          // ONE decision path for EVERY strategy. The reason that drives the
+          // wait is always the target's REAL model-lockout reason, resolved
+          // through the helper's allow-list — never a hardcoded literal.
+          //
+          // SECURITY (see comboCooldownRetry.ts header): the allow-list is the
+          // PRIMARY barrier and `maxWaitMs` only the SECOND one. Hardcoding
+          // reason:"rate_limit" for non-quota-share strategies would drop the
+          // primary barrier and leave only the ceiling — which does NOT cover a
+          // quota_exhausted lock carrying a SHORT upstream retry-after (e.g.
+          // 3s < maxWaitMs): the combo would wait, redispatch against a model
+          // locked until midnight, and burn the attempt. Model lockouts are
+          // recorded for all strategies (recordModelLockoutFailure above is not
+          // gated on quota-share), so the real reason is always available.
+          const decision: ResolveComboCooldownDecisionResult = resolveComboCooldownWaitDecision({
             targets: orderedTargets,
             earliestRetryAfter,
             attempt: comboCooldownAttempt,
             budgetLeftMs: comboCooldownBudgetLeftMs,
             settings: resilienceSettings.comboCooldownWait,
-            lookupLock: (provider, connectionId) => {
-              const rawModel = parseModel(orderedTargets[0]?.modelStr ?? "").model || "";
+            // Key each lookup on the TARGET's own model: quota-share combos are
+            // single-model/multi-account (so this is identical to the previous
+            // orderedTargets[0] behavior), but heterogeneous combos carry a
+            // different model per target.
+            lookupLock: (provider, connectionId, target) => {
+              const rawModel = parseModel(target?.modelStr ?? "").model || "";
+              if (!rawModel) return null;
               return getModelLockoutInfo(provider, connectionId, rawModel);
             },
             computeWaitMs: (retryAfter) => computeClosestRetryAfter(retryAfter).waitMs,
           });
+
           if (decision.wait) {
             log.info(
               "COMBO",
-              `Quota-share cooldown wait: ${msg} — waiting ${Math.ceil(
+              `${strategy} cooldown wait: ${msg} — waiting ${Math.ceil(
                 decision.waitMs / 1000
               )}s (reason=${decision.reason ?? "?"}) then retrying (attempt ${
                 comboCooldownAttempt + 1
               }/${resilienceSettings.comboCooldownWait.maxAttempts})`
             );
             const completed = await waitForCooldownAwareRetry(decision.waitMs, signal);
             if (!completed) {
-              log.info("COMBO", "Quota-share cooldown wait aborted by client disconnect");
+              log.info("COMBO", `${strategy} cooldown wait aborted by client disconnect`);
               return errorResponse(499, "Request aborted");
             }
             comboCooldownAttempt += 1;