Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions config/quality/file-size-baseline.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"_comment": "Catraca de tamanho (check-file-size.mjs). frozen so pode encolher; arquivos novos <= cap. --update ratcheta.",
"_rebaseline_2026_06_24_per_connection_max_concurrent": "Feature (per-connection max_concurrent enforcement, issue #9 follow-up) own growth: open-sse/services/combo.ts 3225->3238 (+13 net). Provider connections already carry a maxConcurrent ceiling (provider_connections.max_concurrent, migration 029, set in the UI/API/DB) but routing ignored it, so low-concurrency subscription accounts (GLM/MiniMax ~1) got flooded -> 429 + cooldown, worse under quota-share where keys share one account. The new wiring lives at the two combo dispatch chokepoints: (1) the strategy===\"quota-share\" branch resolves each target connection's cap and passes it into selectQuotaShareTarget so an at-cap connection is deprioritized (fail-open, never hard-blocked); (2) handleRoundRobinCombo resolves the target connection's cap and uses it as the semaphore maxConcurrency (fallback = combo-level concurrency). ALL extractable logic was moved into the new pure leaf open-sse/services/combo/concurrencyCaps.ts (resolveMaxConcurrentByConnection + makeConnectionConcurrencyResolver, DB reads deduped per connectionId, fail-open) and the pure effectiveMaxConcurrency resolver into combo/comboPredicates.ts; combo.ts keeps only the irreducible chokepoint wiring (a 5-line import + the two call-site bindings). Cohesive at the existing dispatch boundaries, not further extractable. Structural shrink of combo.ts tracked in #3501. Covered by tests/unit/quota-share-strategy.test.ts (per-connection maxConcurrent gating cases) + tests/unit/combo/effective-max-concurrency.test.ts.",
"_rebaseline_2026_06_24_combo_cooldown_wait_quota_share": "Feature quota-share combo cooldown-aware retry (Variante A) own growth: open-sse/services/combo.ts 3225->3293 (+68 = the cooldown-wait wrap inside handleComboChat's quota-share path. The existing setTry loop body is LEFT at its original indentation: instead of an outer `while (true)` (which would re-indent ~1600 lines and bloat the review), the setTry loop is hoisted into a small recursive closure `dispatchWithCooldownRetry`, and a wait+redispatch is a tail `return dispatchWithCooldownRetry()` — re-running ONLY the set loop (exactly the prior continue-to-top-of-set-loop semantics) while selection/shadow-routing/setup above stay untouched. At the 429 crystallization point the lock reason is resolved via getModelLockoutInfo, the decision via the new pure resolveComboCooldownWaitDecision, then await waitForCooldownAwareRetry (499 on abort), decrement the budget, recurse. Gated to strategy==='quota-share' && comboCooldownWait.enabled. `git diff` == `git diff -w` for combo.ts (zero re-indentation noise). The gating policy + reason resolution are extracted to the new pure leaf open-sse/services/combo/comboCooldownRetry.ts (<cap, unit-tested) to keep combo.ts thin; only the wait orchestration (one getModelLockoutInfo lookup + one await) stays at the chokepoint, not extractable. The quota_exhausted/auth/not-found exclusions defend against isRetryableModelLockoutReason (auth.ts:533) treating quota_exhausted as retryable (locked-until-midnight). Covered by tests/unit/combo-cooldown-retry.test.ts (pure helper, all branches) + tests/unit/combo-quota-share-cooldown-wait.test.ts (integration: rate_limit waits+recovers, quota_exhausted no-wait, abort=499, non-quota-share unchanged) + tests/unit/resilience-settings-combo-cooldown-wait.test.ts (settings round-trip/clamp). Also src/app/(dashboard)/dashboard/settings/components/ResilienceTab.tsx 983->1098 (+115 = the ComboCooldownWaitCard exposing enabled/maxWaitMs/maxAttempts/budgetMs in Settings > Resilience, mirroring WaitForCooldownCard; wired through GET/PATCH in src/app/api/resilience/route.ts + comboCooldownWaitSettingsSchema in src/shared/validation/schemas/settings.ts; new UI labels use the t(key)||English-fallback pattern, en-only). Structural shrink of combo.ts + ResilienceTab tracked in #3501.",
"_rebaseline_2026_06_23_4774_combo_legacy_strip": "PR #4774 (KooshaPari, #4382 round-trip) own growth: src/app/(dashboard)/dashboard/combos/page.tsx 4434->4456 (+22 = the client-side LEGACY_COMBO_RESILIENCE_KEYS Set gains queueTimeoutMs + the 12 v3.8.31-era removed keys (queueDepth/fallbackDelayMs/handoffProviders/maxComboDepth/manifestRouting/complexityAwareRouting/pipeline_enabled/pipelineConcurrency/shadowRouting/evalRouting/resetAwareEnabled/resetAwareWindow) with explanatory comments, mirroring the server-side strip list in src/app/api/combos/[id]/route.ts so the modal never re-introduces removed keys on Save). Functional strip list, not a movable block; combos/page.tsx structural shrink tracked in #3501. Covered by tests/unit/combo-config.test.ts (auto-promote + passthrough + legacy-key round-trip).",
"_rebaseline_2026_06_21_4421_node_lookup": "Issue #4421 own growth: src/lib/db/providers.ts 1050->1063 (+13 = resolveProviderNodeForConnection at the existing provider-node lookup — resolves a connection node by exact id OR the bare derived type when unambiguous, + import). Pure selection logic in the new src/lib/db/providerNodeSelect.ts (<cap, unit-tested); cohesive DB wiring next to getProviderNodeById, not extractable. Covered by tests/unit/provider-node-select-4421.test.ts.",
"_rebaseline_2026_06_21_v3833_release_basered_fixes": "Release v3.8.33: 5 deterministic base-reds inherited from parallel-session merges (fast-gates PR→release não rodam test:unit completo) consertados no PR de release. src/sse/services/auth.ts 2279->2289 (+10): #4530 só fiou maxCooldownMs nos 3 sites de combo.ts; os 4 sites de markAccountUnavailable (per-model quota, grok-web 403, per-model 403, local 404) nunca passavam o cap → resolvo mlSettings uma vez e passo maxCooldownMs em todos. tests/unit/db-core-init.test.ts 864->867 (+3): comentário explicando o cap intencional busy_timeout 5s->2s do v3.8.32. Wiring necessário ao chokepoint de lockout; não extraível. Coberto por model-lockout-max-cooldown.test.ts + db-core-init.test.ts. (Demais base-reds — áudio mp3 #912/#913 dedup de handler em geminiHelper.ts, e closure #3578 src/models/ no package.json files — não cresceram arquivo congelado.)",
Expand Down Expand Up @@ -144,7 +145,7 @@
"_rebaseline_2026_06_24_headroom_strategy": "Headroom-aware connection selection (dario technique): combo.ts 3168->3180 (+12 = a new `else if (strategy === \"headroom\")` dispatch branch in handleComboChat that delegates to orderTargetsByHeadroom + its log line, plus the import). The actual logic lives OUT of the god-file: the pure ranker rankByHeadroom/computeHeadroom is the new leaf open-sse/services/combo/headroomRanking.ts (91 LOC, <cap) and the async orderer orderTargetsByHeadroom is appended to the existing open-sse/services/combo/quotaStrategies.ts (<cap) next to its sibling reset-aware/reset-window orderers (reuses their connection-expansion machinery). headroom = 1 - max(util_5h, util_7d) from getSaturation (src/lib/quota/saturationSignals.ts), prefers the connection with the most free capacity. Only the dispatch wiring is irreducible at the existing combo strategy chokepoint (mirrors the reset-aware/reset-window/context-optimized branches); not extractable without hiding the call site. fill-first stays default; all existing strategies untouched. Covered by tests/unit/combo-headroom-ranking.test.ts (pure helper) + tests/unit/combo-headroom-strategy.test.ts (orderer, saturation injected). Structural shrink of combo.ts tracked in #3501.",
"_rebaseline_2026_06_24_quota_share_strategy": "Dedicated quota-share strategy (Phase 3 #9): combo.ts 3180->3190 (+10 = one new `else if (strategy === \"quota-share\")` dispatch branch in handleComboChat that delegates 100% to selectQuotaShareTarget + its log line, plus the import). All the new logic lives OUT of the god-file in two new leaves under open-sse/services/combo/: quotaShareInflight.ts (in-flight counter with TTL/lease, ~150 LOC <cap) and quotaShareStrategy.ts (per-model bucket gating via isBucketSaturated + DRR proportional to weight + P2C over in-flight, ~240 LOC <cap). Only the dispatch wiring is irreducible at the existing combo strategy chokepoint (mirrors the headroom/reset-aware/reset-window/context-optimized branches); not extractable without hiding the call site. ZERO existing strategy cases were modified — only this branch was added, and the qtSd/ combos switched from fill-first to quota-share in src/lib/quota/quotaCombos.ts. Covered by tests/unit/quota-share-strategy.test.ts (gating, DRR fairness, P2C in-flight, fail-open, activation). Structural shrink of combo.ts tracked in #3501.",
"_rebaseline_2026_06_24_task_aware_routing": "Task-aware routing strategy (port PR #2045, OmniRoute #4945): combo.ts 3190->3225 (+35) = one new `else if (strategy === \"task-aware\")` dispatch branch delegating 100% to selectTaskAwareTarget + its imports/log lines. All scoring/classification logic lives OUT of the god-file in the new leaf open-sse/services/taskAwareRouting.ts (553 LOC <cap). Only the dispatch wiring is irreducible at the existing combo strategy chokepoint (mirrors quota-share/headroom/reset-aware branches). ZERO existing strategy cases modified. Covered by tests/unit/combo-task-aware.test.ts (35 tests). Structural shrink of combo.ts tracked in #3501.",
"open-sse/services/combo.ts": 3239,
"open-sse/services/combo.ts": 3306,
"open-sse/services/compression/strategySelector.ts": 848,
"open-sse/services/rateLimitManager.ts": 1035,
"open-sse/services/tokenRefresh.ts": 2070,
Expand Down Expand Up @@ -184,7 +185,7 @@
"src/app/(dashboard)/dashboard/settings/components/MemorySkillsTab.tsx": 898,
"src/app/(dashboard)/dashboard/settings/components/PricingTab.tsx": 1012,
"src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx": 1089,
"src/app/(dashboard)/dashboard/settings/components/ResilienceTab.tsx": 983,
"src/app/(dashboard)/dashboard/settings/components/ResilienceTab.tsx": 1098,
"src/app/(dashboard)/dashboard/settings/components/RoutingTab.tsx": 1629,
"src/app/(dashboard)/dashboard/settings/components/SystemStorageTab.tsx": 1924,
"src/app/(dashboard)/dashboard/usage/components/BudgetTab.tsx": 1016,
Expand Down
69 changes: 68 additions & 1 deletion open-sse/services/combo.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
classifyLockoutReason,
decayModelFailureCount,
formatRetryAfter,
getModelLockoutInfo,
getRuntimeProviderProfile,
isModelLocked,
recordModelLockoutFailure,
Expand Down Expand Up @@ -124,6 +125,11 @@ import {
recordStickyWeightedSuccess,
} from "./combo/rrState.ts";
import { validateResponseQuality, toRetryAfterDisplayValue } from "./combo/validateQuality.ts";
import { resolveComboCooldownWaitDecision } from "./combo/comboCooldownRetry.ts";
import {
computeClosestRetryAfter,
waitForCooldownAwareRetry,
} from "../../src/sse/services/cooldownAwareRetry.ts";
import { handleFusionChat, type FusionTuning } from "./fusion.ts";
import {
TRANSIENT_FOR_SEMAPHORE,
Expand Down Expand Up @@ -1641,7 +1647,27 @@ export async function handleComboChat({

let globalAttempts = 0;

try {
// Quota-share cooldown-aware retry (Variante A). Only quota-share (qtSd/)
// combos opt in: when the set loop would crystallize a 429 model_cooldown
// because the target hit a SHORT transient cooldown, we wait it out and
// re-run the whole set loop instead of propagating the 429. `globalAttempts`
// persists across these waits so MAX_GLOBAL_ATTEMPTS still bounds total work.
// The wait happens at the crystallization point, which holds no semaphore slot
// (the quota-share path never calls semaphore.acquire) and no explicit
// in-flight lease that must be released here.
//
// The set loop is wrapped in a small recursive closure rather than an extra
// labelled `while (true)` so the loop body keeps its original indentation; a
// wait+redispatch is a tail `return dispatchWithCooldownRetry()`, which
// re-runs ONLY the set loop (selection / shadow routing / setup above stay
// untouched), preserving the pre-existing `continue`-to-top-of-set-loop
// semantics exactly.
const comboCooldownWaitEnabled =
strategy === "quota-share" && resilienceSettings.comboCooldownWait.enabled;
let comboCooldownAttempt = 0;
let comboCooldownBudgetLeftMs = resilienceSettings.comboCooldownWait.budgetMs;

const dispatchWithCooldownRetry = async (): Promise<Response> => {
for (let setTry = 0; setTry <= maxSetRetries; setTry++) {
// #1731: Per-set-iteration set of providers whose quota is fully exhausted.
// Reset each retry so providers excluded in a previous attempt get another chance.
Expand Down Expand Up @@ -2565,6 +2591,43 @@ export async function handleComboChat({
const msg = lastError || "All combo models unavailable";

if (earliestRetryAfter) {
// Quota-share cooldown-aware retry: instead of crystallizing the 429,
// wait out a SHORT transient cooldown and re-run the whole set loop.
// Guarded by the helper (quota_exhausted/auth/not-found excluded,
// ceiling, attempts, budget). MAX_GLOBAL_ATTEMPTS still bounds total
// dispatches.
if (comboCooldownWaitEnabled && status === 429) {
const decision = resolveComboCooldownWaitDecision({
targets: orderedTargets,
earliestRetryAfter,
attempt: comboCooldownAttempt,
budgetLeftMs: comboCooldownBudgetLeftMs,
settings: resilienceSettings.comboCooldownWait,
lookupLock: (provider, connectionId) => {
const rawModel = parseModel(orderedTargets[0]?.modelStr ?? "").model || "";
return getModelLockoutInfo(provider, connectionId, rawModel);
},
computeWaitMs: (retryAfter) => computeClosestRetryAfter(retryAfter).waitMs,
});
if (decision.wait) {
log.info(
"COMBO",
`Quota-share cooldown wait: ${msg} — waiting ${Math.ceil(
decision.waitMs / 1000
)}s (reason=${decision.reason ?? "?"}) then retrying (attempt ${
comboCooldownAttempt + 1
}/${resilienceSettings.comboCooldownWait.maxAttempts})`
);
const completed = await waitForCooldownAwareRetry(decision.waitMs, signal);
if (!completed) {
log.info("COMBO", "Quota-share cooldown wait aborted by client disconnect");
return errorResponse(499, "Request aborted");
}
comboCooldownAttempt += 1;
comboCooldownBudgetLeftMs = Math.max(0, comboCooldownBudgetLeftMs - decision.waitMs);
return dispatchWithCooldownRetry();
}
}
const retryHuman = formatRetryAfter(toRetryAfterDisplayValue(earliestRetryAfter));
log.warn("COMBO", `All models failed | ${msg} (${retryHuman})`);
return unavailableResponse(status, msg, earliestRetryAfter, retryHuman);
Expand All @@ -2578,6 +2641,10 @@ export async function handleComboChat({
}

return errorResponse(503, "Combo routing completed without an upstream response");
};

try {
return await dispatchWithCooldownRetry();
} finally {
// G2: Clean up candidate registry to prevent unbounded memory growth.
_unregisterExecutionCandidates(_registeredExecutionKeys);
Expand Down
Loading
Loading