[defer] feat(combo): universal cooldown-aware retry & auto-strategy combo-ref guard - #7301
Merged
diegosouzapw merged 6 commits intoJul 21, 2026
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request extends the cooldown-aware retry mechanism to all combo strategies (beyond just quota-share) and prevents candidate pool expansion in expandAutoComboCandidatePool when a combo references other combos via combo-ref. The reviewer identified an inconsistency in open-sse/services/combo.ts where the code comments mention handling both 429 and 503 status codes, but the conditional check is restricted to 429, and suggested adding support for 503.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| // the helper (quota_exhausted/auth/not-found excluded, ceiling, | ||
| // attempts, budget). MAX_GLOBAL_ATTEMPTS still bounds total dispatches. | ||
| // Available to ALL combo strategies (not just quota-share). | ||
| if (comboCooldownWaitEnabled && status === 429) { |
Contributor
There was a problem hiding this comment.
O comentário na linha 2369 indica explicitamente a intenção de tratar erros 429/503 no mecanismo de retry ciente de cooldown (Cooldown-aware retry: instead of crystallizing the 429/503, wait out...). No entanto, a condição do if na linha 2374 ainda está restrita apenas a status === 429.
Se um erro 503 (Service Unavailable) ocorrer e retornar um cabeçalho Retry-After válido, ele não acionará o cooldown wait devido a essa restrição, o que diverge do comportamento documentado no comentário.
Sugestão: Atualize a condição para incluir o status 503.
Suggested change
| if (comboCooldownWaitEnabled && status === 429) { | |
| if (comboCooldownWaitEnabled && (status === 429 || status === 503)) { |
Owner
|
Thanks for the universal cooldown-aware retry idea — resilience around combo routing is a great area to improve. 🙏 Holding this one for now rather than merging: the auto-strategy combo-ref guard overlaps with resilience logic that's actively moving on the release branch, and the PR targets |
diegosouzapw
changed the title
… guard
Two changes:
1. Universal cooldown-aware retry (combo.ts):
- Remove strategy==="quota-share" gate from comboCooldownWaitEnabled
- Enables all 18 combo strategies (priority, weighted, round-robin, etc.)
to wait out a short transient cooldown and retry the full set
- Non-quota-share strategies use shouldWaitForComboCooldown directly
with earliestRetryAfter and reason="rate_limit" (no per-model lockout)
- quota-share retains its existing per-target model lockout logic
2. Auto-strategy combo-ref guard (autoStrategy.ts):
- expandAutoComboCandidatePool now detects kind==="combo-ref" entries
- When present, returns eligibleTargets without expanding to ALL providers
- Fixes scenario where an "auto" combo with combo-ref delegates to a
sub-combo but pulls in every model from every active provider
Adds two features to improve combo resilience and debuggability:
1. Global combo timeout (comboTimeoutMs)
- Configurable per-combo via DEFAULT_COMBO_CONFIG (default 0 = disabled)
- After each target completes, checks if total elapsed time exceeds limit
- When exceeded, stops trying further targets and returns 504 immediately
- Backward-compatible: 0 preserves legacy unlimited-iteration behavior
2. Aggregated error diagnostics
- comboErrors array accumulates per-model failure details (model, status, error)
- On combo timeout or all-models-exhausted, returns a message listing the
first (up to 5) model-level errors with their HTTP status codes
- Enables operators to see WHICH models failed and WHY without digging
through individual server logs
…d combo-ref guard Adds/updates automated coverage for this PR's production changes (PR Test Policy requires tests alongside src/open-sse/electron/bin changes): - Update the "preserves the first failure status" expectation in combo-routing-engine.test.ts: the aggregated per-model error-diagnostics suffix is new intended output, not a regression. - Add two new tests for the global comboTimeoutMs feature: the combo stops dispatching further targets and returns 504/COMBO_TIMEOUT with aggregated diagnostics once the ceiling trips, and comboTimeoutMs=0 (default) never trips it. - Rewrite the non-quota-share (priority) cooldown-wait scenario in combo-quota-share-cooldown-wait-timing.test.ts: comboCooldownWait is no longer gated on strategy === "quota-share", so a priority combo now waits out a short 429 and re-dispatches too (via shouldWaitForComboCooldown with reason "rate_limit"), instead of propagating immediately as before. Also adds the disabled-flag counterpart for parity with the quota-share suite. - Add coverage for the #COMBO-REF guard in expandAutoComboCandidatePool: a combo whose models array contains a kind:"combo-ref" entry must not be expanded to every model of every active provider. Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
The two new tests initially copied the neighbouring tests' `any`-typed handleSingleModel params / json() casts. Those neighbours are pre-existing violations frozen in config/quality/eslint-suppressions.json at a count of 261 for this file, so the 7 new occurrences pushed it to 268 and broke `npm run lint` (no-explicit-any is an error in tests/ since diegosouzapw#6218; new violations must be fixed, not re-frozen). Type the new tests properly instead: `unknown`/`string` params, a ComboErrorPayload interface for the parsed body, and drop the unused `relayOptions: null as any` (the sibling combo cooldown suites already omit it). Back to exactly 261 — the suppression file is untouched. Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
…, not a hardcoded "rate_limit"
The universal cooldown-aware retry kept the quota-share path on
resolveComboCooldownWaitDecision but gave every OTHER strategy a shortcut that
hardcoded `reason: "rate_limit"` and fed shouldWaitForComboCooldown the
earliestRetryAfter directly.
comboCooldownRetry.ts documents TWO deliberate barriers ("SECURITY —
quota_exhausted must be excluded"): (1) the reason allow-list, and (2) the
maxWaitMs ceiling, explicitly called the SECOND barrier. Hardcoding the reason
removed barrier 1 for 17 of the 18 strategies and left only the ceiling — which
does NOT cover a quota_exhausted lock whose wait lands under maxWaitMs. In that
case the combo waits, redispatches against a model that is locked until the
quota resets, and burns the retry budget for nothing.
The shortcut's premise ("non-quota-share combos have no per-connection model
lockout tracking") is also false: recordModelLockoutFailure in the target loop
is not gated on quota-share, so every strategy records model lockouts and the
real reason is always available.
Fix: one decision path for every strategy, always through
resolveComboCooldownWaitDecision, so the reason always crosses the allow-list.
The lock lookup is now keyed on each TARGET's own model (via a new third
`target` arg on lookupLock) — quota-share combos are single-model/multi-account
so this is identical to the previous orderedTargets[0] behavior, but
heterogeneous combos (priority, weighted, round-robin, …) carry a different
model per target and would otherwise miss every lock but the first.
Regression guard (tests/unit/serial/combo-quota-share-cooldown-wait-timing.test.ts):
a priority combo where modelLockout.errorCodes=[403] leaves the 403's
quota_exhausted lock as the only one in play while a 429 crystallizes the
status, and the resulting wait is short enough that the ceiling lets it through
— so only the allow-list can stop it. Verified failing-then-passing: with the
hardcoded reason the combo makes 6 dispatches (wait+redispatch x2); with the
real reason it makes 2 and propagates the 429.
Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
…wth (combo +91, combo-routing-engine test +68)
diegosouzapw
force-pushed
the
feat/universal-cooldown-retry
branch
from
July 21, 2026 17:12
79762ef to
0acaadf
Compare
Owner
|
Validated in local merge-train on tomni-proxmox-113 @ db6718332763af295ec8374913f1f5eb7641734f (FAST gates green: static + changed tests + vitest) |
diegosouzapw
added a commit
that referenced
this pull request
Jul 22, 2026
…es (OAuthModal/muse-spark/combo + PricingTab/ComboDefaultsTab) Legitimate own-growth from real features merged this cycle (#7735 grok OAuth chooser, #7528 muse-spark WS rewrite, #7301 combo cooldown-retry, #7972 pricing, #7973/#8008 combo). File-size ratchet is release-captain territory; owner-approved rebaseline. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
This was referenced Jul 23, 2026
Merged
6 tasks done
diegosouzapw
added a commit
to DinonowDev/OmniRoute
that referenced
this pull request
Jul 25, 2026
…file-size caps open-sse/services/combo.ts and tests/unit/combo-config.test.ts crossed the frozen file-size baseline after widening the cooldown-wait gate to every combo strategy. Compact the new inline comments and turn the per-strategy assertions into loops — no coverage lost, same reasons/strategies still asserted individually. Refs diegosouzapw#8541, diegosouzapw#7301. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
mikeiagents
pushed a commit
to mikeiagents/OmniRoute
that referenced
this pull request
Jul 25, 2026
…ls[] is non-empty When an auto-combo has models[] populated by the operator but config.auto.candidatePool is empty (the default for combos created via the dashboard), expandAutoComboCandidatePool silently expands the candidate pool to every model of every active provider connection. This overrides the operator's hand-curated models[] and lets unintended models (e.g. gemini-3.1-flash-lite) win the auto-strategy scoring contest. In omniroute@3.8.48 (npm) only one guard exists before the expansion loop (GUARD A: if (config.auto.candidatePool populated) return eligibleTargets). The upstream release branch release/v3.8.49 added a second guard (combo-ref check, PR diegosouzapw#7301) but it does not cover the common pattern where the operator curates with explicit kind:"model" entries. Both gaps share the same root mechanism and the same fix. The new guard short-circuits whenever models[] is a non-empty array, covering both kind:"model" entries (the dashboard default) and kind:"combo-ref" entries (which diegosouzapw#7301 already handles). With this guard in place, the existing combo-ref check becomes redundant; it is left in place for the minimal-scope surgical fix, and can be removed in a follow-up cleanup. Validation (in isolated Docker, 3 providers + 4 controlled combos): - 3 explicit models, empty candidatePool: pool 60 -> 6 - 1 combo-ref + 2 explicit, empty candidatePool: pool 64 -> 10 - 3 explicit models, populated candidatePool (GUARD A path): 6 -> 6 - empty models[] virtual auto: 57 -> 57 (expansion preserved) Closes diegosouzapw#8597
mikeiagents
pushed a commit
to mikeiagents/OmniRoute
that referenced
this pull request
Jul 25, 2026
…ls[] is non-empty When an auto-combo has models[] populated by the operator but config.auto.candidatePool is empty (the default for combos created via the dashboard), expandAutoComboCandidatePool silently expands the candidate pool to every model of every active provider connection. This overrides the operator's hand-curated models[] and lets unintended models (e.g. gemini-3.1-flash-lite) win the auto-strategy scoring contest. In omniroute@3.8.48 (npm) only one guard exists before the expansion loop (GUARD A: if (config.auto.candidatePool populated) return eligibleTargets). The upstream release branch release/v3.8.49 added a second guard (combo-ref check, PR diegosouzapw#7301) but it does not cover the common pattern where the operator curates with explicit kind:"model" entries. Both gaps share the same root mechanism and the same fix. The new guard short-circuits whenever models[] is a non-empty array, covering both kind:"model" entries (the dashboard default) and kind:"combo-ref" entries (which diegosouzapw#7301 already handles). With this guard in place, the existing combo-ref check becomes redundant; it is left in place for the minimal-scope surgical fix, and can be removed in a follow-up cleanup. Validation (in isolated Docker, 3 providers + 4 controlled combos): - 3 explicit models, empty candidatePool: pool 60 -> 6 - 1 combo-ref + 2 explicit, empty candidatePool: pool 64 -> 10 - 3 explicit models, populated candidatePool (GUARD A path): 6 -> 6 - empty models[] virtual auto: 57 -> 57 (expansion preserved) Closes diegosouzapw#8597
mikeiagents
pushed a commit
to mikeiagents/OmniRoute
that referenced
this pull request
Jul 25, 2026
…ls[] is non-empty When an auto-combo has models[] populated by the operator but config.auto.candidatePool is empty (the default for combos created via the dashboard), expandAutoComboCandidatePool silently expands the candidate pool to every model of every active provider connection. This overrides the operator's hand-curated models[] and lets unintended models (e.g. gemini-3.1-flash-lite) win the auto-strategy scoring contest. In omniroute@3.8.48 (npm) only one guard exists before the expansion loop (GUARD A: if (config.auto.candidatePool populated) return eligibleTargets). The upstream release branch release/v3.8.49 added a second guard (combo-ref check, PR diegosouzapw#7301) but it does not cover the common pattern where the operator curates with explicit kind:"model" entries. Both gaps share the same root mechanism and the same fix. The new guard short-circuits whenever models[] is a non-empty array, covering both kind:"model" entries (the dashboard default) and kind:"combo-ref" entries (which diegosouzapw#7301 already handles). With this guard in place, the existing combo-ref check becomes redundant; it is left in place for the minimal-scope surgical fix, and can be removed in a follow-up cleanup. Validation (in isolated Docker, 3 providers + 4 controlled combos): - 3 explicit models, empty candidatePool: pool 60 -> 6 - 1 combo-ref + 2 explicit, empty candidatePool: pool 64 -> 10 - 3 explicit models, populated candidatePool (GUARD A path): 6 -> 6 - empty models[] virtual auto: 57 -> 57 (expansion preserved) Closes diegosouzapw#8597
mikeiagents
pushed a commit
to mikeiagents/OmniRoute
that referenced
this pull request
Jul 25, 2026
…ls[] is non-empty When an auto-combo has models[] populated by the operator but config.auto.candidatePool is empty (the default for combos created via the dashboard), expandAutoComboCandidatePool silently expands the candidate pool to every model of every active provider connection. This overrides the explicit list in models[] and lets unintended models (e.g. gemini-3.1-flash-lite) win the auto-strategy scoring contest. In omniroute@3.8.48 (npm) only one guard exists before the expansion loop (GUARD A: if (config.auto.candidatePool populated) return eligibleTargets). The upstream release branch release/v3.8.49 added a second guard (combo-ref check, PR diegosouzapw#7301) but it does not cover the common pattern where models[] holds explicit kind:"model" entries. Both gaps share the same root mechanism and the same fix. The new guard short-circuits whenever models[] is a non-empty array, covering both kind:"model" entries (the dashboard default) and kind:"combo-ref" entries (which diegosouzapw#7301 already handles). With this guard in place, the existing combo-ref check becomes redundant; it is left in place for the minimal-scope surgical fix, and can be removed in a follow-up cleanup. Validation (in isolated Docker, 3 providers + 4 controlled combos): - 3 explicit models, empty candidatePool: pool 60 -> 6 - 1 combo-ref + 2 explicit, empty candidatePool: pool 64 -> 10 - 3 explicit models, populated candidatePool (GUARD A path): 6 -> 6 - empty models[] virtual auto: 57 -> 57 (expansion preserved) Closes diegosouzapw#8597
diegosouzapw
added a commit
to Prudhvivuda/OmniRoute
that referenced
this pull request
Jul 25, 2026
…ture.ts check:file-size was red for this PR's own growth: combo.ts grew 3640->3693 (+53, the capability-filter fail-closed guard + compatFilterFailOpen escape hatch at both call sites) and combo/comboStructure.ts crossed the 800-line new-file cap at 918 (describeCapabilityFilterExhaustion + providerSupportsEmulatedToolCalling for the diegosouzapw#5240 emulated-tool-calling exemption). Both are irreducible orchestration wiring at the existing combo filter chokepoint (same precedent as diegosouzapw#7301's cooldown-retry generalization). Companion test tests/unit/combo-routing-engine.test.ts frozen at its own grown size (3409->3449). No logic change; 95/95 tests pass. Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
diegosouzapw
added a commit
that referenced
this pull request
Jul 26, 2026
#8494) * fix(backend): fail closed when capability filters empty the combo pool Tools/vision/structured_output filters no longer re-admit the full pool when every target is incompatible. Opt-in via combo config compatFilterFailOpen. Closes #8488 * fix(backend): keep tool-emulation providers under fail-closed filters Carve out providers with toolCalling:"emulated" (#5240) from tools capability_mismatch so chatgpt-web combos still reach the prompt shim. Align round-robin compatFilterFailOpen with settings fallback and drop new any-typed params from the #8488 combo-routing tests. * chore(lint): prune stale combo-routing-engine any suppressions Test cleanup in #8488 dropped four no-explicit-any hits; sync the freeze file. * chore(quality): rebaseline combo.ts + freeze new-above-cap comboStructure.ts check:file-size was red for this PR's own growth: combo.ts grew 3640->3693 (+53, the capability-filter fail-closed guard + compatFilterFailOpen escape hatch at both call sites) and combo/comboStructure.ts crossed the 800-line new-file cap at 918 (describeCapabilityFilterExhaustion + providerSupportsEmulatedToolCalling for the #5240 emulated-tool-calling exemption). Both are irreducible orchestration wiring at the existing combo filter chokepoint (same precedent as #7301's cooldown-retry generalization). Companion test tests/unit/combo-routing-engine.test.ts frozen at its own grown size (3409->3449). No logic change; 95/95 tests pass. Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com> --------- Co-authored-by: ikelvingo <im.kelvinwong@gmail.com> Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com> Co-authored-by: Prudhvivuda <Prudhvivuda@users.noreply.github.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
diegosouzapw
pushed a commit
that referenced
this pull request
Jul 27, 2026
…ls[] is non-empty (#8598) When an auto-combo has models[] populated by the operator but config.auto.candidatePool is empty (the default for combos created via the dashboard), expandAutoComboCandidatePool silently expands the candidate pool to every model of every active provider connection. This overrides the explicit list in models[] and lets unintended models (e.g. gemini-3.1-flash-lite) win the auto-strategy scoring contest. In omniroute@3.8.48 (npm) only one guard exists before the expansion loop (GUARD A: if (config.auto.candidatePool populated) return eligibleTargets). The upstream release branch release/v3.8.49 added a second guard (combo-ref check, PR #7301) but it does not cover the common pattern where models[] holds explicit kind:"model" entries. Both gaps share the same root mechanism and the same fix. The new guard short-circuits whenever models[] is a non-empty array, covering both kind:"model" entries (the dashboard default) and kind:"combo-ref" entries (which #7301 already handles). With this guard in place, the existing combo-ref check becomes redundant; it is left in place for the minimal-scope surgical fix, and can be removed in a follow-up cleanup. Validation (in isolated Docker, 3 providers + 4 controlled combos): - 3 explicit models, empty candidatePool: pool 60 -> 6 - 1 combo-ref + 2 explicit, empty candidatePool: pool 64 -> 10 - 3 explicit models, populated candidatePool (GUARD A path): 6 -> 6 - empty models[] virtual auto: 57 -> 57 (expansion preserved) Closes #8597 Co-authored-by: Michael de Souza Marcos <michael.smarcos@hotmail.com>
HouMinXi
pushed a commit
to HouMinXi/OmniRoute
that referenced
this pull request
Aug 2, 2026
…ombo-ref guard (diegosouzapw#7301) * feat(combo): universal cooldown-aware retry & auto-strategy combo-ref guard Two changes: 1. Universal cooldown-aware retry (combo.ts): - Remove strategy==="quota-share" gate from comboCooldownWaitEnabled - Enables all 18 combo strategies (priority, weighted, round-robin, etc.) to wait out a short transient cooldown and retry the full set - Non-quota-share strategies use shouldWaitForComboCooldown directly with earliestRetryAfter and reason="rate_limit" (no per-model lockout) - quota-share retains its existing per-target model lockout logic 2. Auto-strategy combo-ref guard (autoStrategy.ts): - expandAutoComboCandidatePool now detects kind==="combo-ref" entries - When present, returns eligibleTargets without expanding to ALL providers - Fixes scenario where an "auto" combo with combo-ref delegates to a sub-combo but pulls in every model from every active provider * feat(combo): global comboTimeoutMs + aggregated error diagnostics Adds two features to improve combo resilience and debuggability: 1. Global combo timeout (comboTimeoutMs) - Configurable per-combo via DEFAULT_COMBO_CONFIG (default 0 = disabled) - After each target completes, checks if total elapsed time exceeds limit - When exceeded, stops trying further targets and returns 504 immediately - Backward-compatible: 0 preserves legacy unlimited-iteration behavior 2. Aggregated error diagnostics - comboErrors array accumulates per-model failure details (model, status, error) - On combo timeout or all-models-exhausted, returns a message listing the first (up to 5) model-level errors with their HTTP status codes - Enables operators to see WHICH models failed and WHY without digging through individual server logs * test(combo): cover universal cooldown-aware retry, comboTimeoutMs, and combo-ref guard Adds/updates automated coverage for this PR's production changes (PR Test Policy requires tests alongside src/open-sse/electron/bin changes): - Update the "preserves the first failure status" expectation in combo-routing-engine.test.ts: the aggregated per-model error-diagnostics suffix is new intended output, not a regression. - Add two new tests for the global comboTimeoutMs feature: the combo stops dispatching further targets and returns 504/COMBO_TIMEOUT with aggregated diagnostics once the ceiling trips, and comboTimeoutMs=0 (default) never trips it. - Rewrite the non-quota-share (priority) cooldown-wait scenario in combo-quota-share-cooldown-wait-timing.test.ts: comboCooldownWait is no longer gated on strategy === "quota-share", so a priority combo now waits out a short 429 and re-dispatches too (via shouldWaitForComboCooldown with reason "rate_limit"), instead of propagating immediately as before. Also adds the disabled-flag counterpart for parity with the quota-share suite. - Add coverage for the #COMBO-REF guard in expandAutoComboCandidatePool: a combo whose models array contains a kind:"combo-ref" entry must not be expanded to every model of every active provider. Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com> * test(combo): keep the new comboTimeoutMs tests free of no-explicit-any The two new tests initially copied the neighbouring tests' `any`-typed handleSingleModel params / json() casts. Those neighbours are pre-existing violations frozen in config/quality/eslint-suppressions.json at a count of 261 for this file, so the 7 new occurrences pushed it to 268 and broke `npm run lint` (no-explicit-any is an error in tests/ since diegosouzapw#6218; new violations must be fixed, not re-frozen). Type the new tests properly instead: `unknown`/`string` params, a ComboErrorPayload interface for the parsed body, and drop the unused `relayOptions: null as any` (the sibling combo cooldown suites already omit it). Back to exactly 261 — the suppression file is untouched. Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com> * fix(combo): gate the universal cooldown retry on the REAL lock reason, not a hardcoded "rate_limit" The universal cooldown-aware retry kept the quota-share path on resolveComboCooldownWaitDecision but gave every OTHER strategy a shortcut that hardcoded `reason: "rate_limit"` and fed shouldWaitForComboCooldown the earliestRetryAfter directly. comboCooldownRetry.ts documents TWO deliberate barriers ("SECURITY — quota_exhausted must be excluded"): (1) the reason allow-list, and (2) the maxWaitMs ceiling, explicitly called the SECOND barrier. Hardcoding the reason removed barrier 1 for 17 of the 18 strategies and left only the ceiling — which does NOT cover a quota_exhausted lock whose wait lands under maxWaitMs. In that case the combo waits, redispatches against a model that is locked until the quota resets, and burns the retry budget for nothing. The shortcut's premise ("non-quota-share combos have no per-connection model lockout tracking") is also false: recordModelLockoutFailure in the target loop is not gated on quota-share, so every strategy records model lockouts and the real reason is always available. Fix: one decision path for every strategy, always through resolveComboCooldownWaitDecision, so the reason always crosses the allow-list. The lock lookup is now keyed on each TARGET's own model (via a new third `target` arg on lookupLock) — quota-share combos are single-model/multi-account so this is identical to the previous orderedTargets[0] behavior, but heterogeneous combos (priority, weighted, round-robin, …) carry a different model per target and would otherwise miss every lock but the first. Regression guard (tests/unit/serial/combo-quota-share-cooldown-wait-timing.test.ts): a priority combo where modelLockout.errorCodes=[403] leaves the 403's quota_exhausted lock as the only one in play while a 429 crystallizes the status, and the resulting wait is short enough that the ceiling lets it through — so only the allow-list can stop it. Verified failing-then-passing: with the hardcoded reason the combo makes 6 dispatches (wait+redispatch x2); with the real reason it makes 2 and propagates the 429. Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com> * chore(quality): rebaseline file-size for PR diegosouzapw#7301 own growth (combo +91, combo-routing-engine test +68) --------- Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
HouMinXi
pushed a commit
to HouMinXi/OmniRoute
that referenced
this pull request
Aug 2, 2026
…es (OAuthModal/muse-spark/combo + PricingTab/ComboDefaultsTab) Legitimate own-growth from real features merged this cycle (diegosouzapw#7735 grok OAuth chooser, diegosouzapw#7528 muse-spark WS rewrite, diegosouzapw#7301 combo cooldown-retry, diegosouzapw#7972 pricing, diegosouzapw#7973/diegosouzapw#8008 combo). File-size ratchet is release-captain territory; owner-approved rebaseline. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
HouMinXi
pushed a commit
to HouMinXi/OmniRoute
that referenced
this pull request
Aug 2, 2026
diegosouzapw#8494) * fix(backend): fail closed when capability filters empty the combo pool Tools/vision/structured_output filters no longer re-admit the full pool when every target is incompatible. Opt-in via combo config compatFilterFailOpen. Closes diegosouzapw#8488 * fix(backend): keep tool-emulation providers under fail-closed filters Carve out providers with toolCalling:"emulated" (diegosouzapw#5240) from tools capability_mismatch so chatgpt-web combos still reach the prompt shim. Align round-robin compatFilterFailOpen with settings fallback and drop new any-typed params from the diegosouzapw#8488 combo-routing tests. * chore(lint): prune stale combo-routing-engine any suppressions Test cleanup in diegosouzapw#8488 dropped four no-explicit-any hits; sync the freeze file. * chore(quality): rebaseline combo.ts + freeze new-above-cap comboStructure.ts check:file-size was red for this PR's own growth: combo.ts grew 3640->3693 (+53, the capability-filter fail-closed guard + compatFilterFailOpen escape hatch at both call sites) and combo/comboStructure.ts crossed the 800-line new-file cap at 918 (describeCapabilityFilterExhaustion + providerSupportsEmulatedToolCalling for the diegosouzapw#5240 emulated-tool-calling exemption). Both are irreducible orchestration wiring at the existing combo filter chokepoint (same precedent as diegosouzapw#7301's cooldown-retry generalization). Companion test tests/unit/combo-routing-engine.test.ts frozen at its own grown size (3409->3449). No logic change; 95/95 tests pass. Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com> --------- Co-authored-by: ikelvingo <im.kelvinwong@gmail.com> Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com> Co-authored-by: Prudhvivuda <Prudhvivuda@users.noreply.github.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
HouMinXi
pushed a commit
to HouMinXi/OmniRoute
that referenced
this pull request
Aug 2, 2026
…ls[] is non-empty (diegosouzapw#8598) When an auto-combo has models[] populated by the operator but config.auto.candidatePool is empty (the default for combos created via the dashboard), expandAutoComboCandidatePool silently expands the candidate pool to every model of every active provider connection. This overrides the explicit list in models[] and lets unintended models (e.g. gemini-3.1-flash-lite) win the auto-strategy scoring contest. In omniroute@3.8.48 (npm) only one guard exists before the expansion loop (GUARD A: if (config.auto.candidatePool populated) return eligibleTargets). The upstream release branch release/v3.8.49 added a second guard (combo-ref check, PR diegosouzapw#7301) but it does not cover the common pattern where models[] holds explicit kind:"model" entries. Both gaps share the same root mechanism and the same fix. The new guard short-circuits whenever models[] is a non-empty array, covering both kind:"model" entries (the dashboard default) and kind:"combo-ref" entries (which diegosouzapw#7301 already handles). With this guard in place, the existing combo-ref check becomes redundant; it is left in place for the minimal-scope surgical fix, and can be removed in a follow-up cleanup. Validation (in isolated Docker, 3 providers + 4 controlled combos): - 3 explicit models, empty candidatePool: pool 60 -> 6 - 1 combo-ref + 2 explicit, empty candidatePool: pool 64 -> 10 - 3 explicit models, populated candidatePool (GUARD A path): 6 -> 6 - empty models[] virtual auto: 57 -> 57 (expansion preserved) Closes diegosouzapw#8597 Co-authored-by: Michael de Souza Marcos <michael.smarcos@hotmail.com>
4 tasks done
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…ombo-ref guard (diegosouzapw#7301) * feat(combo): universal cooldown-aware retry & auto-strategy combo-ref guard Two changes: 1. Universal cooldown-aware retry (combo.ts): - Remove strategy==="quota-share" gate from comboCooldownWaitEnabled - Enables all 18 combo strategies (priority, weighted, round-robin, etc.) to wait out a short transient cooldown and retry the full set - Non-quota-share strategies use shouldWaitForComboCooldown directly with earliestRetryAfter and reason="rate_limit" (no per-model lockout) - quota-share retains its existing per-target model lockout logic 2. Auto-strategy combo-ref guard (autoStrategy.ts): - expandAutoComboCandidatePool now detects kind==="combo-ref" entries - When present, returns eligibleTargets without expanding to ALL providers - Fixes scenario where an "auto" combo with combo-ref delegates to a sub-combo but pulls in every model from every active provider * feat(combo): global comboTimeoutMs + aggregated error diagnostics Adds two features to improve combo resilience and debuggability: 1. Global combo timeout (comboTimeoutMs) - Configurable per-combo via DEFAULT_COMBO_CONFIG (default 0 = disabled) - After each target completes, checks if total elapsed time exceeds limit - When exceeded, stops trying further targets and returns 504 immediately - Backward-compatible: 0 preserves legacy unlimited-iteration behavior 2. Aggregated error diagnostics - comboErrors array accumulates per-model failure details (model, status, error) - On combo timeout or all-models-exhausted, returns a message listing the first (up to 5) model-level errors with their HTTP status codes - Enables operators to see WHICH models failed and WHY without digging through individual server logs * test(combo): cover universal cooldown-aware retry, comboTimeoutMs, and combo-ref guard Adds/updates automated coverage for this PR's production changes (PR Test Policy requires tests alongside src/open-sse/electron/bin changes): - Update the "preserves the first failure status" expectation in combo-routing-engine.test.ts: the aggregated per-model error-diagnostics suffix is new intended output, not a regression. - Add two new tests for the global comboTimeoutMs feature: the combo stops dispatching further targets and returns 504/COMBO_TIMEOUT with aggregated diagnostics once the ceiling trips, and comboTimeoutMs=0 (default) never trips it. - Rewrite the non-quota-share (priority) cooldown-wait scenario in combo-quota-share-cooldown-wait-timing.test.ts: comboCooldownWait is no longer gated on strategy === "quota-share", so a priority combo now waits out a short 429 and re-dispatches too (via shouldWaitForComboCooldown with reason "rate_limit"), instead of propagating immediately as before. Also adds the disabled-flag counterpart for parity with the quota-share suite. - Add coverage for the #COMBO-REF guard in expandAutoComboCandidatePool: a combo whose models array contains a kind:"combo-ref" entry must not be expanded to every model of every active provider. Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com> * test(combo): keep the new comboTimeoutMs tests free of no-explicit-any The two new tests initially copied the neighbouring tests' `any`-typed handleSingleModel params / json() casts. Those neighbours are pre-existing violations frozen in config/quality/eslint-suppressions.json at a count of 261 for this file, so the 7 new occurrences pushed it to 268 and broke `npm run lint` (no-explicit-any is an error in tests/ since diegosouzapw#6218; new violations must be fixed, not re-frozen). Type the new tests properly instead: `unknown`/`string` params, a ComboErrorPayload interface for the parsed body, and drop the unused `relayOptions: null as any` (the sibling combo cooldown suites already omit it). Back to exactly 261 — the suppression file is untouched. Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com> * fix(combo): gate the universal cooldown retry on the REAL lock reason, not a hardcoded "rate_limit" The universal cooldown-aware retry kept the quota-share path on resolveComboCooldownWaitDecision but gave every OTHER strategy a shortcut that hardcoded `reason: "rate_limit"` and fed shouldWaitForComboCooldown the earliestRetryAfter directly. comboCooldownRetry.ts documents TWO deliberate barriers ("SECURITY — quota_exhausted must be excluded"): (1) the reason allow-list, and (2) the maxWaitMs ceiling, explicitly called the SECOND barrier. Hardcoding the reason removed barrier 1 for 17 of the 18 strategies and left only the ceiling — which does NOT cover a quota_exhausted lock whose wait lands under maxWaitMs. In that case the combo waits, redispatches against a model that is locked until the quota resets, and burns the retry budget for nothing. The shortcut's premise ("non-quota-share combos have no per-connection model lockout tracking") is also false: recordModelLockoutFailure in the target loop is not gated on quota-share, so every strategy records model lockouts and the real reason is always available. Fix: one decision path for every strategy, always through resolveComboCooldownWaitDecision, so the reason always crosses the allow-list. The lock lookup is now keyed on each TARGET's own model (via a new third `target` arg on lookupLock) — quota-share combos are single-model/multi-account so this is identical to the previous orderedTargets[0] behavior, but heterogeneous combos (priority, weighted, round-robin, …) carry a different model per target and would otherwise miss every lock but the first. Regression guard (tests/unit/serial/combo-quota-share-cooldown-wait-timing.test.ts): a priority combo where modelLockout.errorCodes=[403] leaves the 403's quota_exhausted lock as the only one in play while a 429 crystallizes the status, and the resulting wait is short enough that the ceiling lets it through — so only the allow-list can stop it. Verified failing-then-passing: with the hardcoded reason the combo makes 6 dispatches (wait+redispatch x2); with the real reason it makes 2 and propagates the 429. Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com> * chore(quality): rebaseline file-size for PR diegosouzapw#7301 own growth (combo +91, combo-routing-engine test +68) --------- Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com>
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…es (OAuthModal/muse-spark/combo + PricingTab/ComboDefaultsTab) Legitimate own-growth from real features merged this cycle (diegosouzapw#7735 grok OAuth chooser, diegosouzapw#7528 muse-spark WS rewrite, diegosouzapw#7301 combo cooldown-retry, diegosouzapw#7972 pricing, diegosouzapw#7973/diegosouzapw#8008 combo). File-size ratchet is release-captain territory; owner-approved rebaseline. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
diegosouzapw#8494) * fix(backend): fail closed when capability filters empty the combo pool Tools/vision/structured_output filters no longer re-admit the full pool when every target is incompatible. Opt-in via combo config compatFilterFailOpen. Closes diegosouzapw#8488 * fix(backend): keep tool-emulation providers under fail-closed filters Carve out providers with toolCalling:"emulated" (diegosouzapw#5240) from tools capability_mismatch so chatgpt-web combos still reach the prompt shim. Align round-robin compatFilterFailOpen with settings fallback and drop new any-typed params from the diegosouzapw#8488 combo-routing tests. * chore(lint): prune stale combo-routing-engine any suppressions Test cleanup in diegosouzapw#8488 dropped four no-explicit-any hits; sync the freeze file. * chore(quality): rebaseline combo.ts + freeze new-above-cap comboStructure.ts check:file-size was red for this PR's own growth: combo.ts grew 3640->3693 (+53, the capability-filter fail-closed guard + compatFilterFailOpen escape hatch at both call sites) and combo/comboStructure.ts crossed the 800-line new-file cap at 918 (describeCapabilityFilterExhaustion + providerSupportsEmulatedToolCalling for the diegosouzapw#5240 emulated-tool-calling exemption). Both are irreducible orchestration wiring at the existing combo filter chokepoint (same precedent as diegosouzapw#7301's cooldown-retry generalization). Companion test tests/unit/combo-routing-engine.test.ts frozen at its own grown size (3409->3449). No logic change; 95/95 tests pass. Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com> --------- Co-authored-by: ikelvingo <im.kelvinwong@gmail.com> Co-authored-by: diegosouzapw <diegosouza.pw@gmail.com> Co-authored-by: Prudhvivuda <Prudhvivuda@users.noreply.github.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…ls[] is non-empty (diegosouzapw#8598) When an auto-combo has models[] populated by the operator but config.auto.candidatePool is empty (the default for combos created via the dashboard), expandAutoComboCandidatePool silently expands the candidate pool to every model of every active provider connection. This overrides the explicit list in models[] and lets unintended models (e.g. gemini-3.1-flash-lite) win the auto-strategy scoring contest. In omniroute@3.8.48 (npm) only one guard exists before the expansion loop (GUARD A: if (config.auto.candidatePool populated) return eligibleTargets). The upstream release branch release/v3.8.49 added a second guard (combo-ref check, PR diegosouzapw#7301) but it does not cover the common pattern where models[] holds explicit kind:"model" entries. Both gaps share the same root mechanism and the same fix. The new guard short-circuits whenever models[] is a non-empty array, covering both kind:"model" entries (the dashboard default) and kind:"combo-ref" entries (which diegosouzapw#7301 already handles). With this guard in place, the existing combo-ref check becomes redundant; it is left in place for the minimal-scope surgical fix, and can be removed in a follow-up cleanup. Validation (in isolated Docker, 3 providers + 4 controlled combos): - 3 explicit models, empty candidatePool: pool 60 -> 6 - 1 combo-ref + 2 explicit, empty candidatePool: pool 64 -> 10 - 3 explicit models, populated candidatePool (GUARD A path): 6 -> 6 - empty models[] virtual auto: 57 -> 57 (expansion preserved) Closes diegosouzapw#8597 Co-authored-by: Michael de Souza Marcos <michael.smarcos@hotmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Descrição
Duas correções no sistema de combos do Omniroute que afetam diretamente a experiência com combos
priorityeauto.1. Cooldown-aware retry universal (
combo.ts)Problema: O cooldown-aware retry (esperar um rate-limit curto e retentar o set inteiro) estava exclusivamente ativado para a estratégia
quota-share— as outras 17 estratégias (priority,weighted,round-robin,auto,lkgp,cost-optimized, etc.) não se beneficiavam.Solução: Removeu o gate
strategy === "quota-share"da variávelcomboCooldownWaitEnabled. Estratégias não-quota-share usamshouldWaitForComboCooldowndiretamente comearliestRetryAfterereason: "rate_limit"(sem lookupLock per-model). A lógica quota-share existente permanece inalterada.2. Guard combo-ref em auto-strategy (
autoStrategy.ts)Problema: Um combo
autocomcombo-ref → sub-combo(ex.: Coder → Deepseek) expandia para todos os modelos de todos os providers ativos (~1600 modelos) em vez de respeitar o escopo do sub-combo.Solução:
expandAutoComboCandidatePoolagora verifica se o combo contém entrieskind: "combo-ref". Se sim, retornaeligibleTargetssem expandir — os targets já representam o pool intencionado pelo operador.Impacto
shouldWaitForComboCooldownjá filtra razões não-retryáveis (quota_exhausted,auth_error,not_found).maxWaitMs=5000,maxAttempts=2,budgetMs=8000.Arquivos modificados
open-sse/services/combo.ts(+53/-22 linhas)open-sse/services/combo/autoStrategy.ts(+9 linhas)Tipo de mudança
Testes
tsc --noEmit)dist/) e serviço restartado em produção