Skip to content

fix(combo): guarantee combo loops terminate instead of hanging silently - #10463

Merged
diegosouzapw merged 2 commits into
diegosouzapw:release/v3.8.50from
herjarsa:fix/combo-loop-silent-stop
Aug 21, 2026
Merged

diegosouzapw merged 2 commits into
diegosouzapw:release/v3.8.50from
herjarsa:fix/combo-loop-silent-stop

Conversation

@herjarsa

Copy link
Copy Markdown
Contributor

Summary

Guarantees that OmniRoute's combo/target loops always terminate with an actionable error instead of dying silently (no response, no log) when an upstream hangs.

Fixes 10 silent-stop gaps (G1–G10) found by auditing every loop that iterates models/combos to resolve a task:

Critical — combo target loop (open-sse/services/combo.ts)

  • G1: globalPromise in handleComboChat had no timer — a hung target (per-model timeout disabled) froze the request forever. Added loopSafetyPromise (uses comboTimeoutMs, else a 10-minute hard ceiling COMBO_LOOP_SAFETY_TIMEOUT_MS) to all 3 Promise.race sites; on fire: marks comboExpired, aborts in-flight upstreams, drains in-flight tasks (COMBO_SAFETY_DRAIN_MS), returns 504 with the same aggregated ried: a (500) diagnostics as the regular timeout path.
  • G2: task wrapper .catch() now resolves globalPromise with 502 so an unexpected task throw can never leave the post-loop race hanging.
  • G3: argetTimeoutRunner logs a warning when comboTargetTimeoutMs <= 0 (per-model timeout disabled).
  • G4: handleRoundRobinCombo had no global timeout and no safety net — added
    rSafetyPromise timer (same ceiling),
    rExpired loop guard, Promise.race around each model dispatch, try/catch → 500, and a post-loop 504.
  • G9: round-robin retry delay now honors client disconnect (499).

Localized silent exits

  • G5 (�utoCombo/chaosEngine.ts): dispatchOnePanelModel hardcoded ok: true — a panel where every model returned 503 was treated as success and the error streamed as a valid answer. Now honors
    es.ok; all-failed branch logs All chaos panel models failed with per-model errors.
  • G6 (�utoCombo/pipelineRouter.ts): logs Reflection retries exhausted when the reflection loop burns its budget and still fails.
  • G7 (src/lib/evals/evalRunner.ts): rejects catastrophic regex patterns (ReDoS) via safe-regex before est().
  • G8 (�utoRefreshDaemon.ts): �alidateCredential swallowed network errors in a bare catch {} (fail-open silent) — now logs provider + error.
  • G10 (�atchProcessor.ts): each batch-item dispatch bounded by a 120s wall-clock timeout (withItemDispatchTimeout) so a stuck item fails fast instead of freezing the batch.

Verification

  • New regression suite ests/unit/combo-silent-stop-gaps.test.ts — 10/10 pass (hung upstream → 504 in ~130ms; unexpected throw → 502; chaos all-fail → logged; ReDoS regex rejected; batch timeout).
  • ests/unit/combo-routing-engine.test.ts + combo-target-timeout-runner.test.ts + pipeline-router.test.ts: 105/105 pass (102 existing + new).
  • chaos-executor, �atch-processor, service-batch-processor, �valrunner-builtinsuites-split: 37/37 pass.

pm run typecheck:core clean; ESLint clean on touched files (pre-existing violations in �atchProcessor.ts untouched).

⚠️ base-red inherited: #9985 —
elease/v3.8.50 is currently red upstream; inherited failures are not introduced by this branch.

@herjarsa
herjarsa requested a review from diegosouzapw as a code owner August 15, 2026 10:49
@diegosouzapw

Copy link
Copy Markdown
Owner

Deferido para revalidação do head atual. A rodada anterior registrou um hold de ownership, mas a inspeção atual não confirmou esse worktree; por favor, reconfirme o head e os checks antes da decisão final.

@diegosouzapw diegosouzapw added deferred-v3.8.50 Adiada para o ciclo v3.8.50 (validacao VPS, refactor, ou escopo grande) merge-train-deferred PR ejetada do merge-train — triagem separada and removed merge-train-deferred PR ejetada do merge-train — triagem separada deferred-v3.8.50 Adiada para o ciclo v3.8.50 (validacao VPS, refactor, ou escopo grande) labels Aug 17, 2026
…eadfix

# Conflicts:
#	open-sse/services/autoCombo/chaosEngine.ts
#	open-sse/services/combo.ts
@diegosouzapw
diegosouzapw merged commit d098114 into diegosouzapw:release/v3.8.50 Aug 21, 2026
3 of 6 checks passed
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…instead of hanging silently (diegosouzapw#10463)

Guarantees combo/target loops always terminate with an actionable error instead of hanging when an upstream hangs — fixes 10 silent-stop gaps (G1–G10): combo-loop safety timer + abort on hang → 504, unexpected task throw → 502, round-robin safety net, chaos all-panel-failure visibility, ReDoS-safe eval regex, autoRefreshDaemon swallowed-error logging, batch-item wall-clock timeout.

Validated in an isolated worktree boarded onto origin/release/v3.8.50 (2 real conflicts in combo.ts and chaosEngine.ts, both additive features from concurrently-merged PRs landing at the same insertion point — resolved by combining both, verified no variable-shadowing/scoping issues):
- 10/10 new regression tests pass (combo-silent-stop-gaps.test.ts): hung upstream → 504, unexpected throw → 502, chaos all-fail logged, ReDoS regex rejected, batch timeout.
- 163/163 broader focused tests pass (combo-routing-engine, combo-target-timeout-runner, pipeline-router, chaos-executor, batch-processor ×2, service-batch-processor, evalrunner-builtinsuites-split).
- check-file-size, check-changelog-integrity: OK.
- typecheck:core: clean.
- check-complexity / check-cognitive-complexity: OK, both under baseline.

Co-authored-by: herjarsa <herjarsa@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants