Repository navigation
feat(combo): opaque per-invocation decision trace for priority fallbacks (#10681) - #10730
Merged
diegosouzapw merged 3 commits intoAug 20, 2026
Conversation
…cks (cheaper-inference#10681) Priority combos can be impossible to audit after a mixed fallback: dispatched attempts are persisted in call_logs, but candidates excluded before dispatch (circuit open, provider cooldown, model lockout, quota cutoff, availability, credential gate, concurrency cap, admission lane, predictive TTFT) leave no correlated decision record. Adds: - decisionTrace.ts: opaque invocation id (combo-<uuid>) + ordered per-target decisions (dispatched / skipped_before_dispatch / not_reached) with an allowlisted skip-reason set. Bounded in-memory retention (30min TTL, 2000 invocations). - combo.ts: start/record/finalize hooks at every pre-dispatch skip site, the dispatch point (once per target, retries deduped), and terminal paths (success/timeout/all-failed/all-skipped). - GET /api/usage/combo-trace/[id]: management-auth read of the trace. SAFETY: the trace holds routing metadata only (combo, strategy, provider/model, decision, allowlisted reason, terminal status). Never prompts, request/response bodies, headers, credentials, account ids, or raw upstream error strings. Routing semantics are unchanged; observability is a read-only projection. Tests: unit (id uniqueness, allowlist enforcement, not_reached finalize) + handleComboChat integration (mixed 429 fallback; availability skip). 5/5 pass. Combo suite: 1183/1185 (2 pre-existing base failures, identical on clean release/v3.8.50). typecheck:core failures are the pre-existing OmniGlyph baseline. Closes cheaper-inference#10681
recordComboDecision for the predictive_ttft skip ran after the stopProtectedPriorityTarget return, making it dead code — unlike the other 9 skip sites in combo.ts, which all record before returning. Reorders it to match, adds an integration test driving the skip through handleComboChat, and reformats with prettier. Co-authored-by: stanleytejakusuma <gstanleytejakusuma@gmail.com>
…e log cheaper-inference#10681 egress: the trace store was unreachable in practice for gateway-internal invocations because the combo-<uuid> id surfaced nowhere on the response. - handleComboChat is now a thin wrapper: it pins the invocation id (passing one down if the caller supplied it), stamps X-OmniRoute-Combo-Trace on the final Response at every exit path, and emits one metadata-only log line (id, terminal status, decision count) so finalized traces survive restarts via the existing log pipeline. - decisionTrace: eviction at the MAX_TRACES cap now prefers FINALIZED traces, pinning in-flight invocations against bursts. - tests: header on success and all-failed paths, log-line shape (status yes, request content never), and eviction preference (finalized victim, in-flight survivors). Combo suite 162/162.
Contributor
Author
|
Follow-up c469c26 (after review): stamps the trace id on the receipt so the store is actually reachable post-incident, per the #10681 auditability goal.
|
diegosouzapw
merged commit Aug 20, 2026
bc9a685
into
cheaper-inference:release/v3.8.50
5 checks passed
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…cks (cheaper-inference#10681) (cheaper-inference#10730) Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements #10681: an opaque, per-invocation decision trace for combo fallbacks.
Priority combos can be impossible to audit after a mixed fallback. Dispatched attempts are persisted in
call_logs, but targets excluded before dispatch leave no correlated decision record — you cannot tell whether a configured target was evaluated and gated, reordered, or never reached after a later success.What this adds
open-sse/services/combo/decisionTrace.ts—createInvocationId()(opaquecombo-<uuid>),startComboTrace,recordComboDecision,finalizeComboTrace(marks never-iterated targetsnot_reached),getComboTrace. Bounded in-memory retention: 30min TTL, 2000 invocations (LRU).combo.ts— hooks at every pre-dispatch skip site (circuit open, provider cooldown, request exhaustion, model lockout, quota cutoff, availability, credential gate, concurrency cap, admission lane, predictive TTFT) recordingskipped_before_dispatchwith an allowlisted reason; a singledispatchedrecord per target (per-target retries dedupe); and finalize on success / global timeout / all-failed / all-skipped terminals.GET /api/usage/combo-trace/[id]— management-auth read of the trace for a given invocation.Safety contract
The trace contains routing metadata only: invocation id, combo name, strategy, per-target provider/model, decision, allowlisted skip reason, timestamps, terminal status. Never prompts, request/response bodies, headers, credentials, account ids, or raw upstream error strings. Routing semantics are unchanged — this is a read-only projection of existing dispatch decisions.
Tests
not_reachedfinalize.handleComboChat+ injectedhandleSingleModel): mixed 429 fallback yields[dispatched, dispatched, not_reached]; availability skip yieldsskipped_before_dispatch/availability.combo-runtime-unit-concurrencytmpdir-symlink assertion and the zero-latency compression-opt-in 500 — are identical on cleanrelease/v3.8.50, pre-existing).typecheck:coreerrors are the pre-existing OmniGlyph baseline.Notes
handleComboChataccepts an optionalinvocationIdfor callers that want to seed it.