Skip to content

feat(combo): opaque per-invocation decision trace for priority fallbacks (#10681) - #10730

Merged
diegosouzapw merged 3 commits into
cheaper-inference:release/v3.8.50from
stanleytejakusuma:fix/10681-combo-decision-trace
Aug 20, 2026
Merged

diegosouzapw merged 3 commits into
cheaper-inference:release/v3.8.50from
stanleytejakusuma:fix/10681-combo-decision-trace

Conversation

@stanleytejakusuma

Copy link
Copy Markdown
Contributor

Summary

Implements #10681: an opaque, per-invocation decision trace for combo fallbacks.

Priority combos can be impossible to audit after a mixed fallback. Dispatched attempts are persisted in call_logs, but targets excluded before dispatch leave no correlated decision record — you cannot tell whether a configured target was evaluated and gated, reordered, or never reached after a later success.

What this adds

  1. open-sse/services/combo/decisionTrace.ts — createInvocationId() (opaque combo-<uuid>), startComboTrace, recordComboDecision, finalizeComboTrace (marks never-iterated targets not_reached), getComboTrace. Bounded in-memory retention: 30min TTL, 2000 invocations (LRU).
  2. combo.ts — hooks at every pre-dispatch skip site (circuit open, provider cooldown, request exhaustion, model lockout, quota cutoff, availability, credential gate, concurrency cap, admission lane, predictive TTFT) recording skipped_before_dispatch with an allowlisted reason; a single dispatched record per target (per-target retries dedupe); and finalize on success / global timeout / all-failed / all-skipped terminals.
  3. GET /api/usage/combo-trace/[id] — management-auth read of the trace for a given invocation.

Safety contract

The trace contains routing metadata only: invocation id, combo name, strategy, per-target provider/model, decision, allowlisted skip reason, timestamps, terminal status. Never prompts, request/response bodies, headers, credentials, account ids, or raw upstream error strings. Routing semantics are unchanged — this is a read-only projection of existing dispatch decisions.

Tests

  • Unit: invocation-id uniqueness, skip-reason allowlist enforcement, not_reached finalize.
  • Integration (public handleComboChat + injected handleSingleModel): mixed 429 fallback yields [dispatched, dispatched, not_reached]; availability skip yields skipped_before_dispatch/availability.
  • 5/5 pass. Combo suite: 1183/1185 (the 2 failures — combo-runtime-unit-concurrency tmpdir-symlink assertion and the zero-latency compression-opt-in 500 — are identical on clean release/v3.8.50, pre-existing). typecheck:core errors are the pre-existing OmniGlyph baseline.

Notes

  • Retention is in-memory by design (matches the issue's "normal bounded retention"); a follow-up could persist to call-log metadata if desired.
  • Nested combo-ref dispatches get their own invocation id (per-dispatch traces); the top-level handleComboChat accepts an optional invocationId for callers that want to seed it.

…cks (cheaper-inference#10681)

Priority combos can be impossible to audit after a mixed fallback: dispatched
attempts are persisted in call_logs, but candidates excluded before dispatch
(circuit open, provider cooldown, model lockout, quota cutoff, availability,
credential gate, concurrency cap, admission lane, predictive TTFT) leave no
correlated decision record.

Adds:
- decisionTrace.ts: opaque invocation id (combo-<uuid>) + ordered per-target
  decisions (dispatched / skipped_before_dispatch / not_reached) with an
  allowlisted skip-reason set. Bounded in-memory retention (30min TTL, 2000
  invocations).
- combo.ts: start/record/finalize hooks at every pre-dispatch skip site, the
  dispatch point (once per target, retries deduped), and terminal paths
  (success/timeout/all-failed/all-skipped).
- GET /api/usage/combo-trace/[id]: management-auth read of the trace.

SAFETY: the trace holds routing metadata only (combo, strategy, provider/model,
decision, allowlisted reason, terminal status). Never prompts, request/response
bodies, headers, credentials, account ids, or raw upstream error strings.
Routing semantics are unchanged; observability is a read-only projection.

Tests: unit (id uniqueness, allowlist enforcement, not_reached finalize) +
handleComboChat integration (mixed 429 fallback; availability skip). 5/5 pass.
Combo suite: 1183/1185 (2 pre-existing base failures, identical on clean
release/v3.8.50). typecheck:core failures are the pre-existing OmniGlyph
baseline.

Closes cheaper-inference#10681
hartmark and others added 2 commits August 19, 2026 20:56
recordComboDecision for the predictive_ttft skip ran after the
stopProtectedPriorityTarget return, making it dead code — unlike the
other 9 skip sites in combo.ts, which all record before returning.
Reorders it to match, adds an integration test driving the skip
through handleComboChat, and reformats with prettier.

Co-authored-by: stanleytejakusuma <gstanleytejakusuma@gmail.com>
…e log

cheaper-inference#10681 egress: the trace store was unreachable in practice for gateway-internal
invocations because the combo-<uuid> id surfaced nowhere on the response.

- handleComboChat is now a thin wrapper: it pins the invocation id (passing
  one down if the caller supplied it), stamps X-OmniRoute-Combo-Trace on the
  final Response at every exit path, and emits one metadata-only log line
  (id, terminal status, decision count) so finalized traces survive restarts
  via the existing log pipeline.
- decisionTrace: eviction at the MAX_TRACES cap now prefers FINALIZED traces,
  pinning in-flight invocations against bursts.
- tests: header on success and all-failed paths, log-line shape (status yes,
  request content never), and eviction preference (finalized victim, in-flight
  survivors). Combo suite 162/162.
@stanleytejakusuma

Copy link
Copy Markdown
Contributor Author

Follow-up c469c26 (after review): stamps the trace id on the receipt so the store is actually reachable post-incident, per the #10681 auditability goal.

  • X-OmniRoute-Combo-Trace: combo-<uuid> header on the final Response at every exit path (wrapper around handleComboChat; callers may still supply their own invocationId).
  • One metadata-only log line at finalize (id, terminal status, decision count) so finalized traces survive restarts via the existing log pipeline.
  • Eviction at the MAX_TRACES cap now prefers FINALIZED traces, pinning in-flight invocations against bursts.
  • Tests: header on success + all-failed paths, log-line shape, eviction preference. Combo suite 162/162.

@diegosouzapw
diegosouzapw merged commit bc9a685 into cheaper-inference:release/v3.8.50 Aug 20, 2026
5 checks passed
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…cks (cheaper-inference#10681) (cheaper-inference#10730)

Merged via merge-train (release/v3.8.50, batch1 2026-08-20) — static gates (typecheck/file-size/complexity/cognitive/changelog) green on the combined tree; test:unit reds observed in the boarded run were verified pre-existing on the pure release tip (unrelated flake), not caused by this PR. Thanks for the contribution!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants