Repository navigation
Make zero-latency combo optimizations opt-in - #2917
diegosouzapw merged 2 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces an opt-in mechanism for zero-latency combo optimizations (including hedging, predictive TTFT skips, and proactive fallback compression) to ensure fallback targets receive the original request body exactly by default. The changes include adding a zeroLatencyOptimizationsEnabled configuration flag, updating the backend logic in handleComboChat to respect this flag, introducing a UI toggle in the settings dashboard, updating validation schemas, and adding comprehensive unit tests to verify the behavior. There are no review comments, and I have no additional feedback to provide.
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR makes “zero-latency” combo-routing behaviors explicitly opt-in, wiring the new flag through config validation, defaults, UI, docs, and adding unit tests around fallback request-body preservation vs proactive compression.
Changes:
- Add
zeroLatencyOptimizationsEnabledto combo defaults/config and gate hedging, predictive TTFT skips, and proactive fallback compression behind it. - Extend validation schema + defaults and expose the toggle in the dashboard settings UI.
- Add unit tests covering fallback-body preservation when optimizations are disabled and compression when enabled.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/combo-routing-engine.test.ts | Adds unit tests validating fallback-body behavior with/without zero-latency opt-in. |
| tests/unit/combo-config.test.ts | Expands assertions for new default combo config fields and schema acceptance. |
| src/shared/validation/schemas.ts | Extends runtime combo config schema with zero-latency-related fields. |
| src/app/api/settings/combo-defaults/route.ts | Adds zeroLatencyOptimizationsEnabled to API-provided defaults. |
| src/app/(dashboard)/dashboard/settings/components/ComboDefaultsTab.tsx | Adds a settings toggle for zero-latency optimizations. |
| open-sse/services/comboConfig.ts | Adds opt-in default for zeroLatencyOptimizationsEnabled. |
| open-sse/services/combo.ts | Gates hedging/predictive TTFT/compression behind zeroLatencyOptimizationsEnabled. |
| docs/guides/USER_GUIDE.md | Documents opt-in nature and tradeoffs of zero-latency optimizations. |
| failoverBeforeRetry: z.boolean().optional(), | ||
| maxSetRetries: z.coerce.number().int().min(0).max(10).optional(), | ||
| setRetryDelayMs: z.coerce.number().int().min(0).max(60000).optional(), | ||
| zeroLatencyOptimizationsEnabled: z.boolean().optional(), | ||
| hedging: z.boolean().optional(), | ||
| hedgeDelayMs: z.coerce.number().int().min(0).max(60000).optional(), | ||
| fallbackCompressionMode: compressionModeSchema.optional(), | ||
| fallbackCompressionThreshold: z.coerce.number().int().min(0).max(2_000_000).optional(), | ||
| predictiveTtftMs: z.coerce.number().int().min(0).max(300000).optional(), |
| combo: { | ||
| name: "zero-latency-disabled-preserves-body", | ||
| strategy: "priority", | ||
| models: ["provider-a/model-a", "provider-b/model-b"], | ||
| config: { maxRetries: 0, retryDelayMs: 1, fallbackCompressionThreshold: 1 }, | ||
| }, |
| assert.equal(result.status, 200); | ||
| assert.notEqual(fallbackBody.messages[1].content, longToolOutput); | ||
| assert.match(fallbackBody.messages[1].content, /\.\.\.\[truncated\]$/); |
| {translateOrFallback( | ||
| t, | ||
| "zeroLatencyOptimizationsDesc", | ||
| "Opt in to hedging, predictive TTFT skips, and proactive fallback compression. Leave off to preserve request bodies exactly." | ||
| )} |
| Zero-latency combo optimizations are opt-in. Leave **Zero-latency optimizations** disabled when | ||
| fallback targets must receive the original request body exactly; enabling it allows configured | ||
| hedging, predictive TTFT skips, and proactive fallback compression to trade request fidelity for | ||
| lower tail latency. |
| task.finally(() => runningTasks.delete(task)); | ||
|
|
||
| if (config.hedging && i + 1 < orderedTargets.length) { | ||
| if (zeroLatencyOptimizationsEnabled && config.hedging && i + 1 < orderedTargets.length) { |
Code Review SummaryStatus: No New Issues Found | Recommendation: Merge The PR implements an opt-in mechanism for zero-latency combo optimizations (hedging, predictive TTFT skips, and proactive fallback compression). The implementation is well-structured with:
All existing inline comments have been accounted for. The code follows existing patterns and adds appropriate validation without introducing security, performance, or correctness issues. Files Reviewed (7 files)
Reviewed by laguna-m.1-20260312:free · 1,068,970 tokens |
diegosouzapw
left a comment
diegosouzapw
left a comment
There was a problem hiding this comment.
Approved after local verification, quality checks and merging into release/v3.8.7.
…ombo-toggle Make zero-latency combo optimizations opt-in
Summary
zeroLatencyOptimizationsEnabledcombo runtime switch, defaulting tofalseRationale
@herjarsa The zero-latency work is useful, but latency optimizations must not mutate user prompts, tool outputs, or fallback request bodies by default. Proactive fallback compression can truncate or otherwise change request content, so it needs an explicit operator opt-in instead of being enabled implicitly.
Validation
node --import tsx/esm --test tests/unit/combo-config.test.tsnode --import tsx/esm --test tests/unit/combo-routing-engine.test.tsnpm run typecheck:corenpm run lint(passes with existing warnings only)