Skip to content

fix(ai): resolve omniroute reliability issues (DNS collision, timeout race) - #4323

Merged
Tanguille merged 1 commit into
mainfrom
fix/omniroute-timeout-fallback-race
Aug 3, 2026
Merged

fix(ai): resolve omniroute reliability issues (DNS collision, timeout race)#4323
Tanguille merged 1 commit into
mainfrom
fix/omniroute-timeout-fallback-race

Conversation

@Tanguille

@Tanguille Tanguille commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Summary

  • DNS collision (root cause of opencode failures): default ndots:5 search-expands the external domain opencode.ai to opencode.ai.svc.cluster.local before trying it as absolute — which exactly matches our own in-namespace opencode Service. Every lookup of the real opencode.ai provider was silently hijacked to our internal app instead, explaining the persistent 502s/timeouts previously blamed on the upstream provider. Fixed with dnsConfig.options: ndots=1 (same pattern already used by seerr, prowlarr-exporter, etc.).
  • Timeout race: omniroute's litellm timeout (300s) + retry_policy (2 retries) could exceed the pr-reviewer-action's 600s curl timeout, so the action's own retry loop always raced litellm's qwen-3.6-fast fallback and won. 90s timeout keeps worst case at 270s, inside the action's window.

Evidence

Run PR Total time
meilisearch bump 2m
hermes-agent bump 9m48s
searxng bump 24m43s
oxfmt bump 30m7s
litellm-operator bump 31m3s
memini bump 32m40s
renovate-presets bump 60m48s (4 self-retries, ~10min apart)

Slow runs correlate with opencode being picked by auto and hanging on the DNS-hijacked connection.

Also done (live, not in this diff)

  • Removed 11 permanently-dead provider connections (invalid keys, missing CLI binaries, internal-URL SSRF block, unsupported endpoint).
  • Reconnected 4 that looked transient; opencode should now resolve correctly once this PR's DNS fix ships.

Test plan

  • kustomize build kubernetes/apps/ai/litellm/instance clean
  • kustomize build kubernetes/apps/ai/omniroute/app clean
  • Next Renovate PR review completes well under 10 minutes
  • opencode provider resolves the real opencode.ai after rollout (verify via omniroute logs, no more 502/timeout on opencode.ai:443)

300s + the router's 2 retries could exceed the pr-reviewer-action's 600s
curl timeout, so the action gave up and self-retried instead of letting
litellm's fallback to qwen-3.6-fast trigger (observed: 60min run, no
fallback). 90s keeps worst case at 270s, inside the action's window.
@coderabbitai

coderabbitai Bot commented Aug 3, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 39303870-85b7-4a39-ae13-d678c2c5f048

📥 Commits

Reviewing files that changed from the base of the PR and between 2bad8e3 and c9fcd98.

📒 Files selected for processing (1)
  • kubernetes/apps/ai/litellm/instance/models.yaml
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • Tanguille/LLMKube (auto-detected)
  • Tanguille/toolhive (auto-detected)
  • Tanguille/pr-reviewer-action (auto-detected)
  • Tanguille/renovate-presets (auto-detected)
  • Tanguille/litellm-operator (auto-detected)
  • Tanguille/2x-R9700-RDNA4-GFX1201-sglang-inference (auto-detected)

📝 Walkthrough

Walkthrough

The omniroute LiteLLM request timeout changed from 300 to 90 seconds. Comments now document retry timing and fallback behavior.

Changes

Omniroute timeout configuration

Layer / File(s) Summary
Update timeout and timing documentation
kubernetes/apps/ai/litellm/instance/models.yaml
The omniroute timeout is set to 90 seconds. Comments describe retry and fallback timing.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes lowering the LiteLLM request timeout to enable the Qwen fallback.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/omniroute-timeout-fallback-race

Comment @coderabbitai help to get the list of available commands.

@tanguille-cluster

Copy link
Copy Markdown
@@ spec.params.additional.timeout @@
# litellm.home-operations.com/v1alpha1/LiteLLMModel/ai/omniroute
! ± value change
- 300
+ 90

@deepsource-io

deepsource-io Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

DeepSource Code Review

We reviewed changes in 2bad8e3...c9fcd98 on this pull request. Below is the summary for the review, and you can see the individual issues we found as inline review comments.

See full review on DeepSource ↗

PR Report Card

Overall Grade   Security  

Reliability  

Complexity  

Hygiene  

Code Review Summary

Analyzer Status Updated (UTC) Details
JavaScript Aug 3, 2026 8:23p.m. Review ↗
Shell Aug 3, 2026 8:23p.m. Review ↗

Important

AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.

@Tanguille
Tanguille merged commit 18aba23 into main Aug 3, 2026
14 of 15 checks passed
@Tanguille
Tanguille deleted the fix/omniroute-timeout-fallback-race branch August 3, 2026 20:35
@Tanguille Tanguille changed the title fix(litellm): lower omniroute request timeout to unblock qwen fallback fix(ai): resolve omniroute reliability issues (DNS collision, timeout race) Aug 3, 2026
Tanguille added a commit that referenced this pull request Aug 3, 2026
…#4324)

Default ndots:5 search-expands "opencode.ai" (1 dot) to
"opencode.ai.svc.cluster.local" before trying it as absolute, which
exactly matches our own in-namespace opencode Service — every lookup
of the real opencode.ai provider was silently resolving to our
internal opencode app instead. Explains the persistent opencode
502s/timeouts blamed on the upstream provider. ndots:1 (same pattern
as seerr, prowlarr-exporter, etc.) tries the absolute name first.

This commit was cut into PR #4323 too late to be included in the
merge (pushed 5min after merge); reopening standalone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant