fix(ai): restore mamba SSM dtype, raise context to 148K, add fallback tier - #4520
Conversation
The omniroute fallback chain had a single entry. When qwen36-27b was
scaled to 0 for this cutover, that entry became a dead model group and
omniroute's intermittent 503s ("all upstream accounts are inactive",
free-tier quota exhaustion) turned into hard failures with nowhere left
to go.
qwen35-2b runs on control-2's dri device rather than control-1's GPU, so
it survives the failure mode that took the chain out.
The operator derives a LiteLLMModel from the InferenceService name and exposes no field to override it, so the generated group sent model=qwen38-27b-vllm to a server serving only qwen-3.8. Every call to that group returned "The model `qwen38-27b-vllm` does not exist" (verified against live litellm) while kubectl still listed it as a healthy LiteLLMModel -- the same shape of trap as the fallback that pointed at a scaled-to-zero deployment. --served-model-name takes a list and reports the first entry as `model`, so qwen-3.8 stays primary and the alias only widens what is accepted.
A simplify pass dropped this flag on the theory that it inherits from --mamba-cache-dtype. Production disagreed: KV fell 156,493 -> 149,333 tokens (1.40x -> 1.33x at 112K) and the CPU offload tier halved from 327 to 167 blocks. Conv-state and SSM-state are separate caches and each needs its dtype set.
|
Warning Review limit reached
Next review available in: 35 minutes Limit details: You’ve used all 1 included review currently available under your plan. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan includes up to 1 review per rolling hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe PR adds ChangesAI serving configuration
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to The PR restores the separate cache dtype, adds a fallback on another node, and supports both served-model names. No actionable merge-blocking risk remains beyond normal checks and review. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
@@ spec.routerSettings.fallbacks.0.omniroute @@
# litellm.home-operations.com/v1alpha1/LiteLLMProxy/ai/litellm
! + one list entry added:
+ - qwen35-2b
@@ spec.info.maxInputTokens @@
# litellm.home-operations.com/v1alpha1/LiteLLMModel/ai/qwen-3.8
! ± value change
- 103808
+ 139808
@@ spec.info.maxInputTokens @@
# litellm.home-operations.com/v1alpha1/LiteLLMModel/ai/qwen-3.8-fast
! ± value change
- 103808
+ 139808
@@ spec.extraArgs @@
# inference.llmkube.dev/v1alpha1/InferenceService/ai/qwen38-27b-vllm
! + three list entries added:
+ - qwen38-27b-vllm
+ - bfloat16
+ - "--mamba-ssm-cache-dtype"
@@ spec.vllmConfig.maxModelLen @@
# inference.llmkube.dev/v1alpha1/InferenceService/ai/qwen38-27b-vllm
! ± value change
- 112000
+ 148000
|
AI Automated Review (incremental)Analysis engine: omniroute@http://litellm.ai.svc.cluster.local/v1 (openai) Recommendation: Approve. The incremental delta is comment-only tightening plus the already-verified Change-by-change:
Standards Compliance: No AGENTS.md convention violated; changes are small, focused, and follow existing patterns. Unknowns / Needs Verification: The live-engine KV numbers (158,073 tokens, 5.71 GB free vs Jellyfin's ~4.6 GB reservation) and the Hermes end-to-end |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
maxModelLen is the only per-request cap on KV blocks -- max-num-seqs reserves nothing and max-num-batched-tokens is a compute budget -- so at 112000 a single stream could only ever hold 72% of the pool. Measured live at 148000: pool 158,073 tokens (up 1,580 from better block rounding), 1.07x concurrency, 28.50 GB VRAM used leaving 5.71 GB free -- still above Jellyfin's ~4.6 GB reservation. One stream now reaches 94%. The engine refuses to start if the pool cannot hold one full sequence, so an over-large value fails fast rather than at runtime. The model allows 262,144; KV memory is the binding limit, not the model. litellm maxInputTokens follows at 148000 - 8192.
Tanguille
changed the title
The fallback comment named control-2 as qwen35-2b's node, but its selector is amd.com/igpu which matches control-2 and control-3 -- the guaranteed property is only that it is never on control-1's dGPU, which is what the tier relies on.
| # never shares control-1's dGPU. Degraded, not down. | ||
| fallbacks: | ||
| - omniroute: ["qwen-3.8-fast"] | ||
| - omniroute: ["qwen-3.8-fast", "qwen35-2b"] |
There was a problem hiding this comment.
Info (question): The qwen35-2b fallback target relies on an operator-generated LiteLLMModel whose name/alias wiring is not present in this diff; confirm the generated model group resolves to the llama.cpp qwen3.5-2b alias at runtime.
Automated finding from AI PR review.
1 participant
Follow-ups to #4509, found while reconciling it into production. All four
changes measured on the live engine.
--mamba-ssm-cache-dtypeis a separate pool (regression fix)The simplify pass on #4509 dropped this flag on the theory that it inherits from
--mamba-cache-dtype. It does not, and the mechanism is model-specific:vllm/model_executor/models/config.pyhas a Qwen3.5 override that rewritesautoto the checkpoint'smamba_ssm_dtype, which is float32. The generic"follow the conv-state dtype" path never applies here.
fp32 SSM state forces the attention
block_sizefrom 800 up to 1568, and thecoarser block wastes more pool:
Restoring it recovered exactly the 7,160 tokens. It is silently load-bearing —
the comment now says so.
Context 112K -> 148K, so one session can use the whole pool
maxModelLenis the only per-request cap on KV blocks (max_num_seqsreservesnothing;
max_num_batched_tokensis a compute budget), so at 112000 a singlestream could hold at most 72% of the pool.
Measured at 148000: pool 158,073 tokens (+1,580 from better block rounding),
1.07x concurrency, 28.50 GB VRAM used leaving 5.71 GB free — still above
Jellyfin's ~4.6 GB reservation. One stream now reaches 94% of the pool.
The engine refuses to start if the pool cannot hold one full sequence, so an
over-large value fails fast rather than at runtime. The model allows 262,144
(
max_position_embeddings); KV memory is the binding limit.litellm
maxInputTokensfollows at148000 - 8192.Second fallback tier
omniroutefell back only toqwen-3.8-fast— a single-entry chain on the samenode and GPU. That went hard-down on 2026-08-17 when the 27B was scaled to 0
during the cutover.
qwen35-2bruns on a different node's dri device, so itdegrades instead of dropping.
Served-model-name alias
llmkube auto-registers a
LiteLLMModelunder the InferenceService name, but vLLMonly served
qwen-3.8, so that route 404'd at the vLLM hop.Considered and rejected
--language-model-only(+31,000 KV tokens, ~1.0 GiB reclaimed): it neverloads the vision tower, and karakeep's
INFERENCE_IMAGE_MODELplus hermes'auxiliary.visionboth use it. Product regression traded for context.--prefix-match-unit 100: proposed on a 0.0% prefix-hit reading that turnedout to be a cold-start artifact — real traffic runs 30-58%. It also hard-couples
to
block_size: 800, so it would turn a future mamba-flag change into a startupfailure.
per_token_headKV (potentially ~2.7x concurrency): the backendadvertises support, but correctness/perf on gfx1201 is unverified and int4 KV is
a real quality risk with tool-calling grammar. Needs an eval, not a config edit.
Verification
PIPELINE OK