fix(ai): tune dormant Qwen3.6 vLLM ROCm manifest - #4155
Merged
Conversation
- repin model source; the old sha 404s and CrashLoops the downloader forever - switch to the official vllm/vllm-openai-rocm nightly, which carries upstream PR #40977's RDNAHybridW4A16LinearKernel; the community fork's frozen base image structurally cannot ship it - drop the Model resources block and shrink InferenceService's (8cpu/32-48Gi never actually schedules on this node) to the values that do - add a compile-cache PVC and JIT env vars so torch.compile/triton/inductor survive pod restarts instead of re-paying ~100s + a long warmup each time - tune generation config to match sglang's memory/quantization footprint and widen the startup probe for the observed single-threaded warmup Stays inert (replicas: 0). Benchmarked in isolation against the sglang runtime this replaces: vLLM wins decode at every concurrency level tested, sglang wins prefill: not yet a production cutover.
📝 WalkthroughWalkthroughThe Qwen vLLM manifest adds persistent compilation caching, updates the model revision and quantization, switches to a ROCm nightly image, revises runtime settings and probes, and reduces serving resources. ChangesQwen vLLM runtime
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
@@ spec @@
# inference.llmkube.dev/v1alpha1/Model/ai/qwen36-27b-vllm
! - one map entry removed:
- resources:
- cpu: "8"
- memory: 48Gi
@@ spec.quantization @@
# inference.llmkube.dev/v1alpha1/Model/ai/qwen36-27b-vllm
! ± value change
- AWQ
+ compressed-tensors
@@ spec.source @@
# inference.llmkube.dev/v1alpha1/Model/ai/qwen36-27b-vllm
! ± value change
- hf://cyankiwi/Qwen3.6-27B-AWQ-INT4@8f269fb53eb3fe3be8f01f9755f20570cef0ebe0
+ hf://cyankiwi/Qwen3.6-27B-AWQ-INT4@e5cc0400fb2403c437c2c40a7c52fb5ae93fda18
@@ spec @@
# inference.llmkube.dev/v1alpha1/InferenceService/ai/qwen36-27b-vllm
! + five map entries added:
+ extraArgs:
+ - "--served-model-name"
+ - qwen-3.6
+ - "--limit-mm-per-prompt"
+ - "{\"image\":0,\"video\":0}"
+ - "--mamba-ssm-cache-dtype"
+ - bfloat16
+ - "--trust-remote-code"
+ - "--host"
+ - "0.0.0.0"
+ extraVolumeMounts:
+ - name: compile-cache
+ mountPath: /cache
+ extraVolumes:
+ - name: compile-cache
+ persistentVolumeClaim:
+ claimName: qwen36-27b-vllm-compile-cache
+ parallelSlots: 8
+ probeOverrides:
+ liveness:
+ exec:
+ command:
+ - "true"
+ periodSeconds: 3600
+ readiness:
+ failureThreshold: 6
+ periodSeconds: 30
+ tcpSocket:
+ port: 8000
+ timeoutSeconds: 5
+ startup:
+ failureThreshold: 240
+ httpGet:
+ path: /health
+ port: 8000
+ initialDelaySeconds: 60
+ periodSeconds: 15
@@ spec.env @@
# inference.llmkube.dev/v1alpha1/InferenceService/ai/qwen36-27b-vllm
! + five list entries added:
+ - name: HIP_VISIBLE_DEVICES
+ value: "0"
+ - name: ROCR_VISIBLE_DEVICES
+ value: "0"
+ - name: VLLM_CACHE_ROOT
+ value: /cache/vllm
+ - name: TRITON_CACHE_DIR
+ value: /cache/triton
+ - name: TORCHINDUCTOR_CACHE_DIR
+ value: /cache/inductor
@@ spec.image @@
# inference.llmkube.dev/v1alpha1/InferenceService/ai/qwen36-27b-vllm
! ± value change
- docker.io/tcclaviger/vllm:latest@sha256:dfb67a2c947e995978c03fb7e1d7a0b0601cf3d9958865df1659b165a367153f
+ docker.io/vllm/vllm-openai-rocm:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9@sha256:b72e4aca14bae0f853cca1bb71bcf17dae30ef2e2f9ec39485f4da0506383380
@@ spec.resources.cpu @@
# inference.llmkube.dev/v1alpha1/InferenceService/ai/qwen36-27b-vllm
! ± value change
- 8
+ 2
@@ spec.resources.memory @@
# inference.llmkube.dev/v1alpha1/InferenceService/ai/qwen36-27b-vllm
! ± value change
- 32Gi
+ 24Gi
@@ spec.vllmConfig @@
# inference.llmkube.dev/v1alpha1/InferenceService/ai/qwen36-27b-vllm
! - three map entries removed:
- dtype: bfloat16
- enableChunkedPrefill: true
- quantization: awq
@@ (root level) @@
# v1/PersistentVolumeClaim/ai/qwen36-27b-vllm-compile-cache
! + one document added:
+ apiVersion: v1
+ kind: PersistentVolumeClaim
+ metadata:
+ name: qwen36-27b-vllm-compile-cache
+ namespace: ai
+ labels:
+ kustomize.toolkit.fluxcd.io/name: llmkube-models
+ kustomize.toolkit.fluxcd.io/namespace: ai
+ spec:
+ resources:
+ requests:
+ storage: 10Gi
+ storageClassName: openebs-hostpath
+ accessModes:
+ - ReadWriteOnce
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
kubernetes/apps/ai/llmkube/models/qwen36-27b-vllm.yaml (1)
143-161: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueLiveness probe is effectively disabled — acceptable for this WIP benchmark, but flag before cutover.
exec: command: ["true"]withperiodSeconds: 3600will always succeed regardless of whether the server actually hangs, so a real hang won't trigger a restart for up to an hour, if ever. Givenreplicas: 0and the PR's explicit "not yet a production cutover" framing, this is a reasonable stopgap to avoid killing mid-warmup, but it shouldn't be forgotten before this becomes a live serving path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kubernetes/apps/ai/llmkube/models/qwen36-27b-vllm.yaml` around lines 143 - 161, The liveness probe in probeOverrides is intentionally disabled for the current WIP benchmark; preserve it as-is, but add a clear TODO or cutover reminder near the liveness configuration to replace the always-successful exec check and hourly interval with a real server health probe before production use.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kubernetes/apps/ai/llmkube/models/qwen36-27b-vllm.yaml`:
- Around line 36-40: Update the quantization metadata alongside the source in
the model configuration to identify the checkpoint as compressed-tensors
pack-quantized rather than AWQ. Change only the quantization value and preserve
the existing source pin and format.
---
Nitpick comments:
In `@kubernetes/apps/ai/llmkube/models/qwen36-27b-vllm.yaml`:
- Around line 143-161: The liveness probe in probeOverrides is intentionally
disabled for the current WIP benchmark; preserve it as-is, but add a clear TODO
or cutover reminder near the liveness configuration to replace the
always-successful exec check and hourly interval with a real server health probe
before production use.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 98c87466-c8bb-4e5d-8999-84a7e31fe04b
📒 Files selected for processing (1)
kubernetes/apps/ai/llmkube/models/qwen36-27b-vllm.yaml
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
Tanguille/LLMKube(auto-detected)
AI Automated Review (incremental)Analysis engine: qwen-3.6-fast@http://litellm.ai.svc.cluster.local/v1 (openai) Recommendation: Approve This incremental commit cleans up verbose inline comments from the previous PR iteration without altering any functional YAML. The changes are cosmetic-only, preserving all resource specifications, probe configurations, environment variables, and image references established in the prior review. The manifest remains internally consistent and aligns with repository conventions for comment brevity. Change-by-Change Findings
Standards Compliance
Tool Harness FindingsNo tool harness findings were provided; this section is omitted. Unknowns or Needs VerificationNone — the delta is purely cosmetic and fully verifiable from the diff. |
Tanguille
changed the title
This was referenced Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
replicas: 0) while benchmarking it against the current SGLang runtime.Review follow-up
Validation
git diff --checkSummary by CodeRabbit
New Features
Improvements