feat(llmkube): tune vllm staging config and document VRAM ceiling

[Tanguille]

Records the 2026-07 measurement round on the R9700 (Qwen3.6-27B AWQ) and applies what it settled.

The binding constraint

The R9700 is shared with Jellyfin transcoding, so 0.875 is a hard ceiling, not a tunable. The ~4.6 GB rocm-smi reports free at 0.875 is the transcoding reservation. Both manifests now carry that rationale inline so it does not get "optimized" away later.

Independently, 0.95 is not a valid operating point for SGLang anyway: it grows the pool 183,240 -> 261,019 tokens, then OOMs on a 13.8K-token prefill (tried to allocate 160 MiB, 0 bytes free). It survives short-prompt decode and dies on the shape we actually serve at p50 35K.

vLLM staging config

Still replicas: 0. This makes staging correct and documented, it does not cut production over.

change	why
pin v0.26.0	measured identical to the nightly on KV pool and throughput
parallelSlots 8 -> 32	doubles aggregate throughput at concurrency 16
maxModelLen 180000 -> 64000	180000 needs ~6.55 GiB KV against a ~4.6 GiB ceiling at 0.875; 64000 still covers 74.1% of measured prompts
explicit --kv-cache-memory	vLLM's profiler leaves the pool at 48,000 tokens, explicit sizing gives 121,302 within the same budget
model pin -> QuantTrio AWQ	the variant actually boot-validated; the prior cyankiwi INT4 pin was never booted

Note the model-source change triggers a re-download into the staging cache PVC on next reconcile.

What this overturned

The decode ceiling was KV-pool starvation, not GPU bandwidth. Per-stream throughput collapsed 4.85 -> 2.45 tok/s going from 8 to 16 streams under a small pool, and held at ~4.6 under a large one, nearly doubling aggregate. Earlier rounds read that plateau as a fixed per-step overhead. Growing the pool within 0.875 is the remaining lever, not kernel-level decode work.

Benchmarks taken right after a boot are unreliable. One config read 32.36 tok/s on its first run and 39.04 / 39.09 on two immediate repeats. Some earlier single-run comparisons were reading warmup as a config difference.

Image tags are not version-ordered. The nightly self-reports a stale 0.23.1rc1 base tag, and v0.26.0 was built earlier than the nightly despite the higher number.

llmkube cache-integrity hazard

Hit during this round and documented in the new doc. Patching an InferenceService while its model-downloader is mid-transfer truncates the artifact. The corruption is invisible to every later boot until the engine fails to load, because the downloader writes straight to the final path, persists the ETag from response headers before the body completes, and treats file presence as validity. Recovery needs deleting the artifact and its .etag.

Worth an upstream issue; there is currently no content-integrity check at all since sha256 pinning was rejected on the Model CRD.

Still open

• vLLM prefix cache reports 0% across 889,464 queries (vllm#45238), likely worth more than any decode tuning on this workload
• quant comparison unfinished: vLLM runs QuantTrio, SGLang runs mattbucci, so engine numbers are engine+quant
• how much VRAM transcoding actually needs has not been measured; 0.875 is treated as given
• SGLang #31648 (mamba LRU, prefix hit 0.61 -> 0.83) needs a fork rebuild

Validation

kustomize build kubernetes/apps/ai/llmkube/models clean. SGLang verified serving at 0.875 with a live request; pool 183,240 tokens plus 9.01 GB host KV and 7.14 GB host mamba, 3.71 GB GPU headroom left.

Summary by CodeRabbit

• Documentation
  • Added a July 2026 vLLM vs SGLang benchmark page (Radeon AI PRO R9700, Qwen3.6-27B AWQ) with revised throughput conclusions, measurement caveats, and guidance for recovering from persistent cache truncation after download interruptions.
• Deployment Updates
  • Updated Qwen3.6-27B vLLM model/inference settings (AWQ, more safetensors shards, new vLLM image, higher parallel slots, lower max model length, explicit KV cache sizing); refreshed SGLang config constraint documentation.
• New Tools
  • Added local scripts to measure decode throughput under concurrency and to check deterministic completion behavior.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 396c5027-7a42-43dd-aa35-c9913c94348d

📥 Commits

Reviewing files that changed from the base of the PR and between 6641982 and 990992c.

📒 Files selected for processing (1)

• docs/llm-hosting/bench/spectest.py

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

• Tanguille/LLMKube (auto-detected)

---

📝 Walkthrough

Walkthrough

Changes

The PR adds July 2026 vLLM/SGLang measurement documentation and benchmark scripts, documents model-cache recovery procedures, and updates Qwen36 vLLM deployment settings plus SGLang hardware-constraint comments.

Qwen36 runtime update

Layer / File(s)	Summary
Measurement constraints and comparison docs/llm-hosting/vllm-vs-sglang-2026-07.md	Records the 0.875 VRAM constraint, revised benchmark conclusions, engine version guidance, workload profile, open investigations, and reproducibility notes.
Benchmark tooling and engine experiments docs/llm-hosting/bench/*, docs/llm-hosting/vllm-vs-sglang-2026-07.md	Adds concurrency and speculative-reproduction benchmark scripts, and documents speculative-decoding results plus SGLang patch details.
Model-download failure and recovery docs/llm-hosting/vllm-vs-sglang-2026-07.md	Describes truncated cached artifacts after service updates during downloads and the required artifact and ETag cleanup.
Qwen36 service configuration kubernetes/apps/ai/llmkube/models/qwen36-27b-vllm.yaml, kubernetes/apps/ai/llmkube/models/qwen36-27b-sglang.yaml	Repins the vLLM model and image, adjusts runtime limits and KV-cache sizing, and updates SGLang hardware-constraint comments.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• Tanguille/cluster#4146: Overlaps on the vLLM container image digest.
• Tanguille/cluster#4155: Overlaps on vLLM model and runtime tuning.
• Tanguille/cluster#4163: Overlaps on the vLLM container image digest.

Suggested labels: area/docs, area/kubernetes, type/digest

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly reflects the main changes: vLLM staging config tuning and documenting the VRAM ceiling.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/llm-serving-tuning-2026-07

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

docs/llm-hosting/vllm-vs-sglang-2026-07.md (1)

163-166: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Version the benchmark harness or record immutable provenance.

Because .vllm-opt/ is untracked and local-only, future readers cannot reproduce or audit these measurements. Commit the scripts, or record their hashes together with exact commands and configuration snapshots.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/llm-hosting/vllm-vs-sglang-2026-07.md` around lines 163 - 166, Update
the benchmark documentation around concsweep.py and spectest.py to provide
immutable provenance: either version the .vllm-opt harness scripts in the
repository or record each script’s hash, exact execution command, and
configuration snapshot used for the measurements. Ensure future readers can
reproduce and audit both benchmarks.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/llm-hosting/vllm-vs-sglang-2026-07.md`:
- Around line 75-83: The paragraph incorrectly presents findings from the
invalid 0.95 configuration as applicable to 0.875 production configs. Update the
conclusions around the cited SGLang single-stream comparison and OOM behavior to
either add supporting 0.875 measurements or explicitly scope both findings to
the 0.95 configuration, and remove the claim that they carry over.
- Around line 123-136: Update the downloader flow described here to verify the
completed "$dest" file size against the response Content-Length, using the check
only as a truncation guard. Ensure failed or mismatched downloads are not
treated as valid by the existing fallback, while preserving the current behavior
unless the artifact and its .etag are both validated and published atomically.

---

Nitpick comments:
In `@docs/llm-hosting/vllm-vs-sglang-2026-07.md`:
- Around line 163-166: Update the benchmark documentation around concsweep.py
and spectest.py to provide immutable provenance: either version the .vllm-opt
harness scripts in the repository or record each script’s hash, exact execution
command, and configuration snapshot used for the measurements. Ensure future
readers can reproduce and audit both benchmarks.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: fc4d7aef-8810-4b2d-bd98-1305c3618f2f

📥 Commits

Reviewing files that changed from the base of the PR and between 3166668 and 873c840.

📒 Files selected for processing (3)

• docs/llm-hosting/vllm-vs-sglang-2026-07.md
• kubernetes/apps/ai/llmkube/models/qwen36-27b-sglang.yaml
• kubernetes/apps/ai/llmkube/models/qwen36-27b-vllm.yaml

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

• Tanguille/LLMKube (auto-detected)

[tanguille-cluster]

@@ spec.files @@
# inference.llmkube.dev/v1alpha1/Model/ai/qwen36-27b-vllm
! - four list entries removed:
- - model-00001-of-00004.safetensors
- - model-00002-of-00004.safetensors
- - model-00003-of-00004.safetensors
- - model-00004-of-00004.safetensors
! + eight list entries added:
+ - model-00001-of-00008.safetensors
+ - model-00002-of-00008.safetensors
+ - model-00003-of-00008.safetensors
+ - model-00004-of-00008.safetensors
+ - model-00005-of-00008.safetensors
+ - model-00006-of-00008.safetensors
+ - model-00007-of-00008.safetensors
+ - model-00008-of-00008.safetensors

@@ spec.quantization @@
# inference.llmkube.dev/v1alpha1/Model/ai/qwen36-27b-vllm
! ± value change
- compressed-tensors
+ AWQ

@@ spec.source @@
# inference.llmkube.dev/v1alpha1/Model/ai/qwen36-27b-vllm
! ± value change
- hf://cyankiwi/Qwen3.6-27B-AWQ-INT4@e5cc0400fb2403c437c2c40a7c52fb5ae93fda18
+ hf://QuantTrio/Qwen3.6-27B-AWQ@9b507bdc9afafb87b7898700cc2a591aa6639461

@@ spec.extraArgs @@
# inference.llmkube.dev/v1alpha1/InferenceService/ai/qwen36-27b-vllm
! + two list entries added:
+ - "--kv-cache-memory-bytes"
+ - "4294967296"

@@ spec.image @@
# inference.llmkube.dev/v1alpha1/InferenceService/ai/qwen36-27b-vllm
! ± value change
- vllm/vllm-openai-rocm:nightly@sha256:bd34cea1fe22f0c9f8fef0e362eac2d8b9292971a114eb95d2cfa1026862f5cd
+ vllm/vllm-openai-rocm:v0.26.0@sha256:5709fafe47123becb2f5e61c32d0b97beff1a629bb40bb753c15464f69a97a18

@@ spec.parallelSlots @@
# inference.llmkube.dev/v1alpha1/InferenceService/ai/qwen36-27b-vllm
! ± value change
- 8
+ 32

@@ spec.vllmConfig.maxModelLen @@
# inference.llmkube.dev/v1alpha1/InferenceService/ai/qwen36-27b-vllm
! ± value change
- 180000
+ 64000

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/llm-hosting/bench/concsweep.py`:
- Around line 46-49: Update the per-stream throughput calculation in the
concsweep aggregation block to divide agg by len(ok), so it reflects only
successful streams; retain the zero-success guard to avoid division by zero.
- Line 10: Validate the command-line port before constructing the request URL:
in docs/llm-hosting/bench/concsweep.py at lines 10-10 and
docs/llm-hosting/bench/spectest.py at lines 7-7, parse sys.argv[1] as an integer
and reject values outside 1..65535, then use the validated integer when building
the localhost URL.

In `@docs/llm-hosting/bench/spectest.py`:
- Around line 13-16: Update the benchmark request and validation around the
`prompt` construction and response handling so the expected `handler_0` through
`handler_12` prefix is defined explicitly and the measured result fails unless
the model output begins with that exact verbatim prefix. Ensure generation can
complete the reproduction by requesting a fixed prefix or increasing
`max_tokens` and permitting EOS, rather than reporting throughput for formatted,
divergent, or truncated output.

In `@kubernetes/apps/ai/llmkube/models/qwen36-27b-vllm.yaml`:
- Around line 131-134: Replace the invalid --kv-cache-memory argument in the
vLLM command configuration with --kv-cache-memory-bytes, preserving the existing
4294967296 value and surrounding profiler sizing settings.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 09a11340-435a-40c3-9b72-7db6d3ecc8e9

📥 Commits

Reviewing files that changed from the base of the PR and between 873c840 and 227008e.

📒 Files selected for processing (5)

• docs/llm-hosting/bench/concsweep.py
• docs/llm-hosting/bench/spectest.py
• docs/llm-hosting/vllm-vs-sglang-2026-07.md
• kubernetes/apps/ai/llmkube/models/qwen36-27b-sglang.yaml
• kubernetes/apps/ai/llmkube/models/qwen36-27b-vllm.yaml

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

• Tanguille/LLMKube (auto-detected)

🚧 Files skipped from review as they are similar to previous changes (1)

• docs/llm-hosting/vllm-vs-sglang-2026-07.md

[Tanguille]

Resolved the remaining review remarks in 2c8e9ee:

• --kv-cache-memory invalid on v0.26.0 — confirmed and fixed. Upstream v0.26.0 arg_utils.py only registers --kv-cache-memory-bytes; our config booted because argparse accepts unambiguous prefixes. That is fragile, so it now uses the documented name. The value is unchanged and the service runs at replicas: 0, so nothing restarts on merge.
• per-stream divided by requested concurrency — now divides by len(ok), so a failed stream no longer deflates the figure.
• Unvalidated port argument — both bench scripts parse it as an int and reject anything outside 1..65535.
• spectest.py reported throughput without validating the continuation — it now compares the output against the handler_0..12 source and prints verbatim: OK/DIVERGED at char N. Prefix comparison rather than equality, since ignore_eos lets generation run past the expected text.

Separately, the -CT open item was corrected: it is scoped to vLLM only, the unverified size figure is dropped, and the doc now records that the model card describes it as the raw GPTQ output before AWQ repack, so it is upstream of what SGLang already serves rather than an upgrade to it.

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

docs/llm-hosting/bench/spectest.py (1)

40-40: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Make zip strictness explicit.

Ruff B905 flags this call. Since unequal lengths are intentional for prefix comparison, use strict=False to document that behavior and keep the lint check clean; verify the repository’s Python target supports this parameter.

Proposed fix

-n = next((i for i, (a, b) in enumerate(zip(expect, got)) if a != b), min(len(expect), len(got)))
+n = next((i for i, (a, b) in enumerate(zip(expect, got, strict=False)) if a != b), min(len(expect), len(got)))

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/llm-hosting/bench/spectest.py` at line 40, Update the zip call in the
comparison expression computing n to pass strict=False explicitly, preserving
intentional prefix comparison for unequal expect and got lengths. Confirm the
repository’s supported Python target includes the strict parameter before
applying the change.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/llm-hosting/bench/spectest.py`:
- Around line 38-42: Update the success condition in the verbatim comparison
around expect, got, and n so it reports OK only when got starts with the
complete expect string. Ensure truncated or empty continuations are reported as
DIVERGED, while preserving the existing divergence-position calculation and
output format.

---

Nitpick comments:
In `@docs/llm-hosting/bench/spectest.py`:
- Line 40: Update the zip call in the comparison expression computing n to pass
strict=False explicitly, preserving intentional prefix comparison for unequal
expect and got lengths. Confirm the repository’s supported Python target
includes the strict parameter before applying the change.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 97ef0099-a6f8-4b1c-a32f-67851d78beaa

📥 Commits

Reviewing files that changed from the base of the PR and between 227008e and 2c8e9ee.

📒 Files selected for processing (4)

• docs/llm-hosting/bench/concsweep.py
• docs/llm-hosting/bench/spectest.py
• docs/llm-hosting/vllm-vs-sglang-2026-07.md
• kubernetes/apps/ai/llmkube/models/qwen36-27b-vllm.yaml

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

• Tanguille/LLMKube (auto-detected)

🚧 Files skipped from review as they are similar to previous changes (3)

• kubernetes/apps/ai/llmkube/models/qwen36-27b-vllm.yaml
• docs/llm-hosting/bench/concsweep.py
• docs/llm-hosting/vllm-vs-sglang-2026-07.md

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/llm-hosting/bench/spectest.py`:
- Around line 44-45: Update the mismatch-position fallback in the comparison
expression before the “verbatim” print so that, when no differing character is
found, it uses the shorter of expect and got rather than len(got). Preserve the
existing first-mismatch calculation and DIVERGED state behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1a2fde44-b128-4ed9-8027-4d42f1929e46

📥 Commits

Reviewing files that changed from the base of the PR and between 2c8e9ee and 6641982.

📒 Files selected for processing (1)

• docs/llm-hosting/bench/spectest.py

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

• Tanguille/LLMKube (auto-detected)