feat(llmkube)!: cut Qwen 3.8 over to production, retire Qwen 3.6 SGLang

.github/workflows/agent-pr-review.yaml

@@ -43,22 +43,22 @@ jobs:
           ai_base_url: http://litellm.ai.svc.cluster.local/v1
           ai_api_key: ${{ secrets.LITELLM_API_KEY }}
           # omniroute routes to a larger free-tier model than the self-hosted 27B
-          # (see litellm routerSettings.fallbacks); falls back to qwen-3.6-fast
+          # (see litellm routerSettings.fallbacks); falls back to qwen-3.8-fast
           # automatically if the free provider fails or times out.
           ai_model: omniroute
           # v2.1.10 inherits ai_fallback_base_url from ai_base_url when unset, then
           # requires ai_fallback_model whenever ai_fallback_base_url is set (even
           # inherited) — fails fast with no model call otherwise. Point this at
-          # qwen-3.6-fast directly (same litellm endpoint) rather than duplicating
+          # qwen-3.8-fast directly (same litellm endpoint) rather than duplicating
           # ai_model: litellm's routerSettings.fallbacks already retries omniroute
           # failures internally, so this only fires if that whole chain fails —
           # a real second attempt via a distinct model, not a no-op retry.
-          ai_fallback_model: qwen-3.6-fast
-          # omniroute's proxied models don't share SGLang's sampling_defaults, so
+          ai_fallback_model: qwen-3.8-fast
+          # omniroute's proxied models don't share the self-hosted model's defaults, so
           # an empty temperature has no known-good fallback here — pin explicitly.
           ai_temperature: "0.2"
           ai_response_format: json_schema
-          # Covers both omniroute (observed 2-74s) and, on fallback, qwen-3.6-fast's
+          # Covers both omniroute (observed 2-74s) and, on fallback, qwen-3.8-fast's
           # ~10 tok/s cold prefill on this GPU (155-220s near ~100K ctx, validated).
           ai_request_timeout_sec: "600"
           verdict_policy: findings_severity_gated

.renovaterc.json5

@@ -42,8 +42,12 @@
       description: "HuggingFace model files revision-pinned in llmkube Model sources (HF model repos are git repos; digest = repo commit sha)",
       customType: "regex",
       managerFilePatterns: ["/kubernetes/apps/ai/llmkube/models/.+\\.ya?ml$/"],
+      // Two source forms in use: the resolve/ URL for single-file GGUFs, and
+      // hf://repo@sha for multi-file repos (qwen38, muse-glimmer). Without the
+      // second, the primary serving model's weights digest tracks nothing.
       matchStrings: [
-        "source:\\s+https://huggingface\\.co/(?<depName>[^/\\s]+/[^/\\s]+)/resolve/(?<currentDigest>[a-f0-9]{40})/"
+        "source:\\s+https://huggingface\\.co/(?<depName>[^/\\s]+/[^/\\s]+)/resolve/(?<currentDigest>[a-f0-9]{40})/",
+        "source:\\s+hf://(?<depName>[^/\\s]+/[^/\\s]+)@(?<currentDigest>[a-f0-9]{40})"
       ],
       packageNameTemplate: "https://huggingface.co/{{depName}}",
       currentValueTemplate: "main",
@@ -52,7 +56,9 @@
     {
       description: "opencode npm plugins (executed in-process in an agent pod, so they stay pinned)",
       customType: "regex",
-      managerFilePatterns: ["/kubernetes/apps/ai/opencode/app/config/opencode\\.jsonc$/"],
+      managerFilePatterns: [
+        "/kubernetes/apps/ai/opencode/app/config/opencode\\.jsonc$/"
+      ],
       matchStrings: [
         '"(?<depName>@?[\\w.-]+(?:/[\\w.-]+)?)@(?<currentValue>\\d+\\.\\d+\\.\\d+)"'
       ],
@@ -70,12 +76,6 @@
       automergeType: "pr"
     },
     // Grouping rules
-    {
-      description: "sglang-rdna4 deployed image digest — built by our fork of the upstream RDNA4 image, not a pipeline in this repo; review before the HelmRelease rolls the serving pod (Recreate on the single GPU)",
-      matchDatasources: ["docker"],
-      matchPackageNames: ["ghcr.io/tanguille/sglang-rdna4"],
-      automerge: false
-    },
     {
       description: "Actions Runner Controller Group",
       groupName: "actions-runner-controller",
@@ -99,7 +99,7 @@
       matchDatasources: ["docker", "github-releases"],
       // matches all siderolabs/* packages (talos, installer, etc.); both slashes are regex
       // delimiters, so the inner one must stay escaped AND closed or config validation fails
-      matchPackageNames: ["/siderolabs\\//"],
+      matchPackageNames: ["/siderolabs\\//"]
     },
     {
       description: "Rook-Ceph Group",

docs/llm-hosting/bench/concsweep.py

@@ -17,7 +17,9 @@ def _port():
 
 
 URL = f"http://127.0.0.1:{_port()}/v1/completions"
-MODEL = "qwen-3.6"
+# 2nd arg overrides the served-model-name, since the port-forward target decides
+# which engine answers. Hardcoding it once cost a whole run of silent 0.00 tok/s.
+MODEL = sys.argv[2] if len(sys.argv) > 2 else "qwen-3.8"
 GEN = 96
 
 

docs/llm-hosting/bench/spectest.py

@@ -14,14 +14,17 @@ def _port():
 
 
 URL = f"http://127.0.0.1:{_port()}/v1/completions"
+# 2nd arg overrides the served-model-name, since the port-forward target decides
+# which engine answers. Hardcoding it once cost a whole run of silent 0.00 tok/s.
+MODEL = sys.argv[2] if len(sys.argv) > 2 else "qwen-3.8"
 CODE = "\n".join(f"def handler_{i}(request, context):\n"
                  f"    payload = request.get('payload_{i}')\n"
                  f"    if payload is None:\n"
                  f"        raise ValueError('missing payload_{i}')\n"
                  f"    return {{'status': 200, 'body': payload}}\n" for i in range(240))
 prompt = (f"Here is a Python module:\n\n{CODE}\n\n"
           "Reproduce handler_0 through handler_12 EXACTLY as written above, verbatim:\n\n")
-b = json.dumps({"model": "qwen-3.6", "prompt": prompt, "max_tokens": 400,
+b = json.dumps({"model": MODEL, "prompt": prompt, "max_tokens": 400,
                 "temperature": 0, "ignore_eos": True}).encode()
 t0 = time.perf_counter()
 rq = urllib.request.Request(URL, data=b, headers={"Content-Type": "application/json"})

docs/llm-hosting/engine-benchmarks-gfx1201.md

@@ -4,10 +4,10 @@ Multi-engine measurement record for Qwen3.6-27B on RDNA4 — vLLM, SGLang and ll
 Measurements are dated and kept as a historical series; the open experiments at the
 bottom are still outstanding.
 
-**Current production engine: SGLang**, `ghcr.io/tanguille/sglang-rdna4`, built by our fork
-of the upstream RDNA4 image rather than a pipeline in this repo — see
-`docs/llm-hosting/sglang-blockers.md` "Current approach" for the build source and the
-upstream cutover plan; outstanding upstream gaps are tracked in the same doc. The
+**Current production engine: vLLM**, serving Qwen 3.8 as `qwen38-27b-vllm` (2026-08-16).
+SGLang and the whole Qwen 3.6 lane were retired with that cutover;
+`docs/llm-hosting/sglang-blockers.md` is kept as the record of why SGLang never
+displaced vLLM here. The
 2026-06-21 round below
 concluded in favour of vLLM and has since been superseded — read its numbers as a
 snapshot of that date, not as current guidance.

docs/llm-hosting/sglang-blockers.md

@@ -2,7 +2,7 @@
 
 Tracking what needs to land upstream before SGLang can replace vLLM in production without depending on the mattbucci RDNA4 fork.
 
-**Current approach:** running `ghcr.io/tanguille/sglang-rdna4:sha-cb7b7605...` (torch 2.11+rocm7.2, Triton 3.6, SGLang v0.5.16), built by our fork's own `build-image.yaml` `publish` job (`Tanguille/2x-R9700-RDNA4-GFX1201-sglang-inference`, `main`), not by a pipeline in this repo — `docker/sglang-rdna4/` and `.github/workflows/build-sglang-rdna4.yaml` are retired. That fork branch carries mattbucci's full patch series plus our HiCache OpenSSL-headers fix (`fix/hicache-openssl-headers`, upstream PR mattbucci#6, still open/unmerged there — this is our own fork build, not the upstream package). `SGLANG_RDNA4_DISABLE_STORE_CACHE=1` is required in the HelmRelease env because LLMKube bypasses the image's entrypoint (see the InferenceService manifest). Once mattbucci#6 merges upstream, re-point at `ghcr.io/mattbucci/sglang-rdna4` directly and drop the fork build. The retired PVC-rebuild recipe is gone with the `sglang` app directory; recover it from git history if ever needed.
+**Historical approach (retired 2026-08-16, superseded by vLLM/Qwen 3.8):** ran `ghcr.io/tanguille/sglang-rdna4:sha-cb7b7605...` (torch 2.11+rocm7.2, Triton 3.6, SGLang v0.5.16), built by our fork's own `build-image.yaml` `publish` job (`Tanguille/2x-R9700-RDNA4-GFX1201-sglang-inference`, `main`), not by a pipeline in this repo — `docker/sglang-rdna4/` and `.github/workflows/build-sglang-rdna4.yaml` are retired. That fork branch carries mattbucci's full patch series plus our HiCache OpenSSL-headers fix (`fix/hicache-openssl-headers`, upstream PR mattbucci#6, still open/unmerged there — this is our own fork build, not the upstream package). `SGLANG_RDNA4_DISABLE_STORE_CACHE=1` was required in the HelmRelease env because LLMKube bypasses the image's entrypoint (see the InferenceService manifest). Once mattbucci#6 merges upstream, re-point at `ghcr.io/mattbucci/sglang-rdna4` directly and drop the fork build. The retired PVC-rebuild recipe is gone with the `sglang` app directory; recover it from git history if ever needed.
 
 **Fork state check (2026-08-01):** the deployed image `sha-cb7b76050cbf` already sits on the fork's v0.5.16 rebase (`689339d`, 2026-07-27), so blocker 6's #31648 (mamba LRU) is live in production — it ships in the v0.5.16 release. Fork patches 086 (AMD Triton `num_kv_splits` 16→64, claimed 2.14× @256K) and 087 (bf16 page-vector attention, claimed +21% @256K) are also in that image; both are model-agnostic (verified via patch content) but were only A/B'd on `coder-reap-25b`, not Qwen3.6-27B. A clean TP=1 re-bench of the deployed image is still outstanding (the 2026-08-01 attempt ran under production load and is recorded as a floor in `engine-benchmarks-gfx1201.md`). `diff cb7b760..origin/main` is docs/eval-only, so a rebuild on current fork main gains no runtime changes today.
 

docs/llm-hosting/vllm-vs-sglang-2026-07.md

@@ -236,9 +236,10 @@ Both drive the OpenAI `/v1/completions` endpoint, so they run unmodified against
 engine. Port-forward the service and pass the port:
 
 ```sh
-kubectl -n ai port-forward svc/qwen36-27b 30000:30000 &
-python3 concsweep.py 30000
-python3 spectest.py 30000
+kubectl -n ai port-forward svc/qwen38-27b-vllm 8000:8000 &
+python3 concsweep.py 8000
+python3 spectest.py 8000
+# optional 2nd arg overrides the served-model-name (default qwen-3.8)
 ```
 
 **Discard the first run after a pod restart** (see above), and note `concsweep.py` reports