feat(sglang): enable HiCache L3 file backend

docs/llm-hosting/sglang-blockers.md

@@ -195,10 +195,9 @@ the pinned fork tree carries stock v0.5.14 hicache code (no fork patch touches i
 population request took 113.3s. After a clean pod restart, the token-exact post-restart extension took
 70.7s, but the file-backend metric recorded only 4,538 backed-up tokens, with zero storage
 prefetches and zero `storage_HiCacheFile` hits. Aggregate completion throughput for C1/C4/C8 was
-11.16/8.74/14.09 tok/s; the C8 sample overlapped real traffic. The validated direct I/O, ratio 1.5
-L2, and `write_through_selective` policy remain enabled; do not propose unverified cache tweaks.
-Rollback does not remove `/cache/sglang/hicache`; it can contain prompt-derived data and requires
-explicit operator approval before deletion.
+11.16/8.74/14.09 tok/s; the C8 sample overlapped real traffic. Direct I/O and ratio 1.5 L2 remain
+validated and enabled; `write_through_selective` was replaced on 07-27, see the retest below. Do
+not propose unverified cache tweaks.
 
 **Correction (2026-07-27): "failed restart recovery" was the wrong conclusion.** Two defects sat
 upstream of that measurement, and neither is a property of L3.
@@ -220,10 +219,47 @@ upstream of that measurement, and neither is a property of L3.
    4,538-of-61,600 backup figure. Any L3 test must run under plain `write_through`.
 
 The 07-15 run did not crash, so it cannot have been on an image with defect 1 — meaning it is not
-comparable to current builds and should not be quoted as evidence either way. **L3 is untested, not
-failed.** Retest sequence: rebuild with `libssl-dev` → `write_through` → synthetic prefill to prove
-the backend initialises → populate/restart/extension probe. Not on the production replica: the
-07-27 attempt took prod down mid-traffic and cost ~40 min across two reloads.
+comparable to current builds and should not be quoted as evidence either way. **L3 was untested,
+not failed.**
+
+**Retest (2026-07-27): L3 works.** Run on image `sha256:72d934d3` under plain `write_through`,
+with `--hicache-storage-backend file` and `SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR`.
+
+| stage | evidence |
+|---|---|
+| backend initialises | synthetic prefill served; pod ran 22 min under live traffic, 0 restarts |
+| L3 writes | 268,176 files / 14 GB on disk, `backuped_tokens_total` 267,978 |
+| L2 saturates | `hicache_host_used_tokens` 267,978 of 274,861 |
+| survives restart | after a cold start, `prefetched_tokens_total{storage_backend="file"}` 63,205 |
+
+The restart figures close exactly: 73,120 newly backed + 63,205 prefetched = 136,325
+`hicache_host_used_tokens`, so every L2 token is accounted for as either a fresh write or a
+read-back from disk. Populate (58,071 tok) PP 196.9 tok/s; extension probe (58,209 tok) PP
+252.9 tok/s, TG 0.56 tok/s. Both ran with production traffic on the GPU, so both are floors,
+not clean numbers.
+
+Three operational notes.
+
+**Sizing.** The evictor is unbounded by default and `openebs-hostpath` enforces no quota, so
+`MAX_SIZE` (64Gi) and `MIN_FREE_SPACE` (100Gi) are the only things between L3 and control-1's
+shared 500G root; the free-space floor is what keeps a full cache from pushing the node under
+kubelet's ~50G nodefs eviction threshold. L3 lives on its own `qwen36-27b-hicache` PVC rather
+than sharing the triton cache, so its footprint is visible where capacity is planned.
+
+**The cache key does not cover the weights or the engine.** `get_hash_str` is
+`sha256(token_ids, prior_hash, page_size)` (`mem_cache/utils.py:106`) and the on-disk name only
+appends `config_suffix = f"_{model_name}"` plus TP/PP/CP ranks (`hicache_storage.py:336-346`).
+Nothing in it encodes the weights revision, the KV dtype or the KV layout, while
+`served-model-name` stays stable across Renovate bumps of both the AWQ revision and the sglang
+image. Either bump would otherwise serve KV pages computed by the previous one, which is a
+wrong-output risk rather than a cache miss. `STORAGE_DIR` therefore carries a revision suffix
+(`/hicache/sglang-v0.5.15_awq-f541031d`) that **must be bumped by hand with either pin**. It sits
+directly above them in the manifest to keep the three visible together; the durable fix is
+upstream folding a model/engine fingerprint into `config_suffix`.
+
+**Deletion needs approval.** Rollback does not remove the directory; it holds prompt-derived
+data. The 07-27 test data still sits at the old `/cache/sglang/hicache` path, orphaned by the
+move to a dedicated PVC.
 
 **`hicache-write-policy write_back` trialled and reverted (2026-07-09):** kept alongside the L3 removal
 above as a still-valid L1→L2 (GPU→host) optimization — synthetic testing showed 0 aborts and lower

docs/llm-hosting/vllm-vs-sglang-2026-07.md

@@ -129,8 +129,9 @@ Three things combine:
    impossible.
 2. **ETag is not integrity.** The etag file is written from response headers, which
    arrive before the body, so it can persist for a transfer that later dies. The next
-   boot sends `If-None-Match`, gets `304`, and prints `Model artifact ... revalidated`
-   over a corrupt file. Nothing checks size against `Content-Length`.
+   boot sends `If-None-Match` and prints `Model artifact ... revalidated` over a corrupt
+   file. Nothing checks size against `Content-Length`. Note the message is printed on any
+   curl exit 0, not on a `304`, so it carries no information about what actually happened.
 3. **The fallback keeps bad data.** `elif [ -f "$dest" ]` treats presence as validity.
 
 There is currently no content-integrity mechanism at all: sha256 pinning was rejected on
@@ -175,9 +176,21 @@ served over a truncated weight file. For LFS artifacts that cannot happen, since
 always re-fetched. The stale-etag trap is real for config files and for the *bookkeeping*,
 but the practical failure mode on weights is cost, not silent corruption.
 
+A third defect compounds both: `-o "$dest"` opens the destination for writing at request
+time, so the valid cached copy is truncated at the start of every restart before curl knows
+whether it needs the body. The cache is not merely ignored, it is destroyed.
+
+Confirmed again on the 07-27 L3 restart, which is the clearest single reading: all nine
+artifacts logged `revalidated` while `model-downloader` ran 14m21s (18:08:38Z to 18:22:59Z),
+which at the observed rate is the whole weight set.
+
 This is operator-side (the downloader script is generated by llmkube), so it needs an
-upstream fix: follow redirects with the conditional intact, or range/size check the local
-file before deciding to transfer.
+upstream fix: download to `$dest.tmp` + `mv` so a fetch can never truncate the copy it is
+revalidating, and log `revalidated` only on an actual `304`. A `HEAD` `Content-Length`
+compare is worth adding as a cheap truncation preflight, but it is not cache validation:
+equal size proves neither freshness nor integrity, so it supplements the conditional GET
+rather than replacing it. Filed as
+[defilantech/LLMKube#1309](https://github.com/defilantech/LLMKube/issues/1309).
 
 ## Open items
 

kubernetes/apps/ai/llmkube/models/qwen36-27b-sglang.yaml

@@ -25,6 +25,20 @@ spec:
     requests:
       storage: 10Gi
 ---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: qwen36-27b-hicache
+  namespace: ai
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: openebs-hostpath
+  resources:
+    # hostpath enforces no quota, so the real bound is the evictor's MAX_SIZE below.
+    requests:
+      storage: 64Gi
+---
 # yaml-language-server: $schema=https://k8s-schemas.home-operations.com/inference.llmkube.dev/model_v1alpha1.json
 apiVersion: inference.llmkube.dev/v1alpha1
 kind: Model
@@ -63,6 +77,7 @@ spec:
   runtime: sglang
   # Operator default binds :: (IPv6-only on this IPv4 cluster); override to v4.
   bindAddress: 0.0.0.0
+  # L3 needs OpenSSL headers in the runtime stage; earlier digests kill the scheduler.
   image: ghcr.io/tanguille/sglang-rdna4:v0.5.15-gfx1201@sha256:72d934d335529934bf73cd480b728cd08ae61178c9204dc37a025302b31fd61e
   containerPort: 30000
   endpoint:
@@ -107,17 +122,37 @@ spec:
       value: "0"
     - name: TRITON_CACHE_DIR
       value: /cache/sglang/triton
+    # Correctness knob, not a layout preference. The on-disk page name is
+    # sha256(token_ids, prior_hash, page_size) + "_{model_name}_{tp_rank}_{tp_size}",
+    # so nothing in it encodes the weights revision, the KV dtype or the KV layout.
+    # Same served-model-name plus same tokens hits the same file, which means a
+    # Renovate bump of either the AWQ revision below or the sglang image would serve
+    # KV pages computed by the previous one. Bump this suffix with either of them.
+    - name: SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR
+      value: /hicache/sglang-v0.5.15_awq-f541031d
+    # Unset defaults to unbounded; a full L2 measured 14G on 2026-07-27.
+    - name: SGLANG_HICACHE_FILE_BACKEND_MAX_SIZE
+      value: 64Gi
+    # MAX_SIZE alone is blind to the other tenants of control-1's shared 500G root, so
+    # hold a floor well above kubelet's ~50G nodefs eviction threshold.
+    - name: SGLANG_HICACHE_FILE_BACKEND_MIN_FREE_SPACE
+      value: 100Gi
   extraVolumes:
     - name: triton-cache
       persistentVolumeClaim:
         claimName: qwen36-27b-triton-cache
+    - name: hicache
+      persistentVolumeClaim:
+        claimName: qwen36-27b-hicache
     - name: dshm
       emptyDir:
         medium: Memory
         sizeLimit: 8Gi
   extraVolumeMounts:
     - name: triton-cache
       mountPath: /cache
+    - name: hicache
+      mountPath: /hicache
     - name: dshm
       mountPath: /dev/shm
   probeOverrides:
@@ -181,5 +216,8 @@ spec:
     - --schedule-conservativeness
     - "0.1"
     - --enable-mixed-chunk
+    # write_through_selective gates promotion on hit_count >= 2, which left L3 empty.
     - --hicache-write-policy
-    - write_through_selective
+    - write_through
+    - --hicache-storage-backend
+    - file