diff --git a/docs/llm-hosting/sglang-blockers.md b/docs/llm-hosting/sglang-blockers.md index 676e976801..f2d288693d 100644 --- a/docs/llm-hosting/sglang-blockers.md +++ b/docs/llm-hosting/sglang-blockers.md @@ -195,10 +195,9 @@ the pinned fork tree carries stock v0.5.14 hicache code (no fork patch touches i population request took 113.3s. After a clean pod restart, the token-exact post-restart extension took 70.7s, but the file-backend metric recorded only 4,538 backed-up tokens, with zero storage prefetches and zero `storage_HiCacheFile` hits. Aggregate completion throughput for C1/C4/C8 was -11.16/8.74/14.09 tok/s; the C8 sample overlapped real traffic. The validated direct I/O, ratio 1.5 -L2, and `write_through_selective` policy remain enabled; do not propose unverified cache tweaks. -Rollback does not remove `/cache/sglang/hicache`; it can contain prompt-derived data and requires -explicit operator approval before deletion. +11.16/8.74/14.09 tok/s; the C8 sample overlapped real traffic. Direct I/O and ratio 1.5 L2 remain +validated and enabled; `write_through_selective` was replaced on 07-27, see the retest below. Do +not propose unverified cache tweaks. **Correction (2026-07-27): "failed restart recovery" was the wrong conclusion.** Two defects sat upstream of that measurement, and neither is a property of L3. @@ -220,10 +219,47 @@ upstream of that measurement, and neither is a property of L3. 4,538-of-61,600 backup figure. Any L3 test must run under plain `write_through`. The 07-15 run did not crash, so it cannot have been on an image with defect 1 — meaning it is not -comparable to current builds and should not be quoted as evidence either way. **L3 is untested, not -failed.** Retest sequence: rebuild with `libssl-dev` → `write_through` → synthetic prefill to prove -the backend initialises → populate/restart/extension probe. Not on the production replica: the -07-27 attempt took prod down mid-traffic and cost ~40 min across two reloads. +comparable to current builds and should not be quoted as evidence either way. **L3 was untested, +not failed.** + +**Retest (2026-07-27): L3 works.** Run on image `sha256:72d934d3` under plain `write_through`, +with `--hicache-storage-backend file` and `SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR`. + +| stage | evidence | +|---|---| +| backend initialises | synthetic prefill served; pod ran 22 min under live traffic, 0 restarts | +| L3 writes | 268,176 files / 14 GB on disk, `backuped_tokens_total` 267,978 | +| L2 saturates | `hicache_host_used_tokens` 267,978 of 274,861 | +| survives restart | after a cold start, `prefetched_tokens_total{storage_backend="file"}` 63,205 | + +The restart figures close exactly: 73,120 newly backed + 63,205 prefetched = 136,325 +`hicache_host_used_tokens`, so every L2 token is accounted for as either a fresh write or a +read-back from disk. Populate (58,071 tok) PP 196.9 tok/s; extension probe (58,209 tok) PP +252.9 tok/s, TG 0.56 tok/s. Both ran with production traffic on the GPU, so both are floors, +not clean numbers. + +Three operational notes. + +**Sizing.** The evictor is unbounded by default and `openebs-hostpath` enforces no quota, so +`MAX_SIZE` (64Gi) and `MIN_FREE_SPACE` (100Gi) are the only things between L3 and control-1's +shared 500G root; the free-space floor is what keeps a full cache from pushing the node under +kubelet's ~50G nodefs eviction threshold. L3 lives on its own `qwen36-27b-hicache` PVC rather +than sharing the triton cache, so its footprint is visible where capacity is planned. + +**The cache key does not cover the weights or the engine.** `get_hash_str` is +`sha256(token_ids, prior_hash, page_size)` (`mem_cache/utils.py:106`) and the on-disk name only +appends `config_suffix = f"_{model_name}"` plus TP/PP/CP ranks (`hicache_storage.py:336-346`). +Nothing in it encodes the weights revision, the KV dtype or the KV layout, while +`served-model-name` stays stable across Renovate bumps of both the AWQ revision and the sglang +image. Either bump would otherwise serve KV pages computed by the previous one, which is a +wrong-output risk rather than a cache miss. `STORAGE_DIR` therefore carries a revision suffix +(`/hicache/sglang-v0.5.15_awq-f541031d`) that **must be bumped by hand with either pin**. It sits +directly above them in the manifest to keep the three visible together; the durable fix is +upstream folding a model/engine fingerprint into `config_suffix`. + +**Deletion needs approval.** Rollback does not remove the directory; it holds prompt-derived +data. The 07-27 test data still sits at the old `/cache/sglang/hicache` path, orphaned by the +move to a dedicated PVC. **`hicache-write-policy write_back` trialled and reverted (2026-07-09):** kept alongside the L3 removal above as a still-valid L1→L2 (GPU→host) optimization — synthetic testing showed 0 aborts and lower diff --git a/docs/llm-hosting/vllm-vs-sglang-2026-07.md b/docs/llm-hosting/vllm-vs-sglang-2026-07.md index b9d9bbfc89..f298b810c0 100644 --- a/docs/llm-hosting/vllm-vs-sglang-2026-07.md +++ b/docs/llm-hosting/vllm-vs-sglang-2026-07.md @@ -129,8 +129,9 @@ Three things combine: impossible. 2. **ETag is not integrity.** The etag file is written from response headers, which arrive before the body, so it can persist for a transfer that later dies. The next - boot sends `If-None-Match`, gets `304`, and prints `Model artifact ... revalidated` - over a corrupt file. Nothing checks size against `Content-Length`. + boot sends `If-None-Match` and prints `Model artifact ... revalidated` over a corrupt + file. Nothing checks size against `Content-Length`. Note the message is printed on any + curl exit 0, not on a `304`, so it carries no information about what actually happened. 3. **The fallback keeps bad data.** `elif [ -f "$dest" ]` treats presence as validity. There is currently no content-integrity mechanism at all: sha256 pinning was rejected on @@ -175,9 +176,21 @@ served over a truncated weight file. For LFS artifacts that cannot happen, since always re-fetched. The stale-etag trap is real for config files and for the *bookkeeping*, but the practical failure mode on weights is cost, not silent corruption. +A third defect compounds both: `-o "$dest"` opens the destination for writing at request +time, so the valid cached copy is truncated at the start of every restart before curl knows +whether it needs the body. The cache is not merely ignored, it is destroyed. + +Confirmed again on the 07-27 L3 restart, which is the clearest single reading: all nine +artifacts logged `revalidated` while `model-downloader` ran 14m21s (18:08:38Z to 18:22:59Z), +which at the observed rate is the whole weight set. + This is operator-side (the downloader script is generated by llmkube), so it needs an -upstream fix: follow redirects with the conditional intact, or range/size check the local -file before deciding to transfer. +upstream fix: download to `$dest.tmp` + `mv` so a fetch can never truncate the copy it is +revalidating, and log `revalidated` only on an actual `304`. A `HEAD` `Content-Length` +compare is worth adding as a cheap truncation preflight, but it is not cache validation: +equal size proves neither freshness nor integrity, so it supplements the conditional GET +rather than replacing it. Filed as +[defilantech/LLMKube#1309](https://github.com/defilantech/LLMKube/issues/1309). ## Open items diff --git a/kubernetes/apps/ai/llmkube/models/qwen36-27b-sglang.yaml b/kubernetes/apps/ai/llmkube/models/qwen36-27b-sglang.yaml index 57532bdf6a..0a63b0fb75 100644 --- a/kubernetes/apps/ai/llmkube/models/qwen36-27b-sglang.yaml +++ b/kubernetes/apps/ai/llmkube/models/qwen36-27b-sglang.yaml @@ -25,6 +25,20 @@ spec: requests: storage: 10Gi --- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: qwen36-27b-hicache + namespace: ai +spec: + accessModes: + - ReadWriteOnce + storageClassName: openebs-hostpath + resources: + # hostpath enforces no quota, so the real bound is the evictor's MAX_SIZE below. + requests: + storage: 64Gi +--- # yaml-language-server: $schema=https://k8s-schemas.home-operations.com/inference.llmkube.dev/model_v1alpha1.json apiVersion: inference.llmkube.dev/v1alpha1 kind: Model @@ -63,6 +77,7 @@ spec: runtime: sglang # Operator default binds :: (IPv6-only on this IPv4 cluster); override to v4. bindAddress: 0.0.0.0 + # L3 needs OpenSSL headers in the runtime stage; earlier digests kill the scheduler. image: ghcr.io/tanguille/sglang-rdna4:v0.5.15-gfx1201@sha256:72d934d335529934bf73cd480b728cd08ae61178c9204dc37a025302b31fd61e containerPort: 30000 endpoint: @@ -107,10 +122,28 @@ spec: value: "0" - name: TRITON_CACHE_DIR value: /cache/sglang/triton + # Correctness knob, not a layout preference. The on-disk page name is + # sha256(token_ids, prior_hash, page_size) + "_{model_name}_{tp_rank}_{tp_size}", + # so nothing in it encodes the weights revision, the KV dtype or the KV layout. + # Same served-model-name plus same tokens hits the same file, which means a + # Renovate bump of either the AWQ revision below or the sglang image would serve + # KV pages computed by the previous one. Bump this suffix with either of them. + - name: SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR + value: /hicache/sglang-v0.5.15_awq-f541031d + # Unset defaults to unbounded; a full L2 measured 14G on 2026-07-27. + - name: SGLANG_HICACHE_FILE_BACKEND_MAX_SIZE + value: 64Gi + # MAX_SIZE alone is blind to the other tenants of control-1's shared 500G root, so + # hold a floor well above kubelet's ~50G nodefs eviction threshold. + - name: SGLANG_HICACHE_FILE_BACKEND_MIN_FREE_SPACE + value: 100Gi extraVolumes: - name: triton-cache persistentVolumeClaim: claimName: qwen36-27b-triton-cache + - name: hicache + persistentVolumeClaim: + claimName: qwen36-27b-hicache - name: dshm emptyDir: medium: Memory @@ -118,6 +151,8 @@ spec: extraVolumeMounts: - name: triton-cache mountPath: /cache + - name: hicache + mountPath: /hicache - name: dshm mountPath: /dev/shm probeOverrides: @@ -181,5 +216,8 @@ spec: - --schedule-conservativeness - "0.1" - --enable-mixed-chunk + # write_through_selective gates promotion on hit_count >= 2, which left L3 empty. - --hicache-write-policy - - write_through_selective + - write_through + - --hicache-storage-backend + - file