[GLM-5.3-Flash][HiCache] L3 storage backend support for compressed hybrid DSA (span mode)

[d3lavar]

Motivation

Compressed hybrid DSA models (GLM-5.3-Flash / glm5_next family) cannot use
the L3 HiCache storage tier today. sgl-project#38212 (correctly) refuses to combine the
storage tier with compressed-index pools:

ValueError: Compressed DSA currently supports L2 HiCache only;
storage hashes and transfers require matching page sizes.

The reason is a real granularity mismatch, not a missing flag. For compressed
k-pools, _compressed_index_tree_params (sgl-project#38212) inflates the radix-tree page
to lcm(page_size, pool.page_size * index_kpool) (e.g. 64 → 256) so that one
compressed index row stays atomic: a split inside the group would let children
overwrite a parent's index row that may already be backed up to host memory.
The L3 storage tier, however, hashes keys at tree-page granularity while
transferring one host page per key — with the inflated tree page a single key
corresponds to a span of several physical pages, which the current storage
contract cannot express.

Meanwhile, L3 already works for every other DSA configuration:

• non-compressed hybrid DSA: Mooncake [HiCache & HybridModel] mooncake backend support DSA & mamba model sgl-project/sglang#21259, 3FS [HiCache & HybridModel] 3FS backend support DSA & mamba model sgl-project/sglang#23241, prefetch result sync
  [HiCache]: Optimize hybrid/DSA L3 prefetch result sync and usable-prefix clamping sgl-project/sglang#31443, nightly CI coverage [UnifiedTree]: Add nightly hicache ci for dsa model sgl-project/sglang#25348;
• layer-split CP path: Support Mooncake L3 storage under DSA cache layer split sgl-project/sglang#37122 / [Kernel] Add per-layer LF-to-PF KV transfer for LayerSplit HiCache sgl-project/sglang#37231.

The compressed-kpool family is the only gap. This PR closes it with a
"span mode": the radix-tree page becomes the storage page, and one storage
object covers several consecutive host pages.

Design

Storage page size = radix-tree page size. HiCacheController gains
storage_page_size (defaults to page_size). In span mode it equals the
inflated tree page, so the controller's hash chain, hit queries, host-index
slicing and token accounting align with the tree's node.hash_value by
construction — tree-produced backup keys and controller-computed prefetch keys
are the same keys, no translation layer.

Per-pool strides, derived not assumed. One storage key maps to different
per-pool footprints:

Pool	Per-key entries in a PoolTransfer	Storage object
KV (anchor)	storage_page_size token slots (e.g. 4 × 64)	K concatenated pages
KV-derived sidecars (DSA indexer, indices_from_pool=KV)	same span slots	K concatenated pages
independent state pools (Mamba)	1 checkpoint slot per tree page	single state

The stride is derived from the transfer itself (numel // len(keys)) instead
of being predicted; KV/KV-derived pools must match storage_page_size exactly,
other pools only need a len(keys)-divisible count. This contract was
validated the hard way: the first production run caught Mamba transfers
(one state per tree page) being rejected by an earlier uniform-stride check,
which silently produced KV-only L3 objects that could never satisfy the
component-aware hit check — the derived-stride rule plus a dedicated mixed
KV+state roundtrip test (test_mixed_kv_and_state_pool_roundtrip) close that
hole.

Capability-gated. Backends declare supports_page_spans;
StorageBackendFactory.backend_supports_page_spans() gates both startup
(init_hicache) and runtime attach. The blanket rejection remains for
span-incapable backends — the gate is lifted only where the contract is
actually implemented (file backend in this PR). sgl-project#37122's multi-buffer
_batch_io_v2 packing is the natural path for Mooncake to opt in later.

No behavior change otherwise. For every non-compressed model
storage_page_size == page_size, spans degenerate to one page per key, and
every touched code path executes its original instructions. For non-span
hybrid file storage the derived stride keeps each pool's natural granularity
(KV: page slots per key; independent state pools: one checkpoint slot per
key), matching pre-PR restore behavior — verified by the non-span mixed
roundtrip.

Implementation

• managers/cache_controller.py — storage_page_size on the controller;
  hash chain (_storage_hit_query), hit accounting, host-index slicing and
  completed-token arithmetic run at the storage page; span-aware generic
  page get/set (multi-page dummies, per-page restore) + span propagation in
  _generate_storage_config.
• mem_cache/hybrid_cache/hybrid_cache_controller.py — hybrid
  _storage_hit_query aligned to the storage page; backup_skip token
  accounting counts one storage page per hash key on both sidecar branches
  (ok / failed), so backup-token metrics agree across MLA TP ranks.
• mem_cache/hicache_storage.py — HiCacheStorageConfig.storage_page_size;
  supports_page_spans capability; HiCacheFile span objects: K host pages
  concatenated on batch_set_v2, split per page on batch_get_v2;
  batch_exists_v2 stays key-granular (unchanged).
• mem_cache/storage/backend_factory.py — capability lookup.
• mem_cache/unified_radix_cache.py — the gate becomes a capability check;
  span mode injected at init (live backend patched, since the hybrid stack
  attaches its startup backend inside the controller constructor) and at
  runtime attach.
• Tests: test_hicache_file_span.py (new) and test_hybrid_dsa_hicache.py
  (rejection tests now use a non-span backend; new acceptance tests cover the
  lifted gate for startup and runtime attach).

Relationship to prior work

• Preserve DSA indexes and recurrent checkpoints in HiCache sgl-project/sglang#38212 — this PR's base. It introduces the compressed-index tree-page
  alignment (the lcm) whose existence makes span mode necessary, and the
  L2-only gate whose lift this PR implements. The restore-completeness
  discipline from [Bug] GLM-5.3-Flash (DSA): HiCache host-tier load-back corrupts generation even without speculative decoding — dropped tool calls, degenerate repetition loops (8×H100, TP8) sgl-project/sglang#38031/Preserve DSA indexes and recurrent checkpoints in HiCache sgl-project/sglang#38212 is what the adequacy validation below probes.
• Unified Full KV Cache Layout in L3 sgl-project/sglang#33651 (Unified Full KV Cache Layout in L3) — orthogonal and compatible:
  the unified key namespace digest already includes page size, so spanned and
  non-spanned deployments land in disjoint keyspaces. Unified keys are a
  natural follow-up on top of span mode.
• Support Mooncake L3 storage under DSA cache layer split sgl-project/sglang#37122 / [Kernel] Add per-layer LF-to-PF KV transfer for LayerSplit HiCache sgl-project/sglang#37231 (layer-split Mooncake L3) — solves the layer-split CP
  configuration; this PR covers the non-split compressed path. The
  multi-buffer packing from Support Mooncake L3 storage under DSA cache layer split sgl-project/sglang#37122 is the template for Mooncake to declare
  supports_page_spans = True.
• [HiCache]: Optimize hybrid/DSA L3 prefetch result sync and usable-prefix clamping sgl-project/sglang#31443, [HiCache & HybridModel] mooncake backend support DSA & mamba model sgl-project/sglang#21259, [HiCache & HybridModel] 3FS backend support DSA & mamba model sgl-project/sglang#23241, [UnifiedTree]: Add nightly hicache ci for dsa model sgl-project/sglang#25348 — the working non-compressed DSA L3
  (backends, prefetch sync, CI) that this PR extends to compressed k-pools.
• [HiCache] L3 storage prefetch lifecycle metrics and cross-tier attribution fixes sgl-project/sglang#37503 — per-tier hit metrics; used as the primary production
  verification signal below.
• fix(hicache): accept direct DSA MTP draft pools sgl-project/sglang#36590 — GLM-5.3-Flash MTP draft pools + HiCache compatibility context.

Validation

Unit tests

• span roundtrip through HiCacheFile (2 keys × 4 pages, byte-exact restore
  after wiping the host buffers);
• mixed KV + state-pool backup/restore in a single batch (the stride
  regression described above);
• mixed KV + Mamba roundtrips driven through the controller's own
  _generate_storage_config path (parallel getters stubbed, every config
  field produced by the controller), for both wirings — degenerate non-span
  (storage_page_size == page_size) and span (4x inflation);
• backup_skip completed-token accounting: one storage page per hash on the
  sidecar-ok branch, zero on the sidecar-failed branch;
• batch_exists_v2 prefix semantics under spans;
• transfer length-mismatch guard;
• backend capability lookup;
• hash-chain alignment: get_hash_str(tokens, last_hash, page_size=tree_page)
  chaining equals per-node tree hashing — the invariant that makes
  tree-produced backup keys and controller-computed prefetch keys
  interchangeable.

Full test/registered/unit/mem_cache/ suite: identical results to the
pre-change baseline (1244 passed; the failure set before and after the change
is byte-identical, i.e. zero regressions; the only delta is this PR's own
tests).

Production validation methodology

Ran live on an 8xH100 fleet node (GLM-5.3-Flash, TP8 / DP4 dp-attention /
EP8, EAGLE 5/1/6, fp8 KV), driven end-to-end through the production API
gateway — first under synthetic staged traffic, then left running on real
production fleet traffic (Stage 5). Validation builds were assembled from
this PR's feature branch; the tree matches the stacked commits including
the stride-derivation fix, which re-lands here as 2cb24950 (plus the fp8
KV patch, sgl-project#36904). Three deliberate choices shaped the test:

1. Small pools on purpose. --hicache-size 8 shrinks the host pool to
   670,400 tokens/rank (device pool stays 1,803,584/rank). With production
   sizes the first L3 write needs ~16M unique tokens (~25 min of load); with
   the test sizes the full write→evict→read cycle completes in minutes, so
   bugs surface in minutes too — which is exactly how the Mamba stride bug
   was caught live.
2. Traffic through the real gateway, not a synthetic loopback client:
   the cache-aware router must return repeated prefixes to the node holding
   them for any hit to be observable at all.
3. Verification by counters + content, not by "it answers" — the failure
   mode this whole effort guards against (the [Bug] GLM-5.3-Flash (DSA): HiCache host-tier load-back corrupts generation even without speculative decoding — dropped tool calls, degenerate repetition loops (8×H100, TP8) sgl-project/sglang#38031/Preserve DSA indexes and recurrent checkpoints in HiCache sgl-project/sglang#38212 class) is
   confident garbage from a partially restored context. A request that
   returns HTTP 200 proves nothing; only content-sensitive oracles do.

Stage 1 — startup gate

With --hicache-storage-backend file the server initializes with
Creating storage backend 'file' on all 8 TP ranks and reaches serving
state. On the pre-fix code this exact combination aborts at init with the
L2 HiCache only ValueError, so a clean boot is itself a gate assertion.

Stage 2 — write path under eviction pressure

1800 unique ~20K-token prompts (~40M tokens) driven through the gateway.
sglang:hicache_host_used_tokens climbed 0 → 670,400 and pinned at capacity —
the steady state where every new page displaces an old one into L3. The
pre-fix run had produced _write_span indices length mismatch for mamba: expected 256, got 1 on all ranks at exactly this stage (the Mamba stride
bug, fixed in this PR); post-fix the stage runs clean.

Stage 3 — L3 read path, counted

20 prompts taken from the start of the fill were re-sent. Their eviction is
not assumed but forced and checkable: 9.4M subsequent tokens churned through
per-rank device pools of 1.8M (~2.35M/rank, > capacity) while the host pool
sat pinned at capacity — the re-sent prefixes physically could not survive in
L2. prefill_effective_tokens_total{mode="storage_hit"} moved
0.002M → 0.267M: ~100% of the re-sent prefixes (≈265K tokens) were served
from L3. storage_prefetch_unfulfilled_tokens_total stayed 0 for the entire
session. Node logs independently confirm the restore machinery:

HiCache prefetch success ... completed=25600 matched=25088

completed/matched are exact multiples of the 256-token tree page — span
granularity is directly observable in production logs.

Stage 4 — answer adequacy (the sgl-project#38212 corruption class, probed directly)

The dangerous failure mode of a wrong restore is not an error — it is fluent
garbage: the model answers confidently from a context with holes. HTTP 200
and even low perplexity prove nothing, so the probe is content-sensitive:

• 10 unique ~32K-token contexts (high-entropy filler), each with a secret
  code XZ-NNNN embedded at depths 10%, 20%, …, 90%, 95% — deliberately
  sweeping every span-boundary region, since a partial restore corrupts
  regions of the context, not answers uniformly; a ~32K-token context
  crosses ~125 span objects;
• the question asks only for the code, temperature=0 (greedy),
  max_tokens=256; answers compared across content and
  reasoning_content (the model is a reasoning model);
• cold pass: full prefill, 10/10 codes retrieved — establishes the model
  can solve the probe at all;
• churn: ~9.4M tokens of fresh traffic forces the needle prompts out of
  device and host pools (both bounded, both exceeded);
• L3 pass: identical requests, context restorable only from L3:
  10/10 answers byte-identical to the cold ones, 0 corrupt.

A hole at any span boundary would make the needle at that depth unrecoverable
or the answer incoherent — the probe covers the full depth axis precisely
because restore corruption is positional, not uniform.

Caveat recorded for reproducibility: prefill_effective_tokens_total is
updated on scheduler log intervals, so on idle traffic the counters lag the
events; verdicts were therefore anchored on the content oracle and on
counter deltas measured across stages, not on live scrape values.

Stage 5 — production traffic soak (8xH100 node on live fleet load)

After the staged tests the node went back to serving real production traffic
with the production host-pool size restored (--hicache-size 170 →
14.48M tokens/rank) and carried the morning fleet load through the gateway.
The cache stack under real traffic:

• host pool pinned at 99.98% of capacity (57.9M tokens across the 4 DP
  ranks) with backups and evictions churning continuously — the steady state
  span writes are designed for;
• cache window hit rate reached ~93% under real requests
  (device_hit +34.8M tokens vs +2.8M cold recompute over a 28-minute
  window); TTFT p90 settled at 1.8 s after warm-up;
• the L3 tier registered storage hits on organic traffic (cumulative
  storage_hit doubled within the window) — real request repetition is
  already reaching the disk tier, not only the synthetic replay;
• served outputs stayed clean across the soak: no corruption artifacts,
  stable outputs on repeated requests, no repetition loops, and no malformed
  tool calls in agentic traffic — the exact failure classes a partially
  restored context produces (the [Bug] GLM-5.3-Flash (DSA): HiCache host-tier load-back corrupts generation even without speculative decoding — dropped tool calls, degenerate repetition loops (8×H100, TP8) sgl-project/sglang#38031/Preserve DSA indexes and recurrent checkpoints in HiCache sgl-project/sglang#38212 class);
• zero HTTP 4xx/5xx and zero restarts (cumulative counters strictly
  monotonic across the soak).

Pending

• Restart persistence (L3's headline feature): docker restart wipes
  device + host while the NVMe L3 directory persists; re-running Stage 3/4
  after a restart must reproduce the same storage_hit signature. Mechanics
  are already proven live; this closes the loop on durability.
• Mooncake/NIXL adoption of supports_page_spans (transport-side multi-
  buffer packing per Support Mooncake L3 storage under DSA cache layer split sgl-project/sglang#37122) — next PR in the stack.

Dependency / stacking

Based on pr/hybrid-dsa-hicache-main (sgl-project#38212): the lcm alignment and the gate
live there. fp8 KV (sgl-project#36904) is not required — validation ran on a build that
included it, but the L3 changes are file-disjoint from it.

[d3lavar]

@ormandj hey! This is my first PR to SGLang, so I would like to ask you to do a short review. So once your PR is merged I'll be able to open mine against sglang main branch.

[ormandj]

Thanks for working on this, @d3lavar. I reviewed 07687285 and reproduced two issues:

1. Preserve each pool's transfer granularity. In HiCacheFile._batch_io_v2, storage_page_size applies to every pool, but Mamba supplies one checkpoint slot per key. A 256-token span therefore rejects a valid one-slot checkpoint. This also affects ordinary non-span hybrid file storage: with a 64-token KV page, the controller now passes storage_page_size=64, so the backend expects 64 Mamba slots per key. In a matched CPU roundtrip with two KV keys and two Mamba keys, the base restores both components; this head rejects the Mamba transfers. Please preserve independent state-pool granularity while validating KV and KV-derived spans, and add mixed KV/Mamba roundtrips for both configurations using controller-generated config. Without the checkpoints, the stored KV cannot satisfy a component-complete cache hit.

2. Use storage granularity for skipped-KV backup accounting. The backup_skip branch still multiplies the hash count by self.page_size. With successful sidecar writes, one 256-token storage key reports 64 completed tokens on nonzero MLA TP ranks. Please use storage_page_size and test both branches so the backup-token metrics agree across ranks.

3. Align the description and validation with the submitted revision. The body describes a derived per-pool stride and test_mixed_kv_and_state_pool_roundtrip, but neither is present at this head. These may be in an unpushed revision. Please push the described fix/test and identify the commit used for validation, then update the non-compressed compatibility claim accordingly, so reviewers can connect the reported results to the code being merged.

[d3lavar]

Thanks for the sharp review — all three findings confirmed and fixed. The root cause of (1)/(3): the stack was assembled by cherry-picking the feature branch, and the stride-derivation commit was dropped in transfer — the pushed head predated the fix the production validation had actually run on. My mistake, thanks for catching it.

1. Per-pool granularity — fixed by re-landing the stride-derivation change as 2cb24950: _batch_io_v2 now derives each pool's per-key stride from the transfer itself (KV / KV-derived pools must carry exactly storage_page_size slots per key; independent state pools any len(keys)-divisible count), so Mamba's one-checkpoint-slot transfers are accepted in span mode and in ordinary non-span hybrid file storage.

2. backup_skip accounting — fixed in 3d9aa11b: the sidecar branch counts len(hash) * storage_page_size (was page_size), zero on the failed branch; a two-branch unit test pins both sides.

3. Description/revision alignment — the body now identifies the validation build (feature-branch tree identical to the stacked commits, including 2cb24950, plus the fp8 patch sgl-project#36904) and the non-compressed compatibility claim is reworded: for non-span hybrid file storage the derived stride keeps each pool's natural granularity (KV: page slots per key; state pools: one checkpoint slot per key), matching pre-PR restore behavior.

New tests in 460c8b2f, per your request: mixed KV + Mamba roundtrips driven through the controller's own _generate_storage_config path (parallel getters stubbed, every config field produced by the controller) for both wirings — degenerate non-span and 4x span — plus the backup_skip accounting test for both sidecar branches. Span+hybrid suite green on the new head 460c8b2f (25 passed, 1 skipped; the 3 failures you may see in the run log are host-RAM-pressure flakes of the test runner machine, reproducible on the clean base).