[Feature] Enable DeepSeek V4 IndexCache with PD, CP, and HiCache coverage

[feng397]

Motivation

The upstream implementation is #26274. This PR only attempts to forward-port and merge it onto the current main branch. It does not propose a separate IndexCache design. Credit for the original design, implementation, and benchmark results belongs to the authors and contributors of #26274.

#26274 targets an older version of the DeepSeek V4 code and no longer applies cleanly after the model, attention, cache, and disaggregation paths evolved. This PR keeps its core behavior and adapts it to the current interfaces.

IndexCache lets producer (F) C4 layers compute raw top-k indices and allows following shared (S) C4 layers to reuse them. A shared layer skips its own indexer compressor, query/weight computation, logits, and top-k selection. The cached values are logical C4 indices, so each shared layer still translates them through its own page table before running sparse attention.

Modifications

Forward-port the core implementation

• Add index_topk_freq and index_topk_pattern to DeepSeekV4Config.
• Port the producer/shared execution paths to the current C4 indexer.
• Add PyTorch and Triton raw-index-to-page-index implementations.
• Carry raw top-k state through the current DeepSeek V4 model forward.
• Preserve indexer top-k capture and HiSparse page translation.
• Support the current CUDA and ROCm multi-stream paths.

Adapt to the current runtime

• Validate index_topk_freq as a positive integer and reject floats and
  booleans.
• Normalize index_topk_freq=null to the default value 1.
• Validate explicit F/S patterns and require the first C4 layer to be F.
• Fail fast for IndexCache with TBO or pipeline parallelism (pp_size > 1).
• Exchange a layout signature and producer-layer set during P/D bootstrap.
• Add the current configuration example to the DeepSeek V4 documentation.

Test coverage

• CPU unit tests cover F/S planning, invalid configuration, descriptor
  generation, producer-subset validation, and bootstrap consistency.
• GPU unit tests compare the Triton raw-to-page kernel against the PyTorch
  reference for invalid indices, different page sizes, odd top-k sizes,
  non-contiguous tensors, and empty top-k.
• Hardware-gated integration tests cover:
  • single-node FP8 with index_topk_freq=4;
  • FP4 indexer + attention CP + EAGLE on B200;
  • P/D + HiSparse, including the descriptor handshake;
  • HiCache + unified radix tree cache-hit paths.

Current limitations

Combination	Current behavior
TBO	Rejected at startup
Pipeline parallelism (pp_size > 1)	Rejected at startup
Ascend/NPU	Not claimed by this port

Accuracy Tests

Additional local validation used DeepSeek-V4-Flash with thinking enabled on AIME 2025, 30 questions, avg@4:

Configuration	Accuracy
index_topk_freq=1	94.2%
index_topk_freq=4	90.0%

The original accuracy evaluation and methodology are also available in #26274.

Speed Tests and Profiling

Additional local validation used DeepSeek-V4-Flash in a PD disaggregation setup with random token IDs, 100 output tokens.

Throughput at concurrency=1:

Input length	freq=1 input tok/s	freq=4 input tok/s	Input Δ	freq=1 output tok/s	freq=4 output tok/s	Output Δ
20K	17,909	19,303	+7.8%	89.5	96.5	+7.8%
60K	34,386	36,718	+6.8%	57.3	61.2	+6.8%
100K	36,178	40,694	+12.5%	36.2	40.7	+12.5%
256K	36,761	43,260	+17.7%	14.4	16.9	+17.7%

The gain increases with context length, consistent with #26274, which reports approximately 17% throughput improvement at 256K input length. Full benchmark commands and profiling results are available in the upstream PR.

---

CI States

Latest PR Test (Base): ❌ Run #30601244952
Latest PR Test (Extra): ❌ Run #30601244802

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[ChefWu551]

Thanks for picking this up. Since #26274 introduced the original IndexCache design, I can help review the forward-port and validate the behavior against the original benchmarks. For the new PD/CP/HiCache paths, I can also help check whether the original reuse semantics are preserved.

[Leoyzen]

@feng397
We tested this PR on our DSV4 + DSPARK deployment (index_topk_freq=2, moe-runner-backend=flashinfer_mxfp4, hiSparse decode enabled). It crashes within 1-2 minutes:

CUDA error: an illegal memory access was encountered
  at batch_result_processor.py result.copy_done.synchronize()

Bisected — removing this PR gives 22min+ stable, adding it back crashes in 1-2min every time.

Root cause is in hisparse.cuh's load_cache_to_device_buffer_kernel.

The skip_topk path passes the producer layer's raw_indices (via topk_state.prev) to the consumer layer's swap_in_selected_pages → this CUDA kernel. When sequences are short (c4_seq_len < TOPK=512), the producer's topk_transform_512 fills invalid top-k slots with -1. These -1 values flow through topk_state.prev into the consumer's swap-in call.

The kernel's fast path (seq_len <= HOT_BUFFER_SIZE) guards with if (token_pos >= 0), fine. But the slow path (seq_len > HOT_BUFFER_SIZE) doesn't:

// hisparse.cuh ~line 516, slow path miss-copy:
const int64_t src_loc = req_host_cache_locs[miss_token];
//                                        ↑ miss_token == -1 → OOB READ

The hash insertion (~line 306) also doesn't filter -1 — it gets inserted as a valid key into the shared memory hash table, and downstream miss counting + host cache loc indexing blow up.

Additionally, raw_indices_buffer isn't cleared between CUDA graph replays, so stale -1 rows from one request can contaminate a later long-sequence request's swap-in via topk_state.prev.

Suggestions:

1. Add if (token_idx < 0 || token_idx >= seq_len) guard in the slow path, skip invalid entries
2. Clear raw_indices_buffer[:num_reqs] before each decode forward

Env: 4x L20X TP, SGLANG_RAGGED_VERIFY_MODE=compact, max-running-requests=64, mem-fraction-static=0.82, SGLANG_OPT_USE_ONLINE_COMPRESS not set.

[Leoyzen]

Update: After more on-pod debugging, the -1 sentinel theory in my previous comment turned out to be wrong. HOT_BUFFER_SIZE = 4096 and TOPK = 512 — the -1 entries only appear when c4_seq_len < 512 (short sequences), but the slow path only triggers when c4_seq_len > 4096 (long sequences). The two conditions are mutually exclusive, so -1 values never reach the slow path. The raw_indices_buffer is also fully overwritten by topk_transform_512 on each forward (TOPK == self.top_k == 512), so there's no cross-request stale contamination either. My apologies for the misdirection.

Here's what we actually found:

The crash is in target_verify CUDA graph replay, not decode. The stack trace points to CUDAGraph::replay in the target_verify path. With cuda-graph-max-bs=1 (eager mode), 40/40 requests pass cleanly. The crash only happens with CUDA graph + sustained load (~320 requests with CUDA_LAUNCH_BLOCKING=1).

Most likely trigger: raw_indices allocation order change in forward_c4_indexer. The PR reorders the if/elif branches for raw_indices allocation. In target_verify mode with index_topk_freq=2, producer layers hit the return_topk_indices or capture_enabled branch and allocate raw_indices = torch.empty_like(c4_sparse_page_indices). In base code, this branch is never reached (different order, capture_enabled=False), so raw_indices = None. The extra torch.empty_like + clone() + masked_fill_() + copy_() ops recorded in the CUDA graph perturb the graph memory pool layout. We believe this exposes a latent OOB in one of the base-code kernels — but we couldn't pin down the exact kernel without compute-sanitizer --tool memcheck (ran it but it's too slow with graph replay to reach the crash window).

What we ruled out (all verified on pod):

• DSV4RawVerifyMetadata.copy_() assign-vs-copy: instrumented, chosen and temp always share the same extend_seq_lens_buffer, values correct per-request
• init_forward_metadata_in_graph added at line 1205: no-op, metadata already upgraded
• c_plan.cuh verify_width: traced through kernel0, both versions produce identical write plans
• transform_raw_c4_indices_to_page_indices Triton kernel: clamps outputs, validates inputs, no OOB
• page_table bounds: instrumented, no OOB

Workaround: Skip raw_indices allocation in target_verify mode (return_topk_indices=True but hisparse_decode=False and c4_sparse_raw_indices=None). Consumer falls back to full indexer — same as index_topk_freq=1. The IndexCache benefit in target_verify is minimal since hiSparse swap-in isn't active there anyway.

Would appreciate it if you could run compute-sanitizer on your side to identify the exact overflowing kernel. Happy to share crash logs or pod instrumentation data.

[stewtong]

Independent validation of this branch on 8x NVIDIA B200 SXM (SM100) with DeepSeek-V4-Flash-0731, checkpoint revision 7872f01b. Branch head e9100d6 built editable on top of the sglang:nightly-dev-20260731 base image (embedded sglang 3abbc56, torch 2.11.0+cu130). Launch shape: monolithic single-node TP8, megamoe, context_length 1048576, chunked prefill 8192, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, index_topk_freq 1/2/4 via --json-model-override-args. Reporter context: independent benchmarking on Nebius B200 capacity.

Three findings: an SM100 startup crash in decode CUDA-graph capture (the actionable item, reported first), the perf trend on real prose, and an early-stop quality observation.

1. Startup crash: CUDA illegal memory access in decode CUDA-graph capture on SM100, including at inert freq=1

The branch crashes at startup about 8 minutes in, during decode CUDA-graph capture (decode_cuda_graph_runner.py:853, capture, with the NCCL watchdog abort on the same IMA). This fires even at index_topk_freq=1, where every CSA layer still computes its own top-k. One-variable attribution ladder, same node, same image, same checkpoint, one change per row:

Attempt	branch?	fp4-indexer	decode graphs	Result
1 (17:29Z)	yes	on	on	CUDA illegal memory access in decode CUDA-graph capture; rank-5 scheduler abort at startup (~8 min in)
2 (17:41Z)	yes	off	on	identical crash in the same capture step (fp4-indexer EXCLUDED)
control (17:51Z)	NO (stock base image)	off	on	clean startup 18:00:49Z ("Application startup complete"), control-clean/docker.log
A (18:03Z)	yes	off	off	clean startup 18:09:15Z

This narrows the repro to the branch tree versus the stock base image in this config (SM100, monolithic TP8 + megamoe, fp8 indexer); it does not localize to a specific branch change, and I have not verified a root cause.

Two consequences carry through everything below: all perf and quality legs ran with --disable-cuda-graph (eager decode at bs=1), and on the default fp8 indexer path, since the fp4-indexer flag was dropped after run 1 per the ladder.

Relation to @Leoyzen's report above: same error class, different phase. Their crash is 1-2 minutes into serving under freq=2 with flashinfer_mxfp4 + DSpark + HiSparse decode; ours is at startup capture with none of those flags, and reproduces with DSpark/HiSparse absent. I could not reproduce their runtime crash: freq=2 and freq=4 then served continuously for 23 and 20 minutes with zero illegal-memory hits across all cells and suite traffic. Adjacent but distinct; no shared-root-cause claim.

Full docker logs (both crash attempts and the clean control) and the exact launch command are available on request, and I am happy to rerun on a fix.

2. Perf trend on real prose

Setup difference up front, since it bounds what is comparable: your posted throughput table is PD-disaggregated, random token IDs, 100 output tokens, c=1. Mine is monolithic single-node TP8, real prose (cache-busted long-form text, unique cache-buster per request), max 64 output tokens, c=1, 3 reps, temperature 0. Absolute numbers are not cross-comparable; the qualitative growth trend is the comparable quantity.

Context	freq	TTFT p50	input tok/s (mean)	TTFT vs freq1	tok/s vs freq1
128K (128,539 toks)	1	3.00s	32,184
128K	2	2.87s	34,083	-4.3%	+5.9%
128K	4	2.77s	34,598	-7.7%	+7.5%
512K (522,156 toks)	1	19.59s	25,809
512K	2	15.58s	31,287	-20.5%	+21.2%
512K	4	13.54s	35,603	-30.9%	+38.0%
1M (1,044,301 toks)	1	68.94s	15,161
1M	2	55.45s	19,409	-19.6%	+28.0%
1M	4	45.71s	23,202	-33.7%	+53.0%

The first rep at each context is a cold pass (for example 12.4s TTFT on 128K freq=1 versus about 3.0s warm); the p50 filters it out, the tok/s mean includes it.

On real prose the advantage keeps growing with context, +21%/+38% at 512K and +28%/+53% at 1M for freq 2/4, which matches the direction of your random-ID trend past 256K. Prefill is un-graphed in this model config even on stock (prefill backend='disabled'), so the TTFT/input-throughput methodology is not distorted by the eager-decode workaround; wall-clock decode tails are.

3. Early-stop observation at 512K+

finish_reason=stop at 10-14 completion tokens appears at 512K and 1M and escalates with freq: none at 128K at any freq; 0/3, 2/3, 3/3 at 512K for freq 1/2/4; 2/3, 2/3, 3/3 at 1M. Stopped outputs end in model-native stop-format material and are otherwise coherent and ASCII-clean at temp 0. I am reporting this as an observation, not a defect: the direction is consistent with the AIME25 cost you already report (94.2 to 90.0 at freq=4), and our 7-prompt correctness suite (factual recall, needle, arithmetic, code, multi-turn, 4K decode, JSON+tool call) passes at every freq. Two baseline caveats for fair reading: freq=1 (no CSA reuse) already produced 2/3 early stops at 1M, so a context-length component independent of freq cannot be ruled out; and one freq=1 1M length-bound completion degrades into repeated tokens, so long-context continuation quality in this eager-decode config is marginal at baseline as well.

Available on request: per-request raw response JSONs, suite prompt transcripts, server logs, launch scripts.