[DeepSeek-V4]support dspark c128 online compressor

[BJWang-ant]

Motivation

support dspark c128 online compressor

using env SGLANG_OPT_USE_ONLINE_COMPRESS=1 and SGLANG_EXPERIMENTAL_ONLINE_C128_DSPARK=1 to open dspark c128 online compressor.

this is Precision test results.

  SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2=1 \
  SGLANG_OPT_USE_ONLINE_COMPRESS=1 \
  SGLANG_EXPERIMENTAL_ONLINE_C128_DSPARK=1 \
  sglang serve \
    --trust-remote-code \
    --model-path /upfs/models/deepseek-ai/DeepSeek-V4-Pro-DSpark/ \
    --tp 8 \
    --moe-runner-backend flashinfer_mxfp4 \
    --chunked-prefill-size 8192 \
    --swa-full-tokens-ratio 0.1 \
    --mem-fraction-static 0.85 \
    --tool-call-parser deepseekv4 \
    --reasoning-parser deepseek-v4 \
    --enable-cache-report \
    --speculative-algorithm DSPARK \
    --disable-flashinfer-autotune \
    --enable-metrics \
    --host 127.0.0.1 \
    --port 30001

after using c128-online-compressor full token size is 3167232.

before using c128-online-compressor full token size is 2553344.

---

CI States

Latest PR Test (Base): ❌ Run #32471432842
Latest PR Test (Extra): ❌ Run #32471432471
Latest PR Test (AMD ROCm 7.2): ❌ Run #32471432957

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[BJWang-ant]

@Fridge003 HI could you help me to review code?

[Leoyzen]

Online c128 + DSpark + compact ragged verify crashes on first request.

Setup: DSV4-Flash-0731 FP8, 4×H200, TP4, SGLANG_RAGGED_VERIFY_MODE=compact, all three env vars on. Server boots fine, first verify request dies.

Crashes at c128_online_v2.cuh:687:

RuntimeCheck(0 < extend_len && extend_len <= seq_len);

Compact mode pads the batch to the CUDA graph bucket size, so idle requests have verify_len=0. MTP path is fine (extend_lens are all num_draft_tokens), but DSpark ragged path passes ragged_layout.verify_lens_cpu which has zeros in it, tripping the check.

Fix: either filter out verify_len=0 entries before calling plan_prefill, or skip extend_len==0 in the C++ loop.

Workaround: turn off SGLANG_OPT_USE_ONLINE_COMPRESS. The old NotImplementedError guard was blocking this untested path.

[Leoyzen]

Tested this PR on our DSV4 + DSPARK production environment, crashes after serving traffic for a bit:

NCCL: Cuda failure 'an illegal memory access was encountered' → exit 137

Bisected — confirmed this PR is the trigger. Env: 4x L20X TP, moe-runner-backend=flashinfer_mxfp4, hicache-storage-backend=mooncake, SGLANG_OPT_USE_ONLINE_COMPRESS and SGLANG_EXPERIMENTAL_ONLINE_C128_DSPARK not set.

Spent some time digging in. Quick summary: all online C128 kernel and pool changes are env-gated and don't activate without the env vars. The problem is in a few ungated changes.

speculative_moe_backend_context / speculative_moe_a2a_backend_context are no-ops

speculative_moe_runner_backend defaults back to moe_runner_backend via the override in overrides.py, and speculative_a2a_backend defaults to moe.a2a_backend in initialize_moe_config. Our config is flashinfer_mxfp4 + mooncake, so both context managers just set the same values back — effectively doing nothing.

moe.disable_fp4_allgather = True only affects should_use_flashinfer_cutlass_moe_fp4_allgather(), which requires flashinfer_cutlass + a2a_backend == none. We have neither, so also a no-op.

The actual trigger is the record_stream changes

The PR adds several record_stream calls in dspark_verify.py and dspark_worker_v2.py (previously only batch.seq_lens was recorded; now req_pool_indices, batch.input_ids, out_cache_loc, etc. are too). These are defensive — preventing use-after-free from spec v2 mid-forward rebinds — but they shift the CUDA caching allocator's recycling pattern enough to expose a latent OOB / use-after-free in the base code.

Suspect the overflowing kernel is somewhere in the mooncake A2A dispatch/combine path, flashinfer_mxfp4 MoE workspace, or DSV4 compressor KV-score buffers. Haven't fully pinned it down yet — planning to run compute-sanitizer --tool memcheck to catch the exact kernel.

Workaround for now: revert just the record_stream additions (dspark_verify.py:275-276/281 + dspark_worker_v2.py:528-531), restore the base memory layout, and it's stable. But that hides the bug rather than fixing it — the latent OOB can resurface under different memory pressure or batch shapes.

[BJWang-ant]

Tested this PR on our DSV4 + DSPARK production environment, crashes after serving traffic for a bit:

NCCL: Cuda failure 'an illegal memory access was encountered' → exit 137

Bisected — confirmed this PR is the trigger. Env: 4x L20X TP, moe-runner-backend=flashinfer_mxfp4, hicache-storage-backend=mooncake, SGLANG_OPT_USE_ONLINE_COMPRESS and SGLANG_EXPERIMENTAL_ONLINE_C128_DSPARK not set.

Spent some time digging in. Quick summary: all online C128 kernel and pool changes are env-gated and don't activate without the env vars. The problem is in a few ungated changes.

speculative_moe_backend_context / speculative_moe_a2a_backend_context are no-ops

speculative_moe_runner_backend defaults back to moe_runner_backend via the override in overrides.py, and speculative_a2a_backend defaults to moe.a2a_backend in initialize_moe_config. Our config is flashinfer_mxfp4 + mooncake, so both context managers just set the same values back — effectively doing nothing.

moe.disable_fp4_allgather = True only affects should_use_flashinfer_cutlass_moe_fp4_allgather(), which requires flashinfer_cutlass + a2a_backend == none. We have neither, so also a no-op.

The actual trigger is the record_stream changes

The PR adds several record_stream calls in dspark_verify.py and dspark_worker_v2.py (previously only batch.seq_lens was recorded; now req_pool_indices, batch.input_ids, out_cache_loc, etc. are too). These are defensive — preventing use-after-free from spec v2 mid-forward rebinds — but they shift the CUDA caching allocator's recycling pattern enough to expose a latent OOB / use-after-free in the base code.

Suspect the overflowing kernel is somewhere in the mooncake A2A dispatch/combine path, flashinfer_mxfp4 MoE workspace, or DSV4 compressor KV-score buffers. Haven't fully pinned it down yet — planning to run compute-sanitizer --tool memcheck to catch the exact kernel.

Workaround for now: revert just the record_stream additions (dspark_verify.py:275-276/281 + dspark_worker_v2.py:528-531), restore the base memory layout, and it's stable. But that hides the bug rather than fixing it — the latent OOB can resurface under different memory pressure or batch shapes.

ok, i get it, could you please provide the command that caused the error? Previously, I conducted all my experiments on the B200. For the H200, I need to verify it again.

[Leoyzen]

@BJWang-ant Repro info below.

Note the key finding first: the crash does not need the c128 env vars. Our bisect landed on this PR with a default DSpark config (SGLANG_OPT_USE_ONLINE_COMPRESS and SGLANG_EXPERIMENTAL_ONLINE_C128_DSPARK both unset) — the trigger is the ungated record_stream additions in dspark_verify.py / dspark_worker_v2.py.

They shift the CUDA caching allocator's recycling pattern and expose a latent OOB/use-after-free in base code. So it should reproduce on B200 as well, though allocator layout differences may change timing.

1. Production crash: Illegal memory access → exit 137 (record_stream)

Env: 4×H200, TP4, DeepSeek-V4-Flash FP8 (0731 build).

python3 -m sglang.srt.launch_server \
   --model-path <DSV4-flash-0731-FP8 path> \
   --trust-remote-code \
   --tp 4 \
   --speculative-algorithm DSPARK \
   --moe-runner-backend flashinfer_mxfp4 \
   --hicache-storage-backend mooncake \
   --mem-fraction-static 0.85 \
   --enable-metrics \
   --host 0.0.0.0 --port 30001 

No SGLANG_OPT_USE_ONLINE_COMPRESS, no SGLANG_EXPERIMENTAL_ONLINE_C128_DSPARK, no SGLANG_RAGGED_VERIFY_MODE override.

Load pattern: does not crash on the first request. Died after a few minutes of sustained multi-request traffic. Single curl / one-shot bench won't reproduce it — needs a steady stream of concurrent requests.

Local fix (applied on our integration branch): reverted the group record_stream calls in _forward_prepared_verify (dspark_verify.py) and _forward_decode (dspark_worker_v2.py), restoring the base single-line batch.seq_lens.record_stream. Stable since. That only papers over the latent OOB, which still needs a real fix — I'd suggest reproducing with compute-sanitizer --tool memcheck to pin the actual kernel. Can share the revert patch if useful.

2. Compact ragged verify + c128: crash on first request

Different issue, needs the c128 envs + SGLANG_RAGGED_VERIFY_MODE=compact:

SGLANG_OPT_USE_ONLINE_COMPRESS=1 \
SGLANG_EXPERIMENTAL_ONLINE_C128_DSPARK=1 \
SGLANG_RAGGED_VERIFY_MODE=compact \
python3 -m sglang.srt.launch_server \
  [same flags as above]

Crashes at c128_online_v2.cuh:687 (RuntimeCheck(0 < extend_len && extend_len <= seq_len)). Compact mode pads the batch to the CUDA graph bucket size, so idle requests carry verify_len=0 into the async verify plan. Local fix: skip extend_len==0 rows in _plan_prefill_partial (c128_online_v2.cuh) and plan_compress_prefill (c_plan.cuh). Can open a PR with it.

[BJWang-ant]

@BJWang-ant Repro info below.

Note the key finding first: the crash does not need the c128 env vars. Our bisect landed on this PR with a default DSpark config (SGLANG_OPT_USE_ONLINE_COMPRESS and SGLANG_EXPERIMENTAL_ONLINE_C128_DSPARK both unset) — the trigger is the ungated record_stream additions in dspark_verify.py / dspark_worker_v2.py.

They shift the CUDA caching allocator's recycling pattern and expose a latent OOB/use-after-free in base code. So it should reproduce on B200 as well, though allocator layout differences may change timing.

1. Production crash: Illegal memory access → exit 137 (record_stream)

Env: 4×H200, TP4, DeepSeek-V4-Flash FP8 (0731 build).

python3 -m sglang.srt.launch_server \
   --model-path <DSV4-flash-0731-FP8 path> \
   --trust-remote-code \
   --tp 4 \
   --speculative-algorithm DSPARK \
   --moe-runner-backend flashinfer_mxfp4 \
   --hicache-storage-backend mooncake \
   --mem-fraction-static 0.85 \
   --enable-metrics \
   --host 0.0.0.0 --port 30001 

No SGLANG_OPT_USE_ONLINE_COMPRESS, no SGLANG_EXPERIMENTAL_ONLINE_C128_DSPARK, no SGLANG_RAGGED_VERIFY_MODE override.

Load pattern: does not crash on the first request. Died after a few minutes of sustained multi-request traffic. Single curl / one-shot bench won't reproduce it — needs a steady stream of concurrent requests.

Local fix (applied on our integration branch): reverted the group record_stream calls in _forward_prepared_verify (dspark_verify.py) and _forward_decode (dspark_worker_v2.py), restoring the base single-line batch.seq_lens.record_stream. Stable since. That only papers over the latent OOB, which still needs a real fix — I'd suggest reproducing with compute-sanitizer --tool memcheck to pin the actual kernel. Can share the revert patch if useful.

2. Compact ragged verify + c128: crash on first request

Different issue, needs the c128 envs + SGLANG_RAGGED_VERIFY_MODE=compact:

SGLANG_OPT_USE_ONLINE_COMPRESS=1 \
SGLANG_EXPERIMENTAL_ONLINE_C128_DSPARK=1 \
SGLANG_RAGGED_VERIFY_MODE=compact \
python3 -m sglang.srt.launch_server \
  [same flags as above]

Crashes at c128_online_v2.cuh:687 (RuntimeCheck(0 < extend_len && extend_len <= seq_len)). Compact mode pads the batch to the CUDA graph bucket size, so idle requests carry verify_len=0 into the async verify plan. Local fix: skip extend_len==0 rows in _plan_prefill_partial (c128_online_v2.cuh) and plan_compress_prefill (c_plan.cuh). Can open a PR with it.

1.I rebased the code on the latest main branch. Currently, using the following server and benchmark commands, I have not found the problem of illegal access to video memory as mentioned at b200. The max full token usage is 0.96,the duration of the benchmark test is approximately 70 minutes.
server command:
sglang serve
--trust-remote-code
--model-path /upfs/models/deepseek-ai/DeepSeek-V4-Flash-0731/
--tp 2
--moe-runner-backend flashinfer_mxfp4
--tool-call-parser deepseekv4
--reasoning-parser deepseek-v4
--enable-cache-report
--speculative-algorithm DSPARK
--disable-flashinfer-autotune
--mem-fraction-static 0.55
--enable-metrics
--host 127.0.0.1
--port 30001

benchmark command:
rounds=(30)
seed_ori=42
for i in "${!rounds[@]}"; do
round=${rounds[$i]}
python3 ../sglang/benchmark/hicache/bench_multiturn.py
--model-path /models/deepseek-ai/DeepSeek-V4-Flash-0731
--disable-random-sample
--request-length 8192
--sub-question-input-length 4096
--output-length 2048
--num-clients 64
--num-rounds "${round}"
--max-parallel 32
--request-rate 16
--ready-queue-policy random
--disable-auto-run
--enable-round-barrier
--host "127.0.0.1"
--port "30001"
--seed 422
done

The second issue is still being resolved by me.

[BJWang-ant]

@Leoyzen
#35848 Hi,I also added a pr to fix the bug in the 'extend_len' interpretation.Regarding the second question, after rebasing the latest main branch code, I conducted new tests, including enabling dspark+compact or enabling dspark+no-compact. None of these scenarios reproduced the issue. The log and startup commands are as follows. But indeed, I think what you said is correct. The current code still has the problem of tensor lifecycle errors. It's just that my code increases the likelihood of such issues occurring.
flash-log.zip

I have updated the code. You can test it locally to see if you can still reproduce the result.