[ROCm] Enable DSv4 multi-stream compressor overlap on HIP

[karverma-amd]

Summary

The DSv4 multi-stream compressor overlap (SGLANG_OPT_USE_MULTI_STREAM_OVERLAP) is present but inert/broken on ROCm:

1. arg_groups/model_hook.py force-disables the env on HIP, overriding an explicit user setting.
2. The per-layer alt_streams gate in deepseek_v4.py only wires _is_cuda/_is_npu, so on HIP every layer gets alt_streams = None and the fork never engages.
3. _forward_prepare_multi_stream_hip still writes the non-unified SWA buffer, which crashes on the unified KV pool (swa_kv_pool is None, see fix(rocm): support unified KV in DSV4 multi-stream #38662).

This PR makes the overlap opt-in on HIP (default behavior unchanged — it stays off unless the env is set):

• model_hook: only default-off when the user hasn't set the env (if not is_set(): set(False)).
• deepseek_v4: add _is_hip to the per-layer alt_streams gate.
• _forward_prepare_multi_stream_hip: route Q/KV through the shared _forward_prepare_qkv helper (backend-correct unified_kv store) instead of the drifted SWA write; now returns (q, kv).
• SplitKBufferPool: key the split-K decode scratch per stream, so concurrent side-stream attention cannot clobber the partial-output / partial-LSE tensors.

Depends on the device-resident ordering-edge runtime (ROCm/rocm-systems#11212, see #39282). The overlap forks compressor GEMMs onto side HW queues; without the ordering-edge runtime the per-dispatch multi-queue cost dominates and the concurrent kernels contend (the TP all-reduce alone inflates ~40%), turning the win into a regression. On the ordering-edge runtime the overlap is clean.

Performance

MI355X (gfx950), DeepSeek-V4, TP8, 8k in / 1k out, GPU_MAX_HW_QUEUES=4, DSPARK, fp8 KV. Baseline = overlap off; Patch = overlap on (same image, SGLANG_OPT_USE_MULTI_STREAM_OVERLAP=0/1).

conc	TPOT baseline (ms)	TPOT patch (ms)	TPOT Δ	TTPS/GPU baseline (tok/s)	TTPS/GPU patch (tok/s)	TTPS Δ
1	5.98	5.47	−8.5%	158.5	172.4	+8.8%
4	6.42	6.13	−4.5%	243.6	247.0	+1.4%
8	7.70	7.26	−5.7%	827.3	865.7	+4.6%
16	10.36	10.24	−1.2%	1184.5	1193.7	+0.8%
32	14.89	14.97	+0.5%	1574.8	1566.2	−0.5%

Low-concurrency win (−4.5 to −8.5% TPOT at conc 1‑8, matching the vLLM/ATOM 8k low-conc band); neutral at higher concurrency (the compressor is a smaller fraction of a larger step). No effect on TTFT (decode/verify-only).

Command (perf)

# Baseline: SGLANG_OPT_USE_MULTI_STREAM_OVERLAP=0 ; Patch: =1  (same image)
docker run --rm --device=/dev/kfd --device=/dev/dri --ipc=host --shm-size 32g \
  -e GPU_ARCHS=gfx950 -e GPU_MAX_HW_QUEUES=4 \
  -e SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
  -e SGLANG_USE_AITER=1 -e AITER_BF16_FP8_MOE_BOUND=0 \
  -e SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 -e TORCH_BLAS_PREFER_HIPBLASLT=1 \
  -e SGLANG_OPT_USE_MULTI_STREAM_OVERLAP=1 \
  <image> python3 -m sglang.launch_server \
    --model-path <DeepSeek-V4> --tensor-parallel-size 8 \
    --attention-backend dsv4 --enable-deepseek-v4-fp4-indexer \
    --page-size 256 --swa-full-tokens-ratio 0.15 --kv-cache-dtype fp8_e4m3 \
    --enforce-shared-experts-fusion --context-length 11264 \
    --chunked-prefill-size 16384 --mem-fraction-static 0.90 \
    --max-running-requests 64 --cuda-graph-max-bs-decode 64 \
    --speculative-algorithm DSPARK --speculative-dspark-block-size 6 \
    --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 7

# client: sweep concurrency 1,4,8,16,32 at isl=8192 osl=1024

Accuracy (GSM8K)

benchmark/gsm8k/bench_sglang.py --num-questions 1319 --parallel 1319, greedy (no spec decode), real outputs (no acceptance simulation):

	Accuracy	Invalid
Baseline (overlap off)	0.948	0.000
Patch (overlap on)	0.945	0.000

Δ = 0.3% (≈4/1319) — within run-to-run nondeterminism (fp8 KV / MoE reductions), 0 invalid on both. The overlap only reorders which stream runs the compressor GEMM; the math is unchanged.

Command (accuracy)

docker run --rm --device=/dev/kfd --device=/dev/dri --ipc=host --shm-size 32g \
  -e GPU_ARCHS=gfx950 -e GPU_MAX_HW_QUEUES=4 \
  -e SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton -e SGLANG_USE_AITER=1 \
  -e SGLANG_OPT_USE_MULTI_STREAM_OVERLAP=1 \
  <image> python3 -m sglang.launch_server \
    --model-path <DeepSeek-V4> --tensor-parallel-size 8 \
    --attention-backend dsv4 --enable-deepseek-v4-fp4-indexer \
    --page-size 256 --kv-cache-dtype fp8_e4m3 --enforce-shared-experts-fusion \
    --context-length 16384 --cuda-graph-max-bs-decode 256 --max-running-requests 1319
# then:
python3 benchmark/gsm8k/bench_sglang.py --num-questions 1319 --parallel 1319 --port <port>

Profile (conc 4, 8k/1k) — why it wins on the ordering-edge runtime

	main-stream busy	side-stream busy	concurrency	wall
baseline (off)	1269.5 ms	1.1 ms	1.000x	1665.5 ms
patch (on)	1141.0 ms	161.4 ms	1.095x	1561.1 ms

Enabling the overlap moves ~128 ms of compressor work off the main stream onto 4 active side streams; wall drops −6.3%, mirroring the −4.5% TPOT.

---

CI States

Latest PR Test (Base): ❌ Run #34775797271
Latest PR Test (Extra): ❌ Run #34775797239
Latest PR Test (AMD ROCm 10): ❌ Run #34775797382

[karverma-amd]

Depends on #39282 (device-resident ordering-edge runtime, ROCm/rocm-systems#11212). That runtime is what makes this overlap a win on ROCm; on a stock nightly without it, the multi-queue dispatch contention turns the same code into a regression.