[PoC] Minimal DSpark PD handoff with Prefill PP (no local DSPARK)

[learning-sketch]

Summary

Short-validation PoC for DSpark + PD, allowing Prefill to use --pp-size > 1 without --speculative-algo DSPARK.

Inspired by closed #29705 (metadata-buffer handoff), not the full #31466 RDMA DSPARK_HIDDEN streaming path.

Behavior

Role	Flags	What happens
Prefill	--disaggregation-mode prefill --pp-size > 1, no DSPARK	Capture PP-aware target aux hidden; send last row + short tail via metadata buffers
Decode	--disaggregation-mode decode --speculative-algo DSPARK --pp-size 1	Restore hidden/tail; inject into draft KV before first draft step

Key changes

• New dspark_components/dspark_disaggregation.py helpers (layer-id resolve, metadata sizing, draft-input build, inject)
• MetadataBuffers carries dspark_prefill_tail_*
• Prefill can enable aux capture without local DSPARK worker
• DeepSeek-V4 PP proxy forwards pd_aux_hidden_states_*
• Decode worker injects transferred tail on first decode
• Env knobs: SGLANG_DSPARK_PD_ENABLE_PREFILL_CAPTURE, SGLANG_DSPARK_PD_TARGET_LAYER_IDS, SGLANG_DSPARK_PD_PREFILL_TAIL_TOKENS
• Doc: docs_new/docs/advanced_features/dspark_pd_pp_poc.md
• Unit smoke: test/registered/unit/disaggregation/test_dspark_pd_metadata_poc.py

Limits (intentional)

• Short prompt / short tail only (default tail len = 8)
• No Mooncake streaming hidden / no NIXL-MORI special path
• Decode still cannot enable PP with DSPARK (mainline hard check)
• Prefill/Decode radix cache should stay off / matched

Example

# Prefill
SGLANG_DSPARK_PD_TARGET_LAYER_IDS="<ids>" \
python -m sglang.launch_server \
  --model-path <DeepSeek-V4-Flash-DSpark> \
  --disaggregation-mode prefill --pp-size 2 --disable-radix-cache ...

# Decode
python -m sglang.launch_server \
  --model-path <DeepSeek-V4-Flash-DSpark> \
  --disaggregation-mode decode --speculative-algo DSPARK --pp-size 1 ...

Test plan

• Unit: PYTHONPATH=python python3 -m unittest test.registered.unit.disaggregation.test_dspark_pd_metadata_poc -v
• Manual short-prompt PD: Prefill pp=2 (no DSPARK) + Decode DSPARK
• Compare accept length / output vs Prefill pp=1
• Confirm Prefill+Decode metadata hidden width / tail len match (set SGLANG_DSPARK_PD_TARGET_LAYER_IDS on both if needed)