Repository navigation
[Perf] [NVIDIA] DSpark: Enable the simulated-acceptance fast paths on CUDA - #45421
Conversation
|
[Codex]: Use /tag-and-rerun-ci for UT-only CI on current head f5a3cb8, as requested. Monitor the actual Stage A CPU and single-GPU small partitions and cancel this base workflow after Stage A passes. This does not validate all repository unit tests or full CI. |
|
/tag-and-rerun-ci |
|
[Codex]: UT-only rerun failed on f5a3cb8: Base attempt 3. Stage A CPU partitions 2 and 9 reproduced the same missing post_load_weights errors in the shared-expert-fusion and RunAI loader test doubles. The other 16 Stage A partitions retained their successful results. No further reruns or cancellation were performed; the workflow completed with failure. |
… CUDA Under SGLANG_SIMULATE_ACC_LEN the verify skips the accept sampling whose result is overwritten, folds the accept and the sampled bonus into the verify graph, and treats temperature-only rows as greedy. These were gated to HIP; drop the gate so CUDA takes the same paths. No effect when SGLANG_SIMULATE_ACC_LEN is unset.
…_DSPARK_SIMULATE_ACC_FAST_PATH One platform-independent switch, default off, for the three fast paths under SGLANG_SIMULATE_ACC_LEN (skip the discarded accept sampling, greedy temperature-only rows, the folded accept). SGLANG_SIMULATE_ACC_GREEDY and SGLANG_DSPARK_SIMULATE_FOLD still turn the last two off individually.
…d SGLANG_DSPARK_SIMULATE_ACC_FAST_PATH" This reverts commit 4e515d8.
f5a3cb8 to
c5b9376
Compare
|
[Codex]: UT Stage A passed on c5b9376: all 17 CPU partitions and the single GPU-small partition succeeded. Base run, attempt 1. The workflow was intentionally cancelled after Stage A, and completed cancellation is verified. Some Stage B jobs started before cancellation; this is not a full CI pass. |
…ulated-acceptance fast paths on CUDA Reverted at the top of this branch; kept here only because later commits were written on top of it.
The measured configuration runs the accept sampling as stock does.
…project#45421) Restores the HIP-only guard that sgl-project#45421 lifted: under SGLANG_SIMULATE_ACC_LEN on CUDA, the verify accept runs its sampling path whenever sampling_info is set, and SGLANG_DSPARK_SIMULATE_FOLD / SGLANG_SIMULATE_ACC_GREEDY stay HIP-only. Real acceptance is unchanged. Kept as a separate commit so it can be dropped.
…project#45421) Restores the HIP-only guard that sgl-project#45421 lifted: under SGLANG_SIMULATE_ACC_LEN on CUDA, the verify accept runs its sampling path whenever sampling_info is set, and SGLANG_DSPARK_SIMULATE_FOLD / SGLANG_SIMULATE_ACC_GREEDY stay HIP-only. Real acceptance is unchanged. Kept as a separate commit so it can be dropped.
…project#45421) Restores the HIP-only guard that sgl-project#45421 lifted: under SGLANG_SIMULATE_ACC_LEN on CUDA, the verify accept runs its sampling path whenever sampling_info is set, and SGLANG_DSPARK_SIMULATE_FOLD / SGLANG_SIMULATE_ACC_GREEDY stay HIP-only. Real acceptance is unchanged. Kept as a separate commit so it can be dropped.
Motivation
With
SGLANG_SIMULATE_ACC_LENset (simulated acceptance for benchmarking), DSpark already has three fast paths, but they are gated to HIP:SGLANG_DSPARK_SIMULATE_FOLD);SGLANG_SIMULATE_ACC_GREEDY).On CUDA the same simulated runs still pay for the discarded sampling and the unfolded accept. This PR lets CUDA take the same paths.
Modifications
dspark_verify.py: skip the accept sampling under simulated acceptance on every platform (wassimulate and _is_hip).dspark_worker_v2.py:simulate_foldand_simulate_acc_greedyno longer require HIP; both keep their existing env switches (default on), so either can still be turned off._is_hip/is_hipimports in both files.No effect when
SGLANG_SIMULATE_ACC_LENis unset, so real serving is unchanged.Accuracy Tests
Only the simulated-acceptance path changes. Accuracy runs use real acceptance (
SGLANG_SIMULATE_ACC_LENunset) and take the same code as before.Speed Tests and Profiling
DeepSeek-V4.1-Flash, TP4 GB300, DSpark,
--enable-nvidia-megakernel, AgentX agentic-coding, concurrency 1,--max-running-requests 1,SGLANG_SIMULATE_ACC_LEN=3.51(match-expected, real-draft-token), 3,600 s. Measured on main8e7601ae4a4+ #44945; the two arms differ only by this change (one boot each, different nodes):The two runs reached different context mixes (ISL p50 294K vs 153K), so here is the steady decode step from the server's decode-batch log, binned by context and matched on (conversation, turn):
A rerun on current main will be added.
Checklist
CI States
Latest PR Test (Base): 🚫 Run #39490479896
Latest PR Test (Extra): ❌ Run #39490483441
Latest PR Test (AMD ROCm 10): ⏳ Run #39490480327