Skip to content

[Perf] [NVIDIA] DSpark: Enable the simulated-acceptance fast paths on CUDA - #45421

Merged
Oasis-Git merged 3 commits into
sgl-project:mainfrom
YAMY1234:dspark-sim-acc-cuda
Oct 20, 2026
Merged

Oasis-Git merged 3 commits into
sgl-project:mainfrom
YAMY1234:dspark-sim-acc-cuda

Conversation

@YAMY1234

@YAMY1234 YAMY1234 commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

With SGLANG_SIMULATE_ACC_LEN set (simulated acceptance for benchmarking), DSpark already has three fast paths, but they are gated to HIP:

  • the verify skips the draft / target accept sampling (softmax + rejection), whose result the simulated length overwrites;
  • the accept and the sampled bonus fold into the verify CUDA graph (SGLANG_DSPARK_SIMULATE_FOLD);
  • temperature-only rows are treated as greedy (SGLANG_SIMULATE_ACC_GREEDY).

On CUDA the same simulated runs still pay for the discarded sampling and the unfolded accept. This PR lets CUDA take the same paths.

Modifications

  • dspark_verify.py: skip the accept sampling under simulated acceptance on every platform (was simulate and _is_hip).
  • dspark_worker_v2.py: simulate_fold and _simulate_acc_greedy no longer require HIP; both keep their existing env switches (default on), so either can still be turned off.
  • Remove the now-unused _is_hip / is_hip imports in both files.

No effect when SGLANG_SIMULATE_ACC_LEN is unset, so real serving is unchanged.

Accuracy Tests

Only the simulated-acceptance path changes. Accuracy runs use real acceptance (SGLANG_SIMULATE_ACC_LEN unset) and take the same code as before.

Speed Tests and Profiling

DeepSeek-V4.1-Flash, TP4 GB300, DSpark, --enable-nvidia-megakernel, AgentX agentic-coding, concurrency 1, --max-running-requests 1, SGLANG_SIMULATE_ACC_LEN=3.51 (match-expected, real-draft-token), 3,600 s. Measured on main 8e7601ae4a4 + #44945; the two arms differ only by this change (one boot each, different nodes):

requests tok/s/user p50 p90 TTFT p50
without this change 404 760.0 719.3 102 ms
with this change 474 949.7 926.0 96 ms

The two runs reached different context mixes (ISL p50 294K vs 153K), so here is the steady decode step from the server's decode-batch log, binned by context and matched on (conversation, turn):

context ≤ 64K 64–256K > 256K
without this change 4.581 ms 4.595 ms 4.522 ms
with this change 3.654 ms 3.669 ms 3.723 ms

A rerun on current main will be added.

Checklist


CI States

Latest PR Test (Base): 🚫 Run #39490479896
Latest PR Test (Extra): ❌ Run #39490483441
Latest PR Test (AMD ROCm 10): ⏳ Run #39490480327

@YAMY1234
YAMY1234 marked this pull request as ready for review October 20, 2026 18:31
@YAMY1234 YAMY1234 changed the title [Perf] [NVIDIA] DSpark: Enable the simulated-acceptance fast paths on CUDA [Perf] DSpark: Gate the simulated-acceptance fast paths behind SGLANG_DSPARK_SIMULATE_ACC_FAST_PATH Oct 20, 2026
@YAMY1234 YAMY1234 changed the title [Perf] DSpark: Gate the simulated-acceptance fast paths behind SGLANG_DSPARK_SIMULATE_ACC_FAST_PATH [Perf] [NVIDIA] DSpark: Enable the simulated-acceptance fast paths on CUDA Oct 20, 2026
@Oasis-Git

Copy link
Copy Markdown
Collaborator

[Codex]: Use /tag-and-rerun-ci for UT-only CI on current head f5a3cb8, as requested. Monitor the actual Stage A CPU and single-GPU small partitions and cancel this base workflow after Stage A passes. This does not validate all repository unit tests or full CI.

@Oasis-Git

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 20, 2026
@Oasis-Git

Copy link
Copy Markdown
Collaborator

[Codex]: UT-only rerun failed on f5a3cb8: Base attempt 3. Stage A CPU partitions 2 and 9 reproduced the same missing post_load_weights errors in the shared-expert-fusion and RunAI loader test doubles. The other 16 Stage A partitions retained their successful results. No further reruns or cancellation were performed; the workflow completed with failure.

… CUDA

Under SGLANG_SIMULATE_ACC_LEN the verify skips the accept sampling whose
result is overwritten, folds the accept and the sampled bonus into the
verify graph, and treats temperature-only rows as greedy. These were
gated to HIP; drop the gate so CUDA takes the same paths. No effect when
SGLANG_SIMULATE_ACC_LEN is unset.
…_DSPARK_SIMULATE_ACC_FAST_PATH

One platform-independent switch, default off, for the three fast paths
under SGLANG_SIMULATE_ACC_LEN (skip the discarded accept sampling, greedy
temperature-only rows, the folded accept). SGLANG_SIMULATE_ACC_GREEDY and
SGLANG_DSPARK_SIMULATE_FOLD still turn the last two off individually.
…d SGLANG_DSPARK_SIMULATE_ACC_FAST_PATH"

This reverts commit 4e515d8.
@Oasis-Git
Oasis-Git force-pushed the dspark-sim-acc-cuda branch from f5a3cb8 to c5b9376 Compare October 20, 2026 20:15
@Oasis-Git

Copy link
Copy Markdown
Collaborator

[Codex]: UT Stage A passed on c5b9376: all 17 CPU partitions and the single GPU-small partition succeeded. Base run, attempt 1. The workflow was intentionally cancelled after Stage A, and completed cancellation is verified. Some Stage B jobs started before cancellation; this is not a full CI pass.

@Oasis-Git
Oasis-Git merged commit 7d9bc10 into sgl-project:main Oct 20, 2026
111 of 172 checks passed
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Oct 20, 2026
…ulated-acceptance fast paths on CUDA

Reverted at the top of this branch; kept here only because later commits were written on top of it.
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Oct 20, 2026
The measured configuration runs the accept sampling as stock does.
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Oct 21, 2026
…project#45421)

Restores the HIP-only guard that sgl-project#45421 lifted: under SGLANG_SIMULATE_ACC_LEN on CUDA,
the verify accept runs its sampling path whenever sampling_info is set, and
SGLANG_DSPARK_SIMULATE_FOLD / SGLANG_SIMULATE_ACC_GREEDY stay HIP-only. Real acceptance
is unchanged. Kept as a separate commit so it can be dropped.
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Oct 21, 2026
…project#45421)

Restores the HIP-only guard that sgl-project#45421 lifted: under SGLANG_SIMULATE_ACC_LEN on CUDA,
the verify accept runs its sampling path whenever sampling_info is set, and
SGLANG_DSPARK_SIMULATE_FOLD / SGLANG_SIMULATE_ACC_GREEDY stay HIP-only. Real acceptance
is unchanged. Kept as a separate commit so it can be dropped.
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Oct 21, 2026
…project#45421)

Restores the HIP-only guard that sgl-project#45421 lifted: under SGLANG_SIMULATE_ACC_LEN on CUDA,
the verify accept runs its sampling path whenever sampling_info is set, and
SGLANG_DSPARK_SIMULATE_FOLD / SGLANG_SIMULATE_ACC_GREEDY stay HIP-only. Real acceptance
is unchanged. Kept as a separate commit so it can be dropped.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants