fix: piecewise_cuda_graph get correct qo_indptr

[yyihuang]

Motivation

#21218

Modifications

for padding tokens, append a fake bs+1-th request with pad_tokens extend tokens whose KV indices all point to scratch slot 0. This makes qo_indptr[-1] = static_num_tokens, without affecting causal masks for real requests.

Accuracy Tests

python -m sglang.launch_server --model-path Qwen/Qwen3-14B --attention-backend flashinfer --disable-cuda-graph

python3 benchmark/gsm8k/bench_sglang.py --num-questions 100
100%|███████████████████████████████| 100/100 [00:05<00:00, 18.24it/s]
Accuracy: 0.950
Invalid: 0.000
Latency: 5.524 s
Output throughput: 2255.160 token/s

enable cuda graph:
python3 benchmark/gsm8k/bench_sglang.py --num-questions 100
100%|█████████████████████████████| 100/100 [00:04<00:00, 24.05it/s]
Accuracy: 0.940
Invalid: 0.000
Latency: 4.198 s
Output throughput: 2968.234 token/s

• update after review comment:
  (flashinfer_bench) averyh@umb-b200-238:~/flashinfer-bench/tmp/sglang$ python3 benchmark/gsm8k/bench_sglang.py --num-questions 100
  100%|███████████████████████████████████████████████████████| 100/100 [00:04<00:00, 24.82it/s]
  Accuracy: 0.930
  Invalid: 0.000
  Latency: 4.035 s
  Output throughput: 3180.599 token/s

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review Process

1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
4. After green CI and required approvals, ask Merge Oncalls to merge.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[ispobock]

/tag-and-rerun-ci

[Fridge003]

There is performance regression with this PR
https://github.com/sgl-project/sglang/actions/runs/23578699380/job/68792410119?pr=21452

[Oasis-Git]

Run the test locally with h100 on tp=1 and tp=8 and gsm test passes

[Oasis-Git]

In general the change is reasonable. Here is some suggestions for revision.

[Oasis-Git]

I think we can move num_tokens into the ForwardContext. Also to skip the computation and sync with item(), it is suggested that the var such as num_dummy_pages should be pre-calculated

[yyihuang]

Hi I take your suggestions to update the code:

• Added self.num_tokens: Optional[int] = None field to ForwardContext
• Eliminated both .item() GPU-CPU syncs in the dummy-request block

[Oasis-Git]

with --max-running-requests 1 disable the pcg to pass the ci