[WIP] Fix piecewise cuda graph recompile

[zminglei]

Motivation

Closes #13469

Modifications

Without piecewise cuda graph:

python3 -m sglang.launch_server --model /shared/public/elr-models/xai-org/grok-2/ --tokenizer-path /shared/public/elr-models/xai-org/grok-2/tokenizer.tok.json --tp 8 --quantization fp8 --attention-backend triton --disable-radix-cache

python benchmark/gsm8k/bench_sglang.py --data-path /shared/public/data/gsm8k/test.jsonl --num-questions 1319
Accuracy: 0.929
Invalid: 0.000
Latency: 133.680 s
Output throughput: 1111.674 token/s

python3 -m sglang.bench_serving --backend sglang --dataset-name random-ids --num-prompts 1 --random-input-len 10
24 --random-output-len 1 --random-range-ratio 1 --tokenizer /shared/public/elr-models/xai-org/grok-2/tokenizer.tok.json
---------------Time to First Token----------------
Mean TTFT (ms):                          104.61    
Median TTFT (ms):                        104.61    
P99 TTFT (ms):                           104.61    

With piecewise cuda graph:

python3 -m sglang.launch_server --model /shared/public/elr-models/xai-org/grok-2/ --tokenizer-path /shared/public/elr-models/xai-org/grok-2/tokenizer.tok.json --tp 8 --quantization fp8 --attention-backend triton --enable-piecewise-cuda-graph --disable-radix-cache

python3 -m sglang.bench_serving --backend sglang --dataset-name random-ids --num-prompts 1 --random-input-len 1024 --random-output-len 1 --random-range-ratio 1 --tokenizer /shared/public/elr-models/xai-org/grok-2/tokenizer.tok.json

python benchmark/gsm8k/bench_sglang.py --data-path /shared/public/data/gsm8k/test.jsonl --num-questions 1319
Accuracy: 0.934
Invalid: 0.001
Latency: 121.737 s
Output throughput: 1219.835 token/s

python3 -m sglang.bench_serving --backend sglang --dataset-name random-ids --num-prompts 1 --random-input-len 10
24 --random-output-len 1 --random-range-ratio 1 --tokenizer /shared/public/elr-models/xai-org/grok-2/tokenizer.tok.json
---------------Time to First Token----------------
Mean TTFT (ms):                          79.50     
Median TTFT (ms):                        79.50     
P99 TTFT (ms):                           79.50    

Accuracy Tests

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.
• Work with maintainers to merge your PR. See the PR Merge Process

[hebiao064]

?

[zminglei]

Moved to this PR #13667