[vincentzed]

Motivation

--enable-encoder-swa-bounded-replay keeps each request's sliding-window state in a per-request window instead of the paged cache. Today the server refuses to start if the flag is combined with a prefill CUDA graph, so every prefill runs eagerly, and so does the extra pass that rebuilds a window when a request hits a cached prefix.

Eager prefill costs about 120 ms per batch with this flag on 4x B300. A follow-up turn on a cached conversation pays it twice, once for the rebuild pass and once for the new tokens.

Roadmap #42170 lists "Enable encoder bounded replay" with no owner. This PR builds on the decoder PR #42909 and on the crash fix #42902. Until those merge, its diff includes their commits.

Modifications

1. The request-window layout can be padded. Each graph captures it for its own row count and for one window group per request slot, so a batch of any number of requests fits. Padding rows write to scratch rows and are never committed to a request's window.
2. On each replay the captured layout is refreshed in place and activated again.
3. When a request hits a cached prefix, an extra pass rebuilds its window. That pass now runs through the graph too. It does not need the compressor, and it can still skip it, because under bounded replay the compressor and the indexer run outside the captured graph, at a graph break.
4. Sparse prefill attention reads only the live rows of the padded layout.
5. Only the breakable backend is accepted.

Accuracy Tests

4x B300, deepseek-ai/DeepSeek-V4.1-Flash, TP4 and EP4. Both columns run this branch.

# eager
python3 -m sglang.launch_server --trust-remote-code --model-path deepseek-ai/DeepSeek-V4.1-Flash --tp 4 --ep-size 4 --mem-fraction-static 0.8 --cuda-graph-max-bs-decode 64 --max-running-requests 16 --chunked-prefill-size 8192 --enable-encoder-swa-bounded-replay --disable-prefill-cuda-graph
# graph
python3 -m sglang.launch_server --trust-remote-code --model-path deepseek-ai/DeepSeek-V4.1-Flash --tp 4 --ep-size 4 --mem-fraction-static 0.8 --cuda-graph-max-bs-decode 64 --max-running-requests 16 --chunked-prefill-size 8192 --enable-encoder-swa-bounded-replay --cuda-graph-backend-prefill breakable
# "all three" rows add: --enable-decoder-swa-bounded-replay --speculative-algorithm DSPARK --speculative-dspark-block-size 5
python3 -m sglang.test.few_shot_gsm8k --num-questions 1319 --parallel 32

GSM8K, 1319 questions	eager	graph
encoder replay only	0.900	0.904
encoder + decoder replay + DSpark	0.901	0.910

Each cell is one run. Repeated runs of one configuration moved between 0.900 and 0.915, so these are within noise. The DSpark accept length is 3.40 in both columns.

Speed Tests and Profiling

Time to first token, median of 5 requests sent one at a time, prompts sent as token ids.

Request, encoder replay only	eager	graph	speedup
590 new tokens	120.4 ms	52.0 ms	2.3x
2,060 new tokens	121.0 ms	80.6 ms	1.5x
6,318 new tokens	137.8 ms	131.3 ms	1.05x
14,795 new tokens	291.5 ms	286.8 ms	1.02x
417 new tokens after 13.3K cached tokens	215.6 ms	81.8 ms	2.6x
554 new tokens after 59.9K cached tokens	222.0 ms	93.7 ms	2.4x

Request, encoder + decoder replay + DSpark	eager	graph	speedup
590 new tokens	122.0 ms	48.2 ms	2.5x
2,060 new tokens	121.6 ms	60.1 ms	2.0x
6,318 new tokens	129.7 ms	89.8 ms	1.44x
14,795 new tokens	266.1 ms	185.8 ms	1.43x
417 new tokens after 13.3K cached tokens	216.0 ms	75.2 ms	2.9x
554 new tokens after 59.9K cached tokens	225.6 ms	85.1 ms	2.7x

GSM8K wall time, 32 parallel	eager	graph	speedup
encoder replay only	371.7 s	193.6 s	1.92x
encoder + decoder replay + DSpark	227.5 s	106.8 s	2.13x

48 requests of about 7.4K tokens, 8 concurrent	eager	graph	speedup
encoder replay only	49.7K tok/s	51.4K tok/s	1.03x
encoder + decoder replay + DSpark	56.2K tok/s	80.0K tok/s	1.42x

Capturing all graphs takes about 11 GB per GPU with encoder replay on.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

---

CI States

Latest PR Test (Base): ❌ Run #37667504398
Latest PR Test (Extra): ❌ Run #37667502671
Latest PR Test (AMD ROCm 10): ❌ Run #37667505108

[vincentzed]

@Oasis-Git this is the encoder half for #42170 ("Enable encoder bounded replay"). It is stacked on #42909 (decoder) and #42902 (crash fix), so until those merge the diff here includes their commits. The only commit that belongs to this PR is the last one. I will rebase it as soon as the other two land.

[vincentzed]

Closing this. It was stacked on #42909, which #43736 replaced, and it now conflicts with main.