[aarnetalman]

Motivation

Hybrid Gated DeltaNet models (Qwen3-Next, Qwen3.5, and later) are not batch-invariant under --enable-deterministic-inference when chunked prefill splits a prompt: the Gated DeltaNet prefill kernels process each chunk in 64-token blocks counted from the chunk's start, and the split point depends on what else is in the batch, so the same prompt can get different block boundaries and slightly different outputs. This PR removes that effect by ending every prefill chunk of these models on an absolute 64-token position in deterministic mode, also after a radix-cache hit and for continuation chunks. No change outside deterministic mode; no measurable speed change.

Full attention does not have this problem.

We measured it on MI350X with Qwen3.8-27B. A 3,000-token prompt with --chunked-prefill-size 2048 was sent alone, then together with one other prompt of 100 to 3,500 tokens:

• when the split point moved off a multiple of 64, its output differed from the solo run in 21 of 24 calls (log-probabilities from the first token, and token IDs when the other prompt was 1,000+ tokens);
• when the split stayed on a multiple of 64, 12 of 12 calls matched;
• without splitting, 24 of 24 matched;
• Qwen3-8B (full attention only) matched 24 of 24 either way.

In the end-to-end determinism tests this shows up as occasional prefix and prefix --return-logprob failures (one or two extra distinct outputs in ~300 samples, and log-probability mismatches).

Aligning the chunk length was not enough on its own. The existing deterministic alignment rounds the chunk length, so:

• after a radix-cache hit that ends off the grid, the chunk end is off the grid too;
• continuation chunks of a long prompt are not aligned at all, so their ends move with how much room the batchmates leave.

With only the 64-token rule, the chunk tests passed but the prefix tests, whose shared prefixes are 511 and 4,097 tokens long, still returned extra distinct outputs in 4 of 8 runs at default settings. So the second commit aligns the absolute end of every chunk, first and continuation alike; with it, 0 of 10 did.

This also matches what the other users of this alignment need: FlashInfer is exact when every chunk starts on its q-tile grid (#43612), and DeepSeek DSA asserts that chunked extends start on its kpool boundaries.

The same kernels run on NVIDIA in deterministic mode (FlashInfer GDN prefill is refused there), so this is not specific to ROCm.

Modifications

• Scheduler.init_deterministic_inference_config: for models where hybrid_gdn_config() is set, combine truncation_align_size with the Gated DeltaNet block size (64) using math.lcm, the same way DeepSeek DSA's alignment is combined. So triton attention gets 64, and flashinfer keeps 4096.
• PrefillAdder.add_one_req: when a request is truncated, round prefix_len + extend_len down to the alignment, not extend_len alone.
• PrefillAdder.add_chunked_req: take truncation_align_size and align a truncated continuation's end the same way. A continuation too short to reach the next boundary keeps its length, so the request always makes progress. The scheduler passes its truncation_align_size.
• Without a cached prefix, the first chunk is cut at the same place as before.
• Tests:
  • test_deterministic_chunk_align.py: a hybrid GDN model gets the 64-token grid; other models and normal mode are unchanged.
  • test_prefill_adder.py:
    • a continuation with 1,000 tokens already prefilled and a 4,096-token chunk budget ends its chunk at 5,056 (a multiple of 64);
    • a new request with a 100-token cached prefix and a 200-token budget ends its first chunk at 256 in both add_one_req paths (exact chunk fill and page rounding). Aligning only the length would end it at 292.
• No change outside deterministic mode (truncation_align_size is None there), or for models without Gated DeltaNet layers unless they already use the alignment (FlashInfer, DSA). Mamba2 and KDA models are not covered here.

Related:

• --enable-deterministic-inference accepted and reported enabled, but has no effect on an NVFP4 W4A4 + hybrid-Mamba checkpoint with a DFlash2 drafter (GB10 / sm_121) #36291 reports deterministic mode having no effect on a Qwen3.8-27B NVFP4 checkpoint on GB10; the split-point dependence fixed here is a likely part of it (not verified on that setup).
• [Fix] [AMD] mamba: Keep a chunked prefill cut on the mamba chunk grid so DCP can donate checkpoints #41701 also rounds truncated continuation chunks to the truncation_align_size grid in add_chunked_req (for mamba checkpoint donation with the extra buffer). It rounds the chunk length; this PR rounds the absolute end, which keeps its grid whenever the chunk starts on it. The two touch the same lines; happy to rebase onto whichever lands first.
• [Scheduler] Align deterministic flashinfer chunks to the q-tile grid, not the split-KV tile #43612 moves the alignment value into the attention backends (prefill_chunk_align). It lists "an off-grid radix-cache hit" among the remaining holes; the absolute alignment here closes it for the chunks after the hit. If [Scheduler] Align deterministic flashinfer chunks to the q-tile grid, not the split-KV tile #43612 lands first, the 64-token rule belongs in the hybrid linear-attention backend, as the LCM of the dense child's grid and 64. Happy to rebase either way.

Accuracy Tests

MI350X, rocm/sgl-dev:v0.5.21-rocm724-mi35x-20261012 with this change applied, --attention-backend triton --linear-attn-prefill-backend triton --linear-attn-decode-backend triton --enable-deterministic-inference, SGLANG_USE_AITER=0, default chunked prefill and radix cache. #43788 and #34543 were applied as well, so that the remaining failures come from this path.

Unit tests (test_deterministic_chunk_align.py, test_prefill_adder.py, 57 tests): before, the 4 new tests fail (2 + 2); after, 57 passed.

Split-point test (a long prompt alone vs with one other prompt, --chunked-prefill-size 2048 --disable-radix-cache, output token IDs and log-probabilities compared exactly):

Prompt	Before	After
3,000 tokens, 16 output tokens, other prompt 100-3,500 tokens	3 of 24 identical	16 of 16
8,192 tokens, 1,024 output tokens, other prompt 64-5,120 tokens	10 of 18 (0 of the 8 off the grid)	18 of 18

python3 -m sglang.test.test_deterministic --n-trials 50 for each mode, and a bitwise check of 1,275 responses (batch sizes 1-50, greedy) against batch size 1. For prefix, the number of distinct outputs in each of the 4 prefix groups (~300 samples each; 1 = deterministic; this is what the test returns). With --return-logprob the test also prints a per-sample log-probability comparison, which does not change its return value; it is shown in the column "log-probs". These runs follow the existing deterministic end-to-end tests (TestDeterministicBase, e.g. test_qwen35_deterministic.py), which are CUDA-only and compare decoded text only; here log-probabilities and the bitwise check are compared as well. Happy to add an AMD end-to-end test if wanted.

Model		single	prefix	prefix --return-logprob	log-probs	radix_cache	bitwise (token IDs / log-probs)
Qwen3.8-27B, TP1	main	1	1 1 1 1	1 2 1 1	differ	pass	1275 / 1275
	this PR (2 servers)	1, 1	1 1 1 1, 1 1 1 1	1 1 1 1, 1 1 1 1	differ (2/315), differ (1/307)	pass, pass	1275 / 1275, 1275 / 1275
Qwen3.8-27B-FP8, TP1	main	1	1 1 1 1	1 1 2 1	differ	pass	1275 / 1275
	this PR	1	1 1 1 1	1 1 1 1	identical	pass	1275 / 1275
Qwen3.8-27B, TP2	main	1	1 2 2 1	1 1 1 1	identical	pass	1275 / 48
	this PR	1	1 1 1 1	1 1 1 1	identical	pass	1275 / 50

• This PR removes the extra distinct outputs. It does not remove every log-probability difference: on BF16 TP1, 1-2 samples out of ~300 in one long-prefix group (2,048 or 4,097 tokens) still differ, at the first or second output token. They are not caused by chunk splits. Part of them comes from the Gated DeltaNet output kernel, whose tile depends on the batch's total length for short extends (fixed in [Fix] deterministic: Keep the GDN output tile fixed for varlen prefill #44194); the rest come from radix-cache hits: Gated DeltaNet checkpoints taken mid-prefill were stored rounded to bf16 (fixed in [Fix] GDN: Take prefix-cache checkpoints from the fp32 state in the Triton kernel #44195). With both fixes, every mode passes, log-probabilities included.
• The bitwise log-probability mismatch at TP2 is there with and without this PR and is not related to chunking: the Gated DeltaNet output kernel's tile shrinks with the batch's total length for short prompts. That is fixed in [Fix] deterministic: Keep the GDN output tile fixed for varlen prefill #44194.
• p_vs_d fails before and after; this PR does not target it.

Speed Tests and Profiling

Same setup as above, Qwen3.8-27B TP1, python3 -m sglang.benchmark.serving --backend sglang --dataset-name random --random-input-len 1024 --random-output-len 1024 --random-range-ratio 1.0 --num-prompts 256 --max-concurrency 256 --request-rate inf --seed 1 --warmup-requests 8 --flush-cache, median of 2 runs each, then with the GPUs swapped:

GPU		Output tok/s	ITL p50	TTFT p50	Request latency p50
A	main	1,291.4	59.98 ms	76.8 s	146.3 s
A	this PR	1,291.8 (1.000x)	60.09 ms	76.8 s	146.3 s
B	main	1,300.5	59.69 ms	76.3 s	145.3 s
B	this PR	1,303.2 (1.002x)	59.66 ms	76.1 s	145.1 s

No measurable change; the difference between the two GPUs is larger than the difference between before and after. (TTFT is long at concurrency 256 because the linear-attention state pool limits how many requests run at once, the same before and after.)

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #40674960955
Latest PR Test (Extra): ❌ Run #40674960735
Latest PR Test (AMD ROCm 10): ❌ Run #40674961184

[rkarhila-amd]

Motivation needs a TL;DR - It looks like there are more words in it than in the code changes 😃

[aarnetalman]

Motivation needs a TL;DR - It looks like there are more words in it than in the code changes 😃

Updated the description.

[rkarhila-amd]

(I'm not a real reviewer, but) I don't see any issues in the code beside.