[aarnetalman]

Motivation

Deterministic inference promises the same output for a request whatever else is in the batch, but --enable-mixed-chunk breaks that. With it, the scheduler can put a prefill chunk and decode tokens into one batch, and the whole batch runs the prefill (extend) attention path. So a decode token's result depends on whether a prefill happened to share its step:

• the extend and decode kernels reduce in different orders, so they give slightly different results for the same tokens (the prefill vs decode difference that test_deterministic --test-mode p_vs_d measures);
• with per-phase backends (--prefill-attention-backend / --decode-attention-backend), they can even be different backends.

Mixed chunked prefill is off by default, so this only bites users who turn it on together with --enable-deterministic-inference. Found in the review of #44489 (AITER decode attention in deterministic mode), where a mixed batch would run its decode tokens on the prefill backend.

Modifications

• arg_groups/attention_hook.py (handle_deterministic_inference): in deterministic mode, --enable-mixed-chunk is switched off with a warning, as deterministic mode already does for --enable-aiter-allreduce-fusion.
• test_model_overrides.py: deterministic mode resolves enable_mixed_chunk to False; without it, the flag is kept.
• Unchanged: deterministic mode off; deterministic mode without --enable-mixed-chunk.

Accuracy Tests

MI350X, rocm/sgl-dev:v0.5.21-rocm724-mi35x-20261013, SGLANG_USE_AITER=1, Qwen3-8B TP1. Deterministic mode with Triton attention and --enable-mixed-chunk, with the open fixes #43788 and #34543 applied (deterministic mode on MI350X needs them):

python3 -m sglang.launch_server --model-path Qwen/Qwen3-8B \
  --attention-backend triton --enable-deterministic-inference --enable-mixed-chunk

python3 -m sglang.test.test_deterministic --test-mode {single,prefix,radix_cache} --n-trials 50, --test-mode prefix --return-logprob, and a bitwise batch-size check (the same prompt in batches of 1 to 50, 1,275 responses; token IDs and logprobs compared exactly with the batch-size-1 response), each on a fresh server:

	single	prefix (4 prompts)	prefix logprobs	radix_cache	bitwise
before (enable_mixed_chunk: True)	1 output	1, 2, 1, 1 outputs	identical	pass	1,275 / 1,275
after (server logs "Disable --enable-mixed-chunk because deterministic inference is enabled.", enable_mixed_chunk: False)	1 output	1, 1, 1, 1	identical	pass	1,275 / 1,275

Before, the 511-character prompt of prefix mode, sent with other prompts of different lengths, produced two different outputs across 293 samples: its decode steps sometimes shared a batch with a prefill chunk. single and the bitwise check send one prompt per batch, so nothing is mixed there.

Unit tests, python3 -m pytest test/registered/unit/test_model_overrides.py -k deterministic (same image): before, the new test fails (1 failed, 3 passed); after, 4 passed.

Speed Tests and Profiling

No change without --enable-mixed-chunk. With it, deterministic mode now runs without mixed batches, as when the flag is not given.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #38869885037
Latest PR Test (Extra): ❌ Run #38869885002
Latest PR Test (AMD ROCm 10): ❌ Run #38869885162

[rkarhila-amd]

Another small safety catch to make sure user does not use wrong configuration for deterministic inference. I guess even smallest holes should be blocked. Can't see this causing any harm.

[rkarhila-amd]

Not a stopper, but why a warning instead of raising error?

[aarnetalman]

Not a stopper, but why a warning instead of raising error?

Thanks, good question. I followed what deterministic mode already does with options it can't honour: in the same function, --enable-aiter-allreduce-fusion is switched off with a warning, and the dLLM and UNO paths switch --enable-mixed-chunk off the same way. Mixed chunking only changes how prefill and decode are scheduled, not which kernels compute the results, so deterministic mode can still give the user what they asked for without it. That's different from the attention-backend and RCCL checks, which raise because there's no safe substitute to pick for the user.