[MarinaMackay]

Motivation

Fixes #46446. With --enable-deterministic-inference and --enable-mixed-chunk, Qwen3.5-9B's greedy output for a request changes when another request's prefill is mixed into one of its decode steps.

EagerRunner.execute runs a MIXED batch as EXTEND, so the GDN layers process each running request's decode token as a 1-token sequence through the chunked prefill kernel (chunk_gated_delta_rule). Without mixed chunk, the same token goes through the recurrent decode kernel. The two are mathematically equal but not bitwise equal, and the difference stays in the recurrent state. In the issue's repro only one step is mixed (the 255-token intruder plus the target's 1-token tail at prefix 4120), and that step is enough to flip the target's greedy token about 100 tokens later. The full-attention layers (Triton, deterministic) do not need a change: with only the GDN layers fixed, the outputs match exactly.

Modifications

• ForwardBatch.num_mixed_decode_reqs: the number of running requests that mix_with_running appended to a MIXED batch, set in init_new from mix_running_indices_cpu. This is needed because the eager runner relabels the batch as EXTEND before the attention backends see it. TBO children set it to None.
• GDNAttnBackend: under deterministic inference, init_forward_metadata cuts a mixed batch once per forward into its prefill rows and its decode tail (build_gdn_mixed_decode_split). forward_extend then runs the prefill rows through the unchanged extend path and the tail through forward_decode, the same path a decode step takes. The tails are untracked in a mixed step (as in mix_with_running), so no state tracking changes.
• The split is skipped, and behavior is unchanged, when deterministic inference is off, with MIS, with ReplaySSM (its decode path needs per-row ring metadata that a mixed batch does not build), and when the batch no longer ends in 1-token rows.

Accuracy Tests

1x H200 NVL (SM90), main d7f832d30a, the issue's engine settings (portable/repro.py from the issue: Triton attention and linear attention, deterministic, eager, fp32 lm_head, seed 42).

The issue's two-request repro (pair_T3.jsonl, fresh engine per run):

	mixed chunk on vs off
main	differ, first at generated index 100, 3/3 trials (81 on the reporter's SM120 card)
a99c7d05 (reported revision)	same as main, 3/3
this PR	identical, all 128 target tokens and the intruder's 8, 2/2 trials

Each arm is stable across trials on main, so the difference is between the two scheduling paths, not run-to-run noise. With this PR the mixed-chunk outputs are token-identical to main's mixed-chunk-off outputs.

Client concurrency 1 vs 8, mixed chunk on (32 prompts of 300-3000 tokens cut from the repro's prompt, 64 greedy tokens each, script below). Number of prompts whose outputs differ between the two runs:

	radix on	radix off	mixed off, radix off
main	5/32
this PR	6/32	2/32	3/32
#45545 only	3/32
this PR + #45545	1/32	0/32	0/32

This PR and #45545 fix different things. #45545 aligns where chunked prefill cuts a GDN sequence, which removes the concurrency differences that remain with mixed chunk off. This PR fixes the mixed-in decode tail, which #45545 does not touch, and which the issue's two-request repro needs. (#45545 conflicts with 8252e3af63 in schedule_policy.py. For these runs I kept #45545's side of both hunks, which is equivalent without DCP.) The remaining 1/32 with both fixes only appears with radix cache on, and I have not looked into it. It is out of scope here.

Concurrency script

"""sgl-project/sglang#46446: greedy outputs at client concurrency 1 vs 8.

Same engine settings as the issue's repro.py. 32 prompts of 300-3000 tokens cut
from different offsets of the repro's long prompt, 64 new tokens each.
Usage: python conc.py <model> <pair_T3.jsonl> <port> <out.json>
CONC_MIXED=0 disables mixed chunk, CONC_RADIX=0 disables the radix cache.
"""

import asyncio
import json
import os
import sys

import sglang as sgl

model, pair_path, port, out_path = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
base = json.loads(open(pair_path).readline())["target"]["prompt_token_ids"]
prompts = []
for i in range(32):
    length = 300 + (i * 977) % 2700
    start = (i * 131) % max(1, len(base) - length)
    prompts.append((base * 2)[start : start + length])


async def run(engine, concurrency):
    sem = asyncio.Semaphore(concurrency)

    async def one(ids):
        async with sem:
            out = await engine.async_generate(
                input_ids=ids,
                sampling_params=dict(temperature=0, max_new_tokens=64, ignore_eos=True),
            )
            return out["output_ids"]

    return await asyncio.gather(*(one(p) for p in prompts))


if __name__ == "__main__":
    results = {}
    for concurrency in (1, 8):
        engine = sgl.Engine(
            model_path=model,
            dtype="bfloat16",
            tp_size=1,
            context_length=32768,
            mem_fraction_static=0.65,
            max_running_requests=16,
            chunked_prefill_size=2048,
            enable_mixed_chunk=os.environ.get("CONC_MIXED", "1") == "1",
            disable_radix_cache=os.environ.get("CONC_RADIX", "1") == "0",
            mamba_radix_cache_strategy="extra_buffer",
            max_mamba_cache_size=128,
            attention_backend="triton",
            linear_attn_backend="triton",
            cuda_graph_backend_decode="disabled",
            cuda_graph_backend_prefill="disabled",
            skip_tokenizer_init=True,
            enable_fp32_lm_head=True,
            random_seed=42,
            enable_deterministic_inference=True,
            port=port,
            log_level="warning",
        )
        try:
            results[concurrency] = asyncio.run(run(engine, concurrency))
        finally:
            engine.shutdown()
    diff = [i for i, (a, b) in enumerate(zip(results[1], results[8])) if a != b]
    json.dump(dict(results={str(k): v for k, v in results.items()}, different=diff), open(out_path, "w"))
    print(f"DIFFERENT {len(diff)}/32 {diff}")

Speed Tests and Profiling

The default path is unchanged: the split only runs under --enable-deterministic-inference, and only for a mixed batch. There it adds two shallow ForwardBatch copies and two metadata copies per forward (shared by all GDN layers), and splits each GDN layer's call into one extend launch for the prefill rows and one decode launch for the tail.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Tests run (1x H200 NVL):

• New test_deterministic_mixed_batch_decode_tail_matches_decode in test/registered/attention/unittests/gdn/test_triton.py: a mixed batch (one 40-token prefill plus two decode tails at prefixes 24 and 31) must give the tails the same output, SSM state, and conv state, bit for bit, as a decode step for the same requests; the whole batch is also checked against the kit's PyTorch reference. It fails on main (tail output differs) and passes with this change.
• test/registered/attention/unittests/gdn/test_triton.py: 14 passed, 26 subtests passed.
• pre-commit run --files <changed files>: all hooks passed.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

Developed with AI assistance (Claude Code); I reviewed the change and the test results.

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ❌ Run #40280095968
Latest PR Test (Extra): ❌ Run #40280096033
Latest PR Test (AMD ROCm 10): ❌ Run #40280096144