[knapcio]

v2.6.4

v2.6.3 plus a decode attention that reads the sliding-window (SWA) cache in place, a decode share for running streams while a long prompt prefills, and two prefix-cache changes that keep idle agent sessions resumable without re-prefilling their whole context. Outputs are bit-identical for the decode attention; the other three change only scheduling, eviction order and how a lost resume point is rebuilt. The full KV pool is smaller (about 3.4-3.75M tokens instead of 5.0-5.16M); the rollback is config only. Full details are in CHANGELOG.md.

What's new

• SWA pages read in place by the decode (DSV41_SWA_NATIVE_DECODE=1, now in the production profile). Every decode attention call (up to 64 query rows: the DSpark verify and draft forwards) used to copy each touched 256-token SWA page into four 64-token pages first, 43 copy chains per c1 step, because FlashInfer's decode-dsv4 kernel was instantiated for 64-token pages only. Its address math is generic in the page size, so the same kernel body instantiated at 256-token pages reads the pool in place, with the same tactic and the same output bits. Calls it does not serve keep the copy and are logged once per reason. Two in-boot A/Bs on the four Sparks: greedy c1 +1.36 / +1.86 %, and T = 0 text identical on every prompt that was stable between control runs. The module (runtime/flashinfer_v41c) is built at image build with its .so sha256 checked at load.
• Running streams keep a decode share while a long prompt prefills (DSV41_DECODE_YIELD=33, now in the production profile). Between two 8192-token chunks of a long prompt the engine used to run only the --prefill-decode-interval steps, so the other streams fell to about 8 % of their decode rate for the whole prefill. After each non-final chunk the next one now waits until the running streams have had decode time worth 33/67 of that chunk's time. The decision uses an integer cost model, not a clock, so every rank decides alike, and chunk boundaries, batches and outputs are the engine's own. Four Sparks, two streams on 300K contexts beside a cold 500K prompt: the streams' decode rate during the prefill went from 0.080 to 0.393 of their rate before it, and the 500K prompt's first token came x1.47 later. A lone long prompt prefills exactly as before.
• Idle sessions keep their prefix-cache resume points (DSV41_SWA_THIN=1 with --swa-prefix-tails 384, was 128, now in the production profile). A cached prefix can only be resumed where its last sliding window still holds SWA slots. Every prefill chunk and agent turn leaves a checkpoint, so in stock LRU order an idle session lost its resume point after about 70-100K slots of other traffic and its next turn re-prefilled the whole context. Checkpoints that their own request has moved past are now evicted first, and the SWA pool holds 384 prefix tails. Four Sparks, two long sessions idle for minutes beside sub-agent turns and new 500K sessions: resumed 330K / 300K sessions answered in 3.9 / 6.5 s instead of 61-64 s, with no whole-session re-prefills (2 in each control). Only the eviction order and the pool split change.
• A lost resume point costs a one-chunk replay (DSV41_SWA_CONE_REPLAY=1 DSV41_SWA_CONE_TOKENS=8192, now in the production profile). When a session's SWA window was evicted anyway but its full KV is still cached, the request matches on full KV 8192 tokens earlier and replays only that chunk. The cached latents and index keys are copied into the replay rows and their own writes are masked, so every row from the resume point on computes what a prefix hit there computes. Four Sparks, 16 prompts of 130-400K tokens: the replay's distance from a re-prefill was 1.01x that of a plain prefix hit, with 0 confident token flips, and its first token came in 1.6-2.1 s instead of 21.8-68.3 s. When the first chunk cannot cover the replay, the request falls back to the stock re-prefill and logs it.

Release gate

v2.6.3 / v2.6.4 / v2.6.3: fresh boots in that order, production profile, abliterated 1017 517efab, GPU clock capped at 2200 MHz, 2026-11-05 (run run-20261105-170445-1455369, 274 min including the restore of production). PASS, 0 KILL, 0 INVALID. No KV pin this time (earlier gates pinned 4,194,304), so each boot sized its own pool and the v2.6.4 boot ran with the pool it ships with. Speedbench 0.3 full mode, median of two runs for greedy cells, one run for sampled cells. The v2.6.4 boot ran the v2.6.3 gate image with this release's files overlaid and the swan module built in the image.

metric	v2.6.3	v2.6.4	v2.6.3
greedy c1, all 73 prompts, geomean tok/s	89.34	89.09	88.73
greedy c1, prose, tok/s	65.62	65.95	65.34
greedy c4, aggregate tok/s	131.75	132.62	131.76
load c16, aggregate tok/s	215.47	216.16	216.65
sampled (T 0.7, top-p 0.95) c1, all prompts, tok/s	87.30	85.72	86.01
sampled c1, prose, tok/s	62.76	61.65	62.52
sampled c4, aggregate tok/s	129.03	129.53	129.37
cold prefill 64K, tok/s	6147	6137	6166
cold 388K prompt, first token, s	64.0	63.6	63.4
qeval (75 tasks)	71	71	71
full KV pool, tokens	4,997,632	3,444,224	5,101,824

At T = 0 the v2.6.4 outputs equal the controls' on all 74 of the 75 qeval prompts on which the two controls agree. KLD long 0.00677 and short 0.0045 (PASS). The T = 1 distribution check passed on 12,000 samples, and the 1M needle (969,071 tokens) was found in 195 s. The sampled cells are single runs; the two controls differ by 1.5 % between themselves at c1. Every rank of the v2.6.4 boot logged the swan module, SWA thin, the cone replay and the decode yield (share 33) as installed, with no fallback and no error, and every rank of all three boots logged the FlashInfer #7676 fence as present.

Evicted resume point (v2.6.4 boot): a 300,000-token document (prefill 49.7 s), then 800 distinct 1K-token prompts that pushed its SWA window out, then the document plus 2,048 tokens. Every node replayed one 8,192-token chunk with 0 reverts. The first token came after 2.07 s, with 291,328 of 302,048 prompt tokens cached and 0 errors.

Soak (issue #16 reproduction with a seeded document order plus a c2 companion load, 12 timed minutes): 256 / 318 / 263 requests with 0 failures, head MemAvailable low-water 9.55 / 8.37 / 8.73 GiB. v2.6.4 keeps the soak's documents cached where the controls re-prefill them, so it served more requests in the same time.

KV pool trade-off

The 384 prefix tails reserve 182,784 SWA slots instead of 84,480, and that memory comes out of the full KV pool: 3,444,224 tokens on the v2.6.4 boot against 4,997,632 / 5,101,824 on the controls (earlier A/B windows: 3.60-3.75M against 5.06-5.16M). That still holds three full 1M-token contexts, not four; the production peak in flight over a day was 1.91M tokens. If in-flight context needs more, --swa-prefix-tails 256 is the middle step (about 4.4M tokens).

Upgrade / rollback

The "Production profile" line in .env.tp4.example now ends with DSV41_SWA_NATIVE_DECODE=1 DSV41_DECODE_YIELD=33 DSV41_SWA_THIN=1 DSV41_SWA_CONE_REPLAY=1 DSV41_SWA_CONE_TOKENS=8192, and the production launch line carries --swa-prefix-tails 384. Rebuild Dockerfile.canary-roce to get the swan module. The rollback is config only, and each switch can be unset on its own:

• unset DSV41_SWA_THIN and set --swa-prefix-tails 128 to get back the v2.6.3 prefix cache and the ~5.0-5.1M-token pool;
• unset DSV41_SWA_CONE_REPLAY / DSV41_SWA_CONE_TOKENS to return a lost resume point to the stock re-prefill;
• unset DSV41_DECODE_YIELD to return to the stock prefill / decode cadence;
• unset DSV41_SWA_NATIVE_DECODE to go back to the 64-token page copy.

All the new keys are volatile for DSV41_AUTOTUNE_KEEP, so toggling them reuses the tuned MoE tactics; the tails value is a launch argument, so the first boot after changing it re-tunes once.

Credits

• Gabriel Wu and the contributors to FlashInfer's SM120 sparse-MLA decode kernel (flashinfer#3395 and later): its page-size-generic address math is what lets the stock kernel body read 256-token pages. The 256-token instantiation, its routing and the tests are ours.
• The decode yield defers chunks through SGLang's --prefill-decode-interval path, with Shangming Cai's SGLang #44001 gate already backported. Christopher Owen's prefill yield in spark-ds41f was the starting point of the comparison.
• SWA thin and the cone replay are built on SGLang's unified radix cache (its SWA LRU and tombstones) and on the dsv4.1 branch's decoder bounded replay, whose swa_replay_start window floor the replay reuses. The eviction rule, the replay, the page copy and the write mask are ours.