perf(spec_decode): consume the remote K3 draft reply in stream order - #604

Open

myshytf wants to merge 86 commits into

local-inference-lab:dev/infernal-invocationfrom

myshytf:agent/kimi-k3-draft-deferred-resolve-20260903-pr

Commits on Aug 12, 2026

fix(structured-output): preserve grammar bitmask source widths
voipmonitor
committed
826bb40

Commits on Aug 17, 2026

[Bugfix] Stream Kimi K3 tool-call arguments incrementally

guptaishaan
authored and
voipmonitor
committed
b9eea38
tools: preserve partial Kimi XTML close markers

voipmonitor
and
codex
committed
efbaa95
fix(cache): preserve Mamba CoW after external hits

myshytf
authored and
voipmonitor
committed
9972d3c
test(cache): assert external-hit continuation precondition

myshytf
authored and
voipmonitor
committed
990a63d
Ignore Kimi tool calls after tools section close
voipmonitor
committed
3ebc896
fix(dspark): preserve draft graph capture contract
voipmonitor
committed
c805ebd
spec_decode: preserve DFlash-family checkpoint semantics

voipmonitor
and
codex
committed
3938209
attention: execute replicated KV groups without DCP partitioning

voipmonitor
and
codex
committed
8f7f6c6
models: document DFlash rotary layout detection

voipmonitor
and
codex
committed
f85d3d8

Commits on Aug 18, 2026

[II] Keep Kimi K3 protocol markers out of streamed content
voipmonitor
committed
04a6acf
Fix invalid block handling for hybrid KV cache groups

jungjiyu
authored and
voipmonitor
committed
c09bed0
test(scheduler): cover hybrid KV fail policy across 17 groups
voipmonitor
committed
6bafa63
test(scheduler): preserve hybrid KV recovery edge cases
voipmonitor
committed
3c296be
fix(structured-output): map compacted DSpark grammar rows
voipmonitor
committed
a18a86e

Commits on Aug 19, 2026

test(dspark): define graph-tail mode in capture fixture
voipmonitor
committed
e7ffc38

Commits on Aug 20, 2026

Commits on Aug 21, 2026

[Model][Spec Decode] Tap the pre-norm AttnRes mixture as the Kimi K3 DFlash aux state (#50487)

authored and
voipmonitor
committed
c27d117
test(kimi-k3): strengthen DFlash capture contracts
voipmonitor
committed
0a00dac
Bound Kimi vision RoPE allocation to input grids

voipmonitor
and
codex
committed
71964b3
Bound Kimi projector transients per image
voipmonitor
committed
18d9e27
[II] Let cache specs own DCP block-table geometry
voipmonitor
committed
4f34748
fix(mamba): resume prefix hits on checkpoint grid
voipmonitor
committed
edb1042
fix(multimodal): gather uneven vision shards without TP padding
voipmonitor
committed
832b1c9

Commits on Aug 22, 2026

fix(kimi-k3): preserve final AttnRes block
voipmonitor
committed
d19ef45
fix(attention): preserve aliased LSE during state merge
voipmonitor
committed
7aa4d61
fix(attention): align merge blocks to head groups
voipmonitor
committed
7671c66
fix(kimi-k3): bound chunked MLA output lifetime
voipmonitor
committed
62bead5
[Bugfix][Mamba] Fix overlapping state copy race (#50729)

authored and
voipmonitor
committed
f7180bc
Merge vLLM PR #414
voipmonitor
committed
4cfeb8a
Merge vLLM PR #295
voipmonitor
committed
bf5a9b2
Merge vLLM PR #294
voipmonitor
committed
e62cedb
Merge vLLM PR #320
voipmonitor
committed
6f78b5c
Merge vLLM PR #413
voipmonitor
committed
538ed2e
Merge vLLM PR #422
voipmonitor
committed
8b6e1de
Merge vLLM PR #310
voipmonitor
committed
799c2e7
Merge vLLM PR #415
voipmonitor
committed
bc3dd5a
Merge vLLM PR #418
voipmonitor
committed
afcbf4f
Merge vLLM PR #419
voipmonitor
committed
23e7093
Merge vLLM PR #459
voipmonitor
committed
f52fadc
Merge vLLM PR #460
voipmonitor
committed
94d3833
Merge vLLM PR #463
voipmonitor
committed
dc8f46d
Merge vLLM PR #464
voipmonitor
committed
3c9031e
Merge vLLM PR #467
voipmonitor
committed
7da8bd3
Merge vLLM PR #468
voipmonitor
committed
6ba940c
Merge vLLM PR #469
voipmonitor
committed
89b9db6
Merge upstream vLLM PR #50729 backport
voipmonitor
committed
1c93ca0

Commits on Aug 23, 2026

feat(spec_decode): run the Kimi-K3 DSpark draft on a dedicated remote GPU
myshytf
committed
a653e74
fix(spec_decode): allow scheduler-selected zero draft depth
myshytf
committed
23f9f27
fix(cache): avoid target EAGLE drop for disaggregated DSpark
myshytf
committed
f49f6c3
test(cache): cover DFlash target-cache policy
myshytf
committed
dbf86de
scheduler: skip speculative decoding when all scheduled requests need <=1 output token

malaiwah
authored and
myshytf
committed
1bed2da
fix: address CodeRabbit review — document lookahead reservation is harmless for single-token requests

malaiwah
authored and
myshytf
committed
27e71ca
fix(comm): prewarm FlashInfer PCIe graph shapes
myshytf
committed
74ec7af

Commits on Sep 1, 2026

chore(spec_decode): snapshot the production remote speculator

myshytf
and
claude
committed
faf3b78
perf(spec_decode): defer draft context ingest for mid-prefill steps

myshytf
and
claude
committed
0da4120
perf(spec_decode): send synchronous proposal frames without copying

myshytf
and
claude
committed
27d51ea
perf(comm): serve prefill-size all-reduces on the b12x DMA ring

myshytf
and
claude
committed
ff3c3fa
perf(comm): route in-place TP all-reduces through the custom all-reduce

myshytf
and
claude
committed
d7fda8c

Commits on Sep 2, 2026

Commits on Sep 3, 2026

perf(spec_decode): consume the remote K3 draft reply in stream order

myshytf
and
claude
committed
655c324
fix(spec_decode): keep the context-position staging buffer separate from the reply staging

myshytf
and
claude
committed
af76545
fix(spec_decode): defer remote drafts only with late input ids

myshytf
and
codex
committed
6799804
fix(kimi_k3): consume out-of-place all-reduce results

myshytf
and
codex
committed
26c1512
fix(comm): require explicit all-peer support for DMA

myshytf
and
codex
committed
dc54180
fix(spec_decode): retain every deferred-ingest failure

myshytf
and
codex
committed
4b4beb0
Merge communication correctness fixes into top-k draft transport
myshytf
committed
71ec045
Merge communication correctness fixes into deferred draft resolve
myshytf
committed
a64c318
fix(spec_decode): resolve drafts before request-slot reuse

myshytf
and
codex
committed
84070e4
fix(spec_decode): use effective draft auxiliary geometry

myshytf
and
codex
committed
4d8d493
Merge standalone draft geometry validation into deferred resolve
myshytf
committed
9d24c94
fix(spec_decode): honor the DFlash draft load config

myshytf
and
codex
committed
0a43db3
Merge DFlash load-configuration forwarding into deferred resolve
myshytf
committed
654a8b4
fix(spec_decode): clear every smoke-test cache block

myshytf
and
codex
committed
489b037
Merge complete smoke-cache cleanup into deferred resolve
myshytf
committed
e9f27cc
fix(spec_decode): append capture indexes in constant work

myshytf
and
codex
committed
b9491e0
Merge bounded capture indexes into top-k draft transport
myshytf
committed
638f201
Merge bounded capture indexes into deferred resolve
myshytf
committed
75793e9

Commits on Sep 4, 2026