server: fix checkpoint handling for hybrid/recurrent models (#24055) - #25592
server: fix checkpoint handling for hybrid/recurrent models (#24055)#25592krim404 wants to merge 2 commits into
Conversation
…g#24055) The recurrent state is only valid at the exact position it was saved at, but checkpoints claimed the whole [pos_min, pos_max] range reported by the memory module, and restoring relied on that by accident. Record the actual position instead and only restore a checkpoint when that position is still inside the common prefix of the new prompt. Also erase checkpoints that overlap edited history - once the new tokens are decoded they would look valid again while holding state from content that no longer exists. While at it, make the cache actually useful for agentic clients that strip reasoning from previous turns: always checkpoint near the end of the prompt (that is where the next request diverges), evict the checkpoint closest to its neighbor instead of the oldest one so early anchors survive compaction, try the bounded n_rs_seq rollback before searching checkpoints, and log checkpoint activity at INFO so you can see what the cache is doing without -lv.
…ckpoints The n_rs_seq rollback snapshots are only guaranteed valid for tokens decoded in the last ubatch, so rolling back across decode boundaries could silently restore a stale state - always go through the checkpoint path instead. Adopt a restored checkpoint into the current task so the min-step eviction does not erase it, compare the stale bound in position space (mtmd positions differ from token counts), and keep range (SWA) checkpoints at the exact divergence position since resuming overwrites that entry anyway.
5eb68b9 to
e75a77a
Compare
|
Rebased on current master to resolve the conflict with #25472 and #25649, which landed the near-prompt-end checkpointing and the min-step eviction from this same work package upstream. This PR now carries only the remaining parts: exact-position checkpoints for hybrid/recurrent memory, prefix-based restore, and stale-checkpoint invalidation. Two adjustments on top of the rebase (second commit):
|
|
I am using this + #26004 and the checkpoints and slots are finally usable with Qwen3.6. Previously this was full re-processing. |
Verified working on RTX 3090 + Qwen3.8-27B-IQ4_NL (16GB, qwen35 hybrid)I cherry-picked both commits from this PR onto current master ( Small prompt (~4.8K tokens)
Large prompt (~38.5K tokens)
Observations
Environment
Thanks for the fix — this makes hybrid models actually usable for long agentic sessions. |
|
This is a real and important fix, I've encountered it just now - trying to use qwen 3.8 and not being able to restore anything... Really hope it is merged ASAP |
Apply upstream llama.cpp PR ggml-org#25592 to the fork for Qwen hybrid/recurrent prompt-cache correctness.
llama.cpp PR #25592 Test ReportTested this on Qwen3.6-35B-A3B with This PR appears to fix the checkpoint reuse problem for me. Before applying it, I was frequently seeing: and large portions of the conversation would be prefetched again on later agent turns. After applying this PR, I still see an occasional full reprocess when a slot has no compatible state or the prompt diverges significantly, which seems expected. But once valid checkpoints are established, subsequent turns are consistently reusing the existing context. Some examples from one session: while the slot itself had grown past 18k tokens. So instead of repeatedly processing the entire conversation, normal agent turns are now processing mostly just the new/changed portion. I was seeing generation around 225–230 tok/s and prompt prefill in the several-thousand tok/s range, so this made a very noticeable difference in end-to-end agent latency. I also separately ported the prompt-cache RAM fix from #24649, so I don't want to conflate that with this PR, but the checkpoint restore/reuse behavior specifically looks fixed by #25592. Thanks for working on this — this makes Qwen3.6 much more practical for long-running agentic workloads. I also built a prebuilt CUDA image from my fork with these fixes applied, in case anyone wants to test this without rebuilding llama.cpp themselves: docker pull ghcr.io/tultr/llama.cpp:masterSource: https://github.com/tultr/llama.cpp The image is built directly from the patched |
|
Adding a data point from our side: diagnosing this one was genuinely painful — the symptom was just "full prompt re-processing every turn" with no error, no warning, nothing in the logs pointing at checkpoint invalidation. We spent a lot of time chasing it before landing on the hybrid/recurrent exact-position restore path. We've been running this patch (cherry-picked onto current master) in a production agentic workload since mid-August — long multi-turn sessions, 130K context, RTX 3090 — and it has held up without a single regression. The ~150-400x turn-latency improvement on restored checkpoints is the difference between usable and unusable for agent workloads. Happy to run anything else the maintainers need before merge — longer soak, different model archs, whatever helps. |
|
This is critical for hybrid models like qwen, hope this gets merged asap. Thank you for the work. |
|
Today I built and deployed a local patch of llama.cpp PR #25592 to fix context checkpoint On hybrid SSM+attention models, saved context checkpoints were invalidated when tuned aggressively due to PR #25592 (open/unmerged) sets +1 |
7 participants
Overview
this is a reimplementation of #24797
That PR was rejected because it could reuse recurrent state at positions it
was never valid for (trimming checkpoint metadata in place, resetting rs_idx
when a rollback exceeds the available snapshots). This takes the opposite
approach and keeps the invariant intact: recurrent state is only ever
restored at the exact position it was saved at.
(
pos_min = pos_maxfor hybrid/recurrent) instead of relying on whatseq_pos_minhappens to report - resolves[TAG_CHECKPOINTS_FIX_POS_MIN]for these model types
common prefix of the new prompt
llama_memory_hybridsemantics;seq_rmstill failscleanly on impossible rollbacks (now with a debug log)
On top of that, the cache is made useful for agentic clients, which strip
the reasoning of the previous reply so the next request diverges right at
the end of the previous prompt:
--checkpoint-min-stepoldest one, so early anchors survive edited/compacted histories
n_rs_seqrollback is tried before the checkpoint searchAdditional information
Related: #24055
Tested with Qwen3.6-35B-A3B Q4_K_M on Vulkan (RDNA3.5 APU). Resending
identical 22.8k-token request went from a ~96s full prefill to a 4-tok
cache hit. In a 10-turn agentic session normal turns only process the new
content, a mid-history edit correctly falls back to the last valid
checkpoint, and a compacted history restores an early checkpoint instead of
reprocessing everything. Correctness: at temperature 0 a checkpoint-restored
run produces byte-identical output to a fresh full prefill, and two
independent cold runs are byte-identical to each other.
Requirements