Apply hybrid/recurrent checkpoint fix - #1
Merged
Merged
Conversation
added 2 commits
July 14, 2026 18:21
…g#24055) The recurrent state is only valid at the exact position it was saved at, but checkpoints claimed the whole [pos_min, pos_max] range reported by the memory module, and restoring relied on that by accident. Record the actual position instead and only restore a checkpoint when that position is still inside the common prefix of the new prompt. Also erase checkpoints that overlap edited history - once the new tokens are decoded they would look valid again while holding state from content that no longer exists. While at it, make the cache actually useful for agentic clients that strip reasoning from previous turns: always checkpoint near the end of the prompt (that is where the next request diverges), evict the checkpoint closest to its neighbor instead of the oldest one so early anchors survive compaction, try the bounded n_rs_seq rollback before searching checkpoints, and log checkpoint activity at INFO so you can see what the cache is doing without -lv.
…ckpoints The n_rs_seq rollback snapshots are only guaranteed valid for tokens decoded in the last ubatch, so rolling back across decode boundaries could silently restore a stale state - always go through the checkpoint path instead. Adopt a restored checkpoint into the current task so the min-step eviction does not erase it, compare the stale bound in position space (mtmd positions differ from token counts), and keep range (SWA) checkpoints at the exact divergence position since resuming overwrites that entry anyway.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports upstream llama.cpp PR ggml-org#25592 onto this fork so Qwen hybrid/recurrent models restore context checkpoints only at valid exact positions and avoid unnecessary full prompt reprocessing in agentic workloads.