server: retain recurrent prompt checkpoints for reuse

[ekg]

Summary

This adds server-side checkpoint retention for recurrent-model prompt reuse.

• add recurrent-state rollback snapshots per sequence via --recurrent-snapshots
• retain mid-prompt checkpoints so long shared prefixes can resume from a nearby checkpoint instead of replaying from the beginning
• replace simple FIFO checkpoint pruning with a coverage-oriented policy that keeps old anchors, recent checkpoints, and thins dense interior checkpoints
• add a focused regression test for checkpoint spacing and pruning policy

Motivation

Hybrid/recurrent models can reuse long prompt prefixes, but without a restorable recurrent state the server can only safely reuse exact tail-aligned cache state. For long agent/chat histories, a small rollback window causes otherwise reusable common prefixes to be replayed from the start. Keeping bounded recurrent snapshots plus ladder checkpoints makes long-prefix reuse practical while keeping memory bounded.

Testing

• cmake --build build --config Release -j --target llama-server test-server-checkpoint-policy
• ctest --test-dir build -R test-server-checkpoint-policy --output-on-failure

Opened as draft because the option names and checkpoint-retention policy are intended for maintainer review before this is considered final.

[ekg]

Local live validation on Qwen3.6-35B-A3B-MTP GGUF, using this PR branch (b9757-83d6d2c97) with:

--ctx-checkpoints 32 --checkpoint-min-step 256 --recurrent-snapshots 16 --cache-ram 4096 --spec-type draft-mtp --spec-draft-n-max 2 -c 65536

I sent two chat-completion requests with the same long shared prefix and only a small suffix change, max_tokens=1.

Client-side timing:

request A: elapsed=361.31s prompt_tokens=37817 completion_tokens=1 fingerprint=b9757-83d6d2c97
request B: elapsed=0.72s   prompt_tokens=37817 completion_tokens=1 fingerprint=b9757-83d6d2c97

Relevant server log excerpt:

20.49.196 I slot print_timing: task 619 | prompt eval time = 360867.18 ms / 37816 tokens (104.79 tokens/s)
20.49.196 I slot release: task 619 | stop processing: n_tokens = 37817

20.49.258 I slot operator(): task 645 | Checking checkpoint with [37812, 37812] against 37803, rs rollback = 10...
20.49.274 W slot operator(): task 645 | restored context checkpoint (pos_min = 37812, pos_max = 37812, n_tokens = 37813, n_past = 37803, size = 137.388 MiB)
20.49.341 I slot create_check: task 645 | created context checkpoint 19 of 32 (pos_min = 37802, pos_max = 37802, n_tokens = 37803, size = 137.368 MiB)
20.49.914 I slot print_timing: task 645 | prompt eval time = 655.79 ms / 14 tokens (21.35 tokens/s)
20.49.915 I slot release: task 645 | stop processing: n_tokens = 37817

So the second request avoided replaying the 37.8k-token shared prefix and restored from the near-tail recurrent checkpoint, replaying only the changed suffix.

[ekg]

I'd like to bump this. These updates have been essential for getting useful behavior out of these hybrid recurrent models. If there are other approaches being explored here, I'll be happy to close this and follow those, but if not I request review of this approach.

[pwilkin]

@aldehir it's another one :)

[ggerganov]

I sent two chat-completion requests with the same long shared prefix and only a small suffix change, max_tokens=1.

This already works on master - no changes are needed.

[ekg]

Yup. #24176 fixed this and was merged just after I implemented this. Sorry for the noise.