server : refactor prompt cache state ownership - #25649
Merged
Merged
Conversation
Assisted-by: pi:llama.cpp/Qwen3.6-27B
2 tasks
ggerganov
marked this pull request as ready for review
July 14, 2026 09:31
1 task
1 task
fewtarius
added a commit
to fewtarius/CachyLLama
that referenced
this pull request
Jul 19, 2026
…, OpenCL Q6_K/Adreno, CORS, checkpoint min-step, prompt cache refactor, MoE expert API stays) Upstream highlights since 6be7459: - model: DFlash speculative with KV rotation (ggml-org#25823) - model: Hy3 (hy_v3) with MTP speculative decoding (ggml-org#25395) - model: DeepseekV4 with fused hyper-connection ops (ggml-org#25585) - ggml: 0.17.0, LIGHTNING_INDEXER, out_prod, f16 set_rows - vulkan: Q2_0 support, native e2m1/e4m3 conversions, transfer-queue race fix - CUDA: MMQ kernel config refactor (ggml-org#24127), tighter MMQ src1 buffer for fp4 (ggml-org#25613), CUDA graphs on Volta/Turing, MoE gate/up dedup, CUDA Virtual Devices - ROCm: hexagon L2 cache rework, native fp4, FP16/INT8 coopmat on AMD - SYCL: Battlemage flash attention via oneDNN XMX, XIELU op, fp16 conv2d_dw - OpenCL: Q6_K GEMM/GEMV fix, ragged-tile MoE prefill FP16, Adreno vectorized LD/ST, A7x optimizations, ABS op - kleidiai: SME2 f32 kernel, SME vs SME2 dispatch - server: refactor prompt cache state ownership (ggml-org#25649) - new server_prompt_cache_state separates prompt metadata from KV data - server: evict checkpoints within min-step (ggml-org#25472) - server: text-only slot save/restore with mtmd (ggml-org#25076) - server: --cors-* options (ggml-org#25655) - server: refactored server_stream (ggml-org#25541) - server: respect min-step when splitting prompt batches (ggml-org#25420) - server: move chat-template thinking probe inside init try/catch (ggml-org#24093) - common: auto-download dflash/eagle3 HF sidecars (ggml-org#25811), drop --stdin mutual-exclusion, align tokenize usage - conversion: BitNetForCausalLM, dflash tokenizer fix, split MTP export for HY V3 - llama-quant: exclude i32 ffn_gate_tid2eid routing table, allow manual tensor types with --pure - llama-batch: fix allowed decreasing pos in a seq (ggml-org#25449), n_keep_tail in split_equal for recurrent - llama: refactor fused ops (ggml-org#24646), TP fix for Phi3/Bert/Plamo2/3/ChatGLM - ui: agentic content UX, reasoning effort on mobile add sheet, MCP panel fixes, thinking menu fix - vendor: BoringSSL 0.20250713.0 - tests: actually exercise test-recurrent-state-rollback, ds_v4_hc sentinel init, export-graph-ops graceful exit CachyLLama preservation work (conflict resolution): 1. tools/server/server-task.h: Accept upstream's server_prompt refactor (no data member, clear() method). Move our t_last_used field from server_prompt to server_prompt_cache_state (where it now lives after the refactor). server_prompt_cache_state already has the size() method, so our old size() on server_prompt is no longer needed. 2. tools/server/server-context.cpp (create_checkpoint): Take upstream's min-step eviction pre-filter as the FIRST pass, then keep our existing highest-pos_min eviction as the capacity overflow fallback. These are complementary: min-step removes redundant checkpoints from the same task; highest-pos_min keeps the rec-window-friendly checkpoints when at cap. 3. tools/server/server-context.cpp (handle_completions_impl): Keep our std::vector<server_task> tasks batching for multi-prompt requests and per-user concurrency check, AND take upstream's res->set_req(&req) for spipe ownership transfer. 4. tools/server/server-task.cpp: Fix references to entry.tokens -> entry.prompt.tokens, entry.checkpoints -> entry.prompt.checkpoints, entry.n_tokens() -> entry.prompt.n_tokens(). Update find_eviction_candidate return type from list<server_prompt>::iterator to list<server_prompt_cache_state>::iterator. 5. ggml/src/ggml-cuda/mmq.cuh + new mmq-config-rdna3_5.cuh: Upstream's massive MMQ refactor moved per-architecture config into separate files but did NOT add RDNA3.5 (gfx1150/1/2/3, Strix Halo). Create mmq-config-rdna3_5.cuh (231 CASE entries) derived from rdna2 with nthreads=128 (4 warps) and I=48 (smaller X tile) matching our original Strix Halo tuning. Wire into both host and device dispatch paths before the RDNA4 / RDNA2 fallback. 6. README.md and AGENTS.md: Keep CachyLLama-specific links and project context where upstream added parallel content. Verified: - cmake --build builds clean (Release, CPU-only) - llama-server starts, --help shows all CachyLLama flags preserved: --cache-ssd-hot-ram, --cache-ssd-warm-ram, --cache-ssd-system-prompts, --cache-ssd-system-max-days, --cache-ssd-no-fsync, --cache-ssd-max-conversations, --max-concurrent-per-user - /expert-stats and /expert-tracking endpoints preserved - 55/58 tests pass; 3 failures unrelated to merge: - test-tokenizers-ggml-vocabs: missing model downloads - test-jinja-py: missing jinja2 Python module - test-quant-type-selection: snapshot mismatch on upstream's new MXFP4_MOE heuristic Custom CachyLLama files untouched (no upstream conflicts): - common/kv-ssd-cache.{cpp,h}, common/kv-ssd-posix.h, common/kv-ssd-system-cache.{cpp,h} - common/kv_page_manager.{cpp,h} - tools/server/server-context-page-manager.{cpp,h} - tools/server/server-context-ssd-cache.{cpp,h} - test_kv_page_manager.cpp, tests/test-ssd-cache-caps.cpp - STRIX_HALO_NOTES.md, docs/development/user-isolation-design.md - .github/workflows/build-cpu.yml, build-cuda-windows.yml, build-vulkan.yml
CowboyTim
pushed a commit
to aardbeiplantje/llama.cpp
that referenced
this pull request
Jul 21, 2026
* server : clear checkpoints upon prompt clear * server : move the prompt state data to the server_prompt_cache Assisted-by: pi:llama.cpp/Qwen3.6-27B * server : handle batched slot being cleared
smalinin
pushed a commit
to smalinin/llama.cpp
that referenced
this pull request
Aug 4, 2026
* server : clear checkpoints upon prompt clear * server : move the prompt state data to the server_prompt_cache Assisted-by: pi:llama.cpp/Qwen3.6-27B * server : handle batched slot being cleared
satindergrewal
pushed a commit
to satindergrewal/llama.cpp
that referenced
this pull request
Aug 12, 2026
* server : clear checkpoints upon prompt clear * server : move the prompt state data to the server_prompt_cache Assisted-by: pi:llama.cpp/Qwen3.6-27B * server : handle batched slot being cleared
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
alt #25647
server_prompt_cachenow owns the KV state via a newserver_prompt_cache_statewrapper, whileserver_promptremains purely logical.batched_slotcould end up being cleared in the middle of processing the current batch, causing an ASSERT to fire: https://github.com/ggml-org/llama.cpp/actions/runs/29315070349/job/87030532460Assisted-by
YES. pi:llama.cpp/Qwen3.6-27B