Skip to content

server : respect min-step when splitting prompt batches - #25420

Merged
aldehir merged 1 commit into
ggml-org:masterfrom
aldehir:server-respect-min-step
Jul 9, 2026
Merged

server : respect min-step when splitting prompt batches#25420
aldehir merged 1 commit into
ggml-org:masterfrom
aldehir:server-respect-min-step

Conversation

@aldehir

@aldehir aldehir commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Overview

The server checkpoint logic breaks on every user message start, which does not respect --checkpoint-min-step and negatively impacts prefill speed. This PR corrects that by aligning with min-step.

fixes #25213
fixes #25320

Requirements

@aldehir
aldehir requested a review from a team as a code owner July 7, 2026 22:17
@github-actions github-actions Bot added the server label Jul 7, 2026
@aldehir aldehir mentioned this pull request Jul 9, 2026
Merged
@aldehir
aldehir merged commit 64c8b7d into ggml-org:master Jul 9, 2026
25 checks passed
fewtarius added a commit to fewtarius/CachyLLama that referenced this pull request Jul 19, 2026
@fewtarius
merge upstream/master: 135 commits (DFlash, Hy3 MTP, Q2_0, DeepseekV4…
cf4f42e 
…, OpenCL Q6_K/Adreno, CORS, checkpoint min-step, prompt cache refactor, MoE expert API stays)

Upstream highlights since 6be7459:
- model: DFlash speculative with KV rotation (ggml-org#25823)
- model: Hy3 (hy_v3) with MTP speculative decoding (ggml-org#25395)
- model: DeepseekV4 with fused hyper-connection ops (ggml-org#25585)
- ggml: 0.17.0, LIGHTNING_INDEXER, out_prod, f16 set_rows
- vulkan: Q2_0 support, native e2m1/e4m3 conversions, transfer-queue race fix
- CUDA: MMQ kernel config refactor (ggml-org#24127), tighter MMQ src1 buffer for fp4 (ggml-org#25613), CUDA graphs on Volta/Turing, MoE gate/up dedup, CUDA Virtual Devices
- ROCm: hexagon L2 cache rework, native fp4, FP16/INT8 coopmat on AMD
- SYCL: Battlemage flash attention via oneDNN XMX, XIELU op, fp16 conv2d_dw
- OpenCL: Q6_K GEMM/GEMV fix, ragged-tile MoE prefill FP16, Adreno vectorized LD/ST, A7x optimizations, ABS op
- kleidiai: SME2 f32 kernel, SME vs SME2 dispatch
- server: refactor prompt cache state ownership (ggml-org#25649) - new server_prompt_cache_state separates prompt metadata from KV data
- server: evict checkpoints within min-step (ggml-org#25472)
- server: text-only slot save/restore with mtmd (ggml-org#25076)
- server: --cors-* options (ggml-org#25655)
- server: refactored server_stream (ggml-org#25541)
- server: respect min-step when splitting prompt batches (ggml-org#25420)
- server: move chat-template thinking probe inside init try/catch (ggml-org#24093)
- common: auto-download dflash/eagle3 HF sidecars (ggml-org#25811), drop --stdin mutual-exclusion, align tokenize usage
- conversion: BitNetForCausalLM, dflash tokenizer fix, split MTP export for HY V3
- llama-quant: exclude i32 ffn_gate_tid2eid routing table, allow manual tensor types with --pure
- llama-batch: fix allowed decreasing pos in a seq (ggml-org#25449), n_keep_tail in split_equal for recurrent
- llama: refactor fused ops (ggml-org#24646), TP fix for Phi3/Bert/Plamo2/3/ChatGLM
- ui: agentic content UX, reasoning effort on mobile add sheet, MCP panel fixes, thinking menu fix
- vendor: BoringSSL 0.20250713.0
- tests: actually exercise test-recurrent-state-rollback, ds_v4_hc sentinel init, export-graph-ops graceful exit

CachyLLama preservation work (conflict resolution):

1. tools/server/server-task.h: Accept upstream's server_prompt refactor (no data member, clear() method).
   Move our t_last_used field from server_prompt to server_prompt_cache_state (where it now lives
   after the refactor). server_prompt_cache_state already has the size() method, so our old
   size() on server_prompt is no longer needed.

2. tools/server/server-context.cpp (create_checkpoint): Take upstream's min-step eviction
   pre-filter as the FIRST pass, then keep our existing highest-pos_min eviction as the
   capacity overflow fallback. These are complementary: min-step removes redundant checkpoints
   from the same task; highest-pos_min keeps the rec-window-friendly checkpoints when at cap.

3. tools/server/server-context.cpp (handle_completions_impl): Keep our std::vector<server_task>
   tasks batching for multi-prompt requests and per-user concurrency check, AND take upstream's
   res->set_req(&req) for spipe ownership transfer.

4. tools/server/server-task.cpp: Fix references to entry.tokens -> entry.prompt.tokens,
   entry.checkpoints -> entry.prompt.checkpoints, entry.n_tokens() -> entry.prompt.n_tokens().
   Update find_eviction_candidate return type from list<server_prompt>::iterator to
   list<server_prompt_cache_state>::iterator.

5. ggml/src/ggml-cuda/mmq.cuh + new mmq-config-rdna3_5.cuh: Upstream's massive MMQ refactor
   moved per-architecture config into separate files but did NOT add RDNA3.5 (gfx1150/1/2/3,
   Strix Halo). Create mmq-config-rdna3_5.cuh (231 CASE entries) derived from rdna2 with
   nthreads=128 (4 warps) and I=48 (smaller X tile) matching our original Strix Halo tuning.
   Wire into both host and device dispatch paths before the RDNA4 / RDNA2 fallback.

6. README.md and AGENTS.md: Keep CachyLLama-specific links and project context where upstream
   added parallel content.

Verified:
- cmake --build builds clean (Release, CPU-only)
- llama-server starts, --help shows all CachyLLama flags preserved:
  --cache-ssd-hot-ram, --cache-ssd-warm-ram, --cache-ssd-system-prompts,
  --cache-ssd-system-max-days, --cache-ssd-no-fsync, --cache-ssd-max-conversations,
  --max-concurrent-per-user
- /expert-stats and /expert-tracking endpoints preserved
- 55/58 tests pass; 3 failures unrelated to merge:
  - test-tokenizers-ggml-vocabs: missing model downloads
  - test-jinja-py: missing jinja2 Python module
  - test-quant-type-selection: snapshot mismatch on upstream's new MXFP4_MOE heuristic

Custom CachyLLama files untouched (no upstream conflicts):
- common/kv-ssd-cache.{cpp,h}, common/kv-ssd-posix.h, common/kv-ssd-system-cache.{cpp,h}
- common/kv_page_manager.{cpp,h}
- tools/server/server-context-page-manager.{cpp,h}
- tools/server/server-context-ssd-cache.{cpp,h}
- test_kv_page_manager.cpp, tests/test-ssd-cache-caps.cpp
- STRIX_HALO_NOTES.md, docs/development/user-isolation-design.md
- .github/workflows/build-cpu.yml, build-cuda-windows.yml, build-vulkan.yml
CowboyTim pushed a commit to aardbeiplantje/llama.cpp that referenced this pull request Jul 21, 2026
@aldehir ROCm FP4 Contributor
server : respect min-step when splitting prompt batches (ggml-org#25420)
55cf500
satindergrewal pushed a commit to satindergrewal/llama.cpp that referenced this pull request Aug 12, 2026
@aldehir
server : respect min-step when splitting prompt batches (ggml-org#25420)
db3c2ec
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ggerganov ggerganov ggerganov approved these changes
@ServeurpersoCom ServeurpersoCom ServeurpersoCom approved these changes

Assignees

No one assigned

Labels

server

Projects

None yet

Milestone

No milestone

3 participants

@aldehir @ggerganov @ServeurpersoCom