server: allow accessing /metrics and /slots during llama_decode() by ngxson · Pull Request #27041 · ggml-org/llama.cpp

ngxson · 2026-08-13T21:46:27Z

Overview

Allow access /metrics and /slots during llama_decode() call. Fix #24866

This works by:

Having a worker thread on server_queue
llama_decode() decode will run on the worker thread (via yield_to_queue)
During waiting for llama_decode to finish, new tasks will still be handled, but only metrics task is accepted

Same logic for mtmd processing: yield_to_queue do the heavy task in worker, while main thread continue to process new tasks

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: yes

ServeurpersoCom · 2026-08-14T06:01:01Z

The PR works, I checked it on my side: with a large prompt prefilled in a single batch, the worst /slots latency drops from roughly the whole decode on master to a few ms on your branch.

Heads up on the Python test though: test_metrics_served_while_decoding passes on master too, I ran it against a build of 2606220 without yield_to_queue and it was green 4/4. With n_predict=200 and the default batch the loop already gets back to process_new_tasks between each per-token decode, so /slots answers in a few ms either way and none of the three assertions can tell the two builds apart.

ngxson · 2026-08-14T10:05:33Z

Thanks for testing. I think the timings cannot be guaranteed in testing, we might need to have a fixed delay. But I don't think it matters now.

Removing the e2e test here, I will add a C++ component-level test in a follow-up

…ml-org#27041) * server_queue::worker * call llama_decode inside yield_to_queue * also handle process_mtmd_chunk * clean up * nits * rm test

101 upstream commits (tip 1692f9e). Conflict resolutions keep both sides: paged hybrid/SWA/MSA gates + n_batch refuse, inkling tensors, MUSE_GLIMMER/MIMO2/STEP35 saver+arch fixtures, mtmd inkling+muse+pockettts, server /metrics-during-decode (PR ggml-org#27041) with DS4P reval/quench counters. Automerge audit (control-flow read): Metal champion mask barrier (4b62ac7) and n_batch!=n_ubatch refuse (08f85db) survived. Paged checkout/release and update_slots_paged survived. Incoming yield_to_queue wraps the static decode path only; paged decode stays synchronous. SSM-scan K rollback landed in Metal/CUDA without touching paged kernels. Not pushed. :8082 pid 31896 left running on the pre-merge binary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Upstream ggml-org#27041 moved n_decoded / prompt counters / timing onto server_slot_stats. The paged launch, mirror, decode, draft, and yield-quench sites still named the old members, so llama-server did not compile. Map them onto stats (n_gen, n_prompt_processed, update_prompt_start/last, update_gen_last). Restore has_media() on server_tokens -- the paged media refusal still keys on it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ml-org#27041) * server_queue::worker * call llama_decode inside yield_to_queue * also handle process_mtmd_chunk * clean up * nits * rm test

Brings 218 upstream commits into CachyLLama without losing any of our features. Key carried-over changes from upstream: - llama.cpp v0.2.0 / ggml v0.21.0 version bumps - Vulkan FA MMQ fp32 scaling (ggml-org#27413), PAD_REFLECT_1D (ggml-org#26586), tiled transpose (ggml-org#26585), null checks in queue command pools cleanup (ggml-org#27353) - ggml: rope_set_offset on multiple backends, recurrent state rollback - Vulkan coopmat1 SHMEM_STRIDE_PAD/APPLY_SLM_A_RESHAPE for Intel Xe - server: LLAMA_SERVER_SLOTS_N_DIFF (ggml-org#27600), /metrics during llama_decode (ggml-org#27041), index.html no-cache (ggml-org#27006), make-release workflow - model: MiniMax-M1/Text01 (ggml-org#27018), Kimi-K3 (ggml-org#26185), BailingMoE3 (ggml-org#26608), GraniteSWA (ggml-org#25505), GLM-4.5-Air MTP, DSV4 tensor split (-sm tensor) - ui: Chat Conversation Tabbed navigation, settings refactor - common: --models-dir loading MTP assistant models (ggml-org#24431), --load-mode replacing --mmap (ggml-org#26934), json.h abstraction (ggml-org#27511) - vendor: cpp-httplib 0.53.1, BoringSSL 0.20260813.0, vendor/hash CachyLLama features preserved through conflict resolution: - Persistent SSD-backed KV cache (3-tier hot/warm/cold + system prompt cache) - Per-user isolation (user_id, per-user concurrency cap, slot affinity) - MoE expert residency + co-activation tracking - CachyLLama Vulkan Lightning Indexer (108/108 on Strix Halo) + DSV4 hyper-connection fused ops + DSV4 sparse FA + coopmat shaders - FA quant-KV dequant-once + f16 contiguize (with host-RAM safety gate) - DFlash framework + Laguna-S-2.1 model support - DFlash d2t reduced-vocab draft support (upstream merge) - Context checkpoint ring buffer + SWA skip + memory budget scaling - Stable-prefix LCP gate + prompt_stable_prefix_tokens param - conv_hash conversation-boundary detection - All CachyLLama Vulkan shaders (concat_transpose, lightning_indexer, mmid_row_lists, flash_attn_top_k, dequant_f16_transpose) - common::host_available_ram() utility - llama-moe-residency + llama-moe-coact modules Manual conflict resolution touches: src/models/dflash.cpp (DFlash d2t + aux_norm), src/llama-kv-cache-dsv4.cpp (state snapshot fix), src/llama- memory-recurrent.cpp (rs_idx bounds check), src/llama-model-saver.cpp (DSV4 compress_ratios + swiglu_clamp sizing), ggml/src/ggml-vulkan/ {ggml-vulkan.cpp,vulkan-shaders-gen.cpp,vulkan-shaders/dequant_q8_0. comp,vulkan-shaders/flash_attn.comp,vulkan-shaders/copy_transpose_02. comp} (CachyLLama shader registration + FA scratch gate), ggml/src/ ggml-cuda/mmvq.cu (RDNA3_5 + GB10 enum), gguf-py/gguf/constants.py (DFlash ENC_AUX_NORM + D2T tensors), tests/{CMakeLists.txt,test-backend- ops.cpp,test-llama-archs.cpp,test-recurrent-state-rollback.cpp} (test additions), tools/{CMakeLists.txt,server/*} (server_batch embd support + spec_is_replay + user_id routing + MCP servers + CORS), and docs/{AGENTS.md,README.md} (kept CachyLLama branding). Verified: full build succeeds, test-backend-ops Vulkan LIGHTNING_INDEXER + FLASH_ATTN pass on Strix Halo. Based on a re-merge from the 20260824 (pristine pre-merge) branch after a previous agent's merge attempt produced an unbuildable state from -X ours that wiped shader float-typing and broke the dequant_q8_0 + flash_attn shaders with redefinition errors.

…a dead sampler clone Audit of every synchronization point in the server's speculative decode path, plus the one change that fell out of it. The ordering is already what a single thread can do: llama_decode() does not synchronize, so the target batch stays in flight until the first llama_get_logits_ith(), and common_speculative_process() is a pure submission on ctx_dft whose result nobody reads until the next iteration's common_speculative_draft(). Since the filtered replay landed [TAG_SPEC_AVOID_DRAFT_REEVAL] the draft decode is already issued before all of post_decode()'s sampling, detokenization and JSON work, and - across sub-batches - before the next llama_decode(ctx_tgt). Nothing in between forces a ctx_dft sync except two known cases, both now named in the code. Not changed, deliberately: - the explicit llama_synchronize(ctx_tgt) in decode() stays. It is not there for timing alone: it puts the wait inside yield_to_queue so /metrics and /slots stay answerable while the GPU works (ggml-org#27041, ggml-org#27133), and metrics_post_decode() flushes the queued prompt timings on the promise that the context is synchronized by then. Deferring it would make t_prompt measure submission instead of compute and would move the wait back onto the main thread. - splitting the decode and the sync into two separate yields to open a CPU shadow buys microseconds of bookkeeping at the price of a second thread handoff per batch. Not worth it. - multi-slot draft() interleaving needs a real restructure, out of scope here. What did change: the per-slot common_sampler_clone() in the verification pass is only ever consumed by the checkpoint-restore branch, which is unreachable unless the target context cannot drop the rejected rows by range. For a plain transformer target it was a grammar clone plus an n_vocab-sized candidate-array copy (~384 KiB at 32k vocab, ~1.8 MiB at 151k) executed on every verification step, sitting exactly between the target sync and the ctx_dft submission it delays. It is now taken only when that branch can be reached. Equivalence at temp 0 (stories15M_MOE + stories15M-q4_0 draft, draft-simple, n-min 4 / n-max 8, top_k 1, seed 4242, 3 chained turns, CPU only): identical text, identical per-step acceptance sequence (50 steps, 63/378), identical metrics. The restore path was covered separately with a recurrent target (LFM2-test-ci-80M as both target and draft, temp 1.6 to force rejections): 75 steps, 38 restores, byte-identical output before and after. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019MCRCNAWoP9xTWZNKs5fTx

ngxson added 2 commits August 13, 2026 18:59

server_queue::worker

fca1761

call llama_decode inside yield_to_queue

0b42458

github-actions Bot added the server label Aug 13, 2026

ngxson added 3 commits August 13, 2026 23:54

also handle process_mtmd_chunk

a35d2af

clean up

d809c0f

nits

98e489e

ngxson marked this pull request as ready for review August 13, 2026 22:47

ngxson requested a review from a team as a code owner August 13, 2026 22:47

ServeurpersoCom approved these changes Aug 14, 2026

View reviewed changes

ggerganov approved these changes Aug 14, 2026

View reviewed changes

rm test

3d6761b

ServeurpersoCom approved these changes Aug 14, 2026

View reviewed changes

ngxson merged commit 77918ca into master Aug 14, 2026
27 of 28 checks passed

phattja mentioned this pull request Aug 15, 2026

Misc. bug: Severe generation speed regression (~10-15×) on CPU starting from b10429 (all models) #27126

Closed

Navigation Menu

Uh oh!

server: allow accessing /metrics and /slots during llama_decode() - #27041

server: allow accessing /metrics and /slots during llama_decode()#27041
ngxson merged 6 commits into
masterfrom
xsn/non_blocking_metrics

ngxson commented Aug 13, 2026

Uh oh!

ServeurpersoCom commented Aug 14, 2026

Uh oh!

ngxson commented Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants