Incorporate SSD Caching from CachyLLama (Experimental) - #20
Draft
3rdIteration wants to merge 16 commits into
Draft
3rdIteration wants to merge 16 commits into
3rdIteration wants to merge 16 commits into
Conversation
ci: add Windows MSVC Vulkan CI with downloadable packages
Port the SSD cache feature set from 3rdIteration/CachyLLama (by fewtarius) onto the ROCmFPX base (upstream ~2026-05-14 snapshot): - common/kv-ssd-cache, kv-ssd-system-cache, kv_page_manager: tiered hot/warm/cold KV checkpoint storage with per-user namespaces - kv-ssd-posix.h: Windows pread/pwrite/mkdir shims - server-context-page-manager, server-context-ssd-cache: server glue - llama_memory_seq_rm_attn_only: clear attention cache without touching recurrent state (hybrid models) after checkpoint restore - MTP draft pending_h get/set_state so SSD cold-start restore keeps the cross-token carry-over embedding - --cache-ssd* args, llama_user_id request routing, per-user 429 concurrency cap (--max-concurrent-per-user), overlap-aware prompt cache eviction, deferred final checkpoints Differences from CachyLLama: - MoE expert-tracking API excluded (no consumers in server code) - 429 fast-fail check moved before rd.post_tasks(); upstream iterated a moved-from vector, making the fast path dead code - kept ROCmFPX's checkpoint-restore error handling (ON_DEVICE flags, restore-failure fallback) and wove SSD hooks into it Verified on Windows (MSVC, CPU and Vulkan/gfx1151 builds): checkpoint store + cold-start restore across server restart with -ngl 99 (702 -> 4 prompt tokens reprocessed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On Strix Halo the BIOS carve-out leaves Windows only ~32 GB of host RAM. At long contexts each SSD checkpoint store serializes the full sequence state (~0.8-1.5 GiB) into host buffers and then copied it again into the hot tier, spiking ~2x state size per store. When the allocation failed the bad_alloc was uncaught and the server died silently mid-prefill (reproduced with Qwen3.5-122B-A10B at ~50k tokens: free RAM 0, commit 80/90 GiB, silent exit right after "created context checkpoint"). - kv_ssd_store: hot-tier retention is now best effort - skip it when the blob exceeds the hot budget (previously it evicted the whole tier and hoarded the blob anyway) or when the copy throws bad_alloc; the checkpoint is already durable on disk and stays cold - server_ssd_cache::store/load: catch bad_alloc, skip the store / fall back to prompt reprocessing instead of terminating - create_checkpoint / deferred_create_final_checkpoint: catch bad_alloc, drop the partial checkpoint and continue serving Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The auto-sizer claimed 85% of free RAM (75/25 hot/warm) for tiers whose only benefit is skipping a disk read on restore. On unified-memory machines the host RAM, "VRAM" and these tiers drain the same physical pool, so oversized tiers directly starve the KV cache and prefill staging. Auto-size now only scales the budgets down under memory pressure; explicit -ssd-hot-ram / -ssd-warm-ram still override. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add the missing SSD cache section to the server README (the ported docs only covered user isolation) and a dedicated note for unified-memory machines like Strix Halo, where the BIOS carve-out leaves the host a small slice of RAM and the hot/warm tiers compete with the KV cache for the same physical pool. Suggest explicit small tier budgets there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the unconditional 1 GiB/512 MiB auto-size cap with detection: an integrated GPU with no discrete GPU present (Strix Halo and similar) means "VRAM", host RAM and the cache tiers share one physical pool, so the auto-sizer is capped there and logs the detection. Machines with a discrete GPU keep the original free-RAM based scaling. Also fix two latent config bugs found in the plumbing: - explicit -ssd-hot-ram/-ssd-warm-ram were silently overwritten by auto-sizing; they now disable it (missing counterpart falls back to 1 GiB / 512 MiB) - kv_eviction_config defaults said "6GB/2GB" but were 6 MiB/2 MiB Verified on Strix Halo: startup logs "unified-memory (iGPU) system detected" and first-conversation init auto-sizes to hot=1024 warm=512 (previously 5222/1740 on a 32 GB host). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
github-actions
Bot
added
documentation
…m-2026-07-12 # Conflicts: # tools/server/server-context.cpp
The merge of origin/rocmFPX-cachyllama into upstream-2026-07-12 auto-resolved common/common.h and common/common.cpp to the upstream side of the common_prompt_checkpoint struct. Both branches had added members after data_dft (upstream: storage_tgt/storage_dft; cachyllama: data_spec), and the non-conflicting auto-merge silently dropped the cachyllama data_spec member while keeping all code that references it (server-context.cpp, server-context-ssd-cache.cpp), breaking the CI build. Restore the data_spec member and its handling in the copy/move constructors and assignment operators, size(), clear(), and clear_dft(), matching the rocmFPX-cachyllama definition. Verified with a local llama-server build in both CPU-only and Vulkan (CI-matching) configs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Enhance ROCmFPX with safety profiles, benchmarks, and optimizations
Add a windows-hip job to build-packages.yml producing a runnable Windows ROCm package, modeled on the proven upstream jobs already in this repo (build.yml windows-latest-hip: full clang build on Windows; release.yml windows-hip: HIP SDK 26.Q1 install + rocWMMA headers + ROCm runtime DLL bundling) and on the working Windows ROCm (HIP) job in 3rdIteration/rocmfp4-llama. - GPU targets: RDNA2/3/3.5/4 including gfx1151 (Strix Halo) - Full build (server + CPU variants + ggml-hip) with GGML_BACKEND_DL - Bundles hipblas/rocblas/hipblaslt DLLs and tensile libraries so the artifact runs without a local HIP SDK install - Trigger on push to rocmFPX-cachyllama as well, and save ccache on push so PR builds restore a warm cache from the base branch Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ci: add Windows x64 HIP (ROCm) package build
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incorporates the excellent work of @fewtarius over at https://github.com/fewtarius/CachyLLama currently just leaving as draft while I test some more.
This pull request introduces a new SSD-backed KV cache system with tiered RAM/SSD storage and adds extensive configuration options for it. It also exposes new command-line arguments and configuration fields to control SSD cache behavior and per-user concurrency in the server. Additionally, it updates the build system and workflow to include the new cache code.
The most important changes are:
SSD-Backed KV Cache Implementation:
common/kv-ssd-cache.h).common/CMakeLists.txt).Configuration and Command-Line Interface:
common/arg.cpp).--max-concurrent-per-user) (common/arg.cpp,common/common.h). [1] [2]Core Data Structure Updates:
common_paramsstruct with fields for all SSD cache and prompt cache configuration options, as well as per-user concurrency cap (common/common.h). [1] [2]Build and CI Integration:
.github/workflows/build-packages.yml).Testing & Notes
So far I have been mostly stress testing this with models using a mix of standard quants, rocmFP4 and the rocmFPX version of Step 3.7.
The results so far are quite impressive and it's certainly a massive improvement if switching between tasks, working with large system prompts or even just being able to reboot and continue working on a large job, reloading 100k+ of context in seconds. (Or experiemental llama that crashes ;) )
Requirements