Enhance ROCmFPX with safety profiles, benchmarks, and optimizations - #3
Merged
Merged
Conversation
(cherry picked from commit 221402a)
(cherry picked from commit e32c64d)
(cherry picked from commit 5cdf4b7)
…graph-safe FA, docs ROCmFP4 quantize + CPU dot (rocmfp4.c), all bit-identical (A/B hash verified): - Fuse the code->decode round-trip inside the exhaustive scale search and use a precomputed UE4M3 half-scale table: ~31% faster quantize. - Add an AVX2 CPU vec-dot, runtime-dispatched via __builtin_cpu_supports (ggml-base is built without arch flags): ~3-4x faster q4_0_rocmfp4 dot product. ROCmFP3/6/8 quantize (rocmfpx.c), bit-identical: - Precompute the 127-entry UE4M3 scale table so the scale search and per-element dequant stop calling ldexpf, and fuse the threshold/round + decode step in the fp3/fp6 MSE search: ~9% faster weighted quantize. GPU decode (vecdotq.cuh, mmvq.cu): - Enable FP6 MMVQ HALF_BLOCK_SPLIT by default (+1.85% tg); FP3 MMVQ branch hoist. Quantization routing (llama-quant.cpp): - Let NVFP4 and Q4_0 requantize into ROCmFP4 without --allow-requantize, with an honest note on where each source is near-lossless (NVFP4) vs lossy (Q4_0). - Protect speculative/MTP draft tensors: route the nextn shared-head/embeddings to output/token-embedding precision and floor the small fc / nextn projection heads to Q8_0. Precision only ever increases, so quality cannot regress. - Fix the NEXTN_ENORM op tag (GET_ROWS -> MUL) to match how enorm is consumed. Flash attention (HIP, fattn): - Make the quantized-KV f16 temp buffers HIP graph-capture safe: reuse the ggml pool during capture and small decode batches, and route small quantized-KV batches through the VEC kernel so no cudaMalloc/cudaFree runs mid-capture. Numerical hardening (CUDA/HIP): - Replace -INFINITY reduction/softmax sentinels with -FLT_MAX across softmax, top-k MoE, cross-entropy, dsv4, and the block-reduce max policy so NaN inputs cannot poison the max reductions. Tooling: - Add Q4_0_ROCMFP4_EVEN / Q4_0_ROCMFP4_FAST_EVEN presets (imply --pure) for whole-model even conversions. Docs: - README: add a Quick Start, a "which format should I pick" table, an NVFP4 -> ROCmFP4 conversion guide, and an MTP speculative-decoding section. - gitignore local benchmark plots and internal handoff/convert scratch files. Verified on gfx1151: test-backend-ops MUL_MAT / FLASH_ATTN_EXT / SOFT_MAX pass on ROCm0 + Vulkan0 + CPU; test-quantize-fns passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The --embd-gemma-default, --vision-gemma-4b-default and --vision-gemma-12b-default presets set the model but no runtime flags, so QAT models ran on CPU out of the box. Default them to full GPU offload (n_gpu_layers=-2); the two generative vision presets also enable FlashAttention. Measured on Gemma-4 12B QAT Q4_0 (gfx1151/ROCm0): tg128 13.1 -> 27.3 t/s, pp512 261 -> 321. Embedding preset keeps FA at auto (non-causal encoder). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
eagle3: ggml_concat used the loop index il as the concat axis; it only worked because eagle3 asserts n_layer==1. Use literal dim 0. dflash: the encoder fc input width was computed from the draft n_embd instead of the target hidden size (eagle3 already reads target_hidden_size). Read that key with a fallback to n_embd so existing GGUFs without it load unchanged, and log the resolved value. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Captures the known-good AMD Strix Halo build (HIP + Vulkan + native CPU, gfx1151) as a reproducible preset. The winning decode knobs (GGML_ROCMFP6_MMVQ_HALF_BLOCK_SPLIT=1, GGML_ROCMFPX_RDNA35_NWARPS=1) are already the in-source defaults, so no cache override is needed. New preset only; existing presets untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The DFlash draft converter wrote target_layers but not target_hidden_size, so the loader could only fall back to the draft n_embd for the encoder fc input width. Read the target model's hidden size (via --target-model-dir) and emit it; guarded so a read failure warns and falls back instead of breaking the conversion. Completes the loader-side fix in dflash.cpp. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
QAT models (Gemma etc.) run at full quality as Q4_0 but only reach their speed with GPU offload + FA. When one is loaded with -ngl 0, log a one-line hint to add '-ngl 999 -fa on' (~2x decode measured on Gemma QAT). Log only; no behavior change. Detects via general.name / model path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full backend+spec sweep on gfx1151 (Qwen3.6, ROCm0 vs Vulkan0): dense 27B : Vulkan MTP n6/p0.6 = 22.1 t/s (vs ROCm no-spec 13.5) MoE 35B-A3B: Vulkan no-spec = 79.2 t/s (MTP <=78.5, never wins) - capabilities: add --is-moe (expert_count/expert_used_count markers). - server: dense spec sweet spot n_max 4->6, p_min 0.75->0.6 (base + decode-fast preset); auto-disable draft-mtp for MoE, since the batched verify pass loads the expert union and cancels spec savings (override with FORCE_MTP_MOE=1). DEVICE default was already Vulkan0. Verified assembled command: dense keeps draft-mtp n6/p0.6 on Vulkan0; MoE drops draft-mtp and stays no-spec on Vulkan0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update note summarizing this branch's changes (arch-aware MTP, QAT preset offload, strix-rocmfpx preset, eagle3/dflash fixes) and a rewritten MTP section: dense uses MTP (n6/p0.6), MoE runs no-spec, Vulkan0 is the faster decode backend on gfx1151. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…Cm required) - Add scripts/build-rocmfp4-rocm714-local.sh: auto-downloads ROCm 7.14.0a20260624 toolchain, builds for gfx1200;gfx1201, bundles runtime libs into build-*/lib/ with RPATH set to $ORIGIN/../lib - Update README.md build table and docs with the new option
…branch feat: add self-contained RDNA4 build script for RX 9000 (no system RO…
The experimental branch carried db24788's "MTP safety" rewrite, which added per-step overhead that throttled MTP on fast MoE models (A3B): MoE draft-mtp was net-neutral (76.9 vs 77.6 no-spec). origin/main never had db24788 and has a leaner draft_mtp using the same fork pre_norm API. Swapped main's common_speculative_state_draft_mtp in. Struct-only change; nothing else touched (quant kernels, FA, other spec types all intact). Measured (Qwen3.6-35B-A3B, Vulkan0): ROCmFP4 MoE MTP: 76.9 -> 93.4 t/s (+20% vs no-spec 77.6) Q4_K_M MoE MTP: 66 -> 73.5 t/s Verified: gate exact (n_max 1==4==8 byte-identical); gemma4 assistant MTP runs with 0 decode errors (the vanilla-upstream struct corrupted its KV positions; this does not); dense +58% unchanged; both backends OK. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With the main-lineage draft_mtp (prev commit), MoE models gain +20% from draft-mtp (Qwen3.6-35B-A3B ROCmFP4/Vulkan0: 77.6 -> 93.4 t/s), not a loss. Remove the MoE auto-disable that was added when the db24788 draft_mtp throttled MTP. Dense and MoE both keep n_max 6 / p_min 0.6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…sabled) qwen35 / qwen35moe (and every IMROPE/MROPE arch) map to LLAMA_ROPE_TYPE_IMROPE -> n_pos_per_embd()==4, so the MTP draft/verify batch goes through the M-RoPE branch of the batch position check in llama_batch_allocr::init(). That branch gated the strict "X < Y" rule on `if (batch.token)`, but the MTP hook batch is hybrid: it carries a token id (for the embedding lookup) AND an injected pre-norm hidden-state row in batch.embd. The nextn head legitimately re-decodes at the token's own position (X == Y), so the strict rule failed on every draft step: init: ... for M-RoPE, it is required that the position satisfies: X < Y decode: failed to initialize batch / llama_decode: ret = -1 draft-mtp then silently fell back to plain decode on all M-RoPE models (Gemma4's NEOX + shared-KV path never hit this, which hid the bug). Gate the strict check on `batch.token && !batch.embd`, so hybrid MTP batches take the same lenient "X <= Y" path as pure-embedding batches; a real backward jump (X > Y) is still rejected. Only affects the n_pos_per_embd>1 branch and only batches carrying both token and embd (produced solely by the MTP hook) -> no change to normal M-RoPE decode, vision, or the draft_mtp logic. Measured (gfx1151, Vulkan0, temp0, n_max 6 / p_min 0.6), 0 M-RoPE errors: Qwable-5-27B-Coder (qwen35 dense): no-spec 14.0 -> MTP 47.9 t/s (3.4x) Qwen3.6-35B-A3B (qwen35moe): no-spec 75.7 -> MTP 115.4 t/s (1.5x) README updated: MoE now uses MTP (the earlier "MoE runs no-spec / expert-union ceiling" guidance was an artifact of MTP never engaging on these M-RoPE models). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit c0eb33b)
(cherry picked from commit 5258985)
(cherry picked from commit dcad182)
(cherry picked from commit 547321d)
(cherry picked from commit c3342ee)
(cherry picked from commit eddc62e)
(cherry picked from commit 6e85f2a)
(cherry picked from commit 811bcab)
(cherry picked from commit ea2c49a)
(cherry picked from commit ac9cdf3)
Keep backend-only platform jobs independent from embedded WebUI assets, preserve dedicated WebUI coverage, and stop generation when provisioning is incomplete. Based on upstream llama.cpp commit(s): - 0c3e4fc - fix: Propagate version tag to WebUI asset download in self-hosted CI (#23051) - Aleksander Grygier <aleksander.grygier@gmail.com> - 1348f67 - webui: Use lowercase hash for HF checksum check (#23107) - Omer Ozarslan <omerfaruko@gmail.com> Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Co-authored-by: Omer Ozarslan <omerfaruko@gmail.com>
A backend-sampled token can be valid for the vocabulary without appearing in the current CPU candidate view for another parallel slot. Return that valid token directly and reserve CPU fallback for genuinely invalid token IDs.
Advertise GET_ROWS only for the 25 types instantiated by ggml-metal.metal. This lets unsupported ROCmFPX types fall back safely instead of compiling a nonexistent pipeline and segfaulting during dispatch.
CPU support already advertises quantized OUT_PROD and allocates F32 scratch for it. Add the five ROCmFPX formats to the existing quantized dispatch so high-performance CPU lanes execute them instead of aborting.
The source snapshot imported Apple workflows that execute this root script but omitted the tracked file. Restore the exact executable from the matching upstream lineage. Based on upstream llama.cpp commit(s): - 4d74287 - build : use umbrella Headers directory for XCFramework module map (#23974) - Gerard Martinez <gmarzjr@proton.me> Co-authored-by: Gerard Martinez <gmarzjr@proton.me>
…perimental-to-main-2026-07-11 Promote validated experimental ROCmFPX branch to main
…-a-coffee Add Buy Me a Coffee funding link
github-actions
Bot
added
documentation
…m-2026-07-12 # Conflicts: # tools/server/server-context.cpp
The merge of origin/rocmFPX-cachyllama into upstream-2026-07-12 auto-resolved common/common.h and common/common.cpp to the upstream side of the common_prompt_checkpoint struct. Both branches had added members after data_dft (upstream: storage_tgt/storage_dft; cachyllama: data_spec), and the non-conflicting auto-merge silently dropped the cachyllama data_spec member while keeping all code that references it (server-context.cpp, server-context-ssd-cache.cpp), breaking the CI build. Restore the data_spec member and its handling in the copy/move constructors and assignment operators, size(), clear(), and clear_dft(), matching the rocmFPX-cachyllama definition. Verified with a local llama-server build in both CPU-only and Vulkan (CI-matching) configs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Additional information
Requirements