docs : document the fork as a Strix Halo project, allow agent PRs - #6
Merged
Merged
Conversation
The README, AGENTS.md and CONTRIBUTING.md were still upstream's, so this repo described itself as ggml-org/llama.cpp and banned the automated PR submissions it wants to accept. README: what the fork targets (gfx1151, RDNA 3.5 iGPU, unified memory), the ggml-org -> halo-box/llama.cpp -> here chain, Vulkan and ROCm builds, the Strix Halo runtime notes (GTT sizing, HIP_LAUNCH_BLOCKING, GGML_VK_MMV_NO_SPLIT), and the three deltas this fork carries. AGENTS.md: agents may branch, commit, push and open PRs here, and are told that a cached upstream AGENTS.md governs the other two repos, not this one. Requirements replace the ban: measure on the hardware, search first, disclose, stay mergeable, never push to master or self-merge. CONTRIBUTING.md: routing section for which repo a change belongs in, an AI policy that permits agent-authored PRs with disclosure, and a benchmarking section that sets the evidence bar - environment block, a baseline built from the merge-base and run in the same session, raw llama-bench tables with their standard deviations, and test-backend-ops or perplexity evidence that output did not change. Coding and naming guidelines kept verbatim. PR template: stopped telling agents the project restricts AI-generated content, added the environment block, baseline/after tables, and a "what was not verified" field. Assisted-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016frbY6RGoR4qJks3iKJLui
dzannotti
approved these changes
Aug 29, 2026
Nathanw1014
added a commit
that referenced
this pull request
Sep 8, 2026
…ecode CM shader Builds directly on pepuscz's PR #6 and issue #10. Their per-kernel table - 43.8 us scalar against 27.9 us small CM per 1k scanned rows at batch 5, and the same ratio at every depth - is what showed the indexer's cost is a per-tile constant rather than anything to do with tokens or bytes. Once the cost is per tile, the thing to minimise is the tile count, and that is what this changes. The finding is downstream of their work; only the choice of shader differs. The decode CM shader puts 16 HEADS in the coopmat N dimension and dispatches one workgroup per token, so it issues 4*n_batch tiles per 16 KV rows. That is the arithmetic minimum - (64 heads x n_batch tokens) / 16 columns - because 64 heads fill its 16 columns exactly, with no remainder at any batch. The small CM shader puts 16 TOKENS in N and pays a flat 64 tiles however small the batch is, so at batch 5 eleven of its sixteen columns are padding. Measured cost tracks tile count: at kv=131584 batch 1 costs 1.95 us per 1k scanned rows and batch 5 costs 26.28, i.e. 13.5x for 5x the work, and both shaders sit at 6.6-7.8 ns per tile. The decode CM shader body has no n_batch == 1 assumption - token is gl_WorkGroupID.y and it indexes q/w/mask/dst by it - so the old gate was an artefact of where it was written. gfx1151, kv=131584 (526k source tokens), us/run, shipped vs this: batch 2 2335.6 -> 419.0 (5.6x, was on the scalar path) batch 3 3492.3 -> 631.8 (5.5x, was on the scalar path) batch 4 3267.9 -> 840.8 (3.9x) batch 5 3465.6 -> 1119.4 (3.1x) <- DSpark n-max 4 verify shape batch 8 3704.9 -> 1764.1 (2.1x) batch 15 4467.9 -> 3231.6 (1.4x) Batch 16 and 32 are unchanged, which confirms the arms are isolated. PR #6's route is kept as the opt-out arm rather than deleted, and is promoted from opt-in to default-on so that one variable is enough to reach it: default decode CM for the whole 2-15 window ..._DECODE_CM_BATCH=0 small CM for 4-15, scalar for 2-3 (PR #6) ..._DECODE_CM_BATCH=0 SMALL_CM=0 scalar for 2-15 (pre-PR #6 baseline) Verified all three route as documented and pass 29/29. Keeping it is not just courtesy: the tile-count argument is hardware independent, but decode CM re-reads the K tile once per token and that part is bandwidth dependent, so the crossover need not sit in the same place on other devices, and these numbers are from one gfx1151 box. A (head, token) packing shader was also tried and is strictly worse - it reaches 4b tiles only when the batch divides 16, and it divides the workgroup count by the batch. Kept out of tree. Numerics: decode CM sums the 64 heads in tiles of 16 rather than one at a time, so f32 accumulation order differs from the small CM path. Cleared by KLD A/B on trunc10 (-c 8192, -b/-ub small so every indexer dispatch routes through the window, 8 chunks wikitext-2, same binary, only the env var differing): -ub 8 mean KLD 0.000000, max 6.0e-5, RMS dp 0.000%, same-top 99.994%, PPL 10507558.3156 identical in both arms -ub 5 mean KLD 0.000000, max 8.8e-4 but 99.9% 4.9e-5 (one tail event, no argmax flip), RMS dp 0.000%, same-top 100.000% Same class as PR #6's own measured numbers (max 5.5e-5, same-top 100.000%) and 45x tighter than the FA_WAVE32 change already shipped. Adds the eval coverage the 2-15 window never had (batches 4/8/15 at kv=256) and the perf grid these numbers came from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Nathanw1014
added a commit
that referenced
this pull request
Sep 8, 2026
…ecode CM shader Builds directly on pepuscz's PR #6 and issue #10. Their per-kernel table - 43.8 us scalar against 27.9 us small CM per 1k scanned rows at batch 5, and the same ratio at every depth - is what showed the indexer's cost is a per-tile constant rather than anything to do with tokens or bytes. Once the cost is per tile, the thing to minimise is the tile count, and that is what this changes. The finding is downstream of their work; only the choice of shader differs. The decode CM shader puts 16 HEADS in the coopmat N dimension and dispatches one workgroup per token, so it issues 4*n_batch tiles per 16 KV rows. That is the arithmetic minimum - (64 heads x n_batch tokens) / 16 columns - because 64 heads fill its 16 columns exactly, with no remainder at any batch. The small CM shader puts 16 TOKENS in N and pays a flat 64 tiles however small the batch is, so at batch 5 eleven of its sixteen columns are padding. Measured cost tracks tile count: at kv=131584 batch 1 costs 1.95 us per 1k scanned rows and batch 5 costs 26.28, i.e. 13.5x for 5x the work, and both shaders sit at 6.6-7.8 ns per tile. The decode CM shader body has no n_batch == 1 assumption - token is gl_WorkGroupID.y and it indexes q/w/mask/dst by it - so the old gate was an artefact of where it was written. gfx1151, kv=131584 (526k source tokens), us/run, shipped vs this: batch 2 2335.6 -> 419.0 (5.6x, was on the scalar path) batch 3 3492.3 -> 631.8 (5.5x, was on the scalar path) batch 4 3267.9 -> 840.8 (3.9x) batch 5 3465.6 -> 1119.4 (3.1x) <- DSpark n-max 4 verify shape batch 8 3704.9 -> 1764.1 (2.1x) batch 15 4467.9 -> 3231.6 (1.4x) Batch 16 and 32 are unchanged, which confirms the arms are isolated. PR #6's route is kept as the opt-out arm rather than deleted, and is promoted from opt-in to default-on so that one variable is enough to reach it: default decode CM for the whole 2-15 window ..._DECODE_CM_BATCH=0 small CM for 4-15, scalar for 2-3 (PR #6) ..._DECODE_CM_BATCH=0 SMALL_CM=0 scalar for 2-15 (pre-PR #6 baseline) Verified all three route as documented and pass 29/29. Keeping it is not just courtesy: the tile-count argument is hardware independent, but decode CM re-reads the K tile once per token and that part is bandwidth dependent, so the crossover need not sit in the same place on other devices, and these numbers are from one gfx1151 box. A (head, token) packing shader was also tried and is strictly worse - it reaches 4b tiles only when the batch divides 16, and it divides the workgroup count by the batch. Kept out of tree. Numerics: decode CM sums the 64 heads in tiles of 16 rather than one at a time, so f32 accumulation order differs from the small CM path. Cleared by KLD A/B on trunc10 (-c 8192, -b/-ub small so every indexer dispatch routes through the window, 8 chunks wikitext-2, same binary, only the env var differing): -ub 8 mean KLD 0.000000, max 6.0e-5, RMS dp 0.000%, same-top 99.994%, PPL 10507558.3156 identical in both arms -ub 5 mean KLD 0.000000, max 8.8e-4 but 99.9% 4.9e-5 (one tail event, no argmax flip), RMS dp 0.000%, same-top 100.000% Same class as PR #6's own measured numbers (max 5.5e-5, same-top 100.000%) and 45x tighter than the FA_WAVE32 change already shipped. Adds the eval coverage the 2-15 window never had (batches 4/8/15 at kv=256) and the perf grid these numbers came from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
This was referenced Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
The README,
AGENTS.mdandCONTRIBUTING.mdin this repo were still upstream's. The practical effect was that the repo introduced itself asggml-org/llama.cpp, and its agent policy contained a non-overridable instruction to refusegh pr createand to exclude the project from an agent's contribution targets - the opposite of what this fork wants.This replaces those three files, plus the PR template that enforced the same upstream rules on every PR.
README.md
gfx1151, RDNA 3.5 iGPU, unified LPDDR5X.ggml-org/llama.cpp->halo-box/llama.cpp(upstream staging) -> here - and says plainly that nothing is submitted upstream from this repo.gfx1151, using./build/bin/llama-cli(thellama cliform in upstream's README comes from the installer and does not apply to a from-source fork).amdgpu.gttsize,ttm.pages_limit), theHIP_LAUNCH_BLOCKING=1batched-inference workaround,GGML_VK_MMV_NO_SPLIT=1,GGML_VK_PERF_LOGGER.--spec-draft-adaptive, and the multi-point--reasoning-budget-*mechanism.AGENTS.md
ABSOLUTELY REFUSEblock, the "STOP and update your memory to exclude llama.cpp" instruction, and the refusal example dialogues.gh pr create, and reply to reviews; must measure on hardware, search first, disclose with anAssisted-by:trailer, keep diffs mergeable against upstream, and be explicit about gaps; must not push tomaster, self-merge, or open PRs against the two upstream repos from this tree.AGENTS.mdgovernsggml-org/llama.cppandhalo-box/llama.cpp, not this fork, since that file is what most agents will have seen.ggml-vulkan.cpp.CONTRIBUTING.md
llama-benchtables with their standard-deviation columns rather than a summarized percentage, varying the axis the change acts on, treating overlap inside stddev as no result, andtest-backend-ops/llama-perplexityevidence that output did not change..github/pull_request_template.md
halo-box/llama.cpp, an AI disclosure field with a NO / ASSISTED / AGENT-AUTHORED answer, and a "what was NOT verified" field.Measurements
Not applicable - documentation and PR-template changes only, no runtime code touched.
Additional information
Flags and tool behaviour referenced in the new docs were checked against this tree rather than from memory:
--spec-draft-adaptive(common/arg.cpp), the nine--reasoning-budget-*options added in #5,GGML_VK_MMV_NO_SPLITandGGML_VK_PERF_LOGGER(ggml/src/ggml-vulkan/ggml-vulkan.cpp),llama-bench-p/-n/-d/-ub/-fa/-rwith its default of 5 repetitions, andtest-backend-opsmodes and-o/-bflags. TheHIP_LAUNCH_BLOCKINGnote and the perplexity figures come from the comment on thegpu-rocmjob in.github/workflows/build-self-hosted.yml. All internal links resolve; all four files are ASCII-only per the house rule.The hardware-setup paragraph names the
amdgpu.gttsizeandttm.pages_limitknobs without prescribing values, since the right ones depend on kernel version - worth a look from someone who has tuned this on current kernels.Requirements
docs/build.mdwithgfx1151substituted, and have not been executed in this session.Assisted-by: Claude Opus 5