runtime: the AMD recheck list names the llama.cpp issue, and two comments stop contradicting the file they are in - #1261
Merged
Conversation
…ents stop contradicting the file they are in The Strix Halo arms rest on upstream facts that move without notice, and #1250 wrote the recheck list for them. Three things it did not cover. ggml-org/llama.cpp#27856 (open, checked 2026-09-06) has qwen4exp — the architecture behind the Flash-Next entry #1259 just shipped — collapsing 3.5-4x in decode once context passes ~1k on HIP/gfx1151, plateauing at 5.5-6.1 tok/s where CUDA decays only mildly. It moves with the VENDORED LLAMA.CPP version rather than with ollama's release or its ROCm overlay, so it is on a different recheck axis from the four ollama threads already listed and ollama's release notes will never mention it. It also lands on the arm nobody was watching. The Windows arm already names Vulkan, so it never reaches HIP; the LINUX arm prefers ROCm, and the #290 probe cannot see this failure — it falls back only on positive evidence of CPU residency (size_vram == 0), and a model that is on the GPU but 4x slow at depth is not that. Conservative by design, but the failure shape is outside the arm. No Linux Strix Halo is in the fleet, so this is an upstream report, not an observation here; the note says so. Two stale comments in the same file, both left by #1247 correcting one copy of a figure and not the other: - BackendROCm's doc calls the Windows ROCm overlay ~350 MiB. WantsROCm, sixty lines below it, already carries the measured 247 MB at 0.33.3, as does the knowledge note. - The backend table test names its case "vulkan only (no ROCm on Win APU)". The arm it pins says the opposite in as many words — "Vulkan, because it is FASTER here — not because ROCm is absent" — because ROCm does engage gfx1151 under Windows. The assertion was right; the name was the last copy of the claim #1233 disproved. No behaviour change: comments, one test case name, and a knowledge-note section. Refs #1233 Refs #1247 Refs #1255 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0167iiPnKQz1qcuz2bhNBGep Signed-off-by: gen16k <gen16k@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#1250 wrote the per-bump recheck list for the AMD backend arms. Three things it did not cover, found while re-reading the qwen4exp KV allocation for #1255.
What
A llama.cpp issue on a different recheck axis. New §6 in
docs/knowledges/20260906/1700-what-to-recheck-about-amd-backends.md, plus a matching entry in the!!! MAINTENANCEblock aboveamdROCmSupportedRes.ggml-org/llama.cpp#27856 (open, checked 2026-09-06) —
qwen4exp (Qwen3.8-Flash-Next): severe decode slowdown beyond ~1K context on HIP / gfx1151 (Strix Halo). Decode is 19-21 tok/s below ~512 tokens, falls off a cliff past ~1k, and plateaus at 5.5-6.1 tok/s from 2k onward (4.6 at 45k) — a 3.5-4x collapse. Prefill degrades too. Output stays correct (a 45k needle-in-haystack returns the exact value), so nothing here or in the catalog would notice.-fa offdoes not remove it, which points at the QSA indexer /ggml_top_k/ gather chain rather than the flash-attention kernels. CUDA decays only mildly on the same model and quant.Two things make it worth its own section rather than a fifth row in §1:
resolveBackendWithProbe(feat(cli): enroll with an auth key, through the daemon (#175) #290) cannot catch this: it falls back only on positive evidence of CPU residency (size_vram == 0), and a model that is on the GPU but 4x slow at depth is not that. That is the documented conservatism, not a defect — but the failure shape is outside the arm, and the note says so.The reporter's environment is Ubuntu 24.04 on ROCm 7.2.4, so §2's "7.2.4 is the performance answer" holds here too: the version bump does not remove it. No Linux Strix Halo is in the fleet, so this is recorded as an upstream report and explicitly not as an observation of ours.
Two stale comments in
internal/runtime/ollama_backend.go, both from #1247 correcting one copy of a fact and not its twin:BackendROCmdoc (:24)~350 MiBWantsROCmsixty lines below already carries 247 MB at 0.33.3, read off the asset, as does the knowledge note §4ollama_backend_test.gocase name"strix halo windows: vulkan only (no ROCm on Win APU)"The test's assertion was and is correct; the name was the last surviving copy of the claim #1233 disproved. Renamed, with a comment on the line saying what it used to read so the change is not silently re-derived later.
Verification
Swept the tree for other copies of both figures: every remaining mention of the overlay size is now
~250 MB/247 MB(cmd/waired/runtimes_install_windows.go:80, its test, the note), and every remaining "no ROCm on Windows APU" is past-tense in a record of the correction.No behaviour change — comments, one test case name, and a knowledge-note section.
internal/runtime/is in neithertestnet-relevant-paths.txtnor docs-surface-guard'sSURFACES, anddocs/knowledges/**is in neither, so this PR arms no testnet and needs no docs opt-out.Refs #1233
Refs #1247
Refs #1255
🤖 Generated with Claude Code
https://claude.ai/code/session_0167iiPnKQz1qcuz2bhNBGep