Skip to content

runtime: the AMD recheck list names the llama.cpp issue, and two comments stop contradicting the file they are in - #1261

Merged
gen16k merged 1 commit into
mainfrom
docs/amd-recheck-names-the-qwen4exp-issue
Sep 6, 2026
Merged

gen16k merged 1 commit into
mainfrom
docs/amd-recheck-names-the-qwen4exp-issue

Conversation

@gen16k

@gen16k gen16k commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Why

#1250 wrote the per-bump recheck list for the AMD backend arms. Three things it did not cover, found while re-reading the qwen4exp KV allocation for #1255.

What

A llama.cpp issue on a different recheck axis. New §6 in docs/knowledges/20260906/1700-what-to-recheck-about-amd-backends.md, plus a matching entry in the !!! MAINTENANCE block above amdROCmSupportedRes.

ggml-org/llama.cpp#27856 (open, checked 2026-09-06) — qwen4exp (Qwen3.8-Flash-Next): severe decode slowdown beyond ~1K context on HIP / gfx1151 (Strix Halo). Decode is 19-21 tok/s below ~512 tokens, falls off a cliff past ~1k, and plateaus at 5.5-6.1 tok/s from 2k onward (4.6 at 45k) — a 3.5-4x collapse. Prefill degrades too. Output stays correct (a 45k needle-in-haystack returns the exact value), so nothing here or in the catalog would notice. -fa off does not remove it, which points at the QSA indexer / ggml_top_k / gather chain rather than the flash-attention kernels. CUDA decays only mildly on the same model and quant.

Two things make it worth its own section rather than a fifth row in §1:

  • It moves with the vendored llama.cpp version, not with ollama's release or its ROCm overlay — so the reading that settles §1 and §2 will not answer it, and ollama's release notes never will.
  • It lands on the Linux arm, not the Windows one. Windows Strix Halo already names Vulkan and never reaches HIP. Linux prefers ROCm, and resolveBackendWithProbe (feat(cli): enroll with an auth key, through the daemon (#175) #290) cannot catch this: it falls back only on positive evidence of CPU residency (size_vram == 0), and a model that is on the GPU but 4x slow at depth is not that. That is the documented conservatism, not a defect — but the failure shape is outside the arm, and the note says so.

The reporter's environment is Ubuntu 24.04 on ROCm 7.2.4, so §2's "7.2.4 is the performance answer" holds here too: the version bump does not remove it. No Linux Strix Halo is in the fleet, so this is recorded as an upstream report and explicitly not as an observation of ours.

Two stale comments in internal/runtime/ollama_backend.go, both from #1247 correcting one copy of a fact and not its twin:

site said measured
BackendROCm doc (:24) Windows ROCm overlay is ~350 MiB WantsROCm sixty lines below already carries 247 MB at 0.33.3, read off the asset, as does the knowledge note §4
ollama_backend_test.go case name "strix halo windows: vulkan only (no ROCm on Win APU)" the arm it pins says, verbatim, "Vulkan, because it is FASTER here — not because ROCm is absent" — ROCm does engage gfx1151 under Windows (#1233)

The test's assertion was and is correct; the name was the last surviving copy of the claim #1233 disproved. Renamed, with a comment on the line saying what it used to read so the change is not silently re-derived later.

Verification

go test ./internal/runtime/...        ok (3 packages)
gofmt -l internal/runtime             clean

Swept the tree for other copies of both figures: every remaining mention of the overlay size is now ~250 MB / 247 MB (cmd/waired/runtimes_install_windows.go:80, its test, the note), and every remaining "no ROCm on Windows APU" is past-tense in a record of the correction.

No behaviour change — comments, one test case name, and a knowledge-note section. internal/runtime/ is in neither testnet-relevant-paths.txt nor docs-surface-guard's SURFACES, and docs/knowledges/** is in neither, so this PR arms no testnet and needs no docs opt-out.

Refs #1233
Refs #1247
Refs #1255

🤖 Generated with Claude Code

https://claude.ai/code/session_0167iiPnKQz1qcuz2bhNBGep

…ents stop contradicting the file they are in

The Strix Halo arms rest on upstream facts that move without notice, and
#1250 wrote the recheck list for them. Three things it did not cover.

ggml-org/llama.cpp#27856 (open, checked 2026-09-06) has qwen4exp — the
architecture behind the Flash-Next entry #1259 just shipped — collapsing
3.5-4x in decode once context passes ~1k on HIP/gfx1151, plateauing at
5.5-6.1 tok/s where CUDA decays only mildly. It moves with the VENDORED
LLAMA.CPP version rather than with ollama's release or its ROCm overlay,
so it is on a different recheck axis from the four ollama threads already
listed and ollama's release notes will never mention it.

It also lands on the arm nobody was watching. The Windows arm already
names Vulkan, so it never reaches HIP; the LINUX arm prefers ROCm, and
the #290 probe cannot see this failure — it falls back only on positive
evidence of CPU residency (size_vram == 0), and a model that is on the
GPU but 4x slow at depth is not that. Conservative by design, but the
failure shape is outside the arm. No Linux Strix Halo is in the fleet, so
this is an upstream report, not an observation here; the note says so.

Two stale comments in the same file, both left by #1247 correcting one
copy of a figure and not the other:

  - BackendROCm's doc calls the Windows ROCm overlay ~350 MiB. WantsROCm,
    sixty lines below it, already carries the measured 247 MB at 0.33.3,
    as does the knowledge note.
  - The backend table test names its case "vulkan only (no ROCm on Win
    APU)". The arm it pins says the opposite in as many words — "Vulkan,
    because it is FASTER here — not because ROCm is absent" — because
    ROCm does engage gfx1151 under Windows. The assertion was right; the
    name was the last copy of the claim #1233 disproved.

No behaviour change: comments, one test case name, and a knowledge-note
section.

Refs #1233
Refs #1247
Refs #1255

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0167iiPnKQz1qcuz2bhNBGep
Signed-off-by: gen16k <gen16k@users.noreply.github.com>
@gen16k
gen16k merged commit 92eeffd into main Sep 6, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant