Throwaway: Strix Halo memory repro kit for a 96 GiB graphics carve-out - #10418
danielhanchen wants to merge 2 commits into
Conversation
AMD report that on a Strix Halo box set to 96 GiB of graphics memory, llama.cpp reads the carve-out as shared iGPU memory and large models will not load, and that ROCm refuses allocations above 64 GB. Our CI runner is a 128 GB Strix Halo with a 64 GiB carve-out, and at that size neither claim reproduces: ggml sums the Vulkan heaps to 111.65 GiB, which stays under the 127.65 GiB the machine holds, so nothing impossible is claimed and a 77 GiB model loads; and a single HIP allocation reaches 110.2 GiB rather than stopping at 64. The over-report is a function of the carve-out size, so it needs a box configured the way AMD configured theirs. This kit is what to run on such a box. run.ps1 walks six cells: the host facts, two stock llama.cpp release builds, the memory report, a bounded largest-allocation search, Studio's own Vulkan probe at a pinned commit, and llama-fit-params on both backends. Every cell writes JSON and report.py turns the whole directory into one table with the three statements stated as yes or no against the machine's real memory. The allocation search stops at 92 percent of physical RAM on purpose. AMD's own note warns that an allocation near 100 percent of usable RAM can hang the host or bugcheck it, and the question here is whether an allocation the stock runtime refused now succeeds, not what the new ceiling is. Throwaway: this is a diagnostic to run once, not something to keep.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
for more information, see https://pre-commit.ci
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f93deb9f7a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| except Exception as e: # noqa: BLE001 | ||
| info["error"] = f"{type(e).__name__}: {e}" | ||
|
|
||
| bin_dir = find_bin_dir(args.dest) if args.dest.is_dir() else None |
There was a problem hiding this comment.
When a download or extraction fails, this still searches the persistent destination and returns success if an earlier run left a llama-server there. Because run.ps1 reuses $Work\lcpp\rocm and $Work\lcpp\vulkan even when -Tag changes, a transient failure can silently run an old build while the JSON records the newly requested URL, invalidating the experiment; skip this lookup after an exception or clean/use a tag-specific destination.
Useful? React with 👍 / 👎.
| if h["ram_gib"] is not None: | ||
| physical = round(h["ram_gib"] + (h["registry_vram_gib"] or 0), 2) |
There was a problem hiding this comment.
When the registry probe cannot read qwMemorySize, treating the missing carve-out as zero compares ggml's total against only Windows-visible RAM and can report over_report: yes even for an honest reading. This directly contradicts run.ps1's warning that the statement requires the carve-out; require both ram_gib and registry_vram_gib before constructing physical.
Useful? React with 👍 / 👎.
| info["archive_bytes"] = archive.stat().st_size | ||
| extract(archive, args.dest) | ||
| archive.unlink(missing_ok = True) |
There was a problem hiding this comment.
The release fetch records only the archive size and then deletes it, so the returned directory cannot verify which ROCm or Vulkan release asset was tested if an asset at the pinned tag is replaced. This undermines the README's explicit promise that both builds are hashed and that the exact binaries can be re-checked; record an archive digest, or hashes of the relevant extracted binaries, before unlinking the archive.
Useful? React with 👍 / 👎.
| vk_raw = (sec(doc, "vulkan_raw").get("devices") or [None])[0] | ||
| ggml = (sec(doc, "ggml_vulkan").get("devices") or [None])[0] | ||
| hip = (sec(doc, "hip").get("devices") or [None])[0] |
There was a problem hiding this comment.
On a mixed APU+dGPU host, these first-element selections can evaluate all three statements against the discrete GPU rather than the Strix Halo device, producing an irrelevant negative result even though the raw inventories contain the target. This is a known supported topology: studio/install_llama_prebuilt.py:2438-2440 explicitly documents Strix Halo plus RX 7900 hosts and warns that a bare first match chooses the wrong device; match the collected device name/type/is_igpu fields instead of assuming ordinal zero.
Useful? React with 👍 / 👎.
| if doc.get("_error"): | ||
| continue |
There was a problem hiding this comment.
When a JSON path is missing or malformed, this branch skips the document without adding any labels to undecided, so even report.py missing.json --require over_report exits successfully with an empty statements table. That defeats the purpose of --require and the module's promise that missing required readings return nonzero; mark each requested statement undecidable for this document or return failure immediately.
Useful? React with 👍 / 👎.
Ran the kit on a Strix Halo box at three carve-outs, including 96 GiBHardware: Ryzen AI Max+ PRO 395 / Radeon 8060S (gfx1151), 128 GB, Windows 11, driver 32.0.31041.1004, llama.cpp Headline: The three statements, across three carve-outs
ggml's number never exceeds the machine. This PR's body predicts that at 96 GiB "the same call is expected to report more memory than the machine physically has, which is what AMD saw". Measured, it reports 111.89 GiB against 127.78 GiB. That prediction should be rewritten now that the configuration has an answer. The heap structure does change: at 96 GiB the device-local heap is exactly the carve-out (96.00) and the other is exactly half of visible RAM (31.78 / 2 = 15.89). Cleaner than at the smaller carve-outs, but still summing to less than the machine. Two independent probe runs at 96 GiB produced byte-identical numbers. Allocation cap: no 64 GB clampAt the 32 GiB carve-out, largest single The vendor rule quoted in the body, What actually costs the user: the carve-out, by 3-4xllama-bench, Qwen3-Next-80B-A3B-Instruct UD-Q4_K_XL (42.90 GiB),
At 32 GiB the model does not fit the carve-out and runs from shared system memory; at 96 GiB it is entirely carve-out resident. Whether the model fits the carve-out is worth 3-4x — far more than the backend choice (10-17%) and infinitely more than the memory reporting (nothing). Backend: the advantage is real but small, and reverses on memory
At a small carve-out ROCm starves large models (19.16 GiB less, 4.59x less context). At 96 GiB that inverts: Vulkan advertises more total but reserves more, so ROCm has the larger free pool and fits more context. The ROCm memory disadvantage is a small-carve-out phenomenon. Throughput at 96 GiB on AC, two models,
Prefill is a tie at 96 GiB; Vulkan leads decode by ~16% on both models. Note this is not the +22.9% prefill / +8.3% decode recorded on the CI runner, and at the 32 GiB carve-out on a 313B model ROCm actually decoded 26% faster than Vulkan (3.50 +/- 0.04 vs 2.77 +/- 0.12). The speed advantage is model- and configuration-dependent and should not be quoted as a single number. Lemonade comparisonAMD reported Lemonade working well on this hardware. Lemonade 11.9.0's server on this machine resolves Running the identical benchmark through Lemonade's own bundled build, same model, same flags, same machine:
The Unsloth prebuilt is substantially faster than Lemonade's on the same backend. Lemonade ships b10723 against our b10798, so at least part of this is build vintage rather than anything either project configures. Issues found
Forcing unified memory at a large carve-out is catastrophicNot something Studio does, but worth recording. ROCm at the 96 GiB carve-out with Studio correctly avoids this: This does make unslothai/llama.cpp#149 look worth landing — Not established
|
Follow-up: a correctness result that inverts the Lemonade comparisonSame machine as the previous comment (gfx1151, 128 GB, Windows), now with the carve-out at 96 GiB. This is about output correctness rather than memory, and it changes what "Lemonade works well, Studio does not" can mean on this hardware. There is a known corruption on HIP integrated GPUs from The A/B is one flag on one binary —
~1500x, gone by changing On this hardware, Lemonade's ROCm path produces numerically corrupt output at default settings and the current Unsloth prebuilt does not. The fair qualifier: Lemonade defaults to Combined with the earlier throughput result (Unsloth's Vulkan build +54.9% prefill / +14.0% decode over Lemonade's on the same model), the "Lemonade works, Unsloth does not" framing does not survive measurement on this machine in either dimension. Also confirmed here, relevant to unslothai/llama.cpp#149: Correction to my previous comment's framingMy first perplexity attempt used 512-token chunks against One more trap worth flagging for anyone reproducing this: the ggml-org What is still open on the AMD questionNothing here reproduces a Studio-specific failure at a 96 GiB carve-out. The memory statements are NO at 32, 64 and 96 GiB; the ROCm memory shortfall inverts at 96 GiB; and the corruption above is not Studio-specific, it is upstream and affects Lemonade too. The single most useful thing would still be AMD's carve-out, OS, and what "not optimal" meant numerically. |
1 participant
Throwaway PR. Do not merge. It exists so someone with a Strix Halo box set to a large
graphics carve-out can run one script and send back a directory of JSON.
Why
AMD report two things about Strix Halo, both of which they see at a 96 GiB Variable
Graphics Memory setting:
Our Strix Halo CI runner is a 128 GB machine with a 64 GiB carve-out, and at that size
neither claim reproduces:
HardwareInformation.qwMemorySize(the carve-out)GlobalMemoryStatusExvisible RAMhipMemGetInfototalhipMallocthat allocated and read backThe summation mechanism is real and confirmed to the byte: ggml adds every heap on an
integrated device, not just the device-local ones. But at a 64 GiB carve-out the sum stays
under the machine, so nothing impossible is claimed and the model loads. Windows subtracts
the carve-out from visible RAM, so as the carve-out grows the heap sum grows while the
physical bound does not: at 96 GiB the same call is expected to report more memory than the
machine physically has, which is what AMD saw. That is why this needs their configuration
and not ours.
Same for the allocation cap. The stock rule is not a flat 64 GiB clamp, it is
max(dedicated_VRAM_heap, 0.75 * shared_GART_heap). On the runner the GART branch wins,which is where 110.2 GiB comes from. On a larger carve-out the dedicated branch may dominate
instead, and then the interesting arm is the memory report rather than the cap.
What to do
On a Windows Strix Halo box, set the carve-out to 96 GiB (firmware UMA Frame Buffer Size, or
Adrenalin -> Performance -> Tuning -> Variable Graphics Memory), reboot, then:
It downloads two stock llama.cpp release builds (ROCm gfx1151 and Vulkan) at a pinned tag,
runs six cells, and writes JSON per cell. Nothing is compiled and nothing is installed.
Budget about 30 minutes plus the model download if you include the model cells.
report.pyprints one table of what every layer reports next to the machine's real memory,and three statements as yes or no:
over_report: does ggml claim more memory than the machine physically hassums_heaps: is that number the heaps added together rather than the largest onehip_is_vgm_only: does HIP report only the carve-outOn the 64 GiB runner those come out NO / yes / NO. On a 96 GiB box the first is expected to
flip, and that flip is the bug AMD are describing.
The allocation search is deliberately bounded
Cell 4 stops at 92 percent of physical RAM. AMD's own note with the patched runtime warns
that an allocation near 100 percent of usable RAM can hang the host or trigger a
WIN32K_POWER_WATCHDOG_TIMEOUTbugcheck. The question worth answering is whether anallocation the stock runtime refused now succeeds, not what the new ceiling is, and a
bounded search answers it without going near that condition. Every success is verified by
writing the buffer and reading it back, not by
hipMallocreturning.What to send back
The whole output directory. It contains the host facts, the raw per-heap Vulkan numbers,
the HIP numbers, the allocation boundary, what Studio's own Vulkan probe returns, and
llama-fit-paramson both backends.README.mdin the folder says what each cell measuresand what a wrong-looking answer looks like, so a cell that fails is still informative.