ggml-cuda: avoid direct ROCm_Host compute on HIP integrated GPUs

[Victor-Loos]

Overview

Fixes a regression after #24233, which restored HIP integrated GPU detection. On my AMD APU, this also made the ROCm host-buffer path available for HIP integrated GPUs. With that path enabled, prompt input appears corrupted: chat templates and system prompts can be ignored, and generated output becomes garbage.

This change keeps integrated GPU detection/reporting intact, but disables direct ROCm_Host compute on HIP integrated GPUs. The fix does not disable host-buffer exposure entirely, pinned buffers remain available to preserve functionality, while only the problematic direct compute path is blocked.

Additional information

Tested locally on gfx1151 / RDNA3.5 APU with ROCm 7.14. The corrupted input/output issue is fixed with this patch.
The issue appeared in external harnesses like Pi and OpenCode. It can also be reproduced in the built-in web UI with a sufficiently large combined first input. For example, gemma-4-26B-A4B with 4676 tokens of context or more outputs <unused49> indefinitely. Other models can be way more resilient and their output starts by saying 'Based on the text provided, which appears to be a corrupted or "hallucinated" version of a Wikipedia article'.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - I used AI assistance to review and patch the relevant code. I tested and reviewed the final change myself.

[pwilkin]

@liminfei-amd could you please take a look?

[liminfei-amd]

Thanks for the ping. I tested this on gfx1151 / ROCm 7.2.1. Before #24233, the host-buffer capability and factory were already enabled; the newly reachable behavior was supports_buft(ROCm_Host): false -> true.

The current patch fixes this conservatively, but disabling the factory reduced pp512 from 5857 to 1502 t/s. I suggest a narrower fix:

caps/factory = GGML_CUDA_NO_PINNED is not set
supports_buft(ROCm_Host) = caps/factory enabled && not a HIP integrated device

This keeps pinned staging/output buffers but prevents direct HIP APU computation on ROCm_Host. It retained 5725 t/s (97.7% of baseline). Could you test this variant with the exact Pi/OpenCode reproduction?

[Victor-Loos]

Thanks @liminfei-amd for the testing and for clarifying the narrower fix.

I tested your suggested variant with my setup, and it still fixes the issue. I've updated the PR accordingly.

As for performance, on my setup I didn't see any meaningful difference. Here are my llama-bench results with a Qwen3.6-35B-A3B model:

Version	pp512 (t/s)	tg128 (t/s)
Before	1165.42 ± 29.48	67.57 ± 0.87
Fix v1	1148.87 ± 23.64	67.76 ± 0.70
Fix v2	1144.09 ± 20.46	68.11 ± 0.74

Could you retest the updated version on your side and confirm whether it preserves the pp512 result you mentioned? If there's still a difference, could you share the exact command and model you used so I can try to reproduce it?

[flashburns]

Hello I ran into issues on my strix halo box and this PR fixes it. Step 3.7 Flash IQ4_XS will spam < in a loop at 4k+ context.
I had an agent bisect the issue and it traced it down to the same commit & PR. c7d8722

I have now tested this pull request on my box and its solved.

[liminfei-amd]

Thanks @Victor-Loos Retested fdc1260e against 91d2fc3 on gfx1151 / ROCm 7.2.1. The pp512 regression is gone: exact-parent/head ABBA across ngl=0,10,99 was within -1.84% to +0.87%. Pinned allocation remains available, direct ROCm_Host compute is disabled as intended, GGML_CUDA_NO_PINNED=1 works without an abort, and the gfx1201 dGPU control is unchanged. I could not independently run the Pi/OpenCode setup.

Two small changes are still needed before merge:

1. In non-HIP builds, device is unused in ggml_backend_cuda_device_supports_cuda_host_buft(). Since the CUDA workflow enables LLAMA_FATAL_WARNINGS=ON, please add:

#else
	GGML_UNUSED(device);

2. Please update the PR overview: the new version keeps host-buffer allocation available and only prevents direct ROCm_Host compute on HIP integrated GPUs.

Apart from these two items, the revised implementation looks good to me.

[Victor-Loos]

Thanks, @liminfei-amd. I’ve addressed both remaining points. Could you please take another look when you have time?

[pwilkin]

BTW @liminfei-amd maybe you know the reason behind #15018 ? I've ran into this today as well and I've tried everything, but can't get over the weird 64GB allocation soft limit on ROCm with --mmap.

[liminfei-amd]

@Victor-Loos I rechecked the latest head. The requested code and PR description changes look good.

The exact-parent/head ABBA result remains applicable: pp512 across ngl=0,10,99 stayed within -1.84% to +0.87%. Pinned host allocation remains available, direct ROCm_Host compute is disabled on gfx1151, GGML_CUDA_NO_PINNED=1 no longer aborts, and the gfx1201 dGPU control is unchanged. The latest commit only adds the non-HIP unused-parameter fix.

I still have not independently reproduced the Pi/OpenCode corruption itself, but the requested changes validate cleanly on my side.

[liminfei-amd]

BTW @liminfei-amd maybe you know the reason behind #15018 ? I've ran into this today as well and I've tried everything, but can't get over the weird 64GB allocation soft limit on ROCm with --mmap.

@pwilkin Thanks for the ping. I'm currently trying to reproduce the issue and find a solution.

[0x0013]

Hi!

I also ran into this regression.

Host: Strix Halo 128GB

Behavior before #24233

Models call properly structured tool calls. Malformed tool calls are rare.

Behavior since #24233

Models output intermittent, but persistent, malformed tool calls, which are hard to diagnose. Most often, model would output malformed tool calls after cold boot, and the same behavior would persist after repeated invocations with fresh context. After repeating tests with fresh context and slightly modified prompt (e.g. "please test the brave-search skill" vs "test the brave-search skill" vs "read the brave-search skill" etc), the model would suddenly "flip" to correct tool calls. After this, often even the previously failing prompt, with the exact previously failing context, would consistently pass. Sometimes, invocations would flip back to malformed tool calls.

Non-exhaustive examples of observed malformed tool calls

<read path="/home/user/.pi/agent/skills/brave-search/SKILL.md</read>

<read path="/home/user/.pi/agent/skills/brave-search/SKILL.md" />

<read><parameter>path>/home/user/.pi/agent/skills/brave-search/SKILL.md</parameter></read>

<invoke name="read">
  <parameter name="path">/home/user/.pi/agent/skills/brave-search/SKILL.md</parameter>
</invoke>

<tool_calls><tool_calls><tool_calls><tool_calls><tool_calls><tool_calls><tool_calls>...

Models used

• Qwen3.6-35B-A3B-UD-Q6_K.gguf Unsloth MTP - fails most consistently of the two tested
• Qwen3.6-27B-Q8_0.gguf Unsloth MTP - inconclusive results - sometimes outputs consistently good tool calls, sometimes consistently malformed, sometimes flips between results

Build from this PR outputs consistently structured good tool calls, so far (tested 6 invocations per model, fresh context each time)

[liminfei-amd]

@IMbackK This PR hardens the HIP integrated-GPU support added in #24233 by keeping the correct iGPU classification and pinned ROCm_Host allocation, while preventing unsafe direct computation on ROCm_Host tensors.

I independently reproduced #25992 on gfx1151 with a Qwen3.6-35B model using mmap, -np 4, and --kv-unified. Without this fix, the server produced 2 cross-request replays in 96 complete responses. With the latest PR head, all 96 responses completed without replay, request errors, or nonce misses.

Please review and merge when possible. Thanks!

[walcz-de]

Confirmed fixed on independent gfx1151 hardware (AMD Strix Halo / Ryzen AI Max APU), ROCm 7.14.

I'm the reporter of #25992 (the -np 4 --kv-unified cross-slot response leakage on integrated HIP GPUs, bisected to c7d8722). I built llama-server from this PR (head ce82541) with -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 and ran the exact failing configuration:

llama-server -np 4 --kv-unified -ngl 99 -c 32768

Model fully offloaded to the gfx1151 iGPU (confirmed via rocm-smi --showpids; libs resolve to ROCm 7.14 + gfx1151 Tensile kernels).

Result — with this PR: 12 rounds × 4 concurrent requests, each carrying a unique nonce behind a ~24k-char filler prompt to exercise the host/pinned-buffer path under load → 48/48 responses returned their own nonce, 0 cross-slot leakage. On the affected build the same setup returned other requests' responses verbatim across slots (#25992). Prompt input is also no longer corrupted here — system/user content is honored correctly again (it was ignored/garbled before).

The change is surgical and correct for this case: it disables only the direct ROCm host-buffer compute path on HIP integrated GPUs while keeping integrated-GPU detection and pinned buffers intact. LGTM from a gfx1151 user — would love to see this merged so we can drop our backend pin. Thanks @Victor-Loos.

[pwilkin]

@IMbackK This one's pretty critical, so unless there's something wrong I'd be for merging it ASAP.

[imrehg]

I've tested out this fix with gfx1150 (Radeon 890M), ROCm 7.2.4, and seems to work (with the confirmation that the same tests failed on the current head).

[ORippler]

This looks like it would be a potential data-race/stale pointer. Mind trying #26167 to see if anything pops up?

ggml-cuda: avoid direct ROCm_Host

If I understand the current code correctly, prop.integrated serves only this aspect (i.e. to facilitate use on host-buffers). So we may look to revert #24233 and get this fixed for both CUDA and ROCm. For CUDA, we have limited interest in host-buffer-use atm as the saved copies of inputs/outputs are offset by the cost of full cache-coherency between iGPU and CPU caches on our current UMA devices. Quoting from the CUDA docs on memory coherency for iGPUs:

Since both the CPU and iGPU can cache this memory, it becomes well suited for workloads that benefit from GPU L2 caching and simultaneous access by the CPU or other SoC engines. However, it is important to note that Fully Coherent memory may involve hardware limitations and coherence related overhead. Therefore, it is crucial to analyze the CPU and GPU access patterns or benchmark with and without Full Coherency to determine if the performance benefits outweigh the costs.

I cannot comment on perf for AMD/Intel's iGPUs, but would assume memory coherency is cheap for Apple as they don't even expose the concept of an "exclusively owned pointer) (cf. MLX).

[pwilkin]

@ORippler I'll try to run something with the sanitizer and see if I can find anything.

[Victor-Loos]

@ORippler I built the #26167 sanitizer and ran the same reproduction with GGML_SCHED_SANITIZE=1.

Without this fix, the server aborts with:

ggml-sched-sanitize: RACE (write-after-read) on ROCm_Host[0, 2048)
ggml-sched-sanitize:   read  ROCm0#1      @14174 split 0    inp_tokens (compute)
ggml-sched-sanitize:   write HOST         @415  split -1   inp_tokens (tensor_set)
ggml-sched-sanitize:   no happens-before edge: HOST knows ROCm0#1@14173, needs >=14174

The host rewrites prompt tokens in the ROCm_Host-backed inp_tokens tensor while an asynchronous HIP operation may still be reading from it. There is no synchronization establishing that the HIP read completed before the host write.

With this PR applied, I repeated the same reproduction with the sanitizer enabled, and the race was no longer reported.

[pwilkin]

With #26225 this workaround is no longer needed.

[ggerganov]

With #26225 this workaround is no longer needed.

This does not seem right. If the HIP allocates CPU buffers, they should end up in separate graph splits. If they don't then there is some problem in the HIP backend and should be fixed there.

[pwilkin]

@ggerganov these are not technically CPU buffers, they're GPU_Host buffers that just happen to share the same memory as the CPU buffers (due to memory pin).

[pwilkin]

This is a specific to integrated systems where you can remap the same memory location to be either CPU or GPU.

[ggerganov]

The DGX Spark and the Macs are also integrated systems and don't need this patch.

[pwilkin]

Originally this workaround was done for NVIDIA Jetson Orin and I'm not really sure they "don't need this patch", as the direct host compute was only reenabled for HIP-backed integrated GPUs so far, the other CUDA systems stay on #16308 - so you'd have to remove the gate and test with the sanitizer (#26167) on Spark to see if it indeed doesn't surface.

[ORippler]

Originally this workaround was done for NVIDIA Jetson Orin and I'm not really sure they "don't need this patch",

Yeah CUDA UMA systems should be affected by this as well, see the original thor issue that facilitated disablement.

The DGX Spark and the Macs are also integrated systems and don't need this patch.

DGX Spark doesn't behave like a UMA system atm, and we expect reading from host buffers to be perf-neutral for the aforementioned cache issues: #25863 (comment). Does the metal backend expose/use pinned host memory?

I'd favor a solution where we add a ring-buffer of input/outputs in the scheduler rather than placing more syncs

[jj123451]

I took you PR and applied your commits to latest llama.cpp (https://github.com/jj123451/llama.cpp). I built it locally on evo x2 (Strix Halo) with rocm 7.14 and tested it in parallel on 14 prompts using gemma4. I did the same test with latest original llama.cpp built the same way.

• on original lama.cpp I got gibberish output on all or most prompts
• with your changes all the prompt answers are correct :-)

thx for your effort

[ggerganov]

Does the metal backend expose/use pinned host memory?

No, it only wraps host buffers in Metal buffers, but does not declare a host buffer type. I guess that's the reason to not observe the issue with Metal.

[IMbackK]

I dont have the hardware to test that this solves the problem or to reproduce the original issue, logically the original path should be fine to.

I have to trust in others here that this is the best mitigation.

[IMbackK]

@ggml-org/ggml-cuda a second approval would be welcome.

[HeadCutter]

Confirmed on Strix Halo (gfx1151, Radeon 8060S), ROCm 7.14, Windows.
Bisected the corruption independently to c7d8722.
Pristine upstream 84e908c with only this PR applied, llama-perplexity -m Qwen3.5-0.8B-UD-Q4_K_XL.gguf -f wiki.test.raw -c 4096 -b 2048 -ub 512 --chunks 16 -ngl 99 -t 8 --no-mmap -fa on: 8055.65 without the patch, 15.7509 with it. 15.7509 also matches what the stock official ROCm build (b10359) produces under HIP_LAUNCH_BLOCKING=1

[ORippler]

@ggml-org/ggml-cuda a second approval would be welcome.

If I understand the current code correctly, prop.integrated serves only this aspect (i.e. to facilitate use on host-buffers). So we may look to revert #24233 and get this fixed for both CUDA and ROCm.

@IMbackK are there other use-cases for prop.integrated? If not, this "fix" reverts the only use-case that would want to be enabled. In that case, we should simply revert #24233 in my eyes, to not fragment CUDA/HIP parts unnecessarily

[IMbackK]

@ORippler I like that this pr dosent make the prop "lie" but the point about fragmentation is taken, hip/cuda should both take the same path here. I would be fine with either adding cuda here or reverting #24233

[pwilkin]

@IMbackK why not go the route of adding the ring buffer like we discussed with @ORippler so that this problem is actually fixed and not just hidden?

[IMbackK]

@pwilkin I dont know what is actually wrong with the host buffer usage as is and I dont have a uma system to actually validate any change i would make so i dont actually have any recourse other than to just disable the thing and hope someone else fixes it.

[pwilkin]

@IMbackK aight, let me get back from vacation and I'll propose a proper fix that supersedes #26225

[ORippler]

@IMbackK aight, let me get back from vacation and I'll propose a proper fix that supersedes #26225

@pwilkin happy to pair/assist to ensure it runs smoothly on DGX/RTX Spark also. Note this will involve work on the scheduler, which we are in the process of testing more robustly (@aendk is working on this afaik)