ggml-cuda : fix UMA memory detection for HIP/ROCm on AMD APUs

[hogeheer499-commits]

AMD APUs report prop.integrated == 1, which triggers the UMA memory detection from #17368. This replaces the accurate hipMemGetInfo() value with MemAvailable from /proc/meminfo, which reports significantly less memory on systems with large TTM allocations (e.g. 122 GiB vs 91 GiB on a 128GB Strix Halo system).

For HIP builds, skip the prop.integrated check and only enter the UMA path when GGML_CUDA_ENABLE_UNIFIED_MEMORY is explicitly set. This way hipMemGetInfo() is used by default (which correctly reports TTM-backed memory), while the explicit env var override still works for users who need it.

Verified on AMD Ryzen AI MAX+ 395 (gfx1151, 128GB unified memory, ROCm 7.1) that prop.integrated returns 1 and hipMemGetInfo() returns 122880 MiB while MemAvailable reports ~91 GiB.

Fixes #18159

Related: #19818, #19764, #18650

[hogeheer499-commits]

Bug verification on AMD Ryzen AI MAX+ 395 (gfx1151, 128GB unified memory)

Wrote a test program that simulates the exact code path in ggml_backend_cuda_device_get_memory() to demonstrate the impact:

=== BEFORE UMA override (hipMemGetInfo) ===
  free  = 122879 MiB
  total = 122880 MiB

prop.integrated = 1 (is_uma = true)

=== AFTER UMA override (/proc/meminfo) ===
  free  = 91152 MiB  (from MemAvailable)
  total = 122880 MiB  (unchanged)

=== DIFFERENCE ===
  Lost: 31727 MiB (30 GiB) of usable VRAM!

On AMD APUs, prop.integrated returns 1, triggering the UMA path. This overrides the accurate hipMemGetInfo() value (122879 MiB) with MemAvailable from /proc/meminfo (91152 MiB), losing ~30 GiB of usable GPU memory.

The !defined(GGML_USE_HIP) guard ensures this UMA path only applies to CUDA/NVIDIA builds (DGX Spark) where it was intended, while HIP/ROCm builds continue using hipMemGetInfo() which already reports the correct TTM allocation.

[hogeheer499-commits]

Note on end-to-end testing

I was unable to reproduce the context size reduced behavior described in #18159 because my only available ROCm build environment (ROCm 7.1) segfaults during HIP kernel initialization on gfx1151 — before get_memory() is even called. This is a known ROCm 7.1 + gfx1151 incompatibility unrelated to this fix.

However, the mechanism is clearly demonstrated above: prop.integrated returns 1 on AMD APUs, triggering the UMA path, which replaces hipMemGetInfo() (122879 MiB) with MemAvailable from /proc/meminfo (~91 GiB). This 30 GiB reduction directly feeds into llama_params_fit(), which would reduce context size on systems with less RAM or when loading larger models near the memory limit.

On my 128GB system the 91 GiB reported by MemAvailable is still enough for most models, but users with 64GB or 96GB unified memory (common Strix Halo configs) would see much more severe effects — potentially losing half their usable VRAM.

The fix itself is minimal and clearly correct: hipMemGetInfo() already returns the accurate TTM-backed memory on AMD APUs, so the /proc/meminfo override (designed for DGX Spark) should be skipped for HIP builds.

[moonshadow-25]

Great fix! For Windows users with the same APU, there's a complementary
issue: hipMemAdviseSetCoarseGrain crashes on APU/UMA systems.
PR #20536 addresses that side. There's also an upstream ROCm fix needed:
ROCm/rocm-systems#4077

[JohannesGaessler]

On my Strix Halo system I get the following on master:

ggml_cuda_init: found 1 ROCm devices (Total VRAM: 62206 MiB):
  Device 0: Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32, VRAM: 62206 MiB (62202 MiB free)
build: 8323 (57819b8d4) with GNU 15.2.1 for Linux x86_64
llama_params_fit_impl: projected to use 5438 MiB of device memory vs. 121595 MiB of free device memory
llama_params_fit_impl: will leave 116157 >= 1024 MiB of free device memory, no changes needed
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 0.18 seconds
main: printing fitted CLI arguments to stdout...
-c 0 -ngl -1

With this PR I get:

ggml_cuda_init: found 1 ROCm devices (Total VRAM: 62206 MiB):
  Device 0: Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32, VRAM: 62206 MiB (62202 MiB free)
build: 8325 (e0dace50d) with GNU 15.2.1 for Linux x86_64
llama_params_fit_impl: projected to use 5438 MiB of device memory vs. 62060 MiB of free device memory
llama_params_fit_impl: will leave 56621 >= 1024 MiB of free device memory, no changes needed
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 0.18 seconds
main: printing fitted CLI arguments to stdout...
-c 0 -ngl -1

So at the very least there are some edge cases that this PR does not handle correctly and it cannot be merged like this.

not actually correct

[hogeheer499-commits]

Thanks for testing! I see the issue — on your system hipMemGetInfo() reports only the dedicated VRAM portion (62 GiB) while /proc/meminfo correctly reflects the full TTM-accessible memory (121 GiB).

I think the better fix is actually simpler: instead of adding HIP-specific logic, just change the existing UMA path to take the maximum instead of unconditionally overwriting:

size_t proc_free = (size_t)available_memory_kb * 1024;
if (proc_free > *free) {
    *free = proc_free;
}

This way it works for both CUDA and HIP without any #ifdef:

• On my system (128GB, VRAM maxed): hipMemGetInfo() = 122 GiB > /proc/meminfo = 91 GiB → keeps 122 GiB ✅
• On your system (62 GiB dedicated): hipMemGetInfo() = 62 GiB < /proc/meminfo = 121 GiB → uses 121 GiB ✅ (same as master)

One question: this results in *free > *total on your config. Master already has this behavior, so it should be fine — but should I update *total as well?

I'll push once you confirm.

[JohannesGaessler]

According to the llama.cpp AI usage policy:

It is strictly prohibited to use AI to write your posts for you (bug reports, feature requests, pull request descriptions, Github discussions, responding to humans, ...).

[hogeheer499-commits]

Yeah fair point, I used AI to help structure that comment since English isn't my first language, but I should've just written it myself. Won't happen again. The fix itself (max instead of always overwriting) I do understand and stand behind. Want me to push it?

[JohannesGaessler]

As I said: on my Strix Halo system master correctly reports the full memory wile your PR does not. So even if your PR fixes some cases it breaks other ones and we cannot merge it like this.

[JohannesGaessler]

Sorry, I think I read your post incorrectly. Yes, push your new version, I'll take a look.

[kryoz]

Sorry for pushing but this bug is critical not allowing to use large models. Please make attention to review.

[hogeheer499-commits]

The max() fix has been pushed since March. It handles both configs correctly, your 62 GiB VRAM < 121 GiB meminfo and my 122 GiB > 91 GiB. Ready for review when you get a chance.

[JohannesGaessler]

I'm not convinced that that is the correct way to fix it.

[winmutt]

I am still having this issue on nightly llama-rocm builds as described in lemonade-sdk/llamacpp-rocm#86

[hogeheer499-commits]

Thanks for pushing back on this. I went back through the numbers, and I think you were right that max(hipMemGetInfo, MemAvailable) is not really the right model here.

On my 128GB Strix Halo system I now get:

hipMemGetInfo total / VRAM:  122880 MiB
TTM pages_limit:             122880 MiB
AMDGPU VRAM+GTT free:       ~124971 MiB
MemAvailable:               ~89 GiB

So at least on my machine, hipMemGetInfo lines up exactly with the TTM limit as a total/cap value. /proc/meminfo is quite a bit lower, which makes it look like it is not the right source for the HIP allocation budget here. I also checked the AMDGPU sysfs VRAM+GTT numbers, but those come out slightly above the TTM cap on this system, so they do not look like a clean replacement either.

Because of that, I do not think taking the larger of hipMemGetInfo and MemAvailable is the right fix. It still feels more like a heuristic than something tied to the actual allocation cap.

I think the narrower change is probably the safer one for now: for HIP builds, do not take the /proc/meminfo UMA path just because prop.integrated is true. CUDA/DGX Spark can stay as-is, and GGML_CUDA_ENABLE_UNIFIED_MEMORY can still be the explicit opt-in for HIP.

bool uma_env = getenv(GGML_CUDA_ENABLE_UNIFIED_MEMORY) != nullptr;
#if defined(GGML_USE_HIP)
bool is_uma = uma_env;
#else
bool is_uma = prop.integrated > 0 || uma_env;
#endif // defined(GGML_USE_HIP)

Before I update the branch again, could you check one thing on your Strix Halo system?

cat /sys/module/ttm/parameters/pages_limit
getconf PAGESIZE

I want to compare the converted pages_limit value with the ~62 GiB hipMemGetInfo / VRAM number you are seeing. If those match on your system too, that would be a pretty good sign that hipMemGetInfo is tracking the HIP/TTM cap there as well, and that the 121 GiB value from master is coming from the /proc/meminfo UMA path rather than from HIP's own accounting.

[kryoz]

cat /sys/module/ttm/parameters/pages_limit
32505856

getconf PAGESIZE
4096

ggml_cuda_init: found 1 ROCm devices (Total VRAM: 63720 MiB):
  Device 0: Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32, VRAM: 63720 MiB

free
               total        used        free      shared  buff/cache   available
Mem:       130499696     1589300   111800468        1800    18158996   128910396
Swap:        8388604           0     8388604

1Gb UMA is set in BIOS

[ehoogeveen-medweb]

Note that there was also a change to ROCm which was recently reverted: ROCm/rocm-systems#5204

I'm not sure whether/how that affects llama.cpp, but it might cause misreporting on nightlies released on or after April 14. I don't think the revert has made it into the latest nightly yet (as the latest bump of rocm-systems did not include it).

[JohannesGaessler]

I am still having this issue on nightly llama-rocm builds as described in lemonade-sdk/llamacpp-rocm#86

To be clear: this PR will not fix that issue. All this PR does is change how free and total memory is being reported by the ggml backend but it does not actually allow the ggml backend to allocate and use any additional memory.

[winmutt]

Note that there was also a change to ROCm which was recently reverted: ROCm/rocm-systems#5204

I'm not sure whether/how that affects llama.cpp, but it might cause misreporting on nightlies released on or after April 14. I don't think the revert has made it into the latest nightly yet (as the latest bump of rocm-systems did not include it).

This resolved my issue.

[hogeheer499-commits]

I did more testing on my Strix Halo with ROCm 7.2, because I wanted to understand what is going on before changing the PR again.

The main thing I found is that default hipMalloc and hipMallocManaged behave very differently on this APU.

For default hipMalloc, I tested touched allocations up to 112000 MiB. I also ran an 80000 MiB hipMalloc inside a Docker container limited to 72 GiB, and it still succeeded. GTT usage went up by about the allocation size, and the allocation was not blocked by the cgroup limit. So on this stack default hipMalloc goes through KFD/GTT, not normal system-memory accounting.

hipMallocManaged behaves differently. With the same 72 GiB cgroup, 64000 and 70000 MiB succeed, but 80000 MiB gets OOM-killed. MemAvailable drops by about the allocation size while sysfs GTT usage barely moves. hipMemAdviseSetCoarseGrain does not change that. So managed memory is system/SVM-backed and cgroup-limited here.

                          total (MiB)    free (MiB)
master default             122880         116587
local narrow default       122880         122875
local narrow managed-env   122880         116554
PR/max managed-env         122880         122875

Master replaces hipMemGetInfo with MemAvailable because prop.integrated is set. That loses about 6 GiB on my system even though the default hipMalloc path can use the HIP/GTT budget.

With an 80B Q4 model and --fit-target 70000:

• master default: reduces context to -c 83968 -ngl -1
• local narrow default: no changes needed, -c 0 -ngl -1

This is also why I do not think the max() version was the right model. It fixes default HIP on my machine, but it reports the HIP/GTT number for the managed path too.

I also looked at the Vulkan backend on the same machine. It reports the 8060S with 131584 MiB total / 130528 MiB free because it sums all memory heaps for integrated GPUs. vulkaninfo shows the RADV heaps as about 42.83 GiB + 85.67 GiB. This is not HIP proof, but it is a useful llama.cpp-side sanity check: another backend on the same integrated GPU already treats the memory model differently.

The hard part is systems like yours and Kryoz's where hipMemGetInfo reports around 62 GiB. ROCm/hip#3892 tracks this as a reporting issue, and ROCm/rocm-systems#5596 is adding GTT to APU memory reporting when GTT > VRAM.

I do not want to pretend this covers every case. The conservative change fixes my system and keeps reporting aligned with the allocator, but it stays at 62 GiB on yours until the ROCm fix lands. A GTT-aware bridge could help sooner, but TTM pages_limit is a capacity limit, not live free. After an 80 GiB hipMalloc, hipMemGetInfo.free drops but TTM stays the same, so you cannot just use TTM as free. The missing piece for that bridge is whether hipMemGetInfo.free tracks allocations correctly on the low-reporting systems.

Given that, what scope would you prefer for this PR? I can keep the PR conservative and allocator-aligned, or look into a broader GTT-aware bridge if we can find a reliable live-free source. If you think the underreporting case should wait for the ROCm fix, that is useful to know too.

Disclosure: I used AI assistance to organize my notes for this comment. The tests, results, and conclusions are from my local work.

[hogeheer499-commits]

Small note: I have not pushed another change yet.

The current branch still has the max() version. After the extra tests above, I think the more conservative HIP change is probably the better model, but I wanted to wait for feedback on the scope before updating it.

For a broader GTT-aware fix, I think the missing piece is still whether hipMemGetInfo.free tracks touched hipMalloc allocations correctly on the low-reporting systems. I cannot test that locally because my system already reports the full 120 GiB through hipMemGetInfo

[matt23654]

I have also been wondering how to improve UMA memory reporting. It is not straightforward:

• BIOS allocated VRAM can be used by accelerator but not seen by linux kernel. Windows dual-booters may have substantial allocations of this, even though AMD recommends minimum allocation and GTT instead on linux.
• AMD recommends max about 112GB of GTT available to the accelerator to preserve system stability.

So, ideally we want a function which returns accelerator free memory taking into account different memory pools (dedicated, GTT) and system free memory.

On my system hipMemGetInfo reports 122GB free, which is indeed the GTT amount I have made available to the GPU. It is incorrect because the system only has 114GB MemAvailable at this time. Also, to be robust I wouldn't want to load an AI model even of size 114GB, it would likely crash the system due to competing with system RAM, so the effective amount available as VRAM is even lower still. This is not a concern with dedicated GPUs.

So the memory query functions are not really working as expected, I believe this is also the case on CUDA based UMA as well.

[hogeheer499-commits]

Thanks, this is a good point, and I think I agree with the main concern here.

The more I look at it, the less I think this is just a simple choice between hipMemGetInfo() and MemAvailable. These are not really the same kind of number.

On my Strix Halo system I’m seeing the same kind of split: default hipMalloc and hipMallocManaged do not appear to behave like the same memory path. hipMalloc seems to go through the HIP/KFD/GTT side, while hipMallocManaged behaves much more like system/SVM memory and is affected by the cgroup/system memory limit.

So I agree that even if hipMemGetInfo() reports something like the larger GTT/HIP budget, that does not automatically mean llama.cpp can safely plan a model right up to that number. In my tests, allocations close to the top of that range became unsafe too, so treating the full reported number as a safe loading budget would be too strong.

That also makes me think the current max(hipMemGetInfo, MemAvailable) version is probably not the right final state. It picks the bigger number, but it does not know which allocator path will be used, how much system pressure there is, or what safety margin is needed. It also risks reporting a HIP/GTT-style number for a path that may actually behave more like managed/system memory.

The narrower version of this PR was trying to fix only one specific issue: default HIP allocations should not automatically fall back to the /proc/meminfo UMA path just because prop.integrated is true. I still think that part is defensible, but I do not want to claim that it solves the whole Strix Halo/APU memory accounting problem.

So I think there are two separate things here:

1. a small/narrow fix for the default HIP path, avoiding the implicit MemAvailable override;
2. a broader UMA budgeting design, which probably needs to consider allocator path, GTT/TTM capacity, BIOS VRAM, system MemAvailable, and a safety margin.

For the broader version, I think the missing piece is still whether hipMemGetInfo().free tracks real touched hipMalloc allocations correctly on the low-reporting systems. If it does, it may be useful as part of a live-free signal. If it does not, then I do not think llama.cpp should try to synthesize a safe budget from TTM/sysfs numbers alone.

So I do not think the current max() branch should be treated as the final direction. My preference would be to narrow this PR back down to the one HIP-path issue: default HIP memory reporting should not enter the /proc/meminfo UMA path only because prop.integrated is true.

I realize that this conservative scope still leaves the low-reporting systems at the HIP-reported value. If that is not acceptable for this PR, then I think the right next step is a broader UMA-budgeting design rather than the current max() approach.

[Djip007]

I'll try again to explain what UMA is (but English isn't my native language)...

There are several ways to allocate memory with HIP (and CUDA):

• hipMalloc => allocates memory on the device in VRAM
• hipMallocManaged => allocates memory in RAM and manages automatic transfers to the GPUs when needed.

In the case of an APU, since RAM is directly accessible to the GPU, there is no copying, and when "correctly" configured (hipMemAdviseSetCoarseGrain), there is almost no performance difference compared to allocation with hipMalloc.

Now, the HIP/CUDA backend has two ways to allocate buffers:

• "Normal" with hipMalloc in VRAM
• "UMA" with hipMallocManaged in RAM

Therefore, the reported memory size is currently correct in the backend.

Now, GTT/TTM:

I'll only cover the case of APUs (it's slightly different with an eGPU):
For a long time (kernel versions < 6.14/6.11?), allocation via hipMalloc was done in VRAM only. AMD has since changed this, and for APUs (MI-300A first, then others), allocation can be done either in VRAM (the one defined in the BIOS) or in GTT/TTM.
=> in that case :

• hipMalloc => allocates memory on the device in VRAM+GTT

On recent kernels, allocating VRAM in the BIOS is pointless. For me the best config (the one I use ;) ) is:

• Allocate the VRAM as low as possible (256MB) in Bios. Additional memory will be allocated to the GTT/TTM if needed for Games.
  In this configuration, UMA allocation (with hipMallocManaged) can utilize all of the RAM.

On older kernels where it is necessary (for games?) to define VRAM (16/32 GB), UMA only allows the use of a portion of the memory. A VRAM+GTT configuration with allocation without UMA (hipMalloc) is the only way to utilize a large portion of the memory.

---

Now, getting back to llama.cpp and the hip/cuda backend:
The whole question is when to use UMA (hipMallocManaged):

• By default, not on an eGPU.
• By default on an APU?

The question then becomes: how to force the opposite behavior? At least to force the use of VRAM+GTT on older kernels.

I don't know how it works under Windows (I don't know if there's an equivalent to GTT).

---

Before making a choice, it might be possible, if an APU is detected, to check the two memory sizes and add information/log if there is more VRAM + GTT than RAM, and provide a link(?) to recommendations.

In my configuration (VRAM=256MB, GTT by default), it's more advantageous to leave the current behavior.

However, while this configuration is now the simplest/best, it's difficult to change habits and the now outdated recommendation to define a "large" VRAM size.

[ehoogeveen-medweb]

I think this explanation is largely correct in terms of the primary semantics. However, as of ROCm 6.4.0 the HIP runtime will fall back to system/shared virtual memory (GTT-backed on APUs) when hipMalloc exceeds the available device memory (so with a small reservation, almost always).

With recent kernels this fallback has become a lot more performant and is usually faster than hipMallocManaged (though not quite as fast as a dedicated carve-out). hipMallocManaged (plus HSA_XNACK=1) is mostly useful if you need automatic page migration, oversubscription etc.

[winmutt]

Is this a duplicate of #24906 ?