ggml-cuda: read sysfs VRAM usage on ROCm to fix over-reported free memory

[liminfei-amd]

Fixes #24906.

Problem

On ROCm, hipMemGetInfo (KFD AVAILABLE_MEMORY) only accounts for ROCm/KFD compute allocations and ignores DRM graphics usage (Vulkan/GL/compositor). Free VRAM is over-reported vs rocm-smi, so automatic layer offload can over-commit and OOM at load.

Fix

On GGML_USE_HIP && __linux__, read mem_info_vram_total/used from sysfs (matched by PCI bus id, the same source rocm-smi uses) and report free = total - used. Reuses the existing ctx->pci_bus_id and folds into the existing #if __linux__ memory path (the UMA branch from #17368), so discrete cards take sysfs while UMA/APU stays on /proc/meminfo. Double safety: take min(hip, sysfs) free, so the value can only get more conservative; fall back to hipMemGetInfo if sysfs is unavailable. Non-HIP / non-Linux paths unchanged.

Verified (gfx1100 RX7900GRE, ROCm 7.2.1)

state	hipMemGetInfo free	rocm-smi/sysfs free	patched --list-devices
idle	16332 MiB	16206 MiB	16205 MiB
~11 GiB resident	5630 MiB	~5454 MiB	5454 MiB

Matches rocm-smi. APU falls back via the UMA branch; builds clean on ROCm 7.2.1.

[ardfork]

This is similar to the local patch I have been using to circumvent the issue. I have dropped my local version and switched to this one, it seems to work similarly and resolves the issue too.

However, if this is confirmed to be a bug in the hipMemGetInfo implementation for AMD GPUs, shouldn't it be reported and fixed upstream? If the CUDA function behaves differently by returning all used memory rather than just compute related memory, I would consider that an implementation bug. Downstream projects should not have to implement patches or account for discrepancies in how memory is reported between AMD and NVIDIA cards.

[JohannesGaessler]

I agree that this seems like a HIP bug and should be fixed there. More generally since the default margin for --fit is a relatively conservative 1 GiB anyways I'm not sure a llama.cpp-side patch would be needed in the first place.

[liminfei-amd]

Agreed it is a HIP/ROCm bug. I traced it: hipMemGetInfo → ROCR HSA_AMD_AGENT_INFO_MEMORY_AVAIL → KFD AMDKFD_IOC_AVAILABLE_MEMORY, which does not count DRM graphics allocations (Vulkan/GL/compositor). It is the long-standing ROCm/legacy-rocm-build#1909.

I prototyped the fix in ROCR-Runtime (GpuAgent, MEMORY_AVAIL): on discrete GPUs, cross-check amdgpu's graphics-inclusive VRAM usage and take the conservative value. Verified by LD_PRELOAD of a rebuilt libhsa-runtime64 against an unmodified binary — hipMemGetInfo idle 16332→16170 and 5696→5420 MiB (== rocm-smi); multi-GPU independent; APUs keep the KFD value.

Since that fixes it for every HIP consumer with no application change, I'll take it upstream and close this PR. Thanks for the steer toward fixing it in HIP.

Upstream fix: ROCm/rocm-systems#7990 (tracking issue ROCm/rocm-systems#7993).

[liminfei-amd]

Closing in favor of an upstream ROCm fix.

Root cause is hipMemGetInfo (KFD AMDKFD_IOC_AVAILABLE_MEMORY, surfaced via ROCR HSA_AMD_AGENT_INFO_MEMORY_AVAIL) ignoring DRM graphics usage — ROCm/legacy-rocm-build#1909. I validated a fix in ROCR-Runtime (GpuAgent, MEMORY_AVAIL) that corrects hipMemGetInfo itself for all HIP consumers, so a llama.cpp-side workaround is not needed.

Thanks @ardfork and @JohannesGaessler for the steer toward fixing it in HIP.

[ardfork]

I agree that this seems like a HIP bug and should be fixed there. More generally since the default margin for --fit is a relatively conservative 1 GiB anyways I'm not sure a llama.cpp-side patch would be needed in the first place.

It's quite easy to have more than 1 GiB of VRAM allocated on a desktop system, meaning llama.cpp will try to use all of it. If you run llama.cpp while a video game or something similar is open, the program is completely blind to it, you could have 5 GiB free, but llama.cpp will see your total VRAM and try to overfill it. It's very annoying because, on my system, it often results in a GPU reset or a total system freeze that requires a restart. I wish there were some kind of early OOM killer for VRAM.

[JohannesGaessler]

The code is using a margin of 1 GB on top of all of the memory that is already in use.

[ardfork]

The code is using a margin of 1 GB on top of all of the memory that is already in use.

The whole problem that I shared in #24906 is that currently llama.cpp is unable to see the VRAM in use because hipMemGetInfo only shows VRAM allocated by KFD and a few other things. But will be completely blind to VRAM allocated from any DRM allocations (like your compositor or vulkan). It's not some ~200 MiB difference, it can easily be multiple GiB. Here is a simple example from my desktop:

Idle

Source	Total	Free	Used
hipMemGetInfo()	16384 MiB	16334 MiB	50 MiB
sysfs (mem_info_vram)	16384 MiB	14904 MiB	1480 MiB

With a video game running

Source	Total	Free	Used
hipMemGetInfo()	16384 MiB	16326 MiB	58 MiB
sysfs (mem_info_vram)	16384 MiB	11449 MiB	4935 MiB

So, fit by default will try to fit more than what's available. The sysfs source reports correctly the VRAM in use.