UPSTREAM PR #20536: ggml/hip: fix APU compatibility - soft error handling for hipMemAdviseSetCoarseGrain

[loci-dev]

Note

Source pull request: ggml-org/llama.cpp#20536

Description:

Problem

On AMD APU/iGPU devices (unified memory architecture, e.g. AMD Strix Halo gfx1151), hipMemAdviseSetCoarseGrain returns
hipErrorInvalidValue because this hint is not applicable to UMA systems. The current code wraps this call in CUDA_CHECK(), which treats
it as a fatal error and crashes.

Fix

Treat hipMemAdviseSetCoarseGrain as an optional performance hint:

• Remove CUDA_CHECK() wrapper
• Clear any resulting error with hipGetLastError() to prevent propagation

This matches the intent of the existing comment ("fall back to cudaMalloc if not supported") and is consistent with how optional hints are
handled elsewhere.

Additional Changes

• Add GGML_LOG_DEBUG pre-allocation memory logging to help diagnose memory issues on APU systems
• Store totalGlobalMem in device info struct for future use

Testing

Tested on AMD Strix Halo (gfx1151), 128GB unified memory, Windows 11:

• ✅ Before: crash on hipMemAdviseSetCoarseGrain with hipErrorInvalidValue
• ✅ After: runs successfully, no error propagation

Context: ROCm APU Large BAR Bug

AMD APUs on Windows are currently limited to ~64GB hipMallocManaged allocations due to a ROCm runtime bug where largeBar_ is
unconditionally disabled for all APU devices in HIP mode. This causes the Windows GART allocator's 50%-of-RAM cap to trigger prematurely.

A fix has been submitted to ROCm upstream:
ROCm/rocm-systems#4077

Without that ROCm fix, APUs are still limited to ~64GB regardless of this change. However, this PR is independently valuable:

1. Prevents crashes on APUs with current ROCm versions
2. Enables full functionality once users update to a patched ROCm runtime
3. The debug logging helps users diagnose memory allocation issues

Impact

• ✅ APU/iGPU users: no more crashes when using GGML_CUDA_ENABLE_UNIFIED_MEMORY
• ✅ Discrete GPU users: no change (hint is valid and still applied)
• ✅ No performance regression

[loci-review]

No meaningful performance changes were detected across 120078 analyzed functions in the following binaries: build.bin.llama-tts, build.bin.libmtmd.so, build.bin.libllama.so, build.bin.llama-bench, build.bin.llama-cvector-generator, build.bin.llama-quantize, build.bin.llama-qwen2vl-cli, build.bin.llama-tokenize, build.bin.llama-gemma3-cli, build.bin.llama-gguf-split, build.bin.llama-llava-cli, build.bin.llama-minicpmv-cli, build.bin.libggml-base.so, build.bin.libggml-cpu.so, build.bin.libggml.so.

🔎 Full breakdown: Loci Inspector
💬 Questions? Tag @loci-dev