llama : stage mmap uploads on integrated GPUs

[liminfei-amd]

Overview

Fixes #15018.

Fully offloaded models on integrated GPUs currently upload mmap-backed tensor
data directly from the file mapping. On large models this can enter an extreme
pageable-memory transfer path: in an exact-model reproduction with the reported
82.36 GiB model, loading stopped making progress at 34%, while --no-mmap
completed normally.

For eligible full-offload IGPU loads, read the GGUF data into the model
loader's existing pinned staging buffers and upload it asynchronously with
events. Disable eager whole-file mmap prefetch only for this path so the model
is not read twice.

CPU, discrete-GPU, partial-offload, --mlock, --check-tensors, and backends
without the required async, host-buffer, or event capabilities retain their
existing paths and fallbacks.

Additional information

Validated with the original 82.36 GiB Llama-4 Scout Q6_K model on gfx1151:

• HIP: clean mmap timed out after 600 seconds at 34%; the staged path completed
  in 40.54 seconds with about 493 MiB peak RSS
• Vulkan: the staged path completed in 41.05 seconds with about 206 MiB peak
  RSS
• a separate deterministic small-model control produced unchanged post-load
  logits on HIP and Vulkan
• partial offload, no-pinned fallback, --mlock, and --check-tensors
  controls preserved their existing behavior
• a gfx1201 HIP discrete-GPU control remained unchanged at 4.2587 seconds
• focused model-loader tests passed, 5/5

The triggering workload requires a model larger than 64 GiB and a large-UMA
IGPU, so it is not practical as a normal automated regression test.

@kyuz0 @pwilkin , could you please test this on the original HP Z2 Mini G1a / Fedora 42
setup with default mmap and full offload, preferably using the same Llama-4
Scout model, and confirm whether loading now reaches 100% normally?

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - GitHub Copilot assisted with investigation, implementation, testing, and drafting. The final diff and validation evidence were reviewed before submission.

---

This is an experimental, AI-generated code change, reviewed by AMD engineers before submission.

[0cc4m]

IMO the correct solution here is to not default to mmap on integrated GPUs and to discourage using it for them, since it serves no purpose here unless the backend supports directly working with the mmapped data. We plan to look into that after #20834 lands.

[pwilkin]

@0cc4m you need mmap to use models bigger than your available RAM :)

@liminfei-amd just a note that this sidesteps a ROCm bug: ROCm/ROCm#6501 - I've verified with the new experimental HRX engine with their bindings for HIP and it mmaps the model without problems.

[0cc4m]

you need mmap to use models bigger than your available RAM

It doesn't work unless you access the pages directly from the GPU, which currently you can't, as far as I know, at least on CUDA/ROCm and Vulkan.

[pwilkin]

Wdym "it doesn't work", I just ran a 220GB quant on my Strix Halo :P

[0cc4m]

It doesn't work because the backend currently has to copy the mmapped data into GPU-visible (pinned) memory, instead of directly using the mmapped memory pages.

[pwilkin]

But those are two different meanings of "it doesn't work".

It works in the sense that you can actually run a model that has more than your VRAM, which you can't do with any of the other memory allocation modes.

It doesn't work in the sense that it's slower than it could be with direct memory access.

[0cc4m]

Yes, I suppose that is true, though I do not really know what it does if you try to mmap a model larger than your available shared RAM into an iGPU system. AFAIK it loads the file into RAM cache, then tries to copy each chunk into the same RAM, which is gonna OOM eventually. Did you partially offload when you tried that?

[ORippler]

IMO the correct solution here is to not default to mmap on integrated GPUs and to discourage using it for them, since it serves no purpose here unless the backend supports directly working with the mmapped data. We plan to look into that after #20834 lands.

+1 for default-disabled mmap on UMA systems, as most common use-cases will try to load a model that fits into RAM from disk once instead of weight-streaming. Reducing sysram pressure here is advised, and with -directio one should see competitive loading speeds

[liminfei-amd]

Thanks for the feedback and clarification on the preferred direction. The staged upload path is effective, but I agree that the new load-mode framework from #20834 is the better place to handle the default behavior for UMA systems.

I’m closing this PR in favor of that direction. The underlying ROCm mmap issue remains tracked in ROCm/ROCm#6501, and the validation results here can still be useful if a staged fallback is needed later.