Skip to content

DGX Spark: UMA support - #17368

Merged
ggerganov merged 6 commits into
ggml-org:masterfrom
sfudally-nvidia:dgx_spark_uma
Nov 20, 2025
Merged

DGX Spark: UMA support#17368
ggerganov merged 6 commits into
ggml-org:masterfrom
sfudally-nvidia:dgx_spark_uma

Conversation

@sfudally-nvidia

@sfudally-nvidia sfudally-nvidia commented Nov 18, 2025

Copy link
Copy Markdown
Contributor

Add support for DGX Spark/UMA devices on Linux per this note on detecting available memory:
Memory reporting differences with unified memory architecture

ggerganov
ggerganov reviewed Nov 19, 2025
View reviewed changes

@ggerganov ggerganov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we expect cudaMemGetInfo to be fixed in the future and not need this workaround?

Comment thread ggml/src/ggml-cuda/ggml-cuda.cu
ORippler
ORippler reviewed Nov 19, 2025
View reviewed changes
Comment thread ggml/src/ggml-cuda/ggml-cuda.cu Outdated
slaren
slaren reviewed Nov 19, 2025
View reviewed changes
Comment thread ggml/src/ggml-cuda/ggml-cuda.cu Outdated
Comment thread ggml/src/ggml-cuda/ggml-cuda.cu Outdated
Comment thread ggml/src/ggml-cuda/ggml-cuda.cu Outdated
@github-actions github-actions Bot added Nvidia GPU Issues specific to Nvidia GPUs ggml changes relating to the ggml tensor library for machine learning labels Nov 19, 2025
@sfudally-nvidia @ggerganov
Update ggml/src/ggml-cuda/ggml-cuda.cu
6fcd8ec 
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
@sfudally-nvidia

sfudally-nvidia commented Nov 19, 2025

Copy link
Copy Markdown
Contributor Author

Can we expect cudaMemGetInfo to be fixed in the future and not need this workaround?

Yes, have discussed with CUDA team but do not have an ETA on when that would be released

ggerganov
ggerganov approved these changes Nov 20, 2025
View reviewed changes
Comment thread ggml/src/ggml-cuda/ggml-cuda.cu
@ggerganov
ggerganov requested a review from slaren November 20, 2025 06:18
@ggerganov
ggerganov merged commit a7784a8 into ggml-org:master Nov 20, 2025
1 check passed
@matt23654 matt23654 mentioned this pull request Nov 27, 2025
Closed
@AmesianX AmesianX mentioned this pull request Dec 18, 2025
Closed
Anico2 added a commit to Anico2/llama.cpp that referenced this pull request Jan 15, 2026
@Anico2 @ggerganov
DGX Spark: UMA support (ggml-org#17368)
2dc8aab 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
blime4 referenced this pull request in blime4/llama.cpp Feb 5, 2026
@sfudally-nvidia @ggerganov shaobo.xie
DGX Spark: UMA support (#17368)
fb01587 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
hogeheer499-commits added a commit to hogeheer499-commits/llama.cpp that referenced this pull request Mar 12, 2026
@hogeheer499-commits
ggml-cuda: skip UMA memory detection for HIP/ROCm builds
8674daa 
AMD APUs (e.g., Strix Halo with gfx1151) report prop.integrated=1,
which triggers the UMA memory detection path added in ggml-org#17368.
This causes the available GPU memory to be read from /proc/meminfo
(MemAvailable) instead of hipMemGetInfo().

On AMD APUs with large TTM allocations, hipMemGetInfo() correctly
reports the full GPU-accessible memory (e.g., 122880 MiB on a
128GB system), while /proc/meminfo reports significantly less
(~90 GiB), as it only reflects available system RAM.

The UMA detection path was designed for NVIDIA DGX Spark systems
and is not appropriate for AMD APUs, where hipMemGetInfo() already
returns accurate values.

Fix: guard the UMA detection block with !defined(GGML_USE_HIP).

Tested on: AMD Ryzen AI MAX+ 395, Radeon 8060S (gfx1151), 128GB
unified memory, ROCm 7.1, Ubuntu 24.04, kernel 6.18.14.

Fixes: ggml-org#18159
@hogeheer499-commits hogeheer499-commits mentioned this pull request Mar 12, 2026
Closed
hogeheer499-commits added a commit to hogeheer499-commits/llama.cpp that referenced this pull request Mar 12, 2026
@hogeheer499-commits
ggml-cuda: fix UMA memory detection for HIP/ROCm on AMD APUs
73357da 
AMD APUs report prop.integrated=1 which triggers the UMA memory
path from ggml-org#17368. This overrides hipMemGetInfo() (accurate) with
/proc/meminfo MemAvailable (too low), losing ~30 GiB on a 128GB
Strix Halo system.

For HIP builds, only enter the UMA path when GGML_CUDA_ENABLE_UNIFIED_MEMORY
is explicitly set. This preserves correct behavior for both cases:
- Default: hipMemGetInfo() reports accurate TTM-backed memory
- GGML_CUDA_ENABLE_UNIFIED_MEMORY=1: /proc/meminfo is used (system RAM mode)

Tested on AMD Ryzen AI MAX+ 395, Radeon 8060S (gfx1151), 128GB, ROCm 7.1.

Fixes: ggml-org#18159
hogeheer499-commits added a commit to hogeheer499-commits/llama.cpp that referenced this pull request Mar 19, 2026
@hogeheer499-commits hogeheer499-commits
ggml-cuda: fix UMA memory detection for HIP/ROCm on AMD APUs
a65750f 
AMD APUs report prop.integrated=1 which triggers the UMA memory
path from ggml-org#17368. This overrides hipMemGetInfo() (accurate) with
/proc/meminfo MemAvailable (too low), losing ~30 GiB on a 128GB
Strix Halo system.

For HIP builds, only enter the UMA path when GGML_CUDA_ENABLE_UNIFIED_MEMORY
is explicitly set. This preserves correct behavior for both cases:
- Default: hipMemGetInfo() reports accurate TTM-backed memory
- GGML_CUDA_ENABLE_UNIFIED_MEMORY=1: /proc/meminfo is used (system RAM mode)

Tested on AMD Ryzen AI MAX+ 395, Radeon 8060S (gfx1151), 128GB, ROCm 7.1.

Fixes: ggml-org#18159
@ehfd ehfd mentioned this pull request Mar 23, 2026
Merged
Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (ggml-org#17368)
dda6356 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
ljubomirj pushed a commit to ljubomirj/llama.cpp that referenced this pull request May 6, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (ggml-org#17368)
355a08d 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
my-other-github-account pushed a commit to my-other-github-account/llama.cpp that referenced this pull request May 15, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (ggml-org#17368)
39436d1 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
my-other-github-account pushed a commit to my-other-github-account/llama.cpp that referenced this pull request May 15, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (ggml-org#17368)
ea71f95 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
@ServeurpersoCom ServeurpersoCom mentioned this pull request May 23, 2026
Open
phibya pushed a commit to ziee-ai/llama.cpp that referenced this pull request May 29, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (ggml-org#17368)
87a1aec 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
fewtarius pushed a commit to fewtarius/CachyLLama that referenced this pull request May 30, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (ggml-org#17368)
112b6aa 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
@KI3P KI3P mentioned this pull request Jun 19, 2026
Open
This was referenced Jun 29, 2026
Closed
Open
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 5, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (ggml-org#17368)
3111cb1 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
@123-code 123-code mentioned this pull request Jul 7, 2026
Open
MrLordCat referenced this pull request in MrLordCat/llama.cpp-rdna-lab Jul 16, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (#17368)
4bb28e8 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
@Speedway1 Speedway1 mentioned this pull request Aug 11, 2026
Draft
@superm1 superm1 mentioned this pull request Aug 14, 2026
Merged
zommiommy pushed a commit to zommiommy/llama.cpp that referenced this pull request Aug 18, 2026
@sfudally-nvidia @ggerganov @zommiommy
DGX Spark: UMA support (ggml-org#17368)
4f90252 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ORippler ORippler ORippler left review comments
@ggerganov ggerganov ggerganov approved these changes
@slaren slaren slaren approved these changes

Assignees

No one assigned

Labels

ggml changes relating to the ggml tensor library for machine learning Nvidia GPU Issues specific to Nvidia GPUs

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sfudally-nvidia @ggerganov @slaren @ORippler