fix: handle GetMemoryInfo NOT_SUPPORTED for unified memory GPUs (cherry-pick #1637 to release-v2.8)

[saiyam1814]

What type of PR is this?

/kind bug

What this PR does / why we need it:

Cherry-picks commit 13009aa (PR #1637) onto the release-v2.8 branch so that an eventual v2.8.3 ships with the unified-memory fix for NVIDIA GB10 (DGX Spark) and other architectures whose GPUs return ERROR_NOT_SUPPORTED from nvmlDeviceGetMemoryInfo.

Without this backport, users on v2.8.1 and v2.8.2 continue to hit the device-plugin panic at register.go:115 on GB10 hardware. See #1838 for the full reproduction and #1511 for the original bug report.

The cherry-pick applies cleanly to release-v2.8 (no merge conflicts):

charts/hami/templates/scheduler/device-configmap.yaml  |  1 +
charts/hami/values.yaml                                |  5 +++++
cmd/vGPUmonitor/metrics.go                             |  4 ++++
.../nvidiadevice/nvinternal/plugin/register.go         | 18 ++++++++++++++++--
pkg/device/nvidia/device.go                            |  7 ++++---
5 files changed, 30 insertions(+), 5 deletions(-)

The diff is byte-identical to the upstream merge in master; commit author is preserved as @jsl9208.

Which issue(s) this PR fixes:

Fixes #1838

(Also indirectly resolves the v2.8.x portion of #1511; master is already covered by #1637.)

Special notes for your reviewer:

• Verified on a real DGX Spark (GB10, aarch64) that the post-fix code:
  • Logs GetMemoryInfo not supported for device <UUID>, using configured PreConfiguredDeviceMemory: 131072 MB instead of panicking.
  • Successfully registers nvidia.com/gpu: 2 with the configured deviceSplitCount.
  • Allows two pods to share the GPU with HAMi's per-pod memory accounting.
• No new tests added: the fix's unit-test coverage is in the original PR fix: handle GetMemoryInfo ERROR_NOT_SUPPORTED for unified memory GPUs #1637 and applies unchanged here.
• Submodule pointer (libvgpu) intentionally left untouched.

Does this PR introduce a user-facing change?:

Yes — a new chart value, devicePlugin.preConfiguredDeviceMemory (in MiB), which the device-plugin uses as the device's memory budget when NVML reports ERROR_NOT_SUPPORTED. Default 0 preserves the previous behaviour for non-unified-memory GPUs.

NVIDIA device-plugin no longer panics on GB10 / unified memory architectures
when `nvmlDeviceGetMemoryInfo` returns ERROR_NOT_SUPPORTED. Set
`devicePlugin.preConfiguredDeviceMemory` (MiB) to enable scheduling on
those devices. (cherry-pick of #1637)

---

AI assistance disclosure: this PR's body and the cherry-pick mechanics were drafted with Claude Code; the underlying code change is the unmodified content of #1637, authored by @jsl9208. End-to-end behaviour of the cherry-picked code was verified by me on a DGX Spark before submission.

[hami-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: saiyam1814
Once this PR has been reviewed and has the lgtm label, please assign wawa0210 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hami-robot]

Welcome @saiyam1814! It looks like this is your first PR to Project-HAMi/HAMi 🎉

[gemini-code-assist]

Code Review

This pull request introduces support for GPUs with unified memory architecture, such as NVIDIA GB10 and DGX Spark, which do not support traditional NVML memory queries. It adds a "preConfiguredDeviceMemory" configuration option to the Helm charts and internal configuration structures, allowing users to manually specify GPU memory. The vGPUmonitor and NVIDIA device plugin are updated to handle "nvml.ERROR_NOT_SUPPORTED" errors by either utilizing the pre-configured memory value or skipping the device with informative logging. I have no feedback to provide as there were no review comments to evaluate.

[mesutoezdil]

@saiyam1814 thx, looks good. and looked at the code before weighing in.
on the 3 options:
for 1 (constrain markannotationstodelete) prevents new stuck states but does nothing for existing ones. wrong end of the pipe.
for 2 (device-plugin re-stamps requesting) won't recover scheduling. the scheduler sees requesting_, returns (true, false), and still skips the cache update. to actually trigger re-registration the device-plugin would need to delete the annotation entirely so the else branch fires (true, true). that's a cross-component change and puts cache correctness at the mercy of device-plugin startup sequencing.

but nr 3 s the right fix. the timestamp is already therea nd it just needs to be used. mirror the requesting logic: treat deleted_ older than 60s as recoverable, stamp requesting_, and return (true, true). i think, single-file change in pkg/device/devices.go, no device-plugin changes, covers all vendors (hygon, ascend, vastAI, etc. have no gPU-count fallback so they're stuck unconditionally today).

and one note: bare "deleted" without a timestamp appears in some tests, make sure the parse-error path stays at return true, false rather than accidentally recovering.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...-plugin/nvidiadevice/nvinternal/plugin/register.go	0.00%	13 Missing ⚠️

Flag	Coverage Δ
unittests	51.08% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
pkg/device/nvidia/device.go	48.46% <ø> (ø)
...-plugin/nvidiadevice/nvinternal/plugin/register.go	0.00% <0.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.