fix(device-plugin): skip devices that do not support event based healthchecking

[Wangmin362]

What type of PR is this?

/kind bug

What this PR does / why we need it:

nvmlDeviceRegisterEvents returning ERROR_NOT_SUPPORTED means the device cannot report health through the event mechanism, not that it is faulty. The two separate ifs let that return code fall through into the second branch, so the device is handed to ListAndWatch as Unhealthy and stops being schedulable. That state is not reversible - ListAndWatch carries a FIXME: there is no way to recover from the Unhealthy state - and a restart only repeats the same failed registration. The logs contradict themselves: one line says the device is "too old to support healthchecking", the next one marks it unhealthy.

A switch makes the two cases exclusive. NVIDIA fixed the same block upstream in k8s-device-plugin (commit 8cd1447, "Fix healthchecking on old devices").

Which issue(s) this PR fixes:

NONE

Special notes for your reviewer:

Added a test covering the three RegisterEvents outcomes. Checked both ways: with health.go reverted to master the test fails on the ERROR_NOT_SUPPORTED assertion, with the fix the package passes.

I have no card that answers ERROR_NOT_SUPPORTED for the mask HAMi asks for, so this is not reproduced end to end on a node. I did probe NVML on our test cluster to confirm the return code is reachable in practice: on a GeForce RTX 4090 D (driver 595.71.05) nvmlDeviceRegisterEvents returns ERROR_NOT_SUPPORTED for the ECC event bits, and on a Tesla V100 (driver 550.54.15) for a bit outside its supported mask.

This PR was written primarily by Claude Code.

Does this PR introduce a user-facing change?:

Devices that do not support event based health checking are no longer reported as unhealthy, so they stay schedulable.

Summary by CodeRabbit

• Bug Fixes

  • Devices remain healthy when event registration is unsupported by the available hardware or driver.
  • Other event registration failures continue to be reported and mark the device unhealthy.

• Tests

  • Added coverage for successful, unsupported, and failed event registration scenarios.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c8d07d50-9c9c-4f74-a80c-89f5a6eb45e8

📥 Commits

Reviewing files that changed from the base of the PR and between e6d0902 and 7522094.

📒 Files selected for processing (2)

• pkg/device-plugin/nvidiadevice/nvinternal/rm/health.go
• pkg/device-plugin/nvidiadevice/nvinternal/rm/health_registerevents_test.go

---

📝 Walkthrough

Walkthrough

The health check now handles nvml.ERROR_NOT_SUPPORTED as a warning during event registration. Other registration failures still mark the device unhealthy. A test covers unsupported, failed, and successful registration results.

Changes

NVML health registration

Layer / File(s)	Summary
Event registration handling and validation pkg/device-plugin/nvidiadevice/nvinternal/rm/health.go, pkg/device-plugin/nvidiadevice/nvinternal/rm/health_registerevents_test.go	Unsupported event registration logs a warning without changing device health. Other failures mark the device unhealthy. The test validates all three registration outcomes and shutdown behavior.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• Project-HAMi/HAMi#2322: Both changes refine NVML health handling in checkHealth, but this PR covers event registration.

Suggested reviewers: lengrongfu, ouyangluwei163

Poem

A rabbit checks the NVML gate,
“Unsupported” waits, but does not seal fate.
Failed events turn health to red,
Success keeps green above the thread.
Three small cases hop through the test.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the change to skip devices that do not support event-based health checks.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag	Coverage Δ
unittests	62.03% <100.00%> (+0.30%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...device-plugin/nvidiadevice/nvinternal/rm/health.go	64.08% <100.00%> (+20.19%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[archlitchi]

/lgtm

[hami-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: archlitchi, Wangmin362

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [archlitchi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment