feat: add hami_gpu_device_health metric to scheduler

[manoj-1407]

Used Claude to help identify the gap and draft the patch.
I went through the code myself and understand it.

What type of PR is this?
/kind feature

What this PR does / why we need it:
DeviceUsage.Health is set by the device plugin via CheckHealth() and
the scheduler already uses it internally to skip unhealthy devices
during scoring. but it was never exposed as a Prometheus metric — if
a gpu goes unhealthy the only way to catch it is checking node
annotations manually. this adds hami_gpu_device_health as a gauge
(1=healthy, 0=unhealthy) inside collectNodeMetrics, same label set
as hami_gpu_memory_limit_bytes.

Which issue(s) this PR fixes:
Fixes #2612

Special notes for your reviewer:
same 0/1 gauge pattern hami_mig_device_info already uses.
nodeGPUDev

Summary by CodeRabbit

• New Features

  • Added a node-level GPU device health metric for monitoring.
  • Reports each GPU as healthy (1) or unhealthy (0) with node, device, and GPU type details.

• Tests

  • Added coverage confirming health values and associated device and node information are reported correctly.

[hami-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: manoj-1407
Once this PR has been reviewed and has the lgtm label, please assign fouof for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 655ae468-3e30-40ea-bb98-0eda0b6b31bf

📥 Commits

Reviewing files that changed from the base of the PR and between bb1a9fa and 693649f.

📒 Files selected for processing (2)

• cmd/scheduler/metrics.go
• cmd/scheduler/metrics_test.go

---

📝 Walkthrough

Walkthrough

The scheduler adds the hami_gpu_device_health Prometheus gauge. It reports each GPU as healthy or unhealthy with node and device labels. Tests verify metric names, labels, values, and collection errors.

Changes

GPU device health metrics

Layer / File(s)	Summary
Health metric collection and validation cmd/scheduler/metrics.go, cmd/scheduler/metrics_test.go	The scheduler defines and emits hami_gpu_device_health with node and device labels. Healthy devices emit 1; unhealthy devices emit 0. Tests validate both outputs and collection error handling.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• Project-HAMi/HAMi#2540: Implements the same device health metric and related test.
• Project-HAMi/HAMi#2170: Changes node-level GPU metric labels in the same scheduler metrics file.
• Project-HAMi/HAMi#2554: Updates device-related labels in scheduler GPU metrics and tests.

Suggested reviewers: kunwarsidhu47, archlitchi

Poem

A rabbit checks each glowing card,
One means healthy, zero means scarred.
Node and device labels hop in line,
Prometheus records each status sign.
“The GPUs now speak!” the rabbit cheers.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes add the required 0/1 health metric, labels, scheduler integration, and unit tests described in issue #2612.
Out of Scope Changes check	✅ Passed	The changes are limited to the requested scheduler metric and its unit test, with no unrelated code changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the addition of the hami_gpu_device_health metric to the scheduler.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}