fix(metrics): resolve high-cardinality leak in memory_allocated_bytes

[Aryanbhargava18]

What happened

The hami_gpu_memory_allocated_bytes metric (and its legacy counterpart GPUDeviceMemoryAllocated) defines a device_cores label but incorrectly injects fmt.Sprint(devs.Device.Usedcores) as its value instead of Totalcore.

Because Usedcores is a dynamic state variable (changing every time a pod is scheduled or unscheduled), this causes a severe Prometheus High Cardinality Leak. Every time the used cores change, Prometheus is forced to create a completely new time series, abandoning the old one. On a cluster with high pod churn, this combinatorial explosion of stale time series will eventually cause Prometheus OOMs.

What you expected to happen

Prometheus best practices dictate that labels must represent stable identity attributes, not dynamic state. The device_cores label should represent the static core capacity of the device (Totalcore), while the dynamic state should remain as metric values (e.g., via hami_gpu_core_allocated_ratio).

How to reproduce

1. Deploy HAMi and scrape :31993/metrics.
2. Schedule a pod that requests GPU cores.
3. Observe hami_gpu_memory_allocated_bytes.
4. Delete the pod.
5. Observe that a new time series is created for hami_gpu_memory_allocated_bytes with a different device_cores label, instead of updating the existing series.

Root cause

In cmd/scheduler/metrics.go:

ch <- prometheus.MustNewConstMetric(
    nodevGPUMemoryAllocatedDesc,
    prometheus.GaugeValue,
    float64(devs.Device.Usedmem)*float64(1024)*float64(1024),
    nodeID, devs.Device.ID, fmt.Sprint(devs.Device.Index), 
    fmt.Sprint(devs.Device.Usedcores), // <-- BUG: Injecting dynamic state into label
    devs.Device.Type,
)

Fix

Changed devs.Device.Usedcores to devs.Device.Totalcore when populating the device_cores label, stabilizing the time series identity.

Note: This aligns directly with the gap analysis currently open in #2126 regarding metric cardinality risks.

Summary by CodeRabbit

• Bug Fixes
  • Updated GPU-related Prometheus metrics to align label definitions and semantics across both current and legacy formats.
  • For GPU memory allocation, the device_cores label now reflects total device cores (while the underlying memory allocation value remains based on used memory).
  • For GPU node overview, the label set was simplified by removing the container-sharing dimension, and the device_cores / device_memory_limit labels now report total cores and memory limits.

[hami-robot]

Welcome @Aryanbhargava18! It looks like this is your first PR to Project-HAMi/HAMi 🎉

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9c96021f-db93-43b1-bfb0-b14ae76f9873

📥 Commits

Reviewing files that changed from the base of the PR and between 4760bda and f3747aa.

📒 Files selected for processing (2)

• cmd/scheduler/metrics.go
• skill/hami-vgpu-metrics-summary/SKILL.md

---

📝 Walkthrough

Walkthrough

GPU allocation and overview metrics now use aligned label sets in current and legacy modes. Overview metrics no longer expose shared-container labels, allocation labels use total device cores, and metric documentation lists the updated core and memory-limit labels.

Changes

GPU metric label alignment

Layer / File(s)	Summary
Align GPU metric descriptors and emissions cmd/scheduler/metrics.go	Current and legacy overview descriptors remove shared-container labels. Allocation and overview emissions use total device cores and total device memory where applicable, while allocated-memory values remain based on used memory.
Document updated overview labels skill/hami-vgpu-metrics-summary/SKILL.md	GPU metrics documentation now lists current and legacy device-core labels alongside the memory-limit labels.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested reviewers: haitwang-cloud

Poem

A bunny trims labels, tidy and bright,
Total cores hop into sight.
Shared containers leave the trail,
Docs now tell the matching tale.
Metrics bounce, aligned just right!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately reflects the main fix for the high-cardinality leak in memory_allocated_bytes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[FouoF]

Better remove these changing labels. Actually other metrics can fully replace them.

[Aryanbhargava18]

Makes sense. I've completely removed the device_cores, shared_containers, and device_memory_limit labels from those metrics, since the other gauges already cover them. Pushed the update.

[archlitchi]

/lgtm

[hami-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: archlitchi, Aryanbhargava18

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [archlitchi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment