feat(dashboards): add importable Grafana dashboard for HAMi GPU metrics - #2301
Conversation
The repo referenced a Grafana dashboard for GPU monitoring but only shipped a screenshot (imgs/hami-vgpu-metrics-dashboard.png); there was no JSON users could import. Add dashboards/hami-vgpu-dashboard.json built on the metrics currently exported by the scheduler and the vGPU monitor, plus a README with import steps and a metric reference. The dashboard uses a templated datasource variable and node/namespace template variables so it is portable across clusters and not tied to a specific Prometheus UID. Panels are grouped into cluster overview, physical host GPUs, scheduler allocation, and per-container vGPU workloads. Percent panels use green/amber/red thresholds at 75%/90%; no dual-axis panels. Link the new dashboards directory from the README. Signed-off-by: imantaba <itn.taba@gmail.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe PR adds a HAMi Grafana dashboard with cluster, host, scheduler, and vGPU workload panels. It documents Prometheus requirements, variables, metrics, and import steps. The main README now links directly to the ChangesHAMi Grafana dashboard
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Operator
participant Grafana
participant Prometheus
Operator->>Grafana: Select datasource, node, and namespace
Grafana->>Prometheus: Query HAMi vGPU metrics
Prometheus-->>Grafana: Return metric data
Grafana-->>Operator: Render dashboard panels
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@dashboards/hami-vgpu-dashboard.json`:
- Around line 240-329: Update the Prometheus expressions for the panels titled
“Host GPU memory used” and “Host GPU utilization” to filter by the selected
$node through a node-aware join on the shared device_uuid label, using
hami_gpu_memory_limit_bytes as the node source. Preserve the existing host
metrics and legends, and validate the joined queries against a live scrape
before merging.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: efa0eef6-0df6-47c1-a0a0-c6d900f8bb2e
📒 Files selected for processing (3)
README.mddashboards/README.mddashboards/hami-vgpu-dashboard.json
The host GPU panels queried hami_host_gpu_memory_used_bytes and
hami_host_gpu_utilization_ratio directly. Those metrics come from the vGPU
monitor and carry no node label, so the $node template variable had no
effect on them.
Filter both through 'and on (device_uuid) hami_gpu_memory_limit_bytes{node=~"$node"}':
the scheduler metric supplies the node to device_uuid mapping, so the host
panels now scope to the selected node. With All selected the match is a
no-op, preserving the previous cluster-wide view.
Signed-off-by: imantaba <itn.taba@gmail.com>
|
@mesutoezdil |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: archlitchi, imantaba The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
3 participants
What
The README points to a "pre-built Grafana dashboard" for GPU monitoring, but the repo only shipped a screenshot (
imgs/hami-vgpu-metrics-dashboard.png) — there was no JSON to import. This addsdashboards/hami-vgpu-dashboard.jsonplus a README with import steps and a metric reference.Details
hami_*names).datasourcevariable +node/namespacetemplate vars, so it is portable across clusters and not pinned to a specific Prometheus UID.dashboards/directory from the README.Testing
Validated the JSON conforms to Grafana's import schema (unique panel ids, complete panel/target fields, no gridPos overlaps). Every PromQL expression was checked against the metric names and label sets in
cmd/scheduler/metrics.goandcmd/vGPUmonitor/metrics.go.Does this PR introduce a user-facing change?
This change was prepared with AI assistance (per CONTRIBUTING.md); all changes were reviewed and verified by me.
Summary by CodeRabbit
New Features
Documentation