docs(monitoring): add GPU metrics reference and monitoring guide

[imantaba]

What type of PR is this?

/kind documentation

What this PR does / why we need it:

Adds docs/monitoring.md, a reference for HAMi's Prometheus metrics and how to
operate them. Today the metrics exist but are only discoverable by reading the
scheduler and vGPU-monitor source, so operators have no single place to learn the
names, labels, and scales.

The doc covers:

• Where metrics come from — the scheduler's allocation view (per node, device,
  container; always carries node) versus the vGPU monitor's runtime usage view
  (host GPU and per-container usage), with each component's /metrics endpoint and
  port.
• A full metric reference — every current hami_-prefixed metric with its
  labels and description, split into the allocation and usage tables. All names and
  label sets were verified against the metric definitions in cmd/scheduler and
  cmd/vGPUmonitor.
• The 0-100 gotcha — *_ratio metrics are reported on a 0-100 (percent) scale,
  not 0-1.
• Enabling scraping — via prometheus.enabled (renders ServiceMonitors) or a
  plain Prometheus pointed at the endpoints.
• Legacy metric names and how to re-enable them.
• Example PromQL, including a device_uuid join between the allocation and
  usage views.

Which issue(s) this PR fixes:

Special notes for your reviewer:

This is the documentation piece of a small observability series:

• feat(dashboards): add importable Grafana dashboard for HAMi GPU metrics #2301 adds the Grafana dashboard under dashboards/
• feat(helm): add opt-in PrometheusRule alerts for GPU metrics #2372 adds the opt-in PrometheusRule alerts (prometheus.alerts.*)

This doc's "Dashboards" and "Alerting" sections link to dashboards/ and the
prometheus.alerts values added by those two PRs, so it reads best merged after
them. The metric reference, scraping, and query sections are self-contained and
accurate against the current master.

This change was prepared with AI assistance (per CONTRIBUTING.md); all changes were
reviewed and verified by me.

Does this PR introduce a user-facing change?:

NONE

Summary by CodeRabbit

• Documentation
  • Added comprehensive monitoring guidance, including Prometheus setup, scheduler and vGPU metrics, legacy compatibility, PromQL examples, Grafana dashboards, and optional alerting rules.

[hami-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: imantaba
Once this PR has been reviewed and has the lgtm label, please assign dsfans2014 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8a3db3dd-ba51-408f-9547-4d8f7895f554

📥 Commits

Reviewing files that changed from the base of the PR and between e6d0902 and e55bf30.

📒 Files selected for processing (1)

• docs/monitoring.md

🚧 Files skipped from review as they are similar to previous changes (1)

• docs/monitoring.md

---

📝 Walkthrough

Walkthrough

The pull request adds a monitoring guide for HAMi. It documents metric sources, Prometheus collection, metric names, legacy compatibility, PromQL queries, Grafana dashboards, and optional PrometheusRule alerts.

Changes

Monitoring documentation

Layer / File(s)	Summary
Monitoring guide and metric contracts docs/monitoring.md	Defines scheduler and vGPU monitor sources, endpoints, labels, units, ratios, and current metric names.
Collection and legacy compatibility docs/monitoring.md	Documents Prometheus Operator setup, direct scraping, and legacy metric configuration.
Queries, dashboards, and alerting docs/monitoring.md	Adds PromQL examples, Grafana dashboard guidance, and optional PrometheusRule alert configuration.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• Project-HAMi/HAMi#2301: Adds related HAMi monitoring and Grafana documentation.
• Project-HAMi/HAMi#2361: Documents related monitoring metrics and PromQL guidance.
• Project-HAMi/HAMi#2372: Adds related PrometheusRule monitoring configuration.

Suggested reviewers: archlitchi

Poem

I’m a rabbit with metrics to share,
Prometheus now finds them there.
Scheduler counts, runtime glow,
Grafana charts what systems know.
Alerts wait by the dashboard light.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the addition of GPU metrics documentation and a monitoring guide.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/monitoring.md`:
- Around line 108-109: Update the Prometheus query comment and expression so
they agree: either add an explicit usage threshold such as >80 to return only
containers near their vGPU memory limit, or rename the comment to describe that
the query calculates usage percentages for all containers with positive limits.
- Around line 11-16: Update the Prometheus scrape configuration guidance in
docs/monitoring.md to include the vGPU monitor’s /metrics target alongside the
scheduler endpoint, specifying how non-Operator Prometheus discovers or reaches
the monitor on port 9394. Ensure the documented target uses the actual vGPU
monitor service or pod discovery method rather than implying port 9394 is
directly accessible through a NodePort.
- Around line 91-99: Update the monitoring documentation around the example
PromQL query to remove the claim that `and on (device_uuid)` attaches or copies
the `node` label. Describe the query only as filtering host GPU series by
matching scheduler metrics, or replace it with a label-preserving arithmetic
join if the example must support downstream grouping by `node`.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 414df542-1c81-4ba8-80eb-3d4c040b3812

📥 Commits

Reviewing files that changed from the base of the PR and between e6d0902 and 8aa1485.

📒 Files selected for processing (1)

• docs/monitoring.md

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[archlitchi]

docs/ now will only accept design document, for user-guide, please submit PR to the hami-website monitoring section, link: https://project-hami.io/docs/userguide/monitoring/device-allocation

let me know if you submit the PR in website

[imantaba]

docs/ now will only accept design document, for user-guide, please submit PR to the hami-website monitoring section, link: https://project-hami.io/docs/userguide/monitoring/device-allocation

let me know if you submit the PR in website

Thanks @archlitchi . I've opened the website PR: (https://github.com/Project-HAMi/website/pull/730).
Rather than move the whole doc over, I trimmed it to what the site did not already cover.