From e55bf30521cff68432096650aeee5d599f60ff66 Mon Sep 17 00:00:00 2001 From: imantaba Date: Wed, 5 Aug 2026 10:03:20 +0330 Subject: [PATCH] docs(monitoring): add GPU metrics reference and monitoring guide Add docs/monitoring.md describing HAMi's Prometheus metrics and how to operate them. It covers: - Which component emits what: the scheduler's allocation view (per node, device, container) and the vGPU monitor's runtime usage view (host GPU and per-container usage), including their metrics endpoints and ports. - A full reference of the current hami_-prefixed metrics with labels and descriptions, verified against the scheduler and vGPU-monitor metric definitions, plus a note that *_ratio metrics use a 0-100 scale. - Enabling scraping via prometheus.enabled (ServiceMonitors) or a plain Prometheus against the /metrics endpoints. - The legacy metric names and how to re-enable them. - Example PromQL, including a device_uuid join between the allocation and usage views, and pointers to the Grafana dashboard and the opt-in PrometheusRule alerts. Signed-off-by: imantaba --- docs/monitoring.md | 153 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 docs/monitoring.md diff --git a/docs/monitoring.md b/docs/monitoring.md new file mode 100644 index 0000000000..de9f3c054e --- /dev/null +++ b/docs/monitoring.md @@ -0,0 +1,153 @@ +# Monitoring HAMi + +HAMi exposes Prometheus metrics for GPU allocation and per-container GPU usage. +This document lists the metrics, explains where they come from, and shows how to +scrape them, alert on them, and visualize them. + +## Where metrics come from + +HAMi emits metrics from two components: + +| Component | What it reports | Default metrics endpoint | +| --- | --- | --- | +| Scheduler | Cluster/node allocation view — how much GPU memory and core the scheduler has handed out, per node, device, and container | container port `9395`, exposed as NodePort `31993` (`scheduler.service.monitorPort`) | +| vGPU monitor (runs in the device-plugin DaemonSet) | Runtime usage view — actual host GPU usage and per-container vGPU usage measured inside containers | container port `9394`, exposed as NodePort `31992` (`devicePlugin.service.httpPort`) | + +Both serve Prometheus text format at `/metrics`. + +- **Scheduler** metrics answer "what has been allocated?" — they come from the + scheduler's view of the cluster and always carry a `node` label. +- **vGPU monitor** metrics answer "what is actually being used?" — the host GPU + metrics carry no `node` label (they are per device), and the container metrics + are keyed by `namespace` / `pod` / `container`. + +> Ratio metrics (names ending in `_ratio`) are reported on a **0-100** scale +> (percent), not the 0-1 scale some Prometheus conventions use. Keep that in mind +> when writing thresholds and dashboards. + +## Enabling scraping + +If you run the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator), +set `prometheus.enabled=true` when installing the chart. That renders a +`ServiceMonitor` for both the scheduler and the vGPU monitor (each is also gated on +the `monitoring.coreos.com/v1` `ServiceMonitor` CRD being present): + +```bash +helm install hami hami-charts/hami -n kube-system \ + --set prometheus.enabled=true +``` + +Without the Prometheus Operator, point your own Prometheus at the two `/metrics` +endpoints directly. Both are exposed as NodePort services by default, so they are +reachable on every node: + +- scheduler: `http://:31993/metrics` (`scheduler.service.monitorPort`) +- vGPU monitor: `http://:31992/metrics` (`devicePlugin.service.httpPort`) + +## Metric reference + +The tables below cover the current (`hami_`-prefixed) metric names. All metrics are +gauges. + +### Scheduler metrics (allocation view) + +| Metric | Labels | Description | +| --- | --- | --- | +| `hami_gpu_memory_limit_bytes` | `node`, `device_uuid`, `device_index`, `device_type` | Total device memory of a physical GPU, in bytes | +| `hami_gpu_memory_allocated_bytes` | `node`, `device_uuid`, `device_index`, `device_cores`, `device_type` | Device memory the scheduler has allocated on a GPU, in bytes | +| `hami_gpu_core_limit_ratio` | `node`, `device_uuid`, `device_index`, `device_type` | Device core capacity of a GPU, against which `hami_gpu_core_allocated_ratio` is measured (0-100) | +| `hami_gpu_core_allocated_ratio` | `node`, `device_uuid`, `device_index`, `device_type` | Percentage of a GPU's cores allocated by the scheduler (0-100) | +| `hami_gpu_shared_count` | `node`, `device_uuid`, `device_index`, `device_type` | Number of containers sharing a GPU | +| `hami_node_gpu_memory_allocated_ratio` | `node`, `device_uuid`, `device_index` | Percentage of a GPU's memory allocated by the scheduler (0-100) | +| `hami_node_gpu_overview` | `node`, `device_uuid`, `device_index`, `device_cores`, `device_memory_limit`, `device_type` | Per-GPU overview series carrying capacity as labels | +| `hami_node_gpu_mig_instance_info` | `node`, `device_uuid`, `device_index`, `mig_name` | GPU sharing mode: `0` hami-core, `1` MIG, `2` MPS | +| `hami_vgpu_memory_allocated_bytes` | `namespace`, `node`, `pod`, `container_index`, `device_uuid` | vGPU memory allocated to a container, in bytes | +| `hami_vgpu_core_allocated_ratio` | `namespace`, `node`, `pod`, `container_index`, `device_uuid` | vGPU cores allocated to a container (0-100) | +| `hami_resource_quota_used` | `namespace`, `quota_name`, `limit` | Device `ResourceQuota` usage | + +### vGPU monitor metrics (runtime usage view) + +| Metric | Labels | Description | +| --- | --- | --- | +| `hami_host_gpu_memory_used_bytes` | `device_index`, `device_uuid`, `device_type` | Physical GPU memory in use, in bytes | +| `hami_host_gpu_utilization_ratio` | `device_index`, `device_uuid`, `device_type` | Physical GPU core utilization (0-100) | +| `hami_vgpu_memory_used_bytes` | `namespace`, `pod`, `container`, `vdevice_index`, `device_uuid` | vGPU memory used by a container, in bytes | +| `hami_vgpu_memory_limit_bytes` | `namespace`, `pod`, `container`, `vdevice_index`, `device_uuid` | vGPU memory limit for a container, in bytes | +| `hami_container_device_utilization_ratio` | `namespace`, `pod`, `container`, `vdevice_index`, `device_uuid` | Container GPU SM utilization (0-100) | +| `hami_container_device_memory_bytes` | `namespace`, `pod`, `container`, `vdevice_index`, `device_uuid` | Container device memory usage, in bytes | +| `hami_container_last_kernel_elapsed_seconds` | `namespace`, `pod`, `container`, `vdevice_index`, `device_uuid` | Seconds since the container last ran a GPU kernel | +| `hami_vgpu_memory_context_bytes` | `namespace`, `pod`, `container`, `vdevice_index`, `device_uuid` | Container device memory context size, in bytes | +| `hami_vgpu_memory_module_bytes` | `namespace`, `pod`, `container`, `vdevice_index`, `device_uuid` | Container device memory module size, in bytes | +| `hami_vgpu_memory_buffer_bytes` | `namespace`, `pod`, `container`, `vdevice_index`, `device_uuid` | Container device memory buffer size, in bytes | +| `hami_mig_device_info` | `namespace`, `pod`, `container`, `vdevice_index`, `device_uuid`, `instance_id` | MIG device information for a container | + +### Legacy metric names + +Earlier HAMi releases used camelCase metric names (for example `GPUDeviceMemoryLimit`, +`vGPUMemoryAllocated`, `nodeGPUMemoryPercentage`). These are disabled by default and +can be re-enabled for backward compatibility with `legacyMetrics: true` (chart value) +or the `--legacy-metrics=true` flag. Prefer the `hami_`-prefixed names above for new +dashboards and alerts; the legacy names may be removed in a future release. + +## Joining allocation and usage + +Scheduler and monitor metrics share the `device_uuid` label, which lets you relate +the two views. The host GPU metrics carry no `node` label, so to look at one node's +devices you **filter** them against a scheduler metric that has both `node` and +`device_uuid`: + +```promql +hami_host_gpu_utilization_ratio + and on (device_uuid) hami_gpu_memory_limit_bytes{node="gpu-node-1"} +``` + +`and` returns the host series unchanged wherever a matching `device_uuid` exists on +the right — it filters, but does not copy any labels. To actually **attach** the +`node` label (for example to group host usage by node), use a label-preserving +arithmetic join instead: + +```promql +hami_host_gpu_utilization_ratio + * on (device_uuid) group_left(node) + (hami_gpu_memory_limit_bytes * 0 + 1) +``` + +The right-hand `* 0 + 1` turns each scheduler series into `1`, so multiplying keeps +the host value intact while `group_left(node)` carries the `node` label across. + +## Useful queries + +```promql +# GPU memory allocated vs. capacity on a node (percent) +100 * sum by (node) (hami_gpu_memory_allocated_bytes) + / sum by (node) (hami_gpu_memory_limit_bytes) + +# Containers using more than 80% of their vGPU memory limit (percent) +100 * hami_vgpu_memory_used_bytes / (hami_vgpu_memory_limit_bytes > 0) > 80 + +# Number of GPUs currently shared by more than one container +count(hami_gpu_shared_count > 1) +``` + +## Dashboards + +A ready-to-import Grafana dashboard for these metrics lives in +[`dashboards/`](../dashboards/). See its [README](../dashboards/README.md) for import +steps and the variables it exposes. + +## Alerting + +The Helm chart can install a `PrometheusRule` with a starter set of GPU alerts +(memory/core exhaustion, vGPU memory near limit, scheduler metrics absent). It is +opt-in: + +```bash +helm upgrade hami hami-charts/hami -n kube-system \ + --set prometheus.enabled=true \ + --set prometheus.alerts.enabled=true +``` + +The rule renders only when the `monitoring.coreos.com/v1` `PrometheusRule` CRD is +present. Each alert has its own `threshold`, `for`, and `severity`, and you can add +labels so your Operator's `ruleSelector` picks up the rule — see the +`prometheus.alerts` section of [`charts/hami/values.yaml`](../charts/hami/values.yaml).