Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ The default monitor port is `31993`. You can change it with Helm values such as
HAMi also provides:

- [HAMi-WebUI](https://github.com/Project-HAMi/HAMi-WebUI) for visual cluster and device management.
- Grafana dashboard examples for accelerator monitoring.
- [Grafana dashboards](dashboards/) for accelerator monitoring.
- Benchmark material for evaluating workload behavior and scheduling effects.

![HAMi WebUI](imgs/hami-webui-overview.png)
Expand Down
76 changes: 76 additions & 0 deletions dashboards/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# HAMi Grafana dashboards

Importable Grafana dashboards for the metrics HAMi exposes through Prometheus.

| File | Dashboard | Data source |
| --- | --- | --- |
| [`hami-vgpu-dashboard.json`](hami-vgpu-dashboard.json) | HAMi vGPU metrics | Prometheus |

![HAMi vGPU metrics dashboard](../imgs/hami-vgpu-metrics-dashboard.png)

## Prerequisites

- A Prometheus instance scraping HAMi's components. HAMi exposes metrics from the
**scheduler** and the **vGPU monitor**; the Helm chart can create `ServiceMonitor`
objects for them when `prometheus.enabled=true` (see `charts/hami/values.yaml`).
- Grafana 9.x or newer with that Prometheus configured as a data source.

## Import

**From the Grafana UI**

1. Go to **Dashboards → New → Import**.
2. Upload `hami-vgpu-dashboard.json` (or paste its contents).
3. When prompted, select your Prometheus data source and click **Import**.

**From the API**

```bash
curl -sS -X POST "$GRAFANA_URL/api/dashboards/db" \
-H "Authorization: Bearer $GRAFANA_TOKEN" \
-H "Content-Type: application/json" \
-d "{\"dashboard\": $(cat hami-vgpu-dashboard.json), \"overwrite\": true}"
```

The dashboard uses a templated data-source variable, so it is not tied to any
specific Prometheus UID — Grafana asks which data source to bind on import.

## Variables

| Variable | Meaning |
| --- | --- |
| `datasource` | The Prometheus data source to query. |
| `node` | Filter host/scheduler panels to one or more nodes (defaults to all). |
| `namespace` | Filter vGPU/container panels to one or more namespaces (defaults to all). |

## Panels

- **Cluster overview** — physical GPU count, total and allocated GPU memory,
cluster memory-allocated %, and shared-container count.
- **Physical GPUs (host)** — per-device memory used and utilization, as measured by
the vGPU monitor.
- **Scheduler / allocation** — allocated vs limit GPU memory, per-node memory and
core allocation ratios, and per-device shared count.
- **vGPU / container workloads** — per-container vGPU memory used vs limit,
container utilization, memory used as a % of limit, and a top-10 table.

## Metrics used

The dashboard is built on these HAMi metrics (the current `hami_*` names, exported
on the metrics port; the scheduler and the vGPU monitor each expose a subset):

| Metric | Source | Notes |
| --- | --- | --- |
| `hami_gpu_memory_limit_bytes` | scheduler | Schedulable GPU memory per device. |
| `hami_gpu_memory_allocated_bytes` | scheduler | GPU memory allocated to pods. |
| `hami_gpu_core_allocated_ratio` | scheduler | Allocated compute cores (0-100). |
| `hami_gpu_shared_count` | scheduler | Containers sharing a device. |
| `hami_node_gpu_memory_allocated_ratio` | scheduler | Per-node memory allocated (0-100). |
| `hami_host_gpu_memory_used_bytes` | vGPU monitor | Real memory in use per device. |
| `hami_host_gpu_utilization_ratio` | vGPU monitor | Physical GPU utilization (0-100). |
| `hami_vgpu_memory_used_bytes` | vGPU monitor | Per-container vGPU memory used. |
| `hami_vgpu_memory_limit_bytes` | vGPU monitor | Per-container vGPU memory limit. |
| `hami_container_device_utilization_ratio` | vGPU monitor | Per-container utilization (0-100). |

> Utilization and allocation-ratio metrics are reported on a 0-100 scale, so the
> percentage panels display them directly without rescaling.
Loading
Loading