Skip to content

WIP feat(cpu-power): Rust ACPI/DCGM exporter with Python fallback for AIPerf server-metrics - #381

Closed
ajcasagrande wants to merge 8 commits into
NVIDIA:kylliang/power_study_20260901from
ajcasagrande:ajc/cpu-power-v2
Closed

WIP feat(cpu-power): Rust ACPI/DCGM exporter with Python fallback for AIPerf server-metrics#381
ajcasagrande wants to merge 8 commits into
NVIDIA:kylliang/power_study_20260901from
ajcasagrande:ajc/cpu-power-v2

Conversation

@ajcasagrande

@ajcasagrande ajcasagrande commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Grafts the Rust cpu-power-exporter binary onto Kyle's CPU power branch, with the original Python exporter retained as a fallback.

  • Rust exporter (primary): aarch64-native, reads ACPI hwmon + DCGM CPU power via FFI, background poller, port 9405
  • Python exporter (fallback): stdlib-only, used when the Rust binary is absent or not executable
  • Kyle's Python ACPI/DCGM readers (cpu_power.py, cpu_power_session.py, gpu_power_limit.py) are preserved — they own samples.csv / manifest.json artifact collection
  • Wiring in telemetry_stage.py tries the Rust binary first, falls back to Python module
  • Port 9405 throughout (default in both Rust and Python)

Test plan

  • Rust binary launches and serves /metrics on aarch64 GB200 nodes
  • DCGM auto-fallback to ACPI when no DCGM daemon running
  • Python fallback launches when binary absent
  • AIPERF_SERVER_METRICS_URLS populated and scraped during benchmark
  • Kyle's samples.csv / manifest.json artifacts still written correctly
  • test_cpu_power.py and test_gpu_power_limit.py pass

🤖 Generated with Claude Code

ajcasagrande and others added 6 commits September 3, 2026 12:32
@ajcasagrande @claude
feat(cpu-power): port Rust cpu-power-exporter crate onto Kyle's branch
f3c933c 
Ships the statically-linked Rust binary (aarch64/x86_64 musl) that reads
/sys/class/hwmon hwmon*/power*_average and serves cpu_power_acpi_watts
on :9405/metrics for AIPerf --server-metrics scraping.

Covers:
- src/cpu-power-exporter/ Rust crate (tokio, tracing, clap, anyhow)
- Workspace member added to Cargo.toml
- docker/Dockerfile.cpu-power-exporter (musl static, two-arch)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
@ajcasagrande @claude
feat(cpu-power): add two-arch CI matrix and make targets for cpu-powe…
e8c0c5c 
…r-exporter

- Makefile: cpu-power-exporter / cpu-power-exporter-download targets,
  CPU_POWER_EXPORTER_RELEASE var, setup now depends on both downloads
- release.yaml: unified change detection (single step, two outputs),
  build-cpu-power-exporter job (linux/amd64 + linux/arm64),
  graceful carry-forward for both binaries from the previous release,
  nullglob-safe gh release create

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
@ajcasagrande @claude
feat(cpu-power): wire Rust exporter into telemetry launch and AIPerf …
3c5ad3e 
…metrics

- telemetry_stage: add _resolve_bundled_binary (generalized from
  _resolve_tachometer_binary), add _start_cpu_power_prometheus_exporters
  which launches bin/cpu-power-exporter via srun on each worker node,
  call it from start_cpu_power_telemetry when prometheus_port > 0,
  fix dcgm_exporter guard (return None instead of raise when None)
- benchmark_stage: inject cpu-power-exporter node URLs into
  AIPERF_SERVER_METRICS_URLS when cpu_power.prometheus_port > 0
- submit.py: validate_setup checks bin/cpu-power-exporter alongside tachometer
- schema.py: add CpuPowerConfig.prometheus_port (default 9405),
  gate _validate_dcgm_power on dcgm_exporter is not None

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
@ajcasagrande @claude
chore(cpu-power): add test recipe for lyris GB200 cpu-power-exporter e2e
2d97915 
Uses trace-replay benchmark type so AIPerf actually consumes the
AIPERF_SERVER_METRICS_URLS endpoints. Qwen3.5-27B BF16 to avoid the
8-minute FP4 JIT autotuning pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
@ajcasagrande @claude
feat(telemetry): add stdlib Python Prometheus exporter for Grace CPU …
1a5af22 
…power

Ports cpu_power_exporter.py from kylliang/power_study_20260901 (commit 2a13cb1).
Default port updated to 9405 (consistent with the rest of the cpu-power stack).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
@ajcasagrande @claude
feat(telemetry): fall back to Python exporter when Rust cpu-power-exp…
56a7165 
…orter is absent

If the bundled Rust binary is not present or not executable (e.g. x86_64 nodes
or a checkout without a pre-built bin/), _start_cpu_power_prometheus_exporters
now falls back to `python3 -m srtctl.core.cpu_power_exporter` so Prometheus
scraping still works everywhere.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
@ajcasagrande ajcasagrande mentioned this pull request Sep 3, 2026
Closed
4 tasks
ajcasagrande and others added 2 commits September 3, 2026 13:08
@ajcasagrande
feat(cpu-power-v2): port schema validation, IPv6 url_host, validate_s…
e11d928 
…etup, and tests from ajc/rust-cpu-exporter

- Add url_host() to ip_utils for IPv6 URL bracket safety
- Use url_host() in benchmark_stage.py CPU power URL construction
- Add arch_from_binary() to dynamo_wheels.py
- Rewrite CpuPowerConfig: drop 'dcgm' source, add storage_subdir, acpi_mandatory
- Add _dynamo_system_ports(), _validate_collector_budget(), _validate_cpu_power(),
  _reject_inert_cpu_power_demand() to schema validation
- Rewrite _validate_telemetry() to support CPU-only, DCGM-only, or combined modes
- Fix _validate_observability() to not conflict CPU-only telemetry with Tachometer DCGM
- Improve telemetry_stage.py: drop permitted_device_keys, use cpu_power.storage_subdir
- Update Makefile: version marker, arch check, warn-on-failure for latest release
- Add cpu-power-exporter-setup target; switch setup to use it
- Update validate_setup() to conditionally require cpu-power-exporter
- Add _cpu_power_exporter_problem() with executability and arch mismatch detection
- Port IPv6 tachometer tests and cpu-only telemetry stage test from rust-cpu-exporter
- Port validate_setup tests: disabled/enabled/arch-mismatch/non-executable cases

Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
@ajcasagrande @claude
ci: add cpu-power-exporter.yaml workflow for PR and push validation
7ec1645 
Mirrors tachometer.yaml: runs cargo fmt/test on source changes, and
a two-arch Docker cross-compile to validate the release Dockerfile.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
@ajcasagrande
ajcasagrande marked this pull request as draft September 3, 2026 21:01
@ajcasagrande ajcasagrande changed the title feat(cpu-power): Rust ACPI/DCGM exporter with Python fallback for AIPerf server-metrics WIP feat(cpu-power): Rust ACPI/DCGM exporter with Python fallback for AIPerf server-metrics Sep 3, 2026
@ishandhanani ishandhanani mentioned this pull request Sep 6, 2026
Closed
@FrankD412

FrankD412 commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

@ajcasagrande -- I ended up cherry-picking these into #410 and it just merged. I'll close this as we can pick up work on main to improve/fix bugs.

@FrankD412 FrankD412 closed this Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alec-flowers alec-flowers Awaiting requested review from alec-flowers
@ishandhanani ishandhanani Awaiting requested review from ishandhanani
@nlevin-ui nlevin-ui Awaiting requested review from nlevin-ui
@csahithi csahithi Awaiting requested review from csahithi

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ajcasagrande @FrankD412