feat(power-agent): per-node power-cap enforcement DaemonSet

[kaim-eng]

Part of the PR #9369 split plan.
This is PR 1 of 6 (PR 1a - Power Agent).

Predecessor: none (base = main)
Successor: #9683 (stacked on pr1a/power-agent; #9682 must land first)

Summary

Standalone Power Agent DaemonSet for per-node GPU power-cap enforcement, packaged as a Helm chart. The PR is reviewable in isolation from planner logic and has zero import dynamo.planner.* coupling.

Current branch shape: 26 files changed: net-new components/power_agent/ code and deploy/helm/charts/power-agent/ chart files, plus CI plumbing edits to 5 existing .github/ files.

• components/power_agent/power_agent.py (566 LOC): NVML clamp helper, cgroup parser, multi-pod policy, SIGTERM restore, fail-safe reconcile (_list_pods_on_node -> Optional[list]), and UUID-gated orphan-cap restoration.
• components/power_agent/Dockerfile: slim Python runtime; libnvidia-ml.so is injected at runtime by runtimeClassName: nvidia.
• components/power_agent/tests/: 47 unit tests covering apply_cap, cgroup parsing, multi-pod policy, reconcile fail-safe, and shutdown.
• deploy/helm/charts/power-agent/: Helm chart with DaemonSet, dev pod, ServiceAccount, and RBAC. RBAC renders namespaces through {{ .Release.Namespace }} and supports cluster-scoped or namespace-restricted mode.
• .github/: power_agent path filter plus dedicated power-agent image build/push wiring in PR, post-merge, and release workflows.

Reviewer context:

• Design context: docs/design-docs/powerplanner-design.md section 7 (lands later in the stack; readable from pr5/docs-devenv).
• Planner contract: the only planner-facing coupling is the dynamo.nvidia.com/gpu-power-limit pod annotation.
• Split-plan context: sections 2.1 and 2.1.2.

Validation

• Required/current GitHub checks pass at the current tip; rust jobs and team-request are intentionally skipped by filters.
• components/power_agent/tests/: 47 unit tests in-tree. The last full dev-pod run before the reconcile fail-safe additions was 43 passed on 2026-05-18; CI is green at the current tip.
• Pre-commit on origin/main..pr1a/power-agent: 14 hooks pass, 4 skipped. pytest-marker-report was reconfirmed in a Linux pod with Missing sets: 0. The Windows host has the known pre-existing POSIX-only fcntl import issue in untouched test utilities; Linux CI is unaffected.
• DCO: commits are signed off.

Merge Strategy

Rebase-and-merge, no squash, preserving the single PR commit in main history per split-plan section 4.3.

Stack Order

Merge #9682 first. #9683 and #9790 are both stacked on pr1a/power-agent and become actionable after #9682 lands.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Walkthrough

This PR introduces a complete Kubernetes DaemonSet agent that enforces per-GPU NVML power caps on worker nodes. The daemon reconciles every 15 seconds, mapping running GPU processes to pod UIDs via cgroup parsing, reading power-limit annotations, resolving multi-pod conflicts, and applying caps via NVML with Prometheus metrics and graceful shutdown support.

Changes

Power Agent: GPU NVML Power Cap Enforcement DaemonSet

Layer / File(s)	Summary
Module foundation and utilities components/power_agent/power_agent.py (lines 1–213)	Module header, optional lazy imports for NVML/Kubernetes/Prometheus with fallbacks, logging, constants, cgroup parsing regex for extracting pod UIDs from /proc/{pid}/cgroup, atomic JSON persistence for managed GPU state, UUID normalization for pynvml version compatibility, and Prometheus metrics infrastructure with noop fallback when unavailable.
NVML cap application and multi-pod policy components/power_agent/power_agent.py (lines 220–404), components/power_agent/tests/test_apply_cap.py, components/power_agent/tests/test_multi_pod_policy.py	NVML helper functions to clamp watts to device min/max constraints and apply power caps with logging and metric updates; multi-pod-per-GPU resolution policy that applies an agreed cap when all pods match, otherwise applies safe default on conflicts or missing/invalid annotations, incrementing error counters; unit tests validating clamping boundaries, NVML error tolerance, UUID handling (bytes vs str), and all policy scenarios (single pod, agreement, conflict, invalid annotations, mixed None/valid).
Shutdown and startup recovery components/power_agent/power_agent.py (lines 280–337), components/power_agent/tests/test_shutdown.py	SIGTERM/SIGINT handler restores managed GPU power limits to NVML defaults and signals main loop exit; startup orphan recovery loads persisted GPU UUIDs, checks idle status, restores default caps for idle GPUs, and updates persisted state; unit tests validating per-GPU restoration, graceful shutdown with no managed GPUs, and NVML error handling.
PowerAgent orchestration components/power_agent/power_agent.py (lines 411–577)	PowerAgent class initializes NVML/Kubernetes clients, loads orphan state, lists node pods, maps running PIDs to pod UIDs, builds UID-to-annotation mappings, resolves GPU cap via policy, applies caps via NVML, and persists managed state; blocking reconcile loop runs every 15 seconds; CLI main() entry point accepts --safe-default-watts, optional node/namespace scope, and Prometheus port, then starts daemon.
Cgroup parser validation components/power_agent/tests/test_cgroup_parser.py	Unit tests for cgroup UID extraction across cgroup v1/v2 with systemd and cgroupfs layouts, QoS classes (Guaranteed/Burstable/BestEffort), and non-Kubernetes inputs; validates OSError handling and first-match-wins behavior when multiple cgroup lines exist.
Kubernetes deployment deploy/power_agent/rbac.yaml, deploy/power_agent/daemonset.yaml, deploy/power_agent/dev-pod.yaml	RBAC ServiceAccount, ClusterRole with pod read-access, and ClusterRoleBinding; production DaemonSet runs on GPU-labeled nodes with hostPID, privileged container, nvidia runtime, environment variables for safe default (500 W) and node name, resource limits, and volume mounts for /proc (read-only) and persisted state at /var/lib/dynamo-power-agent; development Pod harness for manual testing and validation on live clusters.
Documentation and CI integration components/power_agent/README.md, .github/filters.yaml	README documents Power Agent's 15-second reconciliation, cgroup UID extraction, annotation reading, NVML power cap enforcement, troubleshooting steps (annotations, logs), Prometheus metrics (applied limit, conflict/safe-default/failure counters), and shutdown/startup semantics; CI filter update triggers planner jobs on power_agent component and deployment changes.

---

🎯 4 (Complex) | ⏱️ ~60 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 27.38% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: introduction of a per-node power-cap enforcement DaemonSet for the Power Agent.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	PR description is comprehensive with clear overview, detailed summary of changes, specific file guidance, related issues reference, and explicit stack ordering context.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 5

🧹 Nitpick comments (1)

components/power_agent/power_agent.py (1)

537-539: ⚡ Quick win

Move argparse import to module scope.

Line 538 imports inside main(), which violates the repo’s Python import placement rule.

Proposed fix

+import argparse
 import json
 import logging
@@
 def main() -> None:
-    import argparse
-
     parser = argparse.ArgumentParser(description="Dynamo Power Agent DaemonSet")

As per coding guidelines: "Keep all imports at the top of each file (flag any import inside functions/classes)."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/power_agent/power_agent.py` around lines 537 - 539, The argparse
import is inside the main() function; move the import to module scope by adding
"import argparse" at the top of the file and removing the in-function import in
main(), ensuring any type or usage remains correct (adjust other imports if
needed) so linting and the repo rule about top-level imports are satisfied.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/power_agent/power_agent.py`:
- Around line 119-124: The function _load_previously_managed_gpus should
defensively handle malformed or unreadable state files: when reading
_MANAGED_STATE_PATH, catch OSError in addition to
FileNotFoundError/JSONDecodeError, load the JSON into a variable (e.g., data =
json.load(f)), verify isinstance(data, dict) before calling
data.get("managed_uuids", []), and if the root is not a dict (or managed_uuids
is not an iterable of strings) return an empty set; ensure any
TypeError/ValueError from unexpected types is handled and results in returning
set() rather than letting startup crash.
- Around line 294-297: Replace the broad except that swallows errors from the
pynvml.nvmlShutdown() call with a targeted handler: catch pynvml.NVMLError (or
the specific pynvml error class) as e, log the exception with the module/class
logger (e.g., logger.exception("NVML shutdown failed: %s", e) or
logging.exception(...)) and then re-raise the error to avoid silent failures;
modify the try/except around pynvml.nvmlShutdown() accordingly so only
NVML-specific errors are caught, logged, and propagated.
- Around line 494-501: The loop that builds pod_annotations appends one entry
per GPU process, causing pods with multiple processes to be counted multiple
times; change the logic in the procs loop (using _extract_pod_uid_from_cgroup
and uid_to_annotation) to deduplicate by pod UID before applying multi-pod
policy — e.g., track seen UIDs (or build a uid->annotation map) and only append
one (uid, annotation) pair per unique UID so each pod is counted once when
evaluating multi-pod warnings/metrics.

In `@deploy/power_agent/daemonset.yaml`:
- Around line 16-35: The DaemonSet metadata.namespace is hardcoded to "default",
which causes mismatches with the RBAC/service account namespace; update the
manifest to use the parameterized namespace variable (e.g.
${POWER_AGENT_NAMESPACE}) instead of "default" and ensure the ServiceAccount
referenced by spec.serviceAccountName ("dynamo-power-agent") is created/bound in
that same parameterized namespace so RBAC bindings resolve correctly; locate
metadata.name ("dynamo-power-agent"), metadata.namespace, and
spec.serviceAccountName in the template to make the change.
- Around line 55-58: Replace the mutable image tag
"nvcr.io/nvidia/dynamo/power-agent:latest" with an immutable reference (a
specific version tag or an image digest), e.g.
"nvcr.io/nvidia/dynamo/power-agent:vX.Y.Z" or
"nvcr.io/nvidia/dynamo/power-agent@sha256:<digest>", so deployments are
reproducible; keep or verify imagePullPolicy (e.g. IfNotPresent) is appropriate
for pinned images and update the inline comment to note the image is
intentionally pinned.

---

Nitpick comments:
In `@components/power_agent/power_agent.py`:
- Around line 537-539: The argparse import is inside the main() function; move
the import to module scope by adding "import argparse" at the top of the file
and removing the in-function import in main(), ensuring any type or usage
remains correct (adjust other imports if needed) so linting and the repo rule
about top-level imports are satisfied.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: f9faddf8-30e0-4a7b-9055-b45244b2a979

📥 Commits

Reviewing files that changed from the base of the PR and between f20ff4e and 3cde449.

📒 Files selected for processing (11)

• .github/filters.yaml
• components/power_agent/README.md
• components/power_agent/power_agent.py
• components/power_agent/tests/__init__.py
• components/power_agent/tests/test_apply_cap.py
• components/power_agent/tests/test_cgroup_parser.py
• components/power_agent/tests/test_multi_pod_policy.py
• components/power_agent/tests/test_shutdown.py
• deploy/power_agent/daemonset.yaml
• deploy/power_agent/dev-pod.yaml
• deploy/power_agent/rbac.yaml

[kaim-eng]

@nv-anants thanks for the review — all addressed in a65126fb88.

#	Comment	Resolution
1	Dockerfile: switch to python 3.12	Base is now python:3.12-slim-bookworm. This also fixed the Power Agent build failure in the /ok to test run: the ECR Docker Hub mirror returned 403 Forbidden on python:3.11-slim-bookworm — it only serves 3.12 (matches every other python Dockerfile in the repo), so 3.11 literally couldn't build here.
2	add both test jobs to post-merge	power-agent-test + power-agent-helm-tests are now in post-merge-ci.yml too, so post-merge runs the same gates as PR CI.
3	avoid release.yml changes for now	Reverted entirely — release.yml now has a zero diff vs main. Happy to start the public-release-artifact process separately with @dagil-nvidia when the time comes.
4	run the tests inside the built image	Done. The Dockerfile gained a test stage (FROM runtime + pytest) whose RUN python -m pytest tests -v validates the exact shipped layers (same Python 3.12 + baked pynvml/kubernetes/prometheus-client). The power-agent-test job is now just docker build --target test …; the build/push job passes --target runtime so only the slim stage is published. This also fixed the Power Agent Tests failure (the old runner-env approach hit ModuleNotFoundError: No module named 'pydantic' from the monorepo-root filterwarnings=error config — the in-image stage + a standalone deploy/power-agent/pytest.ini rootdir avoid that).
6	name both test jobs Power Agent	Both test jobs are now name: Power Agent, so all three group under one umbrella in the checks UI.

(#5 was the /ok to test trigger — thanks. That run was on 76169f1d which predates these fixes; this push should clear the two red Power Agent jobs. Mind re-running /ok to test on a65126fb88 when you get a chance?)

[nv-anants]

/ok to test a65126f

[nv-anants]

/ok to test a12c61d

[nv-anants]

thanks for all the changes @kaim-eng . I am still seeing some issues in CI, I have added a PR here to fix them - #10929

Feel free to merge it in here or make similar changes in your PR.

[kaim-eng]

Thanks @nv-anants — merged your #10929 commits straight onto pr1a/power-agent (fast-forward, so authorship is preserved): c735aeff12, 3743a2ad4d, 89ced1eeb8, now on top of my library/python fix.

I see what was actually breaking CI now: the standalone docker build --target test step pulled library/python from Docker Hub directly (no DOCKER_PROXY), so it never went through the ECR mirror — your change routes the test build through ${ECR_HOSTNAME}/dockerhub/ on the same builder/cache, which is the real fix. Collapsing the three same-named Power Agent jobs into one sequential job (build → in-image pytest → helm lint/render) and dropping the unused marker registrations all make sense too.

Validated locally before pushing:

• docker build --target test → 57 passed in-image (marker removal is clean under --strict-markers).
• helm lint + helm template on the power-agent chart → green.

Since the commits are now on this branch, #10929 should auto-close on the next sync — feel free to close it if not.

[nv-anants]

/ok to test 2da2530

[kaim-eng]

@nv-anants CI is fully green now — should be good to approve/merge.

After merging your #10929 commits I traced the last remaining red, which #10929 didn't cover. The repo-wide dynamo-runtime pytest run (rootdir = repo root) was hitting 6 collection errors:

ERROR collecting deploy/power-agent/tests/test_annotation_scope.py
  ModuleNotFoundError: No module named 'tests.test_annotation_scope'
  (+ test_apply_cap, test_cgroup_parser, test_multi_pod_policy,
     test_reconcile_failsafe, test_shutdown)

Root cause: deploy/power-agent/tests/__init__.py makes that dir a package named tests, which collides with the repo-root tests/ package during whole-repo collection — so import tests.test_annotation_scope resolves into the root package and raises ModuleNotFoundError. That failed the dynamo-runtime test jobs and the dynamo-status-check / backend-status-check aggregators.

Fix (commit 2da2530272): added --ignore-glob=deploy/power-agent/tests/* to the existing addopts ignore list in pyproject.toml — same pattern already used there to "avoid duplicate-module collection errors". The power-agent suite still runs in full via the Dockerfile test stage in the Power Agent job (57 tests), so no coverage is lost; this only stops the repo-wide run from double-collecting it.

Validation:

• Reproduced the collision + confirmed the ignore resolves it in a minimal repro, and against the real pyproject.toml config.
• Full pipeline now: 90 success / 12 skipped / 0 failing. The previously-red dynamo-runtime / test / parallel|gpu|sequential + both status-check aggregators are green (1829 passed on the runtime suite — it now runs instead of crashing at collection).
• Two unrelated flakes blipped mid-run (test_indexers_sync[file] router event-count race on arm64, and a vLLM DynamoCheckpoint deploy test) and both passed on auto-retry — neither touches power-agent.

Only thing left is a maintainer approval (branch shows REVIEW_REQUIRED). Thanks for the help getting the CI structure sorted.

[nv-anants]

/ok to test 89f58ef