Skip to content

feat(deployments): K8s PVC volumes (AIRCORE-757 phase 2) - #536

Merged
tylersbray merged 4 commits into
mainfrom
757-k8s-phase2-volumes/tbray
Jul 1, 2026
Merged

feat(deployments): K8s PVC volumes (AIRCORE-757 phase 2)#536
tylersbray merged 4 commits into
mainfrom
757-k8s-phase2-volumes/tbray

Conversation

@tylersbray

@tylersbray tylersbray commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Implement create_volume / read_volume_status / delete_volume for K8sDeploymentBackend via backends/k8s/volumes.py
  • Map PVC phases: Pending→PENDING, Bound→BOUND, Lost→FAILED
  • Honor K8sVolumeConfig.storageClass and namespace on create; extend ABC + volume reconciler to pass backend_config on read/delete so custom namespaces work
  • Idempotent create on 409 conflict; delete treats 404 as RELEASED

Test plan

  • uv run pytest plugins/nemo-deployments/tests/unit -v (186 passed)
  • New: tests/unit/backends/k8s/test_volumes.py — emission, status mapping, 409 idempotency, namespace on read

Linear

AIRCORE-757 — Phase 2 of 7

Summary by CodeRabbit

  • New Features
    • Added Kubernetes-backed volume lifecycle (create, status, delete) using PVC helpers, including namespace and storage class resolution.
    • Volume operations now accept optional backend configuration for reconciler-driven behavior.
  • Bug Fixes
    • Improved accuracy for existing/conflict PVCs and missing/not-found PVCs, including “not managed” detection.
    • More robust status handling, including reporting PVC deletion progress and safe handling of malformed configuration.
  • Tests
    • Added/expanded unit tests for PVC phase-to-status mapping and create/read/delete flows.
    • Updated unit test fixtures for request-timeout expectations and reconciler forwarding.

…ase 2)

Add volumes.py for create/read/delete PersistentVolumeClaims with phase
mapping (Pending/Bound/Lost). Wire K8sDeploymentBackend volume methods and
pass entity backend_config through volume reconciler read/delete so custom
namespaces resolve correctly.

Signed-off-by: Tyler Bray <tbray@nvidia.com>
@tylersbray
tylersbray requested review from a team as code owners July 1, 2026 17:39
@github-actions github-actions Bot added the feat label Jul 1, 2026
…ating

Reject 409 idempotency when an existing PVC lacks plugin identity labels.
Report DELETING when metadata.deletionTimestamp is set even if phase is Bound.

Signed-off-by: Tyler Bray <tbray@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 430acc15-52a5-4106-9532-5cb428ef360f

📥 Commits

Reviewing files that changed from the base of the PR and between 433c779 and 7a26545.

📒 Files selected for processing (3)
  • plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py
  • plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/volumes.py
  • plugins/nemo-deployments/tests/unit/reconciler/conftest.py
💤 Files with no reviewable changes (1)
  • plugins/nemo-deployments/tests/unit/reconciler/conftest.py
🚧 Files skipped from review as they are similar to previous changes (2)
  • plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py
  • plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/volumes.py

📝 Walkthrough

Walkthrough

Adds optional backend_config to volume status/delete methods, implements Kubernetes PVC-backed volume operations, and threads backend configuration through reconciler calls and tests.

Changes

Volume backend_config wiring and Kubernetes PVC implementation

Layer / File(s) Summary
Backend signatures
plugins/nemo-deployments/src/nemo_deployments_plugin/backends/base.py, plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py
read_volume_status and delete_volume accept optional backend_config parameters.
Kubernetes PVC operations
plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/volumes.py
Resolves backend config, builds PVC manifests, maps PVC phases, and implements async create/read/delete against the Kubernetes API.
Backend delegation and reconciler config
plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/backend.py, plugins/nemo-deployments/src/nemo_deployments_plugin/reconciler/volume_reconciler.py
Kubernetes backend delegates to volume_ops, and the reconciler passes volume.backend_config into read/delete calls.
Tests and fixtures
plugins/nemo-deployments/tests/unit/backends/k8s/conftest.py, plugins/nemo-deployments/tests/unit/backends/k8s/test_volumes.py, plugins/nemo-deployments/tests/unit/reconciler/conftest.py
Mock clients and backend stubs accept the new signatures, and tests cover PVC status mapping, create/read/delete behavior, and malformed config handling.

Possibly related PRs

Suggested labels: fix

Suggested reviewers: benmccown, mckornfield

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: adding Kubernetes PVC volume support for deployments.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 757-k8s-phase2-volumes/tbray

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
plugins/nemo-deployments/tests/unit/backends/k8s/test_volumes.py (1)

58-80: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Redundant client wiring.

mock_k8s_clients.request_timeout = 60 (line 60) is set but the actual clients object passed to create_volume (line 65-67) is a separate MagicMock(spec=KubernetesClients) with its own request_timeout = 60. The mock_k8s_clients.request_timeout assignment is dead here since only core_v1 is shared.

🧹 Suggested cleanup
 async def test_create_volume_conflict_reads_existing_pvc(mock_k8s_clients: MagicMock) -> None:
-    mock_k8s_clients.request_timeout = 60
     existing = MagicMock()
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-deployments/tests/unit/backends/k8s/test_volumes.py` around
lines 58 - 80, Remove the redundant request_timeout setup on mock_k8s_clients in
test_create_volume_conflict_reads_existing_pvc, since create_volume is called
with the separate clients MagicMock that already has request_timeout set. Keep
the shared core_v1 mock wiring for KubernetesClients, and clean up the test
setup so only the object actually passed into volume_ops.create_volume carries
the timeout.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/volumes.py`:
- Around line 76-97: Wrap the calls to resolve_volume_namespace and
resolve_storage_class in create_volume, read_volume_status, and delete_volume
with the same try/except path used for other volume failures so pydantic
ValidationError from K8sVolumeConfig.model_validate is converted into a
VolumeStatusUpdate(status="FAILED", ...). In create_volume, read_volume_status,
and delete_volume, ensure malformed backend_config["k8s"] never escapes
uncaught, and consolidate the two resolver calls per function so the K8s section
is validated only once.

---

Nitpick comments:
In `@plugins/nemo-deployments/tests/unit/backends/k8s/test_volumes.py`:
- Around line 58-80: Remove the redundant request_timeout setup on
mock_k8s_clients in test_create_volume_conflict_reads_existing_pvc, since
create_volume is called with the separate clients MagicMock that already has
request_timeout set. Keep the shared core_v1 mock wiring for KubernetesClients,
and clean up the test setup so only the object actually passed into
volume_ops.create_volume carries the timeout.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7b11558d-7b72-4fdb-919f-a5096b947e33

📥 Commits

Reviewing files that changed from the base of the PR and between 99604c5 and db696ac.

📒 Files selected for processing (8)
  • plugins/nemo-deployments/src/nemo_deployments_plugin/backends/base.py
  • plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py
  • plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/backend.py
  • plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/volumes.py
  • plugins/nemo-deployments/src/nemo_deployments_plugin/reconciler/volume_reconciler.py
  • plugins/nemo-deployments/tests/unit/backends/k8s/conftest.py
  • plugins/nemo-deployments/tests/unit/backends/k8s/test_volumes.py
  • plugins/nemo-deployments/tests/unit/reconciler/conftest.py

Comment thread plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/volumes.py Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
plugins/nemo-deployments/tests/unit/backends/k8s/test_volumes.py (1)

118-128: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Redundant dict construction.

{MANAGED_BY_KEY: labels[MANAGED_BY_KEY], **labels}labels already contains MANAGED_BY_KEY; the explicit key is redundant.

Simplify
-    pvc.metadata.labels = {MANAGED_BY_KEY: labels[MANAGED_BY_KEY], **labels}
+    pvc.metadata.labels = labels
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-deployments/tests/unit/backends/k8s/test_volumes.py` around
lines 118 - 128, The test setup in
test_status_from_pvc_deleting_reports_deleting builds pvc.metadata.labels with a
redundant MANAGED_BY_KEY entry even though volume_identity_labels already
includes it. Simplify the fixture by assigning labels directly (or by
constructing only the needed overrides) so the test remains clear and avoids
duplicated dict keys in the status_from_pvc setup.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@plugins/nemo-deployments/tests/unit/backends/k8s/test_volumes.py`:
- Around line 118-128: The test setup in
test_status_from_pvc_deleting_reports_deleting builds pvc.metadata.labels with a
redundant MANAGED_BY_KEY entry even though volume_identity_labels already
includes it. Simplify the fixture by assigning labels directly (or by
constructing only the needed overrides) so the test remains clear and avoids
duplicated dict keys in the status_from_pvc setup.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0ea16cf3-f65c-49b9-9a77-aac59a65721f

📥 Commits

Reviewing files that changed from the base of the PR and between db696ac and d505737.

📒 Files selected for processing (2)
  • plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/volumes.py
  • plugins/nemo-deployments/tests/unit/backends/k8s/test_volumes.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/volumes.py

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor
Suite Lines Covered Line Rate Branch Rate
Unit Tests 22436/29615 75.8% 60.7%
Integration Tests 12997/28295 45.9% 19.4%

Consolidate K8sVolumeConfig validation into resolve_volume_config and
run it inside the try blocks in create/read/delete so pydantic
ValidationError surfaces as VolumeStatusUpdate instead of crashing the
reconciler. Address CodeRabbit test nitpicks.

Signed-off-by: Tyler Bray <tbray@nvidia.com>
Comment thread plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py Outdated
Comment thread plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/volumes.py Outdated
Drop del backend_config stubs in favor of a clarifying comment on Docker,
and rewrite resolve_volume_namespace with a positive and guard.

Signed-off-by: Tyler Bray <tbray@nvidia.com>
@tylersbray
tylersbray added this pull request to the merge queue Jul 1, 2026
Merged via the queue into main with commit 87845e0 Jul 1, 2026
54 checks passed
@tylersbray
tylersbray deleted the 757-k8s-phase2-volumes/tbray branch July 1, 2026 21:00
arpitsardhana pushed a commit that referenced this pull request Jul 9, 2026
* feat(deployments): implement K8s PVC volume lifecycle (AIRCORE-757 phase 2)

Add volumes.py for create/read/delete PersistentVolumeClaims with phase
mapping (Pending/Bound/Lost). Wire K8sDeploymentBackend volume methods and
pass entity backend_config through volume reconciler read/delete so custom
namespaces resolve correctly.

Signed-off-by: Tyler Bray <tbray@nvidia.com>

* fix(deployments): verify PVC identity labels on 409 and handle terminating

Reject 409 idempotency when an existing PVC lacks plugin identity labels.
Report DELETING when metadata.deletionTimestamp is set even if phase is Bound.

Signed-off-by: Tyler Bray <tbray@nvidia.com>

* fix(deployments): catch malformed k8s volume backend_config as FAILED

Consolidate K8sVolumeConfig validation into resolve_volume_config and
run it inside the try blocks in create/read/delete so pydantic
ValidationError surfaces as VolumeStatusUpdate instead of crashing the
reconciler. Address CodeRabbit test nitpicks.

Signed-off-by: Tyler Bray <tbray@nvidia.com>

* fix(deployments): address mckornfield review on volume backend_config

Drop del backend_config stubs in favor of a clarifying comment on Docker,
and rewrite resolve_volume_namespace with a positive and guard.

Signed-off-by: Tyler Bray <tbray@nvidia.com>

---------

Signed-off-by: Tyler Bray <tbray@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants