feat(deployments): k8s PodSpec compiler for full workload config (AIRCORE-757 phase 5) by tylersbray · Pull Request #553 · NVIDIA-NeMo/nemo-platform

tylersbray · 2026-07-02T19:32:15Z

Summary

Add compiler.py to compile DeploymentConfig + K8sDeploymentConfig into Kubernetes PodSpec objects: init containers, native sidecars (restartPolicy: Always on init, k8s ≥1.29), multi-container workloads, ConfigMap file mounts, tolerations, affinity, security context, and service account.
Refactor jobs.py and deployments.py to use the shared compiler; wire ConfigMap create/delete lifecycle with ownership checks on 409 conflicts and rollback on partial creates.
Add Container.restart_policy for native sidecar init containers; extend validation for duplicate port names and duplicate listen ports across containers.

Test plan

uv run --frozen pytest plugins/nemo-deployments/tests/unit — 251 passed
uv run ruff check plugins/nemo-deployments/
uv run --frozen ty check plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/
Pre-commit hooks pass on commit
Golden/snapshot-style compiler tests in test_compiler.py
Deployment rollback tests (ConfigMap cleanup, adopted-deployment path preserves ConfigMap)

Summary by CodeRabbit

New Features
- Kubernetes deployments and jobs now support multi-container workloads with sidecar-aware pod construction and improved status reporting.
- Config files are packaged into generated ConfigMaps and mounted automatically.
- Added restartPolicy support on container definitions for init containers, exposed in the API schema.
Bug Fixes
- Deployment/job create/delete cleanup is more robust, with better rollback on partial failures.
- Service ports and in-cluster endpoints are derived from all containers with safer duplicate port handling.
- Status checks now better reflect workloads that include init containers and Always restart behavior.

…(AIRCORE-757 phase 5) Extract shared PodSpec compilation for init containers, native sidecars, multi-container deployments, ConfigMap file mounts, and K8sDeploymentConfig. Wire ConfigMap lifecycle with ownership checks and rollback on partial creates. Signed-off-by: Tyler Bray <tbray@nvidia.com>

github-actions · 2026-07-02T19:42:54Z

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	22524/29649	76.0%	60.9%
Integration Tests	13010/28329	45.9%	19.4%

coderabbitai · 2026-07-02T19:45:06Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 49b732f3-234e-45df-8eb7-882c9df36e59

📥 Commits

Reviewing files that changed from the base of the PR and between 150e27f and 0e34d63.

📒 Files selected for processing (4)

plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/compiler.py
plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/deployments.py
plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/jobs.py
plugins/nemo-deployments/tests/unit/backends/k8s/test_compiler.py

🚧 Files skipped from review as they are similar to previous changes (3)

plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/compiler.py
plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/jobs.py
plugins/nemo-deployments/tests/unit/backends/k8s/test_compiler.py

📝 Walkthrough

Walkthrough

Adds shared k8s workload compilation and ConfigMap lifecycle helpers, updates Deployment and Job flows to use compiled multi-container workloads, adds container restart policy schema support, and changes status reads to use all containers.

Changes

K8s workload compiler and multi-container support

Layer / File(s)	Summary
Schema and naming surface `plugins/nemo-deployments/src/nemo_deployments_plugin/entities.py`, `plugins/nemo-deployments/openapi/openapi.yaml`, `plugins/nemo-deployments/src/nemo_deployments_plugin/backends/labels.py`	Adds `Container.restart_policy`, updates the OpenAPI container schemas, and adds a deployment-scoped ConfigMap naming helper.
Compiler core and ConfigMap lifecycle `plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/compiler.py`	Adds workload compilation, validation, container/pod builders, config-file mounts, init-container ordering, and ConfigMap create/delete helpers.
Deployment body, service, endpoints, and cleanup `plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/deployments.py`, `plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/backend.py`	Reworks deployment/service assembly for compiled artifacts and multi-container inputs, and changes status reads to pass all containers onward.
Job creation and deletion flow `plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/jobs.py`	Moves job compilation to the shared compiler and adds ConfigMap lifecycle handling around create/delete conflicts.
Compiler, deployment, and status tests `plugins/nemo-deployments/tests/unit/backends/k8s/*`	Adds compiler coverage and updates deployment/status tests for tuple-based containers, rollback, and foreign-resource cleanup.

Possibly related PRs

NVIDIA-NeMo/nemo-platform#533: Updates the same K8sDeploymentBackend status path used here.
NVIDIA-NeMo/nemo-platform#538: Changes the same restart_policy == "Always" status validation flow.
NVIDIA-NeMo/nemo-platform#549: Modifies the Always-restart Deployment status wiring that this PR further refactors.

Suggested reviewers: mckornfield

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding a shared k8s PodSpec compiler for full workload config.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch 757-k8s-phase5-podspec/tbray

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🧹 Nitpick comments (1)

plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/deployments.py (1)

67-69: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Weak types on build_deployment_body. k8s_config: Any and -> tuple[Any, Any] lose type safety; jobs.py.build_job_body already uses K8sDeploymentConfig | None. Tighten for consistency.

Proposed change

-from nemo_deployments_plugin.entities import Container, DeploymentConfig
+from nemo_deployments_plugin.entities import Container, DeploymentConfig, K8sDeploymentConfig

     workspace: str,
     deployment_name: str,
-    k8s_config: Any,
-) -> tuple[Any, Any]:
+    k8s_config: K8sDeploymentConfig | None,
+) -> tuple[Any, CompiledWorkload]:

CompiledWorkload is exported from compiler; add it to the existing compiler import.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/deployments.py`
around lines 67 - 69, The `build_deployment_body` signature is too weakly typed,
with `k8s_config: Any` and `-> tuple[Any, Any]`, so update it to match the
stronger typing used in `jobs.py.build_job_body`. Change the function in
`build_deployment_body` to accept `K8sDeploymentConfig | None` and return the
appropriate typed tuple, then add the exported `CompiledWorkload` to the
existing compiler import in this module so the stronger annotations resolve
correctly.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/compiler.py`:
- Around line 369-375: The init container build path in compiler.py is dropping
probes for every entry in init_containers because build_container is always
called with include_probes=False. Update the logic around
_ordered_init_containers and build_container so probes are excluded only for
true init containers, while native sidecars with restartPolicy=Always retain
liveness/readiness probes by gating include_probes on the container restart
policy.

---

Nitpick comments:
In
`@plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/deployments.py`:
- Around line 67-69: The `build_deployment_body` signature is too weakly typed,
with `k8s_config: Any` and `-> tuple[Any, Any]`, so update it to match the
stronger typing used in `jobs.py.build_job_body`. Change the function in
`build_deployment_body` to accept `K8sDeploymentConfig | None` and return the
appropriate typed tuple, then add the exported `CompiledWorkload` to the
existing compiler import in this module so the stronger annotations resolve
correctly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3c8229d8-2507-4131-96c2-89cb48a49715

📥 Commits

Reviewing files that changed from the base of the PR and between bd70579 and bf57df4.

📒 Files selected for processing (9)

plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/backend.py
plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/compiler.py
plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/deployments.py
plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/jobs.py
plugins/nemo-deployments/src/nemo_deployments_plugin/backends/labels.py
plugins/nemo-deployments/src/nemo_deployments_plugin/entities.py
plugins/nemo-deployments/tests/unit/backends/k8s/test_compiler.py
plugins/nemo-deployments/tests/unit/backends/k8s/test_deployments.py
plugins/nemo-deployments/tests/unit/backends/k8s/test_k8s_status_mapping.py

Retain liveness/readiness probes on native sidecar init containers, tighten build_deployment_body typing, and regenerate deployments OpenAPI for Container.restart_policy. Signed-off-by: Tyler Bray <tbray@nvidia.com>

mckornfield

have some style things, but I think you can ship it assuming you address em

Use BuiltJob/BuiltDeployment dataclasses instead of tuple returns, SimpleNamespace for k8s deserialization, _reraise_api_unless helper, clearer validation wording, and structural pod_spec golden asserts. Signed-off-by: Tyler Bray <tbray@nvidia.com>

…CORE-757 phase 5) (#553) * feat(deployments): add k8s PodSpec compiler for full workload config (AIRCORE-757 phase 5) Extract shared PodSpec compilation for init containers, native sidecars, multi-container deployments, ConfigMap file mounts, and K8sDeploymentConfig. Wire ConfigMap lifecycle with ownership checks and rollback on partial creates. Signed-off-by: Tyler Bray <tbray@nvidia.com> * fix(deployments): address CodeRabbit review on phase 5 PodSpec compiler Retain liveness/readiness probes on native sidecar init containers, tighten build_deployment_body typing, and regenerate deployments OpenAPI for Container.restart_policy. Signed-off-by: Tyler Bray <tbray@nvidia.com> * fix(deployments): address mckornfield review on phase 5 compiler Use BuiltJob/BuiltDeployment dataclasses instead of tuple returns, SimpleNamespace for k8s deserialization, _reraise_api_unless helper, clearer validation wording, and structural pod_spec golden asserts. Signed-off-by: Tyler Bray <tbray@nvidia.com> --------- Signed-off-by: Tyler Bray <tbray@nvidia.com>

tylersbray requested review from a team as code owners July 2, 2026 19:32

github-actions Bot added the feat label Jul 2, 2026

coderabbitai Bot reviewed Jul 2, 2026

View reviewed changes

Comment thread plugins/nemo-deployments/src/nemo_deployments_plugin/backends/k8s/compiler.py

mckornfield approved these changes Jul 2, 2026

View reviewed changes

tylersbray added this pull request to the merge queue Jul 2, 2026

Merged via the queue into main with commit efe790a Jul 2, 2026
55 checks passed

tylersbray deleted the 757-k8s-phase5-podspec/tbray branch July 2, 2026 21:23

tylersbray mentioned this pull request Jul 6, 2026

feat(deployments): grant k8s backend RBAC in deploy chart (AIRCORE-757 phase 6) #575

Merged

6 tasks

Navigation Menu

Uh oh!

feat(deployments): k8s PodSpec compiler for full workload config (AIRCORE-757 phase 5) - #553

feat(deployments): k8s PodSpec compiler for full workload config (AIRCORE-757 phase 5)#553
tylersbray merged 3 commits into
mainfrom
757-k8s-phase5-podspec/tbray

tylersbray commented Jul 2, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

github-actions Bot commented Jul 2, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 2, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

mckornfield left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

tylersbray commented Jul 2, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Summary by CodeRabbit

Uh oh!

github-actions Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mckornfield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tylersbray commented Jul 2, 2026 •

edited by coderabbitai Bot

Loading

github-actions Bot commented Jul 2, 2026 •

edited

Loading

coderabbitai Bot commented Jul 2, 2026 •

edited

Loading