Skip to content

feat(operator): add runtimeVersionOverride to DGD components - #10494

Merged
sttts merged 6 commits into
mainfrom
tmonty12/dyn-940-runtime-version
Jul 30, 2026
Merged

feat(operator): add runtimeVersionOverride to DGD components#10494
sttts merged 6 commits into
mainfrom
tmonty12/dyn-940-runtime-version

Conversation

@tmonty12

@tmonty12 tmonty12 commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds runtimeVersionOverride as an explicit declaration of Dynamo runtime compatibility when an image tag does not provide it.

The override controls version-dependent operator behavior only. It never rewrites or pins the image reference.

Changes by layer

API and CRDs

  • Adds runtimeVersionOverride to:
    • DGD component specs
    • DCD specs
    • DGDR specs
  • Supports both v1alpha1 and v1beta1, including conversion.
  • Accepts canonical core semver only, for example 1.4.0.
  • Updates generated CRDs, Helm references, and API documentation.
  • Excludes the field from the legacy worker hash, so changing compatibility metadata alone does not roll Pods.

Admission

  • DGD: requires a main image; a non-semver image requires runtimeVersionOverride.
  • DCD: requires a main image but does not require the override because DCDs are controller-generated.
  • DGD/DCD updates: ratchet legacy objects. An unchanged legacy image/version tuple remains valid, while changing an image to a non-semver value requires an override.
  • DGDR: requires the override when spec.image has no parseable semantic-version tag.
  • Keeps DGDR specs immutable after profiling starts, except for the deferred autoApply: false flow:
    • select the runtime override during Profiling or Ready
    • enable auto-apply once the request is Ready

DGDR controller and profiler

  • Propagates the DGDR override into newly generated DGD components.
  • Preserves explicit component overrides when the controller fills missing values.
  • Applies the override defensively immediately before creating a new DGD.
  • Keeps the stored Ready snapshot immutable; it is not repaired or rewritten.
  • Does not mutate an existing deployed DGD through a DGDR update.

Documentation and test support

  • Documents custom-image, digest, and non-semver-tag usage.
  • Updates DGDR/profiler examples and generated references.
  • Adds focused unit, webhook, conversion, controller, and envtest coverage.
  • Updates deployment fixtures to provide deterministic runtime compatibility versions.

Validation

  • go test ./internal/runtimeversion ./api/v1alpha1 -count=1
  • KUBEBUILDER_ASSETS=<envtest assets> go test ./internal/webhook/validation ./internal/controller -count=1
  • make SHELL=/bin/sh manifests
  • make SHELL=/bin/sh generate-api-docs
  • git diff --check

Tracking

@copy-pr-bot

copy-pr-bot Bot commented Jun 9, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added documentation Improvements or additions to documentation deployment::k8s Relates to dynamo deployment in kubernetes labels Jun 9, 2026
@tmonty12 tmonty12 changed the title Add runtimeVersion field with image tag resolution feat(operator): add runtimeVersion field to DGD with image tag resolution Jun 9, 2026
@github-actions github-actions Bot added the feat label Jun 9, 2026
@tmonty12
tmonty12 marked this pull request as ready for review June 9, 2026 19:55
@tmonty12
tmonty12 requested review from a team as code owners June 9, 2026 19:55
@tmonty12
tmonty12 force-pushed the tmonty12/dyn-940-runtime-version branch from 110cef6 to d522934 Compare June 9, 2026 19:57
@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

This PR adds a new optional runtimeVersion field to DynamoComponentDeployment and DynamoGraphDeployment resources. The field tracks runtime compatibility and is automatically derived from container image semantic version tags via webhook defaulting. The implementation includes parsing utilities, validation logic ensuring version consistency with image tags, and comprehensive test coverage across both v1alpha1 and v1beta1 API groups.

Changes

Runtime Version Tracking for Dynamo Components

Layer / File(s) Summary
API schema types and conversion between v1alpha1/v1beta1
deploy/operator/api/v1alpha1/dynamocomponentdeployment_types.go, deploy/operator/api/v1beta1/dynamocomponentdeployment_types.go, deploy/operator/api/v1alpha1/shared_spec_conversion.go, deploy/operator/api/v1alpha1/dynamocomponentdeployment_conversion_test.go, deploy/operator/api/v1alpha1/dynamographdeployment_conversion_test.go, deploy/operator/api/v1alpha1/conversion_field_coverage_test.go
Added RuntimeVersion string field to DynamoComponentDeploymentSharedSpec in both v1alpha1 and v1beta1 with kubebuilder semantic version pattern validation. Bidirectional conversion copies the field between v1alpha1 and v1beta1. Round-trip and field coverage tests verify conversion preservation.
CRD definitions and API documentation
deploy/helm/charts/platform/components/operator/crds/*.yaml, deploy/operator/config/crd/bases/*.yaml, docs/kubernetes/api-reference.md
Updated CRD YAML in both Helm charts and config bases to include runtimeVersion field with regex validation enforcing x.y.z numeric format and description of defaulting behavior. API reference documentation updated across both v1alpha1 and v1beta1 specs.
Runtime version parsing library
deploy/operator/internal/runtimeversion/runtimeversion.go, deploy/operator/internal/runtimeversion/runtimeversion_test.go
New utility package providing Parse() to validate and parse explicit semantic version strings, and ParseImageVersion() to extract and parse semantic version tags from container image references. Helper functions handle tag extraction, version normalization, and prefix stripping. Comprehensive table-driven tests cover valid/invalid inputs and edge cases.
Webhook defaulting from container image tags
deploy/operator/internal/webhook/defaulting/runtimeversion.go, deploy/operator/internal/webhook/defaulting/dynamocomponentdeployment_handler.go, deploy/operator/internal/webhook/defaulting/dynamocomponentdeployment_handler_test.go, deploy/operator/internal/webhook/defaulting/dynamographdeployment_handler.go, deploy/operator/internal/webhook/defaulting/dynamographdeployment_handler_test.go
Defaulting helpers derive RuntimeVersion from main container image tags for v1alpha (alpha) and v1beta (beta) specs, integrating into DCDDefaulter and DGDDefaulter webhook handlers. Handlers log defaulting decisions. Tests verify tag parsing, preservation of explicit values, and handling of unparseable tags.
Webhook validation of runtime version consistency
deploy/operator/internal/webhook/validation/runtimeversion.go, deploy/operator/internal/webhook/validation/dynamocomponentdeployment.go, deploy/operator/internal/webhook/validation/dynamocomponentdeployment_test.go, deploy/operator/internal/webhook/validation/dynamographdeployment.go, deploy/operator/internal/webhook/validation/dynamographdeployment_test.go
Validation logic ensures RuntimeVersion is set when image tags contain parseable semantic versions, and validates consistency between explicit runtimeVersion and image-derived values with detailed error messages. Integrated into component deployment validators. Tests cover valid/invalid scenarios and version mismatch detection.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 3

❌ Failed checks (3 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 14.81% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check ⚠️ Warning The title mentions runtimeVersionOverride for DGD, but the PR adds runtimeVersion across DCD/DGD/DGDR plus webhook and validation logic. Rename it to reflect the actual change, e.g. 'feat(operator): add runtimeVersion field with image-tag defaulting and validation'.
Description check ⚠️ Warning The description is useful, but it does not match the required template and omits the reviewer-start and required related-issues sections. Add the required Overview, Details, Where should reviewer start?, and Related Issues sections, with one completed issue-link path.
✅ Passed checks (2 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (1)
deploy/helm/charts/platform/components/operator/crds/nvidia.com_dynamographdeployments.yaml (1)

20659-20659: 💤 Low value

Minor capitalization inconsistency in field description.

This description starts with lowercase "runtimeVersion" while the first occurrence (line 11674) starts with uppercase "RuntimeVersion". For consistency, consider using the same capitalization in both locations. The lowercase form aligns better with standard Kubernetes field description conventions.

📝 Suggested consistency fix
-                        description: |-
-                          runtimeVersion is the Dynamo runtime version for this component to determine the
+                        description: |-
+                          RuntimeVersion is the Dynamo runtime version for this component to determine the

Or alternatively, update line 11674 to use lowercase for both.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@deploy/helm/charts/platform/components/operator/crds/nvidia.com_dynamographdeployments.yaml`
at line 20659, Update the field description text so the capitalization of the
field name is consistent: change the lowercase "runtimeVersion" in the
description at the second occurrence to match the preferred style (use
"runtimeVersion" in lower camel case) or alternatively change the first
occurrence "RuntimeVersion" to "runtimeVersion"; locate the two descriptions
referencing the Dynamo runtime version (search for "runtimeVersion" and
"RuntimeVersion" in the CRD) and make both descriptions use the same lowercase
"runtimeVersion".
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@deploy/helm/charts/platform/components/operator/crds/nvidia.com_dynamocomponentdeployments.yaml`:
- Around line 11379-11386: The description for the CRD field currently uses
"RuntimeVersion" (PascalCase) but the actual field name is runtimeVersion
(camelCase); update the description text for the runtimeVersion property (the
description block under the runtimeVersion schema) to use "runtimeVersion"
everywhere instead of "RuntimeVersion", and make the same replacement in the
other occurrence noted (the hunk around lines 20231-20238) so both descriptions
consistently reference the camelCase field name.

In `@deploy/operator/config/crd/bases/nvidia.com_dynamographdeployments.yaml`:
- Line 11673: The description for the CRD field incorrectly capitalizes the
field name as "RuntimeVersion"; update the description text to use the exact
field name "runtimeVersion" (lowercase r) so it matches the actual field
`runtimeVersion` in the CRD and maintain consistency across both occurrences
(the hunk at 11673 and the matching hunk at 20658); find the description lines
associated with the `runtimeVersion` field and replace "RuntimeVersion" with
"runtimeVersion" in both places.

In `@deploy/operator/internal/webhook/defaulting/runtimeversion.go`:
- Around line 40-57: Extract the duplicated image-accessor helpers into a shared
package (e.g., internal/webhook/helpers): create exported functions
AlphaMainContainerImage(spec
*nvidiacomv1alpha1.DynamoComponentDeploymentSharedSpec) string and
BetaMainContainerImage(spec
*nvidiacomv1beta1.DynamoComponentDeploymentSharedSpec) string that contain the
current logic from alphaMainContainerImage and betaMainContainerImage, update
the defaulting code in runtimeversion.go to call helpers.AlphaMainContainerImage
/ helpers.BetaMainContainerImage, remove the duplicate implementations from
validation/runtimeversion.go and any other files, and update imports accordingly
so both defaulting and validation use the single shared helper implementation.

In `@deploy/operator/internal/webhook/validation/dynamocomponentdeployment.go`:
- Around line 49-57: The code currently discards warnings from
sharedValidator.Validate when validateAlphaRuntimeVersion fails; change the
error return to preserve those warnings by returning (warnings, err) instead of
(nil, err). Specifically, in the function that calls
sharedValidator.Validate(ctx) and then
validateAlphaRuntimeVersion(&v.deployment.Spec.DynamoComponentDeploymentSharedSpec,
"spec"), update the error path so any error from validateAlphaRuntimeVersion is
returned together with the previously collected warnings.

---

Nitpick comments:
In
`@deploy/helm/charts/platform/components/operator/crds/nvidia.com_dynamographdeployments.yaml`:
- Line 20659: Update the field description text so the capitalization of the
field name is consistent: change the lowercase "runtimeVersion" in the
description at the second occurrence to match the preferred style (use
"runtimeVersion" in lower camel case) or alternatively change the first
occurrence "RuntimeVersion" to "runtimeVersion"; locate the two descriptions
referencing the Dynamo runtime version (search for "runtimeVersion" and
"RuntimeVersion" in the CRD) and make both descriptions use the same lowercase
"runtimeVersion".
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e1acf20f-0e21-474b-8d33-99230b68c364

📥 Commits

Reviewing files that changed from the base of the PR and between 824458c and d522934.

📒 Files selected for processing (23)
  • deploy/helm/charts/platform/components/operator/crds/nvidia.com_dynamocomponentdeployments.yaml
  • deploy/helm/charts/platform/components/operator/crds/nvidia.com_dynamographdeployments.yaml
  • deploy/operator/api/v1alpha1/conversion_field_coverage_test.go
  • deploy/operator/api/v1alpha1/dynamocomponentdeployment_conversion_test.go
  • deploy/operator/api/v1alpha1/dynamocomponentdeployment_types.go
  • deploy/operator/api/v1alpha1/dynamographdeployment_conversion_test.go
  • deploy/operator/api/v1alpha1/shared_spec_conversion.go
  • deploy/operator/api/v1beta1/dynamocomponentdeployment_types.go
  • deploy/operator/config/crd/bases/nvidia.com_dynamocomponentdeployments.yaml
  • deploy/operator/config/crd/bases/nvidia.com_dynamographdeployments.yaml
  • deploy/operator/internal/runtimeversion/runtimeversion.go
  • deploy/operator/internal/runtimeversion/runtimeversion_test.go
  • deploy/operator/internal/webhook/defaulting/dynamocomponentdeployment_handler.go
  • deploy/operator/internal/webhook/defaulting/dynamocomponentdeployment_handler_test.go
  • deploy/operator/internal/webhook/defaulting/dynamographdeployment_handler.go
  • deploy/operator/internal/webhook/defaulting/dynamographdeployment_handler_test.go
  • deploy/operator/internal/webhook/defaulting/runtimeversion.go
  • deploy/operator/internal/webhook/validation/dynamocomponentdeployment.go
  • deploy/operator/internal/webhook/validation/dynamocomponentdeployment_test.go
  • deploy/operator/internal/webhook/validation/dynamographdeployment.go
  • deploy/operator/internal/webhook/validation/dynamographdeployment_test.go
  • deploy/operator/internal/webhook/validation/runtimeversion.go
  • docs/kubernetes/api-reference.md

Comment thread deploy/operator/internal/webhook/defaulting/runtimeversion.go Outdated
Comment thread deploy/operator/internal/webhook/validation/dynamocomponentdeployment.go Outdated
Comment thread deploy/operator/api/v1alpha1/dynamocomponentdeployment_types.go Outdated
Comment thread deploy/operator/internal/webhook/defaulting/dynamographdeployment_handler.go Outdated
Comment thread deploy/operator/internal/dynamo/hash.go
Comment thread deploy/operator/internal/webhook/validation/dynamographdeployment.go Outdated

@nv-tusharma nv-tusharma left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving .github and deploy pytest changes

sttts and others added 3 commits July 30, 2026 00:40
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
Co-authored-by: Thomas Montfort <tjmontfort12@gmail.com>
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
@sttts

sttts commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

/ok to test 8bc8a19

@copy-pr-bot

copy-pr-bot Bot commented Jul 29, 2026

Copy link
Copy Markdown

/ok to test 8bc8a19

@sttts, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

@sttts

sttts commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

/ok to test 8bc8a19

@copy-pr-bot

copy-pr-bot Bot commented Jul 29, 2026

Copy link
Copy Markdown

/ok to test 8bc8a19

@sttts, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

@sttts

sttts commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

/ok to test f7a03f6

@julienmancuso

Copy link
Copy Markdown
Contributor

/ok to test 84e22b6

Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
@sttts

sttts commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

/ok to test a764b96

@julienmancuso

Copy link
Copy Markdown
Contributor

/ok to test b3d1a9c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions backend::sglang Relates to the sglang backend backend::trtllm Relates to the trtllm backend backend::vllm Relates to the vllm backend container deployment::k8s Relates to dynamo deployment in kubernetes documentation Improvements or additions to documentation feat planner size/XXL xpu

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants