feat(operator): hash effective runtime profiles

[xianlubird]

Summary

• resolve each component's effective Dynamo runtime compatibility version from runtimeVersionOverride or the main image tag
• evaluate the centralized runtime feature gates into a canonical RuntimeProfile
• include non-empty runtime profiles in the v2 DGD worker hash while continuing to exclude the raw override value
• thread the profile through ComponentContext so rendering and hashing can consume the same gate decisions
• preserve the frozen legacy v1alpha1 hash and omit empty profiles to avoid changing existing pre-gate hashes
• clarify that changing runtimeVersionOverride can alter runtime-gated Pod rendering and trigger a rollout

Why

#10494 intentionally excluded runtimeVersionOverride from worker hashing because it did not affect rendered resources at that time. After #12421 introduced runtime-version feature gates, changing only the override can change the rendered PodSpec.

Hashing the raw override would be too sensitive: 1.4.0 and 2.0.0 currently enable the same gates and should not create different worker generations. This change hashes the effective rendering profile instead:

Change	Effective profile	Worker hash
unknown runtime → 1.3.0	empty → empty	unchanged
1.3.0 → 1.4.0	canary disabled → enabled	changed
1.4.0 → 2.0.0	enabled → enabled	unchanged
image tag 1.5.0 → override 1.5.0	equivalent	unchanged

This ensures changes that affect rendered worker behavior use the managed rolling-update path without introducing rollouts for compatibility-version changes that produce identical rendering.

This is an upstream-focused version of Thomas Montfort's follow-up proposal in xianlubird#2, with the unrelated graph_test.go consolidation left out to keep the review surface focused.

Validation

• make test
• make manifests
• make vet
• go test ./internal/runtimeversion ./internal/features/runtime ./internal/dynamo

make lint was also attempted locally, but the repository-pinned golangci-lint v1.64.8 failed while loading the Go 1.26 module with no go files to analyze; it did not report a source finding.

Related Issues

• Builds on feat(operator): add runtime feature gates #12421
• Supports the runtime-gated canary implementation in feat(operator): enable worker canary health checks #11083

Summary by CodeRabbit

• New Features

  • Runtime compatibility versions now determine applicable runtime feature behavior during deployment rendering.
  • Runtime version overrides take precedence over image-derived versions, including for custom images.
  • Runtime-gated changes, such as canary health checks, are reflected in deployment updates and rollouts.

• Documentation

  • Clarified that changing the runtime compatibility version can affect rendered Pods and trigger a rollout, without changing the container image.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi xianlubird! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[devin-ai-integration]

Devin Review found 2 potential issues.

[coderabbitai]

Walkthrough

The operator now resolves runtime versions into feature profiles, propagates profiles through component contexts, and includes effective profiles in worker hashes. API and CRD documentation now describes runtime-gated Pod rendering and rollout behavior.

Changes

Runtime profile and worker hashing

Layer / File(s)	Summary
Runtime version and profile resolution deploy/operator/internal/runtimeversion/..., deploy/operator/internal/features/runtime/..., deploy/operator/internal/dynamo/runtime_profile*	Runtime versions resolve from overrides or image tags. Resolved versions produce runtime profiles with threshold-based canary health-check settings.
Component context integration deploy/operator/internal/dynamo/component_common.go, deploy/operator/internal/dynamo/graph.go, deploy/operator/internal/dynamo/graph_test.go	Generated component contexts now include resolved runtime profiles. Tests cover override-based canary health checks.
Worker hash integration deploy/operator/internal/dynamo/hash.go, deploy/operator/internal/dynamo/hash_test.go	Non-empty effective runtime profiles now affect worker hashes. Alpha-compatible hashing continues to ignore runtime overrides. Tests cover equivalent profiles, gate changes, and newer versions with unchanged profiles.
Runtime override documentation deploy/operator/api/..., deploy/operator/config/crd/bases/*dynamocomponentdeployments.yaml, deploy/operator/config/crd/bases/*dynamographdeployments.yaml	Documentation states that runtime compatibility changes can affect runtime-gated Pod rendering and trigger rollouts without changing the image.

Estimated code review effort: 4 (Complex) | ~35 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 26.09% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: hashing effective runtime profiles for operator workers.
Description check	✅ Passed	The description covers the change, rationale, validation, and related work, but it does not use all template headings or select an issue-link path.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies"

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/operator/config/crd/bases/nvidia.com_dynamographdeployments.yaml`:
- Around line 11758-11760: Update the runtimeVersionOverride description at
deploy/operator/config/crd/bases/nvidia.com_dynamographdeployments.yaml:11758-11760
and the corresponding description at
deploy/operator/config/crd/bases/nvidia.com_dynamographdeployments.yaml:20882-20884
to permit intentional runtime override changes independently of image changes,
while retaining guidance that such changes may alter runtime-gated Pod rendering
and trigger a rollout.

In `@deploy/operator/internal/dynamo/graph_test.go`:
- Around line 1430-1448: Add t.Log headings in the relevant t.Run test flow
before the generateComponentContext call and before the assertion block,
including the runtime-profile case and the additionally referenced case. Use
concise messages describing each test step while leaving the test logic
unchanged.

In `@deploy/operator/internal/dynamo/hash_test.go`:
- Around line 95-116: The test
TestComputeLegacyAlphaDGDWorkersSpecHash_IgnoresRuntimeProfile should cover a
runtime-profile gate transition: initialize RuntimeVersionOverride with the
pre-gate version 1.3.0, then change it to the gate version 1.4.0 before
recomputing the hash. Preserve the existing equal-hash assertion and test
structure.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7982c13e-6b1e-4cf7-8f2b-da48102493b9

📥 Commits

Reviewing files that changed from the base of the PR and between c45d976 and 373eb71.

📒 Files selected for processing (16)

• deploy/operator/api/v1alpha1/dynamocomponentdeployment_types.go
• deploy/operator/api/v1beta1/dynamocomponentdeployment_types.go
• deploy/operator/config/crd/bases/nvidia.com_dynamocomponentdeployments.yaml
• deploy/operator/config/crd/bases/nvidia.com_dynamographdeployments.yaml
• deploy/operator/internal/dynamo/component_common.go
• deploy/operator/internal/dynamo/graph.go
• deploy/operator/internal/dynamo/graph_test.go
• deploy/operator/internal/dynamo/hash.go
• deploy/operator/internal/dynamo/hash_test.go
• deploy/operator/internal/dynamo/runtime_profile.go
• deploy/operator/internal/dynamo/runtime_profile_test.go
• deploy/operator/internal/features/runtime/gates.go
• deploy/operator/internal/features/runtime/profile.go
• deploy/operator/internal/features/runtime/profile_test.go
• deploy/operator/internal/runtimeversion/runtimeversion.go
• deploy/operator/internal/runtimeversion/runtimeversion_test.go

[xianlubird]

/ok to test 373eb71

[sttts]

This needs thought why it is correct to add the gates to the hash. And further, what it means when gates changes, or gates are introduced in a version, but are false (then the hash should not change?).

And we need a plan for when gates become true, whether we tolerate automatic rollouts, in general or only by opt-in.

[tmonty12]

Closing in favor of #12633