feat(operator): enable worker canary health checks

[xianlubird]

Overview

Enable canary worker health checks by default for Kubernetes operator-managed workers. This brings the operator back in line with the published health check documentation, which states that Kubernetes deployments default DYN_HEALTH_CHECK_ENABLED to true: https://docs.nvidia.com/dynamo/user-guides/observability-local/health-checks#worker-liveness-and-health-check

Details

• Set the worker base container default DYN_HEALTH_CHECK_ENABLED value to true.
• Update Grove, DCD, and controller test expectations for newly rendered worker specs.
• Keep upgrade-test persisted child fixtures at the old false value and explicitly assert that upgrading across this default flip changes the child spec hash, while still preserving legacy worker identity labels.

The worker default was disabled in #5038 as a temporary mitigation for DIS-1185 races involving discovery propagation, NATS subscription readiness, and health check requests. The underlying issues have since been addressed:

• fix: health check race condition by waiting for watch stream discovery #4876 added discovery readiness waiting for canary health checks.
• fix(health): make canary sole authority on endpoint readiness when enabled #8165 made the canary the sole authority for endpoint readiness when enabled, avoiding the eager Ready -> NotReady race that could trigger Kubernetes restarts.
• fix(health): use local endpoint registry for canary health checks #8294 moved canary health checks to the local endpoint registry, removing the fragile discovery / PushRouter / instance-id path for self-checks.

With those fixes in place, Kubernetes workers can now use the documented default and require the canary path to validate readiness.

Where should the reviewer start?

Start with deploy/operator/internal/dynamo/component_worker.go, then review the test expectation updates in deploy/operator/internal/dynamo/graph_test.go, deploy/operator/internal/controller/dynamocomponentdeployment_controller_test.go, and deploy/operator/internal/controller/upgrade_test.go.

Related Issues

• DIS-1185
• Follow-up to fix: turn off DYN_HEALTH_CHECK_ENABLED #5038
• Related fixes: fix: health check race condition by waiting for watch stream discovery #4876, fix(health): make canary sole authority on endpoint readiness when enabled #8165, fix(health): use local endpoint registry for canary health checks #8294

Summary

• Enable DYN_HEALTH_CHECK_ENABLED=true by default for operator-managed worker components in Kubernetes.
• Align the operator behavior with the published Kubernetes health check default.
• Document and test that existing persisted worker children with the old default will roll out when reconciled with the new default.

Validation

• go test ./internal/dynamo ./internal/controller from deploy/operator
• git diff --check

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi xianlubird! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[devin-ai-integration]

Devin Review found 2 potential issues.

[coderabbitai]

Walkthrough

GetBaseContainer in component_worker.go changes DYN_HEALTH_CHECK_ENABLED from "false" to "true". All test fixtures and assertions across controller and graph tests are updated to match the new value.

Health Check Env Var Default

Layer / File(s)	Summary
GetBaseContainer env var change deploy/operator/internal/dynamo/component_worker.go	DYN_HEALTH_CHECK_ENABLED set to "true" instead of "false" in GetBaseContainer.
Test fixture updates deploy/operator/internal/controller/dynamocomponentdeployment_controller_test.go, deploy/operator/internal/controller/upgrade_test.go, deploy/operator/internal/dynamo/graph_test.go	All test expectations and embedded YAML fixtures updated to assert DYN_HEALTH_CHECK_ENABLED as "true" across Deployment, LeaderWorkerSet, PodCliqueSet, SGLang, and VLLM scenarios.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the main change: enabling worker health checks by default.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description includes the required sections and key implementation/validation details, with only minor template-format deviations in Related Issues.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/operator/internal/controller/upgrade_test.go`:
- Line 167: The pre-upgrade fixture data in upgrade_test.go is now modeling the
post-change state, so the hash comparison in the rollout compatibility tests can
no longer detect a default flip. Update the persisted child object fixtures used
by the specHash(oldChild) and specHash(newChild) assertions to keep
DYN_HEALTH_CHECK_ENABLED at the old persisted value, or if the rollout is
intended, split the affected cases in the relevant test helpers so they
explicitly assert a rollout instead of no-change behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 378786f7-a274-48a0-9915-366ec256a624

📥 Commits

Reviewing files that changed from the base of the PR and between 61865e9 and edd7f74.

📒 Files selected for processing (4)

• deploy/operator/internal/controller/dynamocomponentdeployment_controller_test.go
• deploy/operator/internal/controller/upgrade_test.go
• deploy/operator/internal/dynamo/component_worker.go
• deploy/operator/internal/dynamo/graph_test.go

[rmccorm4]

/ok to test 422a8d4

[xianlubird]

@rmccorm4
Updated the PR to avoid enabling DYN_HEALTH_CHECK_ENABLED unconditionally for every worker.

The base worker default now stays conservative, and each backend opts in only for modes that are known to provide a non-nil canary health_check_payload. Specifically, vLLM/TRT-LLM decode workers remain disabled because they do not currently provide a canary payload, and SGLang tokenizer mode remains disabled because its unified backend explicitly returns None for health_check_payload. Regular vLLM, TRT-LLM, and SGLang worker modes still default to canary health checks.

This keeps the Kubernetes default aligned with the docs for supported worker modes while avoiding permanent NotReady endpoints for modes that cannot satisfy the generate canary yet.

[datadog-official]

🔄 Datadog auto-retried 1 job - 1 passed on retry

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 48.95% (-8.97%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 8a61cd2 | Docs | Datadog PR Page | Give us feedback!}

[sttts]

/ok to test f53dc29

[tmonty12]

The base worker default now stays conservative, and each backend opts in only for modes that are known to provide a non-nil canary health_check_payload. Specifically, vLLM/TRT-LLM decode workers remain disabled because they do not currently provide a canary payload, and SGLang tokenizer mode remains disabled because its unified backend explicitly returns None for health_check_payload. Regular vLLM, TRT-LLM, and SGLang worker modes still default to canary health checks.

TRT-LLM decode worker does provide a canary payload:

dynamo/components/src/dynamo/trtllm/health_check.py

Line 67 in b4999c4

self._disaggregation_mode = disaggregation_mode

, as does vllm. Since unified backend is not the default, we are not concerned about the SGLang tokenizer mode.

Adding conditional logic in the operator based on flags/env-vars is fragile and best avoided.

[dynamo-ops]

/ok to test 8a61cd2

[xianlubird]

The vLLM deploy CI failure is unrelated to this PR's code: actions/setup-python failed while upgrading pip due to repeated 502 responses from files.pythonhosted.org, before the deploy test started.

[dynamo-ops]

/ok to test 52da0a0

[dynamo-ops]

/ok to test 0f0b07c

[dynamo-ops]

/ok to test 66b0fce