refactor(operator): separate managed Grove components

[sttts]

Summary

• Keep every DynamoGraphDeployment value complete throughout the Grove workload program.
• Define the workload ownership boundary once in groveProgram.Reconcile through the optional groveReconcileRequest.IsDelegated predicate.
• Let Grove reconcilers derive managed and delegated components through ManagedComponents() and DelegatedComponents().
• Pass the complete DGD plus the predicate into the Grove renderer, naming, readiness, and GPU-shape functions instead of passing a second component collection.
• Copy and transform only the component currently being rendered; do not expose a filtered or transformed graph representation.
• Keep the shared DCD, worker-hash, rollout, and scaling-adapter APIs unchanged.
• Document the complete-DGD and single-classification invariants in deploy/operator/internal/AGENTS.md.

Context

PR #15062 adds LPX as a delegated component path inside the Grove workload program. Its initial factoring projected non-LPX components into a second, filtered DGD and passed both the source and projected DGDs through the Grove reconcilers.

This stacked PR keeps the API object and its ownership boundary explicit:

workloadProgramRequest.DGD (complete API object)
                    |
                    v
             groveProgram.Reconcile
                    |
                    +-- sets IsDelegated(component) once
                    |
                    v
             groveReconcileRequest
             ├── DGD                    complete graph
             ├── IsDelegated            optional predicate
             ├── ManagedComponents()    Grove-owned selection
             └── DelegatedComponents()  controller-delegated selection
                    |
                    v
              Grove reconcilers
                    |
                    +-- controller code receives req
                    |
                    `-- internal/dynamo receives DGD + IsDelegated
                        and filters while iterating the complete graph

Nested Grove workload reconcilers receive the request and do not reclassify LPX ownership. The PCS renderer deep-copies one selected component immediately before applying legacy compatibility and worker-hash labels. The temporary copy stays inside that render iteration. Stable-resource selectors use the synchronized PCS labels, while GPU-shape resolution and readiness use the same DGD and predicate as rendering.

Semantic behavior

• The Grove PCS contains only components owned by the Grove path.
• LPX components remain outside that PCS and continue through the LPX handoff and status path.
• PCS naming, Grove child naming, restart observation, topology projection, Grove scaling, rendering, GPU shapes, and readiness derive the managed selection from groveReconcileRequest or its DGD plus predicate.
• Legacy worker selectors and worker-hash labels remain aligned between the PCS and stable Services without carrying a parallel component collection.
• Graph-wide model discovery and DGD scaling-adapter reconciliation consume the complete DGD. Admission prohibits scaling adapters on LPX components.
• Shared DCD generation, worker hashing, and rollout retain their existing APIs and behavior.
• A request without IsDelegated treats every component as managed.

There is no API, CRD, migration, or rollout-policy change. This PR is stacked on #15062 and must be rebased or retargeted when that PR changes or merges.

Relates to #14473.

Validation

• CGO_ENABLED=0 go test ./internal/dynamo -count=1
• CGO_ENABLED=0 go test ./internal/controller/lpx -count=1
• Targeted Grove selection, LPX handoff, restart, stable-resource, rendering, compatibility, readiness, worker-hash, and upgrade tests in ./internal/controller
• Targeted LPX admission validation test in ./internal/webhook/validation
• Compile checks for ./internal/controller and ./internal/webhook/validation
• Commit-time pre-commit hooks
• git diff --check

The complete ./internal/controller and webhook suites also contain envtests; those cannot start locally because the worktree does not contain bin/k8s/1.30.0-darwin-arm64/etcd.

Summary by CodeRabbit

• Bug Fixes
  • Improved deployment handling when components are managed by different controllers. Grove now reconciles and reports readiness only for the components it manages, while retaining the complete deployment configuration for graph-wide settings.
  • Preserved existing worker selectors, restart annotations, and workload hash behavior during Grove rendering.

[andrewpaprotsky]

One P2 correctness finding inline, verified by running the same real-API-server test through the complete Grove program on base 4ad02ea and head 25303c8: the base preserves the unrelated adapter, while the head deletes it.

CI note: the operator job also fails on goconst diagnostics for the repeated lpx and prefill test literals, preventing subsequent operator tests from running. The base passed with the same lint configuration. This is a separate CI blocker, not another behavioral finding: https://github.com/ai-dynamo/dynamo/actions/runs/36437691522/job/108979940999

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f4cac1c0-5b48-4aca-b64e-efa5f92e5162

📥 Commits

Reviewing files that changed from the base of the PR and between 4ad02ea and f770c88.

📒 Files selected for processing (30)

• deploy/operator/internal/AGENTS.md
• deploy/operator/internal/controller/dgd_grove_lpx_restart.go
• deploy/operator/internal/controller/dgd_grove_program.go
• deploy/operator/internal/controller/dgd_grove_restart_progress_resolver.go
• deploy/operator/internal/controller/dgd_grove_scaler.go
• deploy/operator/internal/controller/dgd_grove_scaler_test.go
• deploy/operator/internal/controller/dgd_grove_stable_resources_reconciler.go
• deploy/operator/internal/controller/dgd_grove_stable_resources_reconciler_test.go
• deploy/operator/internal/controller/dgd_grove_topology_condition_reconciler.go
• deploy/operator/internal/controller/dgd_grove_worker_hash_suffix_envtest_test.go
• deploy/operator/internal/controller/dgd_grove_workload_renderer.go
• deploy/operator/internal/controller/dgd_grove_workloads_reconciler.go
• deploy/operator/internal/controller/dgd_grove_workloads_reconciler_test.go
• deploy/operator/internal/controller/dgd_lpx_handoff_test.go
• deploy/operator/internal/controller/dgd_worker_rollout_reconciler_test.go
• deploy/operator/internal/controller/dgd_workload_program_test.go
• deploy/operator/internal/controller/dynamographdeployment_controller_test.go
• deploy/operator/internal/controller/lpx/render_test.go
• deploy/operator/internal/controller/upgrade_test.go
• deploy/operator/internal/dynamo/gpu_shape.go
• deploy/operator/internal/dynamo/gpu_shape_test.go
• deploy/operator/internal/dynamo/graph.go
• deploy/operator/internal/dynamo/graph_lpx_test.go
• deploy/operator/internal/dynamo/graph_test.go
• deploy/operator/internal/dynamo/grove.go
• deploy/operator/internal/dynamo/grove_schema_test.go
• deploy/operator/internal/dynamo/grove_test.go
• deploy/operator/internal/dynamo/hash_test.go
• deploy/operator/internal/webhook/validation/dynamographdeployment.go
• deploy/operator/internal/webhook/validation/dynamographdeployment_lpx.go

💤 Files with no reviewable changes (1)

• deploy/operator/internal/webhook/validation/dynamographdeployment_lpx.go

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

Changes

The Grove reconciliation path now passes the complete DynamoGraphDeployment with a delegation predicate. Renderers, resource reconcilers, naming, readiness checks, and tests use managed-component views without creating filtered DGD copies.

Grove delegation-aware reconciliation

Layer / File(s)	Summary
Reconciliation request and propagation deploy/operator/internal/controller/dgd_grove_program.go, deploy/operator/internal/controller/dgd_grove_*	groveReconcileRequest carries the DGD and delegation predicate through topology, restart, workload, scaling, and stable-resource reconciliation.
Rendering and resource generation deploy/operator/internal/dynamo/*, deploy/operator/internal/controller/dgd_grove_workload_renderer.go	PCS naming and generation skip delegated components. Worker-hash, restart-annotation, GPU-shape, and legacy-selector handling use the existing PCS and managed components.
Readiness and stable resources deploy/operator/internal/controller/dgd_grove_stable_resources_reconciler.go, deploy/operator/internal/dynamo/grove.go	Stable resources and readiness evaluation use the complete DGD and exclude delegated components from component observation, status, revision, and worker-hash checks.
Validation and integration coverage deploy/operator/internal/controller/*_test.go, deploy/operator/internal/dynamo/*_test.go, deploy/operator/internal/webhook/validation/*	Tests replace projected DGD fixtures with delegation predicates and verify PCS naming, rendering, services, restart progress, readiness, and source-DGD immutability.

Priority: ⬇️ Low

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: 🟡 Moderate · up to f770c

Delegated components can cause another graph’s scaling adapter to be deleted when names collide. Protect adapter ownership before merging.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 18.02% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 111 functions across 28 files. (1 skipped…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: separating Grove-managed components from delegated components.
Description check	✅ Passed	The description provides the change overview, implementation details, behavior, validation results, and related issue reference. It does not include the template's explicit “Where should the reviewer …
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 18.02% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 111 functions across 28 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Commit to this branch
• Create a new PR

Warning

Some tools did not complete. Review the errors below.

🔧 golangci-lint (2.13.2)

Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions
The command is terminated due to an error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions

---

_{Comment @coderabbitai help to get the list of available commands.}

[dmitry-tokarev-nv]

Approved. The head writes the same objects as the base, except one new Service label (P3, inline) and the adapter ownership fix from the earlier thread. I read c88e0b6fea.

• [P3] Services for prefill and decode components gain nvidia.com/dynamo-sub-component-type. dgd_grove_stable_resources_reconciler.go:151.
• Open: this PR is stacked on #15062 and must merge after it.

What I ran at 3546a4316e, 0a0f361285 and c88e0b6fea.

• Render: 1,952 reconcile scenarios, as the inline comment describes. In both trees, 66 of them stop at the same first error. Three causes are DRA off (30), no registered InferencePool kind (18) and checkpoints off (6). The other two are a runtime version override that is not a semantic version (6) and a container ResourceClaim without a matching pod claim (6). These scenarios compare only that error.
• Admission: the deleted dynamographdeployment_lpx.go held only the component filter. Its one consumer is the combined Grove name-length rule at dynamographdeployment.go:370. That rule gives the same admission results and PCS names at base and head for 976 fixture variants and a 28,980-row name-length grid. A limit raised by one changes 610 rows of that grid.
• Tests: the four changed packages pass in all three trees with envtest, and Ginkgo ran 75 of 75 specs. At c88e0b6fea, 2,754 tests pass, and none fail or skip. The 3 removed tests have replacements with the same cases.
• Mutations: each of these fails named tests: IsDelegated inverted, a nil predicate, ManagedComponents that returns every component, DelegatedComponents that returns none, and the dynamo filter turned off. Removal of either IsControlledBy guard in the adapter reconciler fails TestDGDScalingAdaptersReconciler_DeletesOnlyObservedOwnedAdapters.
• Merges: four of the five parent merges match a redo with git merge-tree exactly. The fifth, 0a0f361285, differs only in its one conflict, tests/deploy/test_dynamocheckpoint.py, which keeps the version of the parent.
• deploy/operator/internal/CLAUDE.md holds @AGENTS.md.

[dmitry-tokarev-nv]

Approved. I read 9244da86c6. The fix does what the reply on line 151 says, and the new test fails without it. One new P3 nit is inline.

• [P3] A component without type whose pod template sets nvidia.com/dynamo-component-type gets an empty nvidia.com/dynamo-sub-component-type label on its Service. dgd_grove_stable_resources_reconciler.go:149.
• Fixed: the P3 at line 151. Native Services no longer carry the label. My reply on that thread has the numbers.
• Open: this PR is stacked on #15062 and must merge after it.

What I checked on the delta from 75eb210 to 9244da8.

• Shape: 9244da86c6 is one commit on top of 75eb210415. It changes only dgd_grove_stable_resources_reconciler.go and its test file. A redo of the merge 75eb210415 with git merge-tree gives the same tree, and the PR diff is the same before and after that merge.
• Render probe: 1,952 scenarios, the same as last round. The head gives the same objects, events and status as the base 5143224563. Only the write log differs. The head does not send the adapter deletes that got NotFound at the base, because of the ownership fix in d3de18eb52.
• Service label probe: 450 scenarios that create Services in native and legacy mode, with and without user-set labels. Every Service at the head matches the base, except in the P3 case.
• Test: TestGroveStableResourcesReconcilerElasticEPServicesSelectRenderedPods fails 10 of 12 subtests on 75eb210415 and passes 12 of 12 at the head. Three mutants of the fix each fail their own subtests.
• Whole package: Pre Merge job operator (109789020394) ran the full internal/controller suite with envtest assets on b8d3f54071, the head merged into a41283bf39. It passed. I did not run the full suite myself.
• Limits: the label probe runs each fixture as written, in Kubernetes discovery only, with one reconcile after the legacy rewrite.

[dmitry-tokarev-nv]

Approved. I read a247e0c77c. No finding of mine is open.

• Fixed: the P3 at line 149. The fix is the guard that I suggested, so I am partly reviewing my own suggestion. My reply on that thread has the numbers.
• Open: this PR is stacked on #15062 and must merge after it.

What I checked on the delta from 9244da8 to a247e0c.

• Shape: a247e0c77c is one commit on top of 9244da86c6. It changes only dgd_grove_stable_resources_reconciler.go (+1/-1) and its test file (+7/-1). The base 5143224563 did not move.
• Probes: the render probe gives the same output at the head and at 9244da86c6 in all 1,952 scenarios. The labels probe creates the Services of 450 scenarios. Its output differs only by the 141 empty labels that the fix removes.
• Test: TestGroveStableResourcesReconcilerElasticEPServicesSelectRenderedPods passes 14 of 14 subtests at the head. With the guard reverted, only the 2 new type-from-pod-label subtests fail.
• Package: I ran all of ./internal/controller/ with the envtest assets for Kubernetes 1.30.0, at the head and at 9244da86c6. Both runs pass 335 tests and 75 Ginkgo specs. The only difference is the 2 new subtests. With an empty assets folder, the 6 envtest tests fail, so the passing runs used the assets.
• CI: the Pre Merge operator job ran the module tests with envtest on ef5af56078. That commit merges the head with #15062 and main at df8ad66921. The job passed.