fix(operator): normalize equivalent worker hash inputs

[bzsuni]

Overview

Semantically equivalent worker configurations can produce different worker hashes even when they render equivalent Grove specs:

• Omitted Grove options versus empty wrappers or forceScalingGroup: false.
• Omitted checkpoint configuration versus checkpoint.enabled: false.
• An omitted compilation-cache mount path versus the explicit backend default, with the same PVC.

The controller writes this hash to the worker clique's nvidia.com/dynamo-worker-hash label, so representation-only differences can cause unnecessary worker hash changes.

This change canonicalizes these equivalent configurations in the copied hash input. The submitted configuration is unchanged.

Details

Empty Grove wrappers and disabled checkpoint configuration are removed from the hash input. If the experimental wrapper becomes empty after normalization, it is removed as well.

Enabled checkpoint configuration, forceScalingGroup: true, and other experimental features remain significant.

Compilation-cache normalization uses the same backend-default lookup as the renderer. An explicit backend-default mount path is normalized to omission, while custom paths and PVC names remain significant.

Reproduction

Regression tests compare otherwise identical workers using these equivalent forms:

Omitted form	Equivalent explicit form
experimental omitted	experimental: {}
experimental omitted	experimental: {grove: {}}
experimental omitted	experimental: {grove: {forceScalingGroup: false}}
experimental.checkpoint omitted	checkpoint: {enabled: false} under experimental
compilationCache.mountPath omitted	/root/.cache/vllm for vLLM with the same PVC

Before the fixes, each pair produced equivalent rendered Grove specs before controller hash-stamping, but different worker hashes.

With the fixes, the hashes also match.

The tests also verify that effective configuration remains significant: forceScalingGroup: true, enabled checkpointing, custom cache paths, and different PVC names still produce different hashes. Input-preservation checks verify that rendering and normalization do not modify the submitted configuration.

Upgrade behavior

Omission remains the canonical representation.

Configurations already using the omitted forms retain their hashes. Configurations using one of the equivalent explicit forms above may receive a one-time worker hash change after upgrading.

Validation

After combining the fixes and rebasing onto main:

cd deploy/operator
go test -mod=readonly -p 4 ./internal/dynamo -count=1

Where should the reviewer start?

• deploy/operator/internal/dynamo/hash.go: worker hash normalization in workerHashSpec.
• deploy/operator/internal/dynamo/hash_test.go: equivalent-configuration regression tests and input-preservation checks.

Related Issues

🚫 This PR is NOT linked to an issue:

• Confirmed — no related issue

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi bzsuni! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ee42bcb5-61e9-4c2a-b93f-9a59f444bc12

📥 Commits

Reviewing files that changed from the base of the PR and between 0852047 and e99a49e.

📒 Files selected for processing (2)

• deploy/operator/internal/dynamo/hash.go
• deploy/operator/internal/dynamo/hash_test.go

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The change normalizes ineffective Grove and Experimental configuration wrappers before worker hashing. Tests cover equivalent rendered workloads, hash stability, input immutability, explicit true, and preservation of unrelated experimental features.

Changes

Worker hash normalization

Layer / File(s)	Summary
Canonicalize worker hash configuration deploy/operator/internal/dynamo/hash.go	workerHashSpec omits empty Grove and Experimental wrappers while retaining meaningful experimental configuration.
Validate normalized hash behavior deploy/operator/internal/dynamo/hash_test.go	Tests verify equivalent rendered specifications, stable hashes, non-mutating inputs, explicit true changes, and preservation of unrelated experimental features.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~15 minutes

Merge Risk: ⚪ Minimal · up to e99a4

The worker-hash canonicalization is covered for the intended equivalent configurations and preserves effective experimental options, with no actionable merge risk identified.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely describes the main change: normalizing equivalent worker hash inputs.
Description check	✅ Passed	The description includes the required Overview, Details, reviewer-start, and Related Issues sections. It explains the behavior, tests, upgrade impact, and validation command.

• Fix all pre-merge checks with AI

---

_{Comment @coderabbitai help to get the list of available commands.}

[tmonty12]

/ok to test e99a49e

[tmonty12]

Non-blocking: while you’re updating hash normalization, could we also cover these equivalent configurations, here or in a follow-up?

• experimental.checkpoint: omitted versus enabled: false. Both disable checkpointing, but currently produce different worker hashes.
• compilationCache.mountPath: omitted versus the resolved backend default—for vLLM, /root/.cache/vllm, with the same PVC. Both render the same mount and cache environment variable, but currently hash differently.

I reproduced identical rendered Grove specs with different hashes for both cases. We should normalize these on the hash-input copy and add equivalent-render/equal-hash tests, while keeping enabled checkpointing and genuinely different mount paths distinct.

[tmonty12]

/ok to test 558951e