Skip to content

Conversation

@linkvt
Copy link

@linkvt linkvt commented Oct 23, 2025

Hi,

when running kubectl rollout restart deployment, Kubernetes adds the kubectl.kubernetes.io/restartedAt annotation to the deployment's pod template to trigger a rolling restart. However, Knative's reconciler was removing this annotation by completely replacing the deployment spec, causing new pods to be immediately terminated instead of completing the rollout.

This change preserves externally-added metadata at both the deployment level and template level during reconciliation:

  • Deployment-level labels (was already the case) and annotations (new)
  • Pod template labels and annotations (e.g., kubectl.kubernetes.io/restartedAt)

The fix uses kmeta.UnionMaps to merge desired state with existing state, allowing external annotations/labels to take precedence while still permitting Knative to manage its own metadata.

Changes:

  • cruds.go: Added preservation of deployment and template-level metadata
  • table_test.go: Added test case to verify external metadata preservation

Fixes #14705

Proposed Changes

  • Preserve deployment and template annotations and labels during reconcile
    • knative labels and annotations have priority and will overwrite other labels/annotations

Release Note

Preserve deployment and template annotations and labels during reconcile

/kind bug

Thanks for the review!

@knative-prow knative-prow bot added kind/bug Categorizes issue or PR as related to a bug. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Oct 23, 2025
When running `kubectl rollout restart deployment`, Kubernetes adds the
`kubectl.kubernetes.io/restartedAt` annotation to the deployment's pod
template to trigger a rolling restart. However, Knative's reconciler was
removing this annotation by completely replacing the deployment spec,
causing new pods to be immediately terminated instead of completing the
rollout.

This change preserves externally-added metadata at both the deployment
level and template level during reconciliation:
- Deployment-level labels (was already the case) and annotations (new)
- Pod template labels and annotations (e.g., kubectl.kubernetes.io/restartedAt)

The fix uses kmeta.UnionMaps to merge desired state with existing state,
allowing external annotations/labels to take precedence while still
permitting Knative to manage its own metadata.

Changes:
- cruds.go: Added preservation of deployment and template-level metadata
- table_test.go: Added test case to verify external metadata preservation
@linkvt linkvt force-pushed the preserve-deployment-metadata branch from f079377 to 7632241 Compare October 23, 2025 12:03
@codecov
Copy link

codecov bot commented Oct 23, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.08%. Comparing base (910ae9e) to head (7632241).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #16199      +/-   ##
==========================================
- Coverage   80.11%   80.08%   -0.03%     
==========================================
  Files         214      214              
  Lines       16940    16945       +5     
==========================================
  Hits        13571    13571              
- Misses       3011     3014       +3     
- Partials      358      360       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@linkvt linkvt requested a review from Fedosin October 23, 2025 12:14
@Fedosin
Copy link
Contributor

Fedosin commented Oct 23, 2025

LGTM, but I'd like @dprotaso to take a look as well.

@linkvt
Copy link
Author

linkvt commented Oct 23, 2025

/remove-approve

Was added automatically as I'm co-release lead.

@knative-prow
Copy link

knative-prow bot commented Oct 23, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign linkvt for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knative-prow knative-prow bot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 23, 2025
@linkvt
Copy link
Author

linkvt commented Oct 23, 2025

/ok-to-test

@knative-prow knative-prow bot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Oct 23, 2025
@dprotaso
Copy link
Member

/hold

Will revisit after the release next Tuesday

@knative-prow knative-prow bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. kind/bug Categorizes issue or PR as related to a bug. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

During rollout restart of a deployment the new pod gets terminated within seconds

3 participants