Skip to content

Fix infinite reconciliation on webhook resources#1651

Merged
istio-testing merged 1 commit intoistio-ecosystem:mainfrom
dgn:fix-webhook-reconciliation
Mar 6, 2026
Merged

Fix infinite reconciliation on webhook resources#1651
istio-testing merged 1 commit intoistio-ecosystem:mainfrom
dgn:fix-webhook-reconciliation

Conversation

@dgn
Copy link
Copy Markdown
Collaborator

@dgn dgn commented Mar 5, 2026

istiod updates the CABundle and FailurePolicy on its ValidatingWebhookConfigurations and MutatingWebhookConfigurations. The controller needs to ignore those changes to avoid continuous reconciliation of the control plane.

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 5, 2026

Codecov Report

❌ Patch coverage is 0% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.44%. Comparing base (6b151f8) to head (3d2f26c).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
...trollers/istiorevision/istiorevision_controller.go 0.00% 12 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (6b151f8) and HEAD (3d2f26c). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (6b151f8) HEAD (3d2f26c)
2 1
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1651       +/-   ##
===========================================
- Coverage   80.96%   52.44%   -28.53%     
===========================================
  Files          50       50               
  Lines        2474     2479        +5     
===========================================
- Hits         2003     1300      -703     
- Misses        347     1096      +749     
+ Partials      124       83       -41     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Contributor

@nrfox nrfox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One very small nit otherwise LGTM.

Curious how we caught the validating webhook but not the mutating webhook?

@dgn dgn force-pushed the fix-webhook-reconciliation branch from 52e5dff to 5308696 Compare March 6, 2026 08:55
@dgn
Copy link
Copy Markdown
Collaborator Author

dgn commented Mar 6, 2026

Curious how we caught the validating webhook but not the mutating webhook?

Honestly this behavior was known and stable for a looong time.. I'm wondering how we didn't hit this bug earlier.

@dgn
Copy link
Copy Markdown
Collaborator Author

dgn commented Mar 6, 2026

/retest

1 similar comment
@dgn
Copy link
Copy Markdown
Collaborator Author

dgn commented Mar 6, 2026

/retest

@dgn
Copy link
Copy Markdown
Collaborator Author

dgn commented Mar 6, 2026

/retest

}
case *admissionv1.MutatingWebhookConfiguration:
for i := range len(webhookConfig.Webhooks) {
webhookConfig.Webhooks[i].ClientConfig.CABundle = nil
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dont we have to clear FailurePolicy for the MutatingWebhookConfiguration as well?

webhookConfig.Webhooks[i].FailurePolicy = nil

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope. istiod never modifies it, it only does it for the ValidatingWebhookConfiguration

@sridhargaddam
Copy link
Copy Markdown
Contributor

@dgn Added a hold for you to take a look at my review comment. Please feel free to remove it, if changes are not required.

@dgn
Copy link
Copy Markdown
Collaborator Author

dgn commented Mar 6, 2026

/retest

@dgn
Copy link
Copy Markdown
Collaborator Author

dgn commented Mar 6, 2026

/cherry-pick release-1.29
/cherry-pick release-1.28
/cherry-pick release-1.27
/cherry-pick release-1.26
/cherry-pick release-1.0

@istio-testing
Copy link
Copy Markdown
Collaborator

@dgn: once the present PR merges, I will cherry-pick it on top of release-1.0, release-1.26, release-1.27, release-1.28, release-1.29 in new PRs and assign them to you.

Details

In response to this:

/cherry-pick release-1.29
/cherry-pick release-1.28
/cherry-pick release-1.27
/cherry-pick release-1.26
/cherry-pick release-1.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@dgn
Copy link
Copy Markdown
Collaborator Author

dgn commented Mar 6, 2026

/retest

istiod updates the CABundle and FailurePolicy on its
ValidatingWebhookConfigurations and MutatingWebhookConfigurations. The
controller needs to ignore those changes to avoid continuous
reconciliation of the control plane.

Signed-off-by: Daniel Grimm <dgrimm@redhat.com>
@dgn dgn force-pushed the fix-webhook-reconciliation branch from 5308696 to 3d2f26c Compare March 6, 2026 16:08
@istio-testing istio-testing merged commit b3cd952 into istio-ecosystem:main Mar 6, 2026
14 of 15 checks passed
@istio-testing
Copy link
Copy Markdown
Collaborator

In response to a cherrypick label: #1651 failed to apply on top of branch "release-1.0":

Applying: Fix infinite reconciliation on webhook resources
Using index info to reconstruct a base tree...
M	controllers/istiorevision/istiorevision_controller.go
M	tests/integration/api/istiorevision_test.go
Falling back to patching base and 3-way merge...
Auto-merging tests/integration/api/istiorevision_test.go
CONFLICT (content): Merge conflict in tests/integration/api/istiorevision_test.go
Auto-merging controllers/istiorevision/istiorevision_controller.go
CONFLICT (content): Merge conflict in controllers/istiorevision/istiorevision_controller.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 Fix infinite reconciliation on webhook resources

@istio-testing
Copy link
Copy Markdown
Collaborator

In response to a cherrypick label: new issue created for failed cherrypick: #1658

@istio-testing
Copy link
Copy Markdown
Collaborator

In response to a cherrypick label: new pull request created: #1659

@istio-testing
Copy link
Copy Markdown
Collaborator

In response to a cherrypick label: new pull request created: #1660

openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Mar 9, 2026
* upstream/main:
  Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1666)
  Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1665)
  Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1662)
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651)
dgn added a commit to dgn/sail-operator that referenced this pull request Mar 10, 2026
istiod updates the CABundle and FailurePolicy on its
ValidatingWebhookConfigurations and MutatingWebhookConfigurations. The
controller needs to ignore those changes to avoid continuous
reconciliation of the control plane.

Signed-off-by: Daniel Grimm <dgrimm@redhat.com>
dgn added a commit to dgn/sail-operator that referenced this pull request Mar 10, 2026
istiod updates the CABundle and FailurePolicy on its
ValidatingWebhookConfigurations and MutatingWebhookConfigurations. The
controller needs to ignore those changes to avoid continuous
reconciliation of the control plane.

Signed-off-by: Daniel Grimm <dgrimm@redhat.com>
istio-testing pushed a commit that referenced this pull request Mar 10, 2026
istiod updates the CABundle and FailurePolicy on its
ValidatingWebhookConfigurations and MutatingWebhookConfigurations. The
controller needs to ignore those changes to avoid continuous
reconciliation of the control plane.

Signed-off-by: Daniel Grimm <dgrimm@redhat.com>
openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Mar 10, 2026
* upstream/release-1.26:
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651) (istio-ecosystem#1670)
openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Mar 10, 2026
* upstream/release-1.26:
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651) (istio-ecosystem#1670)
openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Mar 10, 2026
* upstream/release-1.26:
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651) (istio-ecosystem#1670)
openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Mar 11, 2026
* upstream/release-1.26:
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651) (istio-ecosystem#1670)
openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Mar 11, 2026
* upstream/release-1.26:
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651) (istio-ecosystem#1670)
openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Mar 11, 2026
* upstream/release-1.26:
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651) (istio-ecosystem#1670)
openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Mar 11, 2026
* upstream/release-1.26:
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651) (istio-ecosystem#1670)
openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Mar 11, 2026
* upstream/release-1.26:
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651) (istio-ecosystem#1670)
openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Mar 11, 2026
* upstream/release-1.26:
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651) (istio-ecosystem#1670)
openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Mar 12, 2026
* upstream/release-1.26:
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651) (istio-ecosystem#1670)
openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Mar 12, 2026
* upstream/release-1.26:
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651) (istio-ecosystem#1670)
openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Mar 12, 2026
* upstream/release-1.26:
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651) (istio-ecosystem#1670)
istio-testing pushed a commit that referenced this pull request Apr 9, 2026
istiod updates the CABundle and FailurePolicy on its
ValidatingWebhookConfigurations and MutatingWebhookConfigurations. The
controller needs to ignore those changes to avoid continuous
reconciliation of the control plane.

Signed-off-by: Daniel Grimm <dgrimm@redhat.com>
openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Apr 9, 2026
* upstream/release-1.0:
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651) (istio-ecosystem#1671)
openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Apr 9, 2026
* upstream/release-1.0:
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651) (istio-ecosystem#1671)
openshift-service-mesh-bot pushed a commit to openshift-service-mesh-bot/sail-operator that referenced this pull request Apr 10, 2026
* upstream/release-1.0:
  Fix infinite reconciliation on webhook resources (istio-ecosystem#1651) (istio-ecosystem#1671)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants