Skip to content

Backport of fix(control-plane): acl tokens deleted while pods in graceful shutdown into release/1.2.x#3743

Merged
DanStough merged 2 commits intorelease/1.2.xfrom
backport/dans/NET-8345/graceful-shutdown-acl-race/privately-regular-cub
Mar 20, 2024
Merged

Backport of fix(control-plane): acl tokens deleted while pods in graceful shutdown into release/1.2.x#3743
DanStough merged 2 commits intorelease/1.2.xfrom
backport/dans/NET-8345/graceful-shutdown-acl-race/privately-regular-cub

Conversation

@hc-github-team-consul-core
Copy link
Copy Markdown
Collaborator

Backport

This PR is auto-generated from #3736 to be assessed for backporting due to the inclusion of the label backport/1.2.x.

🚨

Warning automatic cherry-pick of commits failed. If the first commit failed,
you will see a blank no-op commit below. If at least one commit succeeded, you
will see the cherry-picked commits up to, not including, the commit where
the merge conflict occurred.

The person who merged in the original PR is:
@DanStough
This person should manually cherry-pick the original PR into a new backport PR,
and close this one when the manual backport PR is merged in.

merge conflict error: POST https://api.github.com/repos/hashicorp/consul-k8s/merges: 409 Merge conflict []

The below text is copied from the body of the original PR.


Changes proposed in this PR

  • Users have reported that when GracefulShutdown is enabled, ACL tokens are deleted immediately when a pod goes in to TERMINATING, which can disrupt consul DNS and an app's shutdown procedure if that is required.
  • The new behavior is that if a pod still exists and has GracefulShutdown enabled, we will skip ACL and node cleanup, instead re-queue the event for the longest shutdown period, plus some buffer to cut down on noise.

How I've tested this PR

  • Manual testing in a local kube cluster, deploying an app w/ ACLs and graceful shutdown. I delete a pod and watch the control-plane and consul-dataplane logs to verify when the ACL token is cleaned up.
  • Additional unit tests added.

How I expect reviewers to test this PR

👀 ☕

Checklist


Overview of commits

@hc-github-team-consul-core hc-github-team-consul-core force-pushed the backport/dans/NET-8345/graceful-shutdown-acl-race/privately-regular-cub branch from 3eab26e to 7e4d49c Compare March 14, 2024 15:28
@hashicorp-cla
Copy link
Copy Markdown

hashicorp-cla commented Mar 14, 2024

CLA assistant check
All committers have signed the CLA.

@DanStough DanStough force-pushed the backport/dans/NET-8345/graceful-shutdown-acl-race/privately-regular-cub branch from fd2625f to ec361f3 Compare March 14, 2024 16:18
@DanStough DanStough marked this pull request as ready for review March 14, 2024 16:18
@DanStough DanStough requested a review from ndhanushkodi March 19, 2024 14:54
@DanStough
Copy link
Copy Markdown
Contributor

@ndhanushkodi would you remind reviewing the second commit of this PR where we working on the test fix yesterday. Once you give the 👍 I will come back around to the later release branches and make a PR just for the test fix.

@DanStough DanStough force-pushed the backport/dans/NET-8345/graceful-shutdown-acl-race/privately-regular-cub branch from fd0f726 to 87843dd Compare March 19, 2024 16:32
Copy link
Copy Markdown
Contributor

@ndhanushkodi ndhanushkodi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second commit looks good, just two non-blocking questions

@DanStough DanStough force-pushed the backport/dans/NET-8345/graceful-shutdown-acl-race/privately-regular-cub branch from 87843dd to a02dfea Compare March 19, 2024 19:50
@DanStough DanStough force-pushed the backport/dans/NET-8345/graceful-shutdown-acl-race/privately-regular-cub branch from a02dfea to aa76040 Compare March 19, 2024 20:00
@DanStough DanStough merged commit 8f70c98 into release/1.2.x Mar 20, 2024
@DanStough DanStough deleted the backport/dans/NET-8345/graceful-shutdown-acl-race/privately-regular-cub branch March 20, 2024 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants