Skip to content

Prevent a crash loop caused by upstream changes#237

Closed
ejweber wants to merge 2 commits intokubernetes-csi:masterfrom
ejweber:236-prevent-crashloop
Closed

Prevent a crash loop caused by upstream changes#237
ejweber wants to merge 2 commits intokubernetes-csi:masterfrom
ejweber:236-prevent-crashloop

Conversation

@ejweber
Copy link

@ejweber ejweber commented Dec 19, 2023

What type of PR is this?

/kind bug

What this PR does / why we need it:

See #236 and longhorn/longhorn#7116 for a complete analysis.

Now that csi-lib-utils.Connect returns after a configurable 30 second timeout:

  • We need to ensure livenessprobe and its CSI plugin don't enter an unrecoverable alternating crash loop that prevents liveness probes from ever succeeding. We do this by setting the connection timeout during initialization to five minutes.
  • We can do away with the acquireConnection wrapper function, which previously implemented timeouts at the livenessprobe layer. These timeouts can now simply be passed to the csi-lib-utils layer.

Which issue(s) this PR fixes:

Fixes #236

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. labels Dec 19, 2023
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Dec 19, 2023

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ejweber
Once this PR has been reviewed and has the lgtm label, please assign xing-yang for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 19, 2023
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Dec 19, 2023

Welcome @ejweber!

It looks like this is your first PR to kubernetes-csi/livenessprobe 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-csi/livenessprobe has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Dec 19, 2023
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Dec 19, 2023

Hi @ejweber. Thanks for your PR.

I'm waiting for a kubernetes-csi member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Dec 19, 2023
sunnylovestiramisu added a commit to sunnylovestiramisu/livenessprobe that referenced this pull request Dec 20, 2023
@sunnylovestiramisu
Squashed 'release-tools/' changes from f9d5b9c..f8c8cc4
61715ea 
f8c8cc4 Merge pull request kubernetes-csi#237 from msau42/prow
b36b5bf Merge pull request kubernetes-csi#240 from dannawang0221/upgrade-go-version
adfddcc Merge pull request kubernetes-csi#243 from pohly/git-subtree-pull-fix
c465088 pull-test.sh: avoid "git subtree pull" error
7b175a1 Update csi-test version to v5.2.0
987c90c Update go version to 1.21 to match k/k
2c625d4 Add script to generate patch release notes

git-subtree-dir: release-tools
git-subtree-split: f8c8cc4
@jsafrane
Copy link
Contributor

jsafrane commented Dec 21, 2023

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 21, 2023
k8s-ci-robot and others added 2 commits December 21, 2023 09:41
@k8s-ci-robot @ejweber
Merge pull request kubernetes-csi#238 from sunnylovestiramisu/module-…
07bd11f 
…update-master

Update dependency go modules for k8s v1.29.0
@ejweber
Prevent a crash loop caused by upstream changes
0ec40d8 
csi-lib-utils.Connect now times out and returns an error after 30 seconds by
default. If the CSI plugin we are connecting to is unavailable during
livenessprobe initialization, livenessprobe will crash and potentially enter a
crash loop. If the backoff on both the CSI plugin and livenessprobe becomes
large, the situation is unrecoverable.

Configure the timeout of the csi-lib-utils.Connect function and remove
acquireConnection since the upstream changes have made it obselete.

Signed-off-by: Eric Weber <eric.weber@suse.com>
@ejweber ejweber force-pushed the 236-prevent-crashloop branch from 3b2f368 to 0ec40d8 Compare December 21, 2023 15:43
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 21, 2023
@ejweber ejweber mentioned this pull request Jan 4, 2024
Closed
@ejweber
Copy link
Author

ejweber commented Jan 4, 2024

There is a competing PR in #240 that takes a similar, but improved approach. It is driven by an official maintainer, so it has a good chance of success. Closing this one in favor of that one.

@ejweber ejweber closed this Jan 4, 2024
@ejweber ejweber mentioned this pull request Jan 4, 2024
Merged
rhrmo pushed a commit to rhrmo/livenessprobe that referenced this pull request Jan 13, 2026
@k8s-ci-robot
Merge pull request kubernetes-csi#237 from msau42/prow
f8c8cc4 
Add script to generate patch release notes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ggriffiths ggriffiths Awaiting requested review from ggriffiths

@j-griffith j-griffith Awaiting requested review from j-griffith

Assignees

No one assigned

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

New csi-lib-utils/connection.Connect logic can cause permanent CSI plugin outage

3 participants

@ejweber @k8s-ci-robot @jsafrane