health-server: Do not cleanup health checking result on node updates. #30917
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
marseel
added
kind/bug
|
/cc @christarazi |
danehans
reviewed
Feb 23, 2024
marseel
force-pushed
the
fix_health_checking_even_more
branch
from
b53d562 to
0212cf7
Compare
|
Good idea @danehans I've added a test. I was already thinking about refactoring that part and totally forgot to add the test. |
|
/test |
christarazi
approved these changes
Feb 28, 2024
|
In theory, this code should be simple, I think that we need a better design, I will follow up with the issue and do it as my side-project. |
marseel
force-pushed
the
fix_health_checking_even_more
branch
from
0212cf7 to
ad51c2b
Compare
|
Rebased on the main as |
|
/test |
Whenever node was updated, healtch-checking was removing and re-adding that node. This caused it to lose information about previously performed probes, which resulted in `unknown` status for such nodes. This can happen often especially in ENI mode, where node updates happen each time new pod is scheduled on the node. Signed-off-by: Marcel Zieba <[email protected]>
marseel
force-pushed
the
fix_health_checking_even_more
branch
from
ad51c2b to
fa1bccf
Compare
@marseel when ^ issue is created, can you link it to this PR? |
|
/test |
maintainer-s-little-helper
bot
added
the
ready-to-merge
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Whenever a node was updated, healtch-checking was removing and re-adding that node. This caused it to lose information about previously performed probes, which resulted in
unknownstatus for such nodes. This can happen often, especially in ENI mode, where node updates happen each time a new pod is scheduled on the node.Backstory:
#29566 fixed issue where we were reporting deleted nodes as unhealthy, but it introduced an issue when the updating node was overriding probe status, marking such nodes status as unreachable. Fix in #30504 changed it to mark such nodes as unknown. Now we are preserving probe status on node updates.
Fixes: #29566