fix TestALPNProxyHTTPProxyBasicAuthDial flakiness#26650
Merged
GavinFrazar merged 1 commit intomasterfrom May 22, 2023
Merged
Conversation
* stop the node before resetting http proxy auth credentials, then start the node again which succeeds immediately, so that the test doesn't time out after 10 seconds waiting for the node to join due to random backoff/jitter.
zmb3
approved these changes
May 20, 2023
mdwn
approved these changes
May 22, 2023
public-teleport-github-review-bot
Bot
removed the request for review
from fspmarshall
GavinFrazar
deleted the
gavinfrazar/fix-TestALPNProxyHTTPProxyBasicAuthDial-flakiness
branch
|
@GavinFrazar See the table below for backport results.
|
This was referenced May 22, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #14143
3rd time's the charm! I think the issue was that the test is setup to intentionally prevent a node from joining through an HTTP basic auth proxy when it has the wrong credentials, but the node only has 10 seconds to successfully join before the test code times it out. This meant it would fail the initial dial attempt, then the test gives it correct credentials and waits for it to join, but it only had 10 seconds to do all of that and join the cluster or the test would fail. I suspect the random failures were coming from backoff/jitter in dial attempts.
So I fixed it by stopping the node after verifying that it was unable to join, then setting correct HTTP basic auth credentials and starting the node again. This should allow the node to join on its first attempt to dial the cluster with good credentials and avoid running out the 10 second timeout due to random backoff/jitter.
I was able to reliably reproduce the original test's flakiness locally by reducing the 10 second timeout to 1 second. With this temporarily reduced timeout, I tried this PR's changes and failed to reproduce a test failure after 100 attempts with and without the race detector enabled. So I think I finally got this one fixed for good.