Fix SearchableSnapshotsPersistentCacheIntegTests #80077
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tlrx
added
>test
elasticmachine
added
the
Team:Distributed (Obsolete)
Collaborator
|
Pinging @elastic/es-distributed (Team:Distributed) |
Member
Author
|
ping @original-brownbear @arteam - would love to have your thoughts :) |
Member
Author
|
Thanks Armin! |
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this pull request
Nov 3, 2021
The two tests in SearchableSnapshotsPersistentCacheIntegTests failed few times when they were expecting the persistent cache to be empty after mounted indices deletions. While I wasn't able to reproduce the issue (which happened ~5 times in the last 2 months) I suspect the cause are the same: the tests wait for the mounted index to be green but don't always wait for the prewarming to be completed. I think it is possible that a part of a cache file finishes to be prewarmed after the test is completed, making the next step to fail as the persistent cache will contain a doc that is not related to the subsequent test. This commit changes the testCacheSurviveRestart so that it waits for the prewarming to complete. Similarly it changes t estPersistentCacheCleanUpAfterRelocation to also wait for the prewarming to complete (just in case, this test verifies that the recovery is done). It also logs more information about the emptiness of the persistent cache on data nodes and makes the cluster scope to TEST to ensure that a dedicated test cluster is used for each test. Closes elastic#76159 Closes elastic#76160
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this pull request
Nov 3, 2021
The two tests in SearchableSnapshotsPersistentCacheIntegTests failed few times when they were expecting the persistent cache to be empty after mounted indices deletions. While I wasn't able to reproduce the issue (which happened ~5 times in the last 2 months) I suspect the cause are the same: the tests wait for the mounted index to be green but don't always wait for the prewarming to be completed. I think it is possible that a part of a cache file finishes to be prewarmed after the test is completed, making the next step to fail as the persistent cache will contain a doc that is not related to the subsequent test. This commit changes the testCacheSurviveRestart so that it waits for the prewarming to complete. Similarly it changes t estPersistentCacheCleanUpAfterRelocation to also wait for the prewarming to complete (just in case, this test verifies that the recovery is done). It also logs more information about the emptiness of the persistent cache on data nodes and makes the cluster scope to TEST to ensure that a dedicated test cluster is used for each test. Closes elastic#76159 Closes elastic#76160
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this pull request
Nov 3, 2021
The two tests in SearchableSnapshotsPersistentCacheIntegTests failed few times when they were expecting the persistent cache to be empty after mounted indices deletions. While I wasn't able to reproduce the issue (which happened ~5 times in the last 2 months) I suspect the cause are the same: the tests wait for the mounted index to be green but don't always wait for the prewarming to be completed. I think it is possible that a part of a cache file finishes to be prewarmed after the test is completed, making the next step to fail as the persistent cache will contain a doc that is not related to the subsequent test. This commit changes the testCacheSurviveRestart so that it waits for the prewarming to complete. Similarly it changes t estPersistentCacheCleanUpAfterRelocation to also wait for the prewarming to complete (just in case, this test verifies that the recovery is done). It also logs more information about the emptiness of the persistent cache on data nodes and makes the cluster scope to TEST to ensure that a dedicated test cluster is used for each test. Closes elastic#76159 Closes elastic#76160
Collaborator
elasticsearchmachine
pushed a commit
that referenced
this pull request
Nov 3, 2021
The two tests in SearchableSnapshotsPersistentCacheIntegTests failed few times when they were expecting the persistent cache to be empty after mounted indices deletions. While I wasn't able to reproduce the issue (which happened ~5 times in the last 2 months) I suspect the cause are the same: the tests wait for the mounted index to be green but don't always wait for the prewarming to be completed. I think it is possible that a part of a cache file finishes to be prewarmed after the test is completed, making the next step to fail as the persistent cache will contain a doc that is not related to the subsequent test. This commit changes the testCacheSurviveRestart so that it waits for the prewarming to complete. Similarly it changes t estPersistentCacheCleanUpAfterRelocation to also wait for the prewarming to complete (just in case, this test verifies that the recovery is done). It also logs more information about the emptiness of the persistent cache on data nodes and makes the cluster scope to TEST to ensure that a dedicated test cluster is used for each test. Closes #76159 Closes #76160
elasticsearchmachine
pushed a commit
that referenced
this pull request
Nov 3, 2021
) * Fix SearchableSnapshotsPersistentCacheIntegTests (#80077) The two tests in SearchableSnapshotsPersistentCacheIntegTests failed few times when they were expecting the persistent cache to be empty after mounted indices deletions. While I wasn't able to reproduce the issue (which happened ~5 times in the last 2 months) I suspect the cause are the same: the tests wait for the mounted index to be green but don't always wait for the prewarming to be completed. I think it is possible that a part of a cache file finishes to be prewarmed after the test is completed, making the next step to fail as the persistent cache will contain a doc that is not related to the subsequent test. This commit changes the testCacheSurviveRestart so that it waits for the prewarming to complete. Similarly it changes t estPersistentCacheCleanUpAfterRelocation to also wait for the prewarming to complete (just in case, this test verifies that the recovery is done). It also logs more information about the emptiness of the persistent cache on data nodes and makes the cluster scope to TEST to ensure that a dedicated test cluster is used for each test. Closes #76159 Closes #76160 * fixes
elasticsearchmachine
pushed a commit
that referenced
this pull request
Nov 3, 2021
) * Fix SearchableSnapshotsPersistentCacheIntegTests (#80077) The two tests in SearchableSnapshotsPersistentCacheIntegTests failed few times when they were expecting the persistent cache to be empty after mounted indices deletions. While I wasn't able to reproduce the issue (which happened ~5 times in the last 2 months) I suspect the cause are the same: the tests wait for the mounted index to be green but don't always wait for the prewarming to be completed. I think it is possible that a part of a cache file finishes to be prewarmed after the test is completed, making the next step to fail as the persistent cache will contain a doc that is not related to the subsequent test. This commit changes the testCacheSurviveRestart so that it waits for the prewarming to complete. Similarly it changes t estPersistentCacheCleanUpAfterRelocation to also wait for the prewarming to complete (just in case, this test verifies that the recovery is done). It also logs more information about the emptiness of the persistent cache on data nodes and makes the cluster scope to TEST to ensure that a dedicated test cluster is used for each test. Closes #76159 Closes #76160 * usual suspect
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The two tests in
SearchableSnapshotsPersistentCacheIntegTestsfailed few times when they were expecting the persistent cache to be empty after mounted indices deletions.While I wasn't able to reproduce the issue (which happened ~5 times in the last 2 months) I suspect the cause are the same: the tests wait for the mounted index to be green but don't always wait for the prewarming to be completed. I think it is possible that a part of a cache file finishes to be prewarmed after the test is completed, making the next step to fail as the persistent cache will contain a doc that is not related to the subsequent test.
This pull request changes the
testCacheSurviveRestartso that it waits for the prewarming to complete. Similarly it changestestPersistentCacheCleanUpAfterRelocationto also wait for the prewarming to complete (just in case, this test verifies that the recovery is done). It also logs more information about the emptiness of the persistent cache on data nodes and makes the cluster scope toTESTto ensure that a dedicated test cluster is used for each test.Closes #76159
Closes #76160