[SPARK-23222][SQL] Make DataFrameRangeSuite not flaky #20431
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
Contributor
|
does this significantly increase the test runtime? |
Member
Author
|
I didn't see significant difference. It is about 450~500 milliseconds. |
|
Test build #86796 has finished for PR 20431 at commit
|
Member
Author
|
retest this please. |
viirya
commented
Jan 30, 2018
| val listener = new SparkListener { | ||
| override def onJobStart(jobStart: SparkListenerJobStart): Unit = { | ||
| eventually(timeout(10.seconds)) { | ||
| eventually(timeout(10.seconds), interval(1.millis)) { |
Member
Author
There was a problem hiding this comment.
The default interval is 15millis, IIUC. It is more possibly that the range stage finishes in the interval. So reduce it to 1millis.
|
Test build #86811 has finished for PR 20431 at commit
|
Contributor
|
thanks, merging to master/2.3! |
asfgit
pushed a commit
that referenced
this pull request
Jan 30, 2018
## What changes were proposed in this pull request? It is reported that the test `Cancelling stage in a query with Range` in `DataFrameRangeSuite` fails a few times in unrelated PRs. I personally also saw it too in my PR. This test is not very flaky actually but only fails occasionally. Based on how the test works, I guess that is because `range` finishes before the listener calls `cancelStage`. I increase the range number from `1000000000L` to `100000000000L` and count the range in one partition. I also reduce the `interval` of checking stage id. Hopefully it can make the test not flaky anymore. ## How was this patch tested? The modified tests. Author: Liang-Chi Hsieh <[email protected]> Closes #20431 from viirya/SPARK-23222. (cherry picked from commit 84bcf9d) Signed-off-by: Wenchen Fan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
It is reported that the test
Cancelling stage in a query with RangeinDataFrameRangeSuitefails a few times in unrelated PRs. I personally also saw it too in my PR.This test is not very flaky actually but only fails occasionally. Based on how the test works, I guess that is because
rangefinishes before the listener callscancelStage.I increase the range number from
1000000000Lto100000000000Land count the range in one partition. I also reduce theintervalof checking stage id. Hopefully it can make the test not flaky anymore.How was this patch tested?
The modified tests.