[SPARK-20607][CORE]Add new unit tests to ShuffleSuite#17868
Closed
heary-cao wants to merge 1 commit intoapache:masterfrom
Closed
[SPARK-20607][CORE]Add new unit tests to ShuffleSuite#17868heary-cao wants to merge 1 commit intoapache:masterfrom
heary-cao wants to merge 1 commit intoapache:masterfrom
Conversation
heary-cao
changed the title
srowen
requested changes
May 5, 2017
Member
There was a problem hiding this comment.
You aren't closing the existing sc here
Contributor
Author
There was a problem hiding this comment.
Has been modified.
please review it again.
thanks.
Contributor
Author
There was a problem hiding this comment.
sorry,
I have modify it.
thanks.
Contributor
Author
There was a problem hiding this comment.
Sorry,
When debugging use value, forget the back.
heary-cao
force-pushed
the
ShuffleSuite
branch
2 times, most recently
from
a6c2a1d to
130d597
Compare
|
Test build #3713 has finished for PR 17868 at commit
|
Contributor
Author
|
@srowen |
|
Test build #3720 has started for PR 17868 at commit |
|
Test build #3725 has started for PR 17868 at commit |
|
Test build #3734 has finished for PR 17868 at commit
|
Member
|
merged to master |
lycplus
pushed a commit
to lycplus/spark
that referenced
this pull request
May 24, 2017
## What changes were proposed in this pull request? This PR update to two: 1.adds the new unit tests. testing would be performed when there is no shuffle stage, shuffle will not generate the data file and the index files. 2.Modify the '[SPARK-4085] rerun map stage if reduce stage cannot find its local shuffle file' unit test, parallelize is 1 but not is 2, Check the index file and delete. ## How was this patch tested? The new unit test. Author: caoxuewen <cao.xuewen@zte.com.cn> Closes apache#17868 from heary-cao/ShuffleSuite.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR update to two:
1.adds the new unit tests.
testing would be performed when there is no shuffle stage,
shuffle will not generate the data file and the index files.
2.Modify the '[SPARK-4085] rerun map stage if reduce stage cannot find its local shuffle file' unit test,
parallelize is 1 but not is 2, Check the index file and delete.
How was this patch tested?
The new unit test.