[SPARK-19597][CORE] test case for task deserialization errors #16930
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #72880 has finished for PR 16930 at commit
|
Contributor
Author
|
Jenkins, retest this please |
|
Test build #72891 has finished for PR 16930 at commit
|
Contributor
kayousterhout
left a comment
kayousterhout
left a comment
There was a problem hiding this comment.
One small comment -- otherwise looks good
| val conf = new SparkConf | ||
| val serializer = new JavaSerializer(conf) | ||
| val env = createMockEnv(conf, serializer) | ||
| val serializedTask = serializer.newInstance().serialize(new NonDeserializableTask) |
Contributor
There was a problem hiding this comment.
can you take the error message as a param to NonDeserializableTask, so the magic string can be defined once and used below on line 137? From a quick read I thought the one on line 137 was coming from the Spark core code (not from the test code)
Contributor
|
LGTM assuming tests pass |
|
Test build #73437 has finished for PR 16930 at commit
|
Contributor
|
I merged this into master |
Contributor
Author
|
thanks @kayousterhout ! |
Yunni
pushed a commit
to Yunni/spark
that referenced
this pull request
Feb 27, 2017
Adds a test case that ensures that Executors gracefully handle a task that fails to deserialize, by sending back a reasonable failure message. This does not change any behavior (the prior behavior was already correct), it just adds a test case to prevent regression. Author: Imran Rashid <[email protected]> Closes apache#16930 from squito/executor_task_deserialization.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a test case that ensures that Executors gracefully handle a task that fails to deserialize, by sending back a reasonable failure message. This does not change any behavior (the prior behavior was already correct), it just adds a test case to prevent regression.