[SPARK-19646][CORE][STREAMING] binaryRecords replicates records in scala API #16974
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…he underlying array, which may be reused when repeated reads don't need a different size, as is the case with binaryRecords APIs
Contributor
|
LGTM - pending jenkins |
|
Test build #73047 has finished for PR 16974 at commit
|
Contributor
|
@srowen should we add a regression test? It seems weird that we didn't catch this in tests. |
uncleGen
approved these changes
Feb 17, 2017
Member
Author
|
Agreed, I fixed the tests to actually fix this, and generally cleaned up the relevant test code |
|
Test build #73056 has finished for PR 16974 at commit
|
|
Test build #3578 has finished for PR 16974 at commit
|
asfgit
pushed a commit
that referenced
this pull request
Feb 20, 2017
…ala API ## What changes were proposed in this pull request? Use `BytesWritable.copyBytes`, not `getBytes`, because `getBytes` returns the underlying array, which may be reused when repeated reads don't need a different size, as is the case with binaryRecords APIs ## How was this patch tested? Existing tests Author: Sean Owen <[email protected]> Closes #16974 from srowen/SPARK-19646. (cherry picked from commit d0ecca6) Signed-off-by: Sean Owen <[email protected]>
asfgit
pushed a commit
that referenced
this pull request
Feb 20, 2017
…ala API Use `BytesWritable.copyBytes`, not `getBytes`, because `getBytes` returns the underlying array, which may be reused when repeated reads don't need a different size, as is the case with binaryRecords APIs Existing tests Author: Sean Owen <[email protected]> Closes #16974 from srowen/SPARK-19646. (cherry picked from commit d0ecca6) Signed-off-by: Sean Owen <[email protected]>
Member
Author
|
Merged to master/2.1/2.0 as it's a reasonably important bug |
Member
Author
|
Oops, pick was clean into 2.1 but it actually resulted in an error: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.1-test-sbt-hadoop-2.7/374/consoleFull Fixing now ... |
Yunni
pushed a commit
to Yunni/spark
that referenced
this pull request
Feb 27, 2017
…ala API ## What changes were proposed in this pull request? Use `BytesWritable.copyBytes`, not `getBytes`, because `getBytes` returns the underlying array, which may be reused when repeated reads don't need a different size, as is the case with binaryRecords APIs ## How was this patch tested? Existing tests Author: Sean Owen <[email protected]> Closes apache#16974 from srowen/SPARK-19646.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Use
BytesWritable.copyBytes, notgetBytes, becausegetBytesreturns the underlying array, which may be reused when repeated reads don't need a different size, as is the case with binaryRecords APIsHow was this patch tested?
Existing tests