Skip to content

[SPARK-29918][SQL] RecordBinaryComparator should check endianness when compared by long #26548

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
WangGuangxin wants to merge 6 commits into from
Closed

[SPARK-29918][SQL] RecordBinaryComparator should check endianness when compared by long #26548

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
37ebd38
handle endianness
WangGuangxin Nov 15, 2019
3cb480d
use long.compareUnsigned & add ut
WangGuangxin Nov 18, 2019
a07cfc6
fix style
WangGuangxin Nov 18, 2019
fdbc871
update
WangGuangxin Nov 18, 2019
5cc04fa
style
WangGuangxin Nov 18, 2019
753883d
update comments & ut
WangGuangxin Nov 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions sql/core/src/main/java/org/apache/spark/sql/execution/RecordBinaryComparator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
import org.apache.spark.unsafe.Platform;
import org.apache.spark.util.collection.unsafe.sort.RecordComparator;

import java.nio.ByteOrder;

public final class RecordBinaryComparator extends RecordComparator {

@Override
Expand Down Expand Up @@ -49,9 +51,13 @@ public int compare(
// for architectures that support unaligned accesses, chew it up 8 bytes at a time
if (Platform.unaligned() || (((leftOff + i) % 8 == 0) && ((rightOff + i) % 8 == 0))) {
while (i <= leftLen - 8) {
final long v1 = Platform.getLong(leftObj, leftOff + i);
final long v2 = Platform.getLong(rightObj, rightOff + i);
long v1 = Platform.getLong(leftObj, leftOff + i);
long v2 = Platform.getLong(rightObj, rightOff + i);
if (v1 != v2) {
if (ByteOrder.nativeOrder().equals(ByteOrder.LITTLE_ENDIAN)) {

@srowen srowen Nov 15, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cant his check happen once and make it a constant rather than call each time?

This is also the unfortunately the common case, little-endian architectures. Hm, I'm trying to think of an alternative that would introduce overhead only for big-endian, but not sure if it's possible. that said reverseBytes ought to be very fast. And I guess this only happens once before this terminates.

@srowen srowen Nov 15, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify the issue isn't endianness alone, but aligned access? varying either one could cause the result to change.

v1 = Long.reverseBytes(v1);
v2 = Long.reverseBytes(v2);
}
return v1 > v2 ? 1 : -1;

@srowen srowen Nov 15, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pulling together my comment with #25491 (comment) - maybe this also has to be Long.compareUnsigned?

@WangGuangxin WangGuangxin Nov 15, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, thanks for point out this. This is the same issue with #25491 (comment). And this can be reproduced in our cluster.

}
i += 8;
Expand Down
2 changes: 1 addition & 1 deletion ...e/src/test/java/test/org/apache/spark/sql/execution/sort/RecordBinaryComparatorSuite.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ public void testBinaryComparatorWhenSubtractionIsDivisibleByMaxIntValue() throws
insertRow(row1);
insertRow(row2);

assert(compare(0, 1) < 0);
assert(compare(0, 1) > 0);

@dongjoon-hyun dongjoon-hyun Nov 15, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, do you mean this is wrong before this PR?

@srowen srowen Nov 15, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change definitely changes the ordering, as bytes are compared in a different order.

@WangGuangxin WangGuangxin Nov 15, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, do you mean this is wrong before this PR?

RecordBinaryComparator is used as a local sort comparator in RoundRobinPartition to make sure the order of each records are the same after rerun. But the relative order is not important.

}

@Test
Expand Down