Skip to content

Struct null aware equality comparator <=> support #6306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rwlee merged 13 commits into from
Aug 19, 2022
Merged

Struct null aware equality comparator <=> support #6306

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
36867cd
struct binop support
rwlee Jul 21, 2022
2dfe2f0
Merge remote-tracking branch 'pub/branch-22.08' into rwlee/structbinops
rwlee Jul 21, 2022
1a61988
Merge remote-tracking branch 'pub/branch-22.08' into rwlee/structbinops
rwlee Jul 27, 2022
6388743
Merge remote-tracking branch 'pub/branch-22.08' into rwlee/structbinops
rwlee Jul 28, 2022
663ad20
switch to integration tests
rwlee Aug 1, 2022
e6321b3
Merge remote-tracking branch 'pub/branch-22.08' into rwlee/structbinops
rwlee Aug 1, 2022
0419d31
avoid DayTimeIntervalGen errors with < 330
rwlee Aug 3, 2022
f20dc8f
first pass ns_equal
rwlee Aug 5, 2022
303f5cc
Merge remote-tracking branch 'pub/branch-22.10' into rwlee/struct_eqns
rwlee Aug 5, 2022
d35966e
remove extra whitespace and update docs
rwlee Aug 12, 2022
e76fdb6
Merge remote-tracking branch 'pub/branch-22.10' into rwlee/struct_eqns
rwlee Aug 12, 2022
24fede5
Merge remote-tracking branch 'pub/branch-22.10' into rwlee/struct_eqns
rwlee Aug 12, 2022
a95a417
Merge remote-tracking branch 'pub/branch-22.10' into rwlee/struct_eqns
rwlee Aug 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/supported_ops.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5636,7 +5636,7 @@ are limited.
<td><b>NS</b></td>
<td><b>NS</b></td>
<td> </td>
<td><b>NS</b></td>
<td><em>PS<br/>UTC is only supported TZ for child TIMESTAMP;<br/>unsupported child types BINARY, CALENDAR, ARRAY, UDT</em></td>
<td><b>NS</b></td>
</tr>
<tr>
Expand All @@ -5657,7 +5657,7 @@ are limited.
<td><b>NS</b></td>
<td><b>NS</b></td>
<td> </td>
<td><b>NS</b></td>
<td><em>PS<br/>UTC is only supported TZ for child TIMESTAMP;<br/>unsupported child types BINARY, CALENDAR, ARRAY, UDT</em></td>
<td><b>NS</b></td>
</tr>
<tr>
Expand Down
4 changes: 1 addition & 3 deletions integration_tests/src/main/python/cmp_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,8 @@ def test_func(data_gen):
data_gens = [DayTimeIntervalGen()]
for data_gen in data_gens:
test_func(data_gen)


@pytest.mark.parametrize('data_gen', eq_gens_with_decimal_gen, ids=idfn)
@pytest.mark.parametrize('data_gen', eq_gens_with_decimal_gen + struct_gens_sample_with_decimal128_no_list, ids=idfn)
def test_eq_ns(data_gen):
(s1, s2) = gen_scalars(data_gen, 2, force_no_nulls=not isinstance(data_gen, NullGen))
data_type = data_gen.data_type
Expand Down Expand Up @@ -170,7 +169,6 @@ def test_func(data_gen):
for data_gen in data_gens:
test_func(data_gen)


@pytest.mark.parametrize('data_gen', orderable_gens, ids=idfn)
def test_gt(data_gen):
(s1, s2) = gen_scalars(data_gen, 2, force_no_nulls=not isinstance(data_gen, NullGen))
Expand Down
8 changes: 4 additions & 4 deletions sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1958,11 +1958,11 @@ object GpuOverrides extends Logging {
"Check if the values are equal including nulls <=>",
ExprChecks.binaryProject(
TypeSig.BOOLEAN, TypeSig.BOOLEAN,
("lhs", TypeSig.commonCudfTypes + TypeSig.NULL + TypeSig.DECIMAL_128 +
GpuTypeShims.additionalPredicateSupportedTypes,
("lhs", (TypeSig.commonCudfTypes + TypeSig.NULL + TypeSig.DECIMAL_128 +
GpuTypeShims.additionalPredicateSupportedTypes + TypeSig.STRUCT).nested(),
TypeSig.comparable),
("rhs", TypeSig.commonCudfTypes + TypeSig.NULL + TypeSig.DECIMAL_128 +
GpuTypeShims.additionalPredicateSupportedTypes,
("rhs", (TypeSig.commonCudfTypes + TypeSig.NULL + TypeSig.DECIMAL_128 +
GpuTypeShims.additionalPredicateSupportedTypes + TypeSig.STRUCT).nested(),
TypeSig.comparable)),
(a, conf, p, r) => new BinaryExprMeta[EqualNullSafe](a, conf, p, r) {
override def convertToGpu(lhs: Expression, rhs: Expression): GpuExpression =
Expand Down
4 changes: 2 additions & 2 deletions tools/src/main/resources/supportedExprs.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,8 +176,8 @@ ElementAt,S,`element_at`,None,project,result,S,S,S,S,S,S,S,S,PS,S,S,S,S,NS,PS,PS
EndsWith,S, ,None,project,src,NA,NA,NA,NA,NA,NA,NA,NA,NA,S,NA,NA,NA,NA,NA,NA,NA,NA
EndsWith,S, ,None,project,search,NA,NA,NA,NA,NA,NA,NA,NA,NA,PS,NA,NA,NA,NA,NA,NA,NA,NA
EndsWith,S, ,None,project,result,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
EqualNullSafe,S,`<=>`,None,project,lhs,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,NS,NA,NS,NS
EqualNullSafe,S,`<=>`,None,project,rhs,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,NS,NA,NS,NS
EqualNullSafe,S,`<=>`,None,project,lhs,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,NS,NA,PS,NS
EqualNullSafe,S,`<=>`,None,project,rhs,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,NS,NA,PS,NS
EqualNullSafe,S,`<=>`,None,project,result,S,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
EqualTo,S,`=`; `==`,None,project,lhs,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,NS,NA,PS,NS
EqualTo,S,`=`; `==`,None,project,rhs,S,S,S,S,S,S,S,S,PS,S,S,S,NS,NS,NS,NA,PS,NS
Expand Down