Commit e3ba74b
committed
[SPARK-49200][SQL] Fix null type non-codegen ordering exception
### What changes were proposed in this pull request?
Spark mark `NullType` as orderable, and we return 0 when gen comparing code for `NullType`.
```
object OrderUtils {
def isOrderable(dataType: DataType): Boolean = dataType match {
case NullType => true
```
This pr makes `NullType` ordering work for non-codegen path to avoid exception.
### Why are the changes needed?
Fix exception:
```sql
set spark.sql.codegen.factoryMode=NO_CODEGEN;
set spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.EliminateSorts;
select * from range(10) order by array(null);
```
```
org.apache.spark.SparkIllegalArgumentException: Type PhysicalNullType does not support ordered operations.
at org.apache.spark.sql.errors.QueryExecutionErrors$.orderedOperationUnsupportedByDataTypeError(QueryExecutionErrors.scala:352)
at org.apache.spark.sql.catalyst.types.PhysicalNullType.ordering(PhysicalDataType.scala:246)
at org.apache.spark.sql.catalyst.types.PhysicalNullType.ordering(PhysicalDataType.scala:243)
at org.apache.spark.sql.catalyst.types.PhysicalArrayType$$anon$1.<init>(PhysicalDataType.scala:283)
at org.apache.spark.sql.catalyst.types.PhysicalArrayType.interpretedOrdering$lzycompute(PhysicalDataType.scala:281)
at org.apache.spark.sql.catalyst.types.PhysicalArrayType.interpretedOrdering(PhysicalDataType.scala:281)
at org.apache.spark.sql.catalyst.types.PhysicalArrayType.ordering(PhysicalDataType.scala:277)
at org.apache.spark.sql.catalyst.expressions.InterpretedOrdering.compare(ordering.scala:67)
at org.apache.spark.sql.catalyst.expressions.InterpretedOrdering.compare(ordering.scala:40)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter$RowComparator.compare(UnsafeExternalRowSorter.java:254)
at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter$SortComparator.compare(UnsafeInMemorySorter.java:70)
at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter$SortComparator.compare(UnsafeInMemorySorter.java:44)
```
### Does this PR introduce _any_ user-facing change?
yes, bug fix
### How was this patch tested?
add test
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #47707 from ulysses-you/null-ordering.
Authored-by: ulysses-you <[email protected]>
Signed-off-by: youxiduo <[email protected]>1 parent 2fb8dff commit e3ba74b
File tree
2 files changed
+13
-3
lines changed- sql
- catalyst/src/main/scala/org/apache/spark/sql/catalyst/types
- core/src/test/scala/org/apache/spark/sql
2 files changed
+13
-3
lines changedLines changed: 1 addition & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
242 | 242 | | |
243 | 243 | | |
244 | 244 | | |
245 | | - | |
246 | | - | |
| 245 | + | |
247 | 246 | | |
248 | 247 | | |
249 | 248 | | |
| |||
Lines changed: 12 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
34 | | - | |
| 34 | + | |
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| |||
1430 | 1430 | | |
1431 | 1431 | | |
1432 | 1432 | | |
| 1433 | + | |
| 1434 | + | |
| 1435 | + | |
| 1436 | + | |
| 1437 | + | |
| 1438 | + | |
| 1439 | + | |
| 1440 | + | |
| 1441 | + | |
| 1442 | + | |
| 1443 | + | |
1433 | 1444 | | |
1434 | 1445 | | |
1435 | 1446 | | |
| |||
0 commit comments