Commit 38ad0e7
[SPARK-49476][SQL] Fix nullability of base64 function
### What changes were proposed in this pull request?
Fix the nullability of the `Base64` expression to be based on the child's nullability, and not always be nullable.
### Why are the changes needed?
apache#47303 had a side effect of changing the nullability by the switch to using `StaticInvoke`. This was also backported to Spark 3.5.2 and caused schema mismatch errors for stateful streams when we upgraded. This restores the previous behavior which is supported by StaticInvoke through the `returnNullable` argument. If the child is non-nullable, we know the result will be non-nullable.
### Does this PR introduce _any_ user-facing change?
Restores the nullability of the `Base64` expression to what is was in Spark 3.5.1 and earlier.
### How was this patch tested?
New UT
### Was this patch authored or co-authored using generative AI tooling?
No
Closes apache#47941 from Kimahriman/base64-nullability.
Lead-authored-by: Adam Binford <[email protected]>
Co-authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit c274c5a)
Signed-off-by: Max Gekk <[email protected]>1 parent d5caaaa commit 38ad0e7
File tree
2 files changed
+9
-1
lines changed- sql/catalyst/src
- main/scala/org/apache/spark/sql/catalyst/expressions
- test/scala/org/apache/spark/sql/catalyst/expressions
2 files changed
+9
-1
lines changedLines changed: 2 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2432 | 2432 | | |
2433 | 2433 | | |
2434 | 2434 | | |
2435 | | - | |
| 2435 | + | |
| 2436 | + | |
2436 | 2437 | | |
2437 | 2438 | | |
2438 | 2439 | | |
| |||
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
Lines changed: 7 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
464 | 464 | | |
465 | 465 | | |
466 | 466 | | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
467 | 474 | | |
468 | 475 | | |
469 | 476 | | |
| |||
0 commit comments