Skip to content

Commit 38ad0e7

Browse files
KimahrimanMaxGekk
andcommitted
[SPARK-49476][SQL] Fix nullability of base64 function
### What changes were proposed in this pull request? Fix the nullability of the `Base64` expression to be based on the child's nullability, and not always be nullable. ### Why are the changes needed? apache#47303 had a side effect of changing the nullability by the switch to using `StaticInvoke`. This was also backported to Spark 3.5.2 and caused schema mismatch errors for stateful streams when we upgraded. This restores the previous behavior which is supported by StaticInvoke through the `returnNullable` argument. If the child is non-nullable, we know the result will be non-nullable. ### Does this PR introduce _any_ user-facing change? Restores the nullability of the `Base64` expression to what is was in Spark 3.5.1 and earlier. ### How was this patch tested? New UT ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#47941 from Kimahriman/base64-nullability. Lead-authored-by: Adam Binford <[email protected]> Co-authored-by: Maxim Gekk <[email protected]> Signed-off-by: Max Gekk <[email protected]> (cherry picked from commit c274c5a) Signed-off-by: Max Gekk <[email protected]>
1 parent d5caaaa commit 38ad0e7

File tree

2 files changed

+9
-1
lines changed

2 files changed

+9
-1
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2432,7 +2432,8 @@ case class Base64(child: Expression, chunkBase64: Boolean)
24322432
dataType,
24332433
"encode",
24342434
Seq(child, Literal(chunkBase64, BooleanType)),
2435-
Seq(BinaryType, BooleanType))
2435+
Seq(BinaryType, BooleanType),
2436+
returnNullable = false)
24362437

24372438
override def toString: String = s"$prettyName($child)"
24382439

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -464,6 +464,13 @@ class StringExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
464464
val b = $"b".binary.at(0)
465465
val bytes = Array[Byte](1, 2, 3, 4)
466466

467+
assert(!Base64(Literal(bytes)).nullable)
468+
assert(Base64(Literal.create(null, BinaryType)).nullable)
469+
assert(Base64(Literal(bytes).castNullable()).nullable)
470+
assert(!UnBase64(Literal("AQIDBA==")).nullable)
471+
assert(UnBase64(Literal.create(null, StringType)).nullable)
472+
assert(UnBase64(Literal("AQIDBA==").castNullable()).nullable)
473+
467474
checkEvaluation(Base64(Literal(bytes)), "AQIDBA==", create_row("abdef"))
468475
checkEvaluation(Base64(UnBase64(Literal("AQIDBA=="))), "AQIDBA==", create_row("abdef"))
469476
checkEvaluation(Base64(UnBase64(Literal(""))), "", create_row("abdef"))

0 commit comments

Comments
 (0)