Skip to content

Commit 3d72063

Browse files
JoshRosendongjoon-hyun
authored andcommitted
[SPARK-48081] Fix ClassCastException in NTile.checkInputDataTypes() when argument is non-foldable or of wrong type
### What changes were proposed in this pull request? While migrating the `NTile` expression's type check failures to the new error class framework, PR #38457 removed a pair of not-unnecessary `return` statements and thus caused certain branches' values to be discarded rather than returned. As a result, invalid usages like ``` select ntile(99.9) OVER (order by id) from range(10) ``` trigger internal errors like errors like ``` java.lang.ClassCastException: class org.apache.spark.sql.types.Decimal cannot be cast to class java.lang.Integer (org.apache.spark.sql.types.Decimal is in unnamed module of loader 'app'; java.lang.Integer is in module java.base of loader 'bootstrap') at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:99) at org.apache.spark.sql.catalyst.expressions.NTile.checkInputDataTypes(windowExpressions.scala:877) ``` instead of clear error framework errors like ``` org.apache.spark.sql.catalyst.ExtendedAnalysisException: [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "ntile(99.9)" due to data type mismatch: The first parameter requires the "INT" type, however "99.9" has the type "DECIMAL(3,1)". SQLSTATE: 42K09; line 1 pos 7; 'Project [unresolvedalias(ntile(99.9) windowspecdefinition(id#0L ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())))] +- Range (0, 10, step=1, splits=None) at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.dataTypeMismatch(package.scala:73) at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$7(CheckAnalysis.scala:315) ``` ### Why are the changes needed? Improve error messages. ### Does this PR introduce _any_ user-facing change? Yes, it improves an error message. ### How was this patch tested? Added a new test case to AnalysisErrorSuite. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46333 from JoshRosen/SPARK-48081. Authored-by: Josh Rosen <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit b99a64b) Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent b6b8d61 commit 3d72063

File tree

2 files changed

+36
-2
lines changed

2 files changed

+36
-2
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -850,7 +850,7 @@ case class NTile(buckets: Expression) extends RowNumberLike with SizeBasedWindow
850850
// for each partition.
851851
override def checkInputDataTypes(): TypeCheckResult = {
852852
if (!buckets.foldable) {
853-
DataTypeMismatch(
853+
return DataTypeMismatch(
854854
errorSubClass = "NON_FOLDABLE_INPUT",
855855
messageParameters = Map(
856856
"inputName" -> "buckets",
@@ -861,7 +861,7 @@ case class NTile(buckets: Expression) extends RowNumberLike with SizeBasedWindow
861861
}
862862

863863
if (buckets.dataType != IntegerType) {
864-
DataTypeMismatch(
864+
return DataTypeMismatch(
865865
errorSubClass = "UNEXPECTED_INPUT_TYPE",
866866
messageParameters = Map(
867867
"paramIndex" -> "1",

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -344,6 +344,40 @@ class AnalysisErrorSuite extends AnalysisTest {
344344
"inputType" -> "\"BOOLEAN\"",
345345
"requiredType" -> "\"INT\""))
346346

347+
errorClassTest(
348+
"the buckets of ntile window function is not foldable",
349+
testRelation2.select(
350+
WindowExpression(
351+
NTile(Literal(99.9f)),
352+
WindowSpecDefinition(
353+
UnresolvedAttribute("a") :: Nil,
354+
SortOrder(UnresolvedAttribute("b"), Ascending) :: Nil,
355+
UnspecifiedFrame)).as("window")),
356+
errorClass = "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
357+
messageParameters = Map(
358+
"sqlExpr" -> "\"ntile(99.9)\"",
359+
"paramIndex" -> "first",
360+
"inputSql" -> "\"99.9\"",
361+
"inputType" -> "\"FLOAT\"",
362+
"requiredType" -> "\"INT\""))
363+
364+
365+
errorClassTest(
366+
"the buckets of ntile window function is not int literal",
367+
testRelation2.select(
368+
WindowExpression(
369+
NTile(AttributeReference("b", IntegerType)()),
370+
WindowSpecDefinition(
371+
UnresolvedAttribute("a") :: Nil,
372+
SortOrder(UnresolvedAttribute("b"), Ascending) :: Nil,
373+
UnspecifiedFrame)).as("window")),
374+
errorClass = "DATATYPE_MISMATCH.NON_FOLDABLE_INPUT",
375+
messageParameters = Map(
376+
"sqlExpr" -> "\"ntile(b)\"",
377+
"inputName" -> "`buckets`",
378+
"inputExpr" -> "\"b\"",
379+
"inputType" -> "\"INT\""))
380+
347381
errorClassTest(
348382
"unresolved attributes",
349383
testRelation.select($"abcd"),

0 commit comments

Comments
 (0)