[SPARK-38001][SQL] Replace the error classes related to unsupported features by `UNSUPPORTED_FEATURE` #35302

MaxGekk · 2022-01-24T17:29:52Z

What changes were proposed in this pull request?

In the PR, I propose to re-use one error class UNSUPPORTED_FEATURE in the following Spark's exceptions:

QueryCompilationErrors.unsupportedIfNotExistsError - when IF NOT EXISTS is not supported by INSERT INTO.
QueryExecutionErrors.aesModeUnsupportedError - when an user specify unsupported AES mode and padding.
QueryExecutionErrors.literalTypeUnsupportedError - impossible to create a literal from the input value (some Java class, for instance).
QueryExecutionErrors.transactionUnsupportedByJdbcServerError - the target JDBC server does not support transaction.

And replace the following exceptions by IllegalStateException since they are internal, and should be not visible to users:

QueryExecutionErrors.simpleStringWithNodeIdUnsupportedError - a sub-class of Expression or Block doesn't implements the method simpleStringWithNodeId().
QueryExecutionErrors.dataTypeUnsupportedForWriterFuncError - generating of a writer function for a struct field, array element, map key or map value doesn't support Catalyst's type.

Also, added new base test suite QueryCompilationErrorsDSv2Suite for testing DSv2 specific compilation error classes.

Why are the changes needed?

Reducing the number of error classes should prevent from explode of error-classes.json. Also, using one error class for similar errors should improve user experience with Spark SQL.

Does this PR introduce any user-facing change?

Yes.

How was this patch tested?

By running the affected test suites:

$ build/sbt "test:testOnly *SparkThrowableSuite"
$ build/sbt "test:testOnly *QueryExecutionErrorsSuite"
$ build/sbt "test:testOnly *DataSourceV2SQLSuite"
$ build/sbt "test:testOnly *DataSourceV2DataFrameSessionCatalogSuite"
$ build/sbt "test:testOnly *DataFramePivotSuite"

and the new test suite:

$ build/sbt "test:testOnly *QueryCompilationErrorsDSv2Suite"

…E_ID

…supportedError()

cloud-fan · 2022-01-26T06:09:37Z

...t/src/main/scala/org/apache/spark/sql/catalyst/expressions/InterpretedUnsafeProjection.scala


      case _ =>
-        throw QueryExecutionErrors.dataTypeUnsupportedError(dt)
+        throw QueryExecutionErrors.dataTypeUnsupportedForWriterFuncError(dt)


seems we can never reach here unless we have a bug. maybe we don't need error class here and can throw IllegalStateException directly?

How about to not handle the default case at all (I mean remove it). The compiler should output a warning if we forget to handle a type, and we.will get a match error exception in the case too.

After offline discussion, I'm going to replace the exception by IllegalStateException.

cloud-fan · 2022-01-26T06:10:55Z

sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala

+  test("UNSUPPORTED_FEATURE: unsupported types (map and struct) in Literal.apply") {
+    def checkUnsupportedTypeInLiteral(v: Any): Unit = {
+      val e = intercept[SparkRuntimeException] {
+        Literal(v)


This is not a user-facing API. Can we use functions.lit?

cloud-fan · 2022-01-26T06:28:16Z

sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala

        pivotVal.toString, pivotVal.dataType.simpleString, pivotCol.dataType.catalogString))
  }

  def unsupportedIfNotExistsError(tableName: String): Throwable = {


shall we add a QueryCompilationErrorsSuite and test this error?

InsertIntoSQLOnlyTests has a test for the error already. BTW, the test from InsertIntoSQLOnlyTests runs as part of:

V1WriteFallbackSessionCatalogSuite

DataSourceV2SQLSuite

DataSourceV2SQLSessionCatalogSuite

DataSourceV2DataFrameSuite

DataSourceV2DataFrameSessionCatalogSuite

@cloud-fan How about to create the QueryCompilationErrorsSuiteBase trait/abstract class and include it to those test suites?

I added the base test suites QueryCompilationErrorsSuiteBase + SessionCatalogTestBase and new test suites for compilation errors:

QueryCompilationErrorsDSv2Suite

QueryCompilationErrorsDSv2SessionCatalogSuite

QueryCompilationErrorsV1WriteFallbackSuite

cloud-fan · 2022-01-26T06:28:51Z

sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala

-      messageParameters = Array(nodeName))
+    new SparkUnsupportedOperationException(
+      errorClass = "UNSUPPORTED_FEATURE",
+      messageParameters = Array(s"$nodeName does not implement simpleStringWithNodeId"))


this does not sound like a user-facing error. Can we double-check it?

Seems, it is not. At least, I haven't found a way how to trigger it from user level. I will replace the exception by IllegalStateException as for #35302 (comment).

MaxGekk · 2022-01-27T12:47:12Z

@cloud-fan Could you look at this one more time, please.

cloud-fan · 2022-01-27T13:57:07Z

sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala

+        .agg(sum($"sales.earnings"))
+        .collect()
+    }
+    assert(e2.getMessage === "The feature is not supported: " +


TBH, this message is quite misleading, as the query does not create literals directly. We can improve it later.

I opened the ticket SPARK-38097 to improve the error.

cloud-fan · 2022-01-27T13:57:24Z

thanks, merging to master!

…eatures by `UNSUPPORTED_FEATURE` In the PR, I propose to re-use one error class `UNSUPPORTED_FEATURE` in the following Spark's exceptions: - `QueryCompilationErrors.unsupportedIfNotExistsError` - when `IF NOT EXISTS` is not supported by `INSERT INTO`. - `QueryExecutionErrors.aesModeUnsupportedError` - when an user specify unsupported AES mode and padding. - `QueryExecutionErrors.literalTypeUnsupportedError` - impossible to create a literal from the input value (some Java class, for instance). - `QueryExecutionErrors.transactionUnsupportedByJdbcServerError` - the target JDBC server does not support transaction. And replace the following exceptions by `IllegalStateException` since they are internal, and should be not visible to users: - `QueryExecutionErrors.simpleStringWithNodeIdUnsupportedError` - a sub-class of `Expression` or `Block` doesn't implements the method `simpleStringWithNodeId()`. - `QueryExecutionErrors.dataTypeUnsupportedForWriterFuncError` - generating of a writer function for a struct field, array element, map key or map value doesn't support Catalyst's type. Also, added new base test suite `QueryCompilationErrorsDSv2Suite` for testing DSv2 specific compilation error classes. Reducing the number of error classes should prevent from explode of `error-classes.json`. Also, using one error class for similar errors should improve user experience with Spark SQL. Yes. By running the affected test suites: ``` $ build/sbt "test:testOnly *SparkThrowableSuite" $ build/sbt "test:testOnly *QueryExecutionErrorsSuite" $ build/sbt "test:testOnly *DataSourceV2SQLSuite" $ build/sbt "test:testOnly *DataSourceV2DataFrameSessionCatalogSuite" $ build/sbt "test:testOnly *DataFramePivotSuite" ``` and the new test suite: ``` $ build/sbt "test:testOnly *QueryCompilationErrorsDSv2Suite" ``` Closes apache#35302 from MaxGekk/re-use-unsupported_feature-error-class. Authored-by: Max Gekk <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

MaxGekk added 2 commits January 24, 2022 20:18

Reuse UNSUPPORTED_FEATURE from literalTypeUnsupportedError

bcaa235

Fix style

f026997

github-actions bot added CORE SQL labels Jan 24, 2022

MaxGekk added 6 commits January 25, 2022 10:01

Fix DataFramePivotSuite

a903ad3

Move a test from LiteralExpressionSuite

99c7913

Use UNSUPPORTED_FEATURE instead of UNSUPPORTED_SIMPLE_STRING_WITH_NOD…

32884da

…E_ID

UNSUPPORTED_TRANSACTION_BY_JDBC_SERVER -> UNSUPPORTED_FEATURE

9e67cab

IF_PARTITION_NOT_EXISTS_UNSUPPORTED -> UNSUPPORTED_FEATURE

fc7de5c

Use UNSUPPORTED_FEATURE instead of UNSUPPORTED_DATATYPE in dataTypeUn…

0a0d19f

…supportedError()

MaxGekk changed the title ~~[WIP][SPARK-38001][SQL] Replace the unsupported error classes by UNSUPPORTED_FEATURE~~ [SPARK-38001][SQL] Replace the unsupported error classes by UNSUPPORTED_FEATURE Jan 25, 2022

MaxGekk marked this pull request as ready for review January 25, 2022 16:01

MaxGekk requested a review from cloud-fan January 25, 2022 16:01

MaxGekk changed the title ~~[SPARK-38001][SQL] Replace the unsupported error classes by UNSUPPORTED_FEATURE~~ [SPARK-38001][SQL] Replace the error classes related to unsupported features by UNSUPPORTED_FEATURE Jan 25, 2022

cloud-fan reviewed Jan 26, 2022

View reviewed changes

MaxGekk added 5 commits January 26, 2022 12:47

Use IllegalStateException instead of Spark's exception

27cdaab

Literal -> lit

0702ccc

Remove simpleStringWithNodeIdUnsupportedError

a2af415

Add new test suite for compilation errors

c72f9b2

Add SessionCatalogTestBase

d9c77fd

MaxGekk requested a review from cloud-fan January 26, 2022 20:47

MaxGekk added 4 commits January 27, 2022 11:48

Remove duplicated V2 test suites.

fedac2f

Remove unnecessary changes

c0eee62

Match to entire error message

ffab56d

Move a pivot test to QueryExecutionErrorsSuite

ff9a6a5

cloud-fan reviewed Jan 27, 2022

View reviewed changes

cloud-fan approved these changes Jan 27, 2022

View reviewed changes

cloud-fan closed this in 88e8006 Jan 27, 2022

MaxGekk mentioned this pull request Feb 4, 2022

[SPARK-37936][SQL] Use error classes in the parsing errors of intervals #35390

Closed

ivoson mentioned this pull request Feb 6, 2022

[SPARK-38105][SQL] Use error classes in the parsing errors of joins #35405

Closed

[SPARK-38001][SQL] Replace the error classes related to unsupported features by UNSUPPORTED_FEATURE #35302

[SPARK-38001][SQL] Replace the error classes related to unsupported features by UNSUPPORTED_FEATURE #35302

Uh oh!

Conversation

MaxGekk commented Jan 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaxGekk commented Jan 27, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Jan 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[SPARK-38001][SQL] Replace the error classes related to unsupported features by `UNSUPPORTED_FEATURE` #35302

[SPARK-38001][SQL] Replace the error classes related to unsupported features by `UNSUPPORTED_FEATURE` #35302

MaxGekk commented Jan 24, 2022 •

edited

Loading

cloud-fan commented Jan 27, 2022 •

edited

Loading