[SPARK-36107][SQL] Refactor first set of 20 query execution errors to use error classes #33538

Peng-Lei · 2021-07-27T09:35:43Z

What changes were proposed in this pull request?

Refactor some exceptions in QueryExecutionErrors to use error classes. as follows:

columnChangeUnsupportedError
logicalHintOperatorNotRemovedDuringAnalysisError
cannotEvaluateExpressionError
cannotGenerateCodeForExpressionError
cannotTerminateGeneratorError
castingCauseOverflowError
cannotChangeDecimalPrecisionError
invalidInputSyntaxForNumericError
cannotCastFromNullTypeError
cannotCastError
cannotParseDecimalError
simpleStringWithNodeIdUnsupportedError
evaluateUnevaluableAggregateUnsupportedError
dataTypeUnsupportedError
dataTypeUnsupportedError
failedExecuteUserDefinedFunctionError
divideByZeroError
invalidArrayIndexError
mapKeyNotExistError
rowFromCSVParserNotExpectedError

Why are the changes needed?

SPARK-36107

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existed UT Testcase

AmplabJenkins · 2021-07-27T09:36:57Z

Can one of the admins verify this patch?

Peng-Lei · 2021-07-27T12:19:37Z

@karenfeng FYI

karenfeng

Thanks for working on this! We'll need to create some new SparkThrowable implementations; if we simply throw the existing exception types, the error class and SQLSTATE are not propagated through.

core/src/main/resources/error/error-classes.json

karenfeng · 2021-07-27T17:52:09Z

sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala

Can you add a new exception type that mixes UnsupportedOperationException with SparkThrowable? It'd look like SparkArithmeticException. Then we can catch these exceptions and use their error classes/SQLSTATEs.

Of course, I'll do it. Thank you.

@karenfeng I also add SparkUnsupportedOperationException, SparkIllegalStateException,SparkNumberFormatException,SparkIllegalArgumentException,SparkArrayIndexOutOfBoundsException,SparkNoSuchElementException that with SparkThrowable.

sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala

core/src/main/resources/error/error-classes.json

karenfeng · 2021-07-28T17:48:46Z

core/src/main/resources/error/error-classes.json

I believe that HY represents CLI-specific conditions. This seems to be triggered by overflows; maybe 22023 is a better fit.

Yeah. Maybe 22023 is a better fit. The exception is triggered by invalid parameter value. Thank you.

karenfeng

Thanks for doing this! LGTM. @HyukjinKwon, can you also take a look?

1. add SparkUnsupportedOperationException SparkIllegalStateException SparkNumberFormatException SparkIllegalArgumentException SparkArrayIndexOutOfBoundsException SparkNoSuchElementException 2. change the sqlstate suitable code and order the error in error-class.json 3. others

Peng-Lei · 2021-08-24T07:54:42Z

@HyukjinKwon Could you take a look?

…xecutionErrors ### What changes were proposed in this pull request? When we refactor the query execution errors to use error classes in QueryExecutionErrors, we need define some exception that mix SparkThrowable into a base Exception type. according the example [SparkArithmeticException](https://github.com/apache/spark/blob/f90eb6a5db0778fd18b0b544f93eac3103bbf03b/core/src/main/scala/org/apache/spark/SparkException.scala#L75) Add SparkXXXException as follows: - `SparkClassNotFoundException` - `SparkConcurrentModificationException` - `SparkDateTimeException` - `SparkFileAlreadyExistsException` - `SparkFileNotFoundException` - `SparkNoSuchMethodException` - `SparkIndexOutOfBoundsException` - `SparkIOException` - `SparkSecurityException` - `SparkSQLException` - `SparkSQLFeatureNotSupportedException` Refactor some exceptions in QueryExecutionErrors to use error classes and new exception for testing new exception Some added by [PR](#33538) as follows: - `SparkUnsupportedOperationException` - `SparkIllegalStateException` - `SparkNumberFormatException` - `SparkIllegalArgumentException` - `SparkArrayIndexOutOfBoundsException` - `SparkNoSuchElementException` ### Why are the changes needed? [SPARK-36336](https://issues.apache.org/jira/browse/SPARK-36336) ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? existed ut test Closes #33573 from Peng-Lei/SPARK-36336. Authored-by: PengLei <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

Peng-Lei · 2021-08-25T07:50:06Z

core/src/main/resources/error/error-classes.json

  },
+  "FAILED_EXECUTE_UDF" : {
+    "message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ],
+    "sqlState" : "45000"


The SQLSTATE '45000' is not valid. Because it is not included in README

@karenfeng I delete it.

HyukjinKwon · 2021-08-26T00:43:45Z

seems like the tests fail. Can you fix it up please?

Peng-Lei · 2021-08-26T02:09:14Z

seems like the tests fail. Can you fix it up please?

Thank you. Done

Peng-Lei · 2021-08-27T04:54:35Z

cc @cloud-fan @gengliangwang @HyukjinKwon

HyukjinKwon · 2021-09-12T01:17:40Z

Sorry I saw this late. There have been few changes made. @karenfeng mind double checking before we go ahead please?

karenfeng

Thanks for working on this @Peng-Lei! I left some comments - please tag me once you address them.

core/src/main/resources/error/error-classes.json

Peng-Lei · 2021-09-17T06:17:45Z

@karenfeng I've already address the comments. Please check. Thank you.

karenfeng

LGTM, thanks @Peng-Lei! @HyukjinKwon, can you help take a second look?

HyukjinKwon · 2021-09-20T01:34:06Z

Merged to master.

Peng-Lei · 2021-09-20T09:17:37Z

@karenfeng @HyukjinKwon Thank you！

MaxGekk · 2022-08-23T19:19:56Z

core/src/main/resources/error/error-classes.json

    "message" : [ "Field name %s is ambiguous and has %s matching fields in the struct." ],
    "sqlState" : "42000"
  },
+  "CANNOT_CAST_DATATYPE" : {


Any ideas how to trigger this error class? I haven't found any ways how trigger it from user code. WDYT? @cloud-fan @HyukjinKwon

github-actions bot added CORE SQL labels Jul 27, 2021

Peng-Lei force-pushed the SPARK-36017 branch from 0b4f205 to 4e2f264 Compare July 27, 2021 10:12

Peng-Lei changed the title ~~[WIP][SPARK-36107][SQL] Refactor first set of 20 query execution errors to use error classes~~ [SPARK-36107][SQL] Refactor first set of 20 query execution errors to use error classes Jul 27, 2021

Peng-Lei force-pushed the SPARK-36017 branch 2 times, most recently from 245b6a3 to 6ec85b9 Compare July 27, 2021 12:17

Peng-Lei force-pushed the SPARK-36017 branch from 6ec85b9 to 0f040e5 Compare July 27, 2021 12:57

karenfeng reviewed Jul 27, 2021

View reviewed changes

core/src/main/resources/error/error-classes.json Outdated Show resolved Hide resolved

Peng-Lei force-pushed the SPARK-36017 branch 8 times, most recently from 43d06f0 to 8f1508c Compare July 28, 2021 13:31

karenfeng reviewed Jul 28, 2021

View reviewed changes

Peng-Lei force-pushed the SPARK-36017 branch 2 times, most recently from 98a1563 to b44e8fa Compare July 29, 2021 06:13

Peng-Lei mentioned this pull request Jul 29, 2021

[SPARK-36336][SQL] Add new exception of base exception used in QueryExecutionErrors #33573

Closed

karenfeng approved these changes Jul 30, 2021

View reviewed changes

Peng-Lei force-pushed the SPARK-36017 branch 2 times, most recently from 98d7635 to db6d8d3 Compare August 23, 2021 09:20

use error classes to refact the first 20 errors in QueryExecutionErrors

9025b97

Peng-Lei force-pushed the SPARK-36017 branch from db6d8d3 to baac91f Compare August 23, 2021 12:40

Peng-Lei force-pushed the SPARK-36017 branch from baac91f to 14ec185 Compare August 23, 2021 12:55

Merge branch 'master' into SPARK-36017

0546f96

Peng-Lei force-pushed the SPARK-36017 branch from 758504a to 0546f96 Compare August 25, 2021 07:39

Peng-Lei commented Aug 25, 2021

View reviewed changes

Recheck the SQLSTATE to ensure it is valid.

6bfb10c

Peng-Lei force-pushed the SPARK-36017 branch from defeacb to 6bfb10c Compare August 26, 2021 01:32

mark the exception as private[spark]

13dfe5b

Merge branch 'master' into SPARK-36017

5a8e5f9

karenfeng reviewed Sep 16, 2021

View reviewed changes

Peng-Lei added 2 commits September 17, 2021 10:37

Merge branch 'apache:master' into SPARK-36017

32d2e6f

address comments

1f979c8

karenfeng approved these changes Sep 17, 2021

View reviewed changes

HyukjinKwon closed this in c2881c5 Sep 20, 2021

Peng-Lei deleted the SPARK-36017 branch September 20, 2021 09:16

MaxGekk reviewed Aug 23, 2022

View reviewed changes

[SPARK-36107][SQL] Refactor first set of 20 query execution errors to use error classes #33538

[SPARK-36107][SQL] Refactor first set of 20 query execution errors to use error classes #33538

Uh oh!

Conversation

Peng-Lei commented Jul 27, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

AmplabJenkins commented Jul 27, 2021

Uh oh!

Peng-Lei commented Jul 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

karenfeng left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

karenfeng Jul 27, 2021

Choose a reason for hiding this comment

Uh oh!

Peng-Lei Jul 28, 2021

Choose a reason for hiding this comment

Uh oh!

Peng-Lei Jul 28, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

karenfeng Jul 28, 2021

Choose a reason for hiding this comment

Uh oh!

Peng-Lei Jul 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karenfeng left a comment

Choose a reason for hiding this comment

Uh oh!

Peng-Lei commented Aug 24, 2021

Uh oh!

Peng-Lei Aug 25, 2021

Choose a reason for hiding this comment

Uh oh!

Peng-Lei Aug 25, 2021

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Aug 26, 2021

Uh oh!

Peng-Lei commented Aug 26, 2021

Uh oh!

Peng-Lei commented Aug 27, 2021

Uh oh!

HyukjinKwon commented Sep 12, 2021

Uh oh!

karenfeng left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Peng-Lei commented Sep 17, 2021

Uh oh!

karenfeng left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Sep 20, 2021

Peng-Lei commented Jul 27, 2021 •

edited

Loading

Peng-Lei Jul 29, 2021 •

edited

Loading