[SPARK-43482][SS] Expand QueryTerminatedEvent to contain error class if it exists in exception #41150
Conversation
|
cc. @zsxwing @viirya @HyukjinKwon Please take a look. Thanks! |
| * @param errorClassOnException The error class from the exception if the query was terminated | ||
| * with an exception which is a part of error class framework. |
There was a problem hiding this comment.
Is error class framework a public API that end-users understand or can find document?
There was a problem hiding this comment.
SparkException and SparkThrowable are all public API where we expose to API doc. It's just that we lack about documentation for these classes. That's something we have to fix; we could probably refer to SparkThrowable.getErrorClass here, but it's not documented so wouldn't help unfortunately.
There was a problem hiding this comment.
Do we think it'd be better to not mention about error class framework and say "this gives an error classification if the exception is available and classified"? If then I can avoid mentioning error class framework.
@HyukjinKwon WDYT? cc. @MaxGekk
There was a problem hiding this comment.
I think it's fine .. it's sort of documented ..e.g., https://github.com/apache/spark/blob/master/docs/sql-error-conditions-connect-error-class.md
There was a problem hiding this comment.
https://spark.apache.org/docs/latest/sql-error-conditions.html
Above is the landing page on website. It's in SQL guide doc.
Co-authored-by: Hyukjin Kwon <[email protected]>
|
Merged to master. |
3 participants
What changes were proposed in this pull request?
This adds a new field in QueryTerminatedEvent about "error class" if the information exists in the exception. Note that it doesn't retrieve the information from StreamingQueryException, because it is marked as it's own error class and we want to know the causing exception.
Why are the changes needed?
Custom streaming query listeners can get the "categorized" error as cause of the query termination, which can be used to determine trends of the errors, aggregation, etc.
Does this PR introduce any user-facing change?
Yes, users having customer streaming query listener will have additional information in termination event.
How was this patch tested?
Modified UTs.