[SPARK-54128][CONNECT][SQL] Convert IllegalArgumentException to SparkException with proper error classes in Spark Connect server #52818
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
grundprinzip
force-pushed
the
SPARK-error-messages
branch
from
e3de1b6 to
c95a8da
Compare
grundprinzip
changed the title
grundprinzip
force-pushed
the
SPARK-error-messages
branch
3 times, most recently
from
8f59403 to
2ac3c88
Compare
…h proper error classes in Spark Connect server
This PR converts several IllegalArgumentException and UnsupportedOperationException
instances in the Spark Connect server to proper SparkException with structured
error classes, ensuring consistent error propagation to clients.
Changes include:
1. Added error class definitions in error-conditions.json:
- INVALID_PARAMETER_VALUE.INTERRUPT_TYPE_TAG_REQUIRES_TAG
- INVALID_PARAMETER_VALUE.INTERRUPT_TYPE_OPERATION_ID_REQUIRES_ID
- INVALID_PARAMETER_VALUE.STREAMING_LISTENER_COMMAND_MISSING
- INVALID_ARTIFACT_PATH
- UNSUPPORTED_FEATURE.INTERRUPT_TYPE
2. Updated service handlers:
- SparkConnectInterruptHandler: Converted generic exceptions to SparkSQLException
- SparkConnectAddArtifactsHandler: Converted to SparkRuntimeException for invalid paths
- SparkConnectStreamingQueryListenerHandler: Converted to SparkSQLException
3. Added test coverage in SparkConnectServiceE2ESuite
Previously, the Spark Connect server threw generic Java exceptions that:
- Did not include structured error classes
- Could not be properly categorized by clients
- Provided less actionable error information
- Were inconsistent with Spark's error handling standards
The error handling infrastructure (ErrorUtils.handleError) can only propagate
error classes from SparkThrowable instances. Generic Java exceptions are
converted to generic UNKNOWN errors, losing important context.
Yes, but only to improve error messages.
Before:
java.lang.IllegalArgumentException: INTERRUPT_TYPE_TAG requested, but no operation_tag provided.
After:
[INVALID_PARAMETER_VALUE.INTERRUPT_TYPE_TAG_REQUIRES_TAG] The value of parameter(s)
operation_tag in interrupt is invalid: INTERRUPT_TYPE_TAG requested, but no
operation_tag provided.
Clients can now parse structured error classes for better error handling and recovery.
1. Added new test cases in SparkConnectServiceE2ESuite:
- test("Interrupt with TAG type without operation_tag throws proper error class")
- test("Interrupt with OPERATION_ID type without operation_id throws proper error class")
2. Manual verification of error class propagation through the gRPC layer
3. All modified files pass Scala style checks
Generated-by: Claude Sonnet 4.5
grundprinzip
force-pushed
the
SPARK-error-messages
branch
from
2ac3c88 to
abfa696
Compare
HyukjinKwon
approved these changes
Nov 2, 2025
dongjoon-hyun
pushed a commit
that referenced
this pull request
Nov 3, 2025
…Exception with proper error classes in Spark Connect server
### What changes were proposed in this pull request?
This PR converts several IllegalArgumentException and UnsupportedOperationException instances in the Spark Connect server to proper SparkException with structured error classes, ensuring consistent error propagation to clients.
Changes include:
1. Added error class definitions in error-conditions.json:
- INVALID_PARAMETER_VALUE.INTERRUPT_TYPE_TAG_REQUIRES_TAG
- INVALID_PARAMETER_VALUE.INTERRUPT_TYPE_OPERATION_ID_REQUIRES_ID
- INVALID_PARAMETER_VALUE.STREAMING_LISTENER_COMMAND_MISSING
- INVALID_ARTIFACT_PATH
- UNSUPPORTED_FEATURE.INTERRUPT_TYPE
2. Updated service handlers:
- SparkConnectInterruptHandler: Converted generic exceptions to SparkSQLException
- SparkConnectAddArtifactsHandler: Converted to SparkRuntimeException for invalid paths
- SparkConnectStreamingQueryListenerHandler: Converted to SparkSQLException
3. Added test coverage in SparkConnectServiceE2ESuite
### Why are the changes needed?
Previously, the Spark Connect server threw generic Java exceptions that:
- Did not include structured error classes
- Could not be properly categorized by clients
- Provided less actionable error information
- Were inconsistent with Spark's error handling standards
The error handling infrastructure (ErrorUtils.handleError) can only propagate error classes from SparkThrowable instances. Generic Java exceptions are converted to generic UNKNOWN errors, losing important context.
### Does this PR introduce _any_ user-facing change?
Yes, but only to improve error messages.
Before:
java.lang.IllegalArgumentException: INTERRUPT_TYPE_TAG requested, but no operation_tag provided.
After:
[INVALID_PARAMETER_VALUE.INTERRUPT_TYPE_TAG_REQUIRES_TAG] The value of parameter(s)
operation_tag in interrupt is invalid: INTERRUPT_TYPE_TAG requested, but no
operation_tag provided.
Clients can now parse structured error classes for better error handling and recovery.
### How was this patch tested?
1. Added new test cases in SparkConnectServiceE2ESuite:
- test("Interrupt with TAG type without operation_tag throws proper error class")
- test("Interrupt with OPERATION_ID type without operation_id throws proper error class")
2. Manual verification of error class propagation through the gRPC layer
3. All modified files pass Scala style checks
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.5
Closes #52818 from grundprinzip/SPARK-error-messages.
Authored-by: Martin Grund <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit a830dd2)
Signed-off-by: Dongjoon Hyun <[email protected]>
|
Merged to master/4.1 for Apache Spark 4.1.0. Thank you, @grundprinzip and @HyukjinKwon . |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR converts several IllegalArgumentException and UnsupportedOperationException instances in the Spark Connect server to proper SparkException with structured error classes, ensuring consistent error propagation to clients.
Changes include:
Added error class definitions in error-conditions.json:
Updated service handlers:
Added test coverage in SparkConnectServiceE2ESuite
Why are the changes needed?
Previously, the Spark Connect server threw generic Java exceptions that:
The error handling infrastructure (ErrorUtils.handleError) can only propagate error classes from SparkThrowable instances. Generic Java exceptions are converted to generic UNKNOWN errors, losing important context.
Does this PR introduce any user-facing change?
Yes, but only to improve error messages.
Before:
java.lang.IllegalArgumentException: INTERRUPT_TYPE_TAG requested, but no operation_tag provided.
After:
[INVALID_PARAMETER_VALUE.INTERRUPT_TYPE_TAG_REQUIRES_TAG] The value of parameter(s)
operation_tag in interrupt is invalid: INTERRUPT_TYPE_TAG requested, but no
operation_tag provided.
Clients can now parse structured error classes for better error handling and recovery.
How was this patch tested?
Added new test cases in SparkConnectServiceE2ESuite:
Manual verification of error class propagation through the gRPC layer
All modified files pass Scala style checks
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.5