refactor(core): use DataFusionError::External for structured task errors - #2276
Merged
milenkovicm merged 4 commits intoAug 11, 2026
Merged
Conversation
Member
Author
|
@milenkovicm when you have a moment please review. |
avantgardnerio
approved these changes
Aug 11, 2026
avantgardnerio
left a comment
Contributor
There was a problem hiding this comment.
Nice cleanup! I could not find any faults with it.
Contributor
|
thanks @villebro and @avantgardnerio, will merge this |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2202 (review discussion r3737946864).
Rationale for this change
#2202 kept shuffle-fetch failures structural until task-failure classification, but carried the structured
BallistaErroracross DataFusion asArrowError::ExternalError(Box<BallistaError>). Because DataFusion'sRecordBatchStreamyieldsResult<RecordBatch, DataFusionError>, that Arrow error auto-converts (viaFrom<ArrowError> for DataFusionError) intoDataFusionError::ArrowError(ArrowError::ExternalError(..))— a double wrap that recovery had to unwind through both the DataFusion and Arrow layers.DataFusionError::Externalis the purpose-built variant for errors originating outside DataFusion, andfind_root()already sees through it (traversing anyShared/Context/Diagnosticlayers). Using it removes the Arrow intermediate and collapses recovery to a single downcast:This is an internal refactor: errors are classified into the
FailedTaskproto inside the executor process before crossing any wire, so the wrapping convention can change without affecting the protocol.What changes are included in this PR?
BallistaError::into_datafusionis the single helper for the carry step, used at the shuffle-reader fetch-stream boundary and the shuffle-writer drain handoff.find_fetch_failed/fetch_failed_in_arrow/fetch_failed_in_datafusioncollapse into onefind_root()+Externaldowncast, andis_retryable_iosees through the same wrapper so IO errors stay retryable.DataFusionError(summaries_to_batchin both writers, theexecute_query_pull/execute_query_pushclient result streams) no longer wrap it in Arrow just to have it flattened back.distributed_queryfetch path now routes through the sameinto_datafusionhelper instead of hand-buildingDataFusionError::External.External,Shared, andContextshapes in one place instead of a separate test per shape.Are there any user-facing changes?
No API or documentation changes.
Free-text task error messages lose the redundant wrapper layer: the
Arrow error: External error:prefix collapses toExternal error:, and the two IO-error messages (Task failed due to Ballista IO error/Task failed due to DataFusion IO error) are unified toTask failed due to IO error. Structured failure classification — the fetch-partition, retryable-IO, task-killed, and execution-error reasons the scheduler acts on — is unchanged, including underShared/Contextlayers.