fix: [branch-54] see through Shared wrapper when classifying task IO errors (backport of #2119) - #2239
Merged
andygrove merged 1 commit intoAug 6, 2026
Conversation
A retryable IO error raised on a join's shared build side arrives at the FailedTask classifier wrapped in DataFusionError::Shared, so the shallow matches! on the outermost variant never sees the IoError inside and the task is marked non-retryable. Classify on find_root() instead so the retryability decision is based on the root error regardless of wrapping. Co-authored-by: goingforstudying-ctrl <goingforstudying-ctrl@users.noreply.github.com> (cherry picked from commit b162708)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Backport of #2119 to
branch-54. The issue it fixes is #2028.Rationale for this change
A transient IO error on the build side of a hash join kills the whole job instead of being retried.
The retry classifier in
ballista/core/src/error.rsdoes a shallowmatches!(*e, DataFusionError::IoError(_))on the outermost variant. Errors coming off a join's shared build side get wrapped inDataFusionError::Shared(anArc, for sharing across consumers), so theIoErrorinside is never seen and the task falls through to the catch-allretryable: falsearm. With AQE off the error arrives unwrapped and retry works, which is how the wrapping was narrowed down as the cause.This turns recoverable object-store flakiness into job failures, so it is worth having on the release branch.
What changes are included in this PR?
A clean cherry-pick of b162708, unmodified.
Classifies on
find_root()instead of the outermost variant, so the retryability decision is based on the root error regardless of wrapping. One-line change plus unit tests covering bare,Shared-wrapped,Context-wrapped-Shared, and non-IO cases.Are there any user-facing changes?
No API changes. Tasks that fail with an IO error wrapped in
DataFusionError::Sharedare now retried rather than failing the job.Verified locally on the
branch-54base:cargo fmt --all -- --checkis clean, andcargo check --workspace --all-targets --lockedcompletes with no warnings on a combined stack of the six backports being proposed together. Test execution is left to CI.