Skip to content

Conversation

@hiloboy0119
Copy link

This adds two options for the S3 client used by S3FileIO

  • s3.retry-on-unchecked-io-exception - This will configure the client to retry on an UncheckedIOException. At Netflix we have seen this happen during metadata commits which cause an entire job to fail.
  • s3.max-retries - The maximum number of retries

@github-actions github-actions bot added the AWS label Jul 11, 2023
@jzhuge
Copy link
Member

jzhuge commented Jul 12, 2023

@jfz's PR 3539 in AWS SDK added UncheckedIOException to RETRYABLE_EXCEPTIONS. It is in 2.19.2 and later. We can just upgrade AWS SDK v2 to pick up UncheckedIOException
changes.

@nastra
Copy link
Contributor

nastra commented Jul 12, 2023

thanks for pointing this out @jzhuge. So I think we should change the PR to just update the AWS SDK version. @hiloboy0119 could you do that please?

@singhpk234
Copy link
Contributor

[doubt] Will be be required then considering we are already as SDK version 2.20.18 @nastra @jzhuge

@nastra
Copy link
Contributor

nastra commented Jul 12, 2023

ah right, I forgot that we're already on 2.20.18, thanks for pointing that out @singhpk234

@github-actions
Copy link

github-actions bot commented Sep 6, 2024

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

@github-actions github-actions bot added the stale label Sep 6, 2024
@github-actions
Copy link

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions bot closed this Sep 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants