HADOOP-19295. S3A: fs.s3a.connection.request.timeout too low: quick fix #7087
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
...for large uploads over slow links * New default value of 15 minutes in source * Updated docs * Troubleshooting: new stack trace and details of the problem Change-Id: If3319268a8df35b5f0377f31ea9df2c2f49186ea
steveloughran
changed the title
|
|
||
| /** | ||
| * Default duration of a request before it is timed out: 60s. | ||
| * Default duration of a request before it is timed out: 16m. |
Member
There was a problem hiding this comment.
16m seems to mismatch with the actual code, 15 minutes, and performance.md. Maybe, 15m?
| * (default 15s) allowed for a fast failure of all other operations. | ||
| * In V2 SDK this now applies to all operations, including uploads. | ||
| * A large timeout is now needed, even though it means that some service/network | ||
| * failures may now take a long time to surface. |
Member
There was a problem hiding this comment.
However, 15minutes sounds like too big as a timeout, doesn't it? What is the background of this huge value?
Technically, 15m means 1 hour when we retry 3 times.
Contributor
Author
There was a problem hiding this comment.
problem is we want a duration which is valid with many other streams competing for that bandwidth, so a single block upload of, say, 64M needs time. I don't know what a good time is.
Contributor
Author
There was a problem hiding this comment.
we could maybe cut to something a bit lower, like say 10m?
side issue: why don't you try it as is for a remote upload of the rc1 tar.gz and see how you get on without this patch.
|
💔 -1 overall
This message was automatically generated. |
4 tasks
Contributor
Author
|
In #7089 I'm doing a real fix with different timings but
for this reason, I think we should target a 3.4.2 with it |
steveloughran
commented
Oct 2, 2024
|
|
||
| <!--- global properties --> | ||
|
|
||
| <property> |
Contributor
Author
There was a problem hiding this comment.
not sure how i deleted this; will revert
steveloughran
commented
Oct 2, 2024
Contributor
Author
steveloughran
left a comment
steveloughran
left a comment
There was a problem hiding this comment.
going to switch to 10m if that is enough for the problem to go away locally
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
...for large uploads over slow links
I don't think this should be the final patch.
Because I've just discovered how we can set different timeouts on each request, so
a long timeout for data put/post seems straightforward, as well as testing.
How was this patch tested?
big bulk uploads
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?