HADOOP-18104: S3A: Add configs to configure minSeekForVectorReads and maxReadSizeForVectorReads#3964
Merged
mukund-thakur merged 7 commits intoapache:feature-vectored-iofrom Apr 29, 2022
Conversation
Contributor
Author
|
CC @mehakmeet |
mukund-thakur
force-pushed
the
feature-vectored-io
branch
from
ac08a25 to
87682c9
Compare
… maxReadSizeForVectorReads Part of HADOOP-18103. Introducing fs.s3a.min.seek.vectored.read and fs.s3a.max.readsize.vectored.read to configure min seek and max read during a vectored IO operation in S3A connector. These propeties actually define how the ranges will be merged. To completely disable merging set fs.s3a.max.readsize.vectored.read to 0.
mukund-thakur
force-pushed
the
HADOOP-18104
branch
from
48ba464 to
99c983e
Compare
|
💔 -1 overall
This message was automatically generated. |
steveloughran
requested changes
Feb 23, 2022
Contributor
steveloughran
left a comment
steveloughran
left a comment
There was a problem hiding this comment.
minor changes, name of the options the most significant
...mon-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
Outdated
Show resolved
Hide resolved
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MoreAsserts.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
Outdated
Show resolved
Hide resolved
|
|
||
| /** | ||
| * What is the largest size that we should group ranges | ||
| * together during vectored read? |
Contributor
There was a problem hiding this comment.
not sure that "readsize" is the best term here
Contributor
Author
There was a problem hiding this comment.
we can say merged.size?
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AReadOpContext.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AReadOpContext.java
Outdated
Show resolved
Hide resolved
| import java.util.ArrayList; | ||
| import java.util.List; | ||
|
|
||
| import org.junit.Test; |
Contributor
There was a problem hiding this comment.
i think the ordering of imports is already mixed up here; should be
java
other
org.apache
Contributor
Author
There was a problem hiding this comment.
Not sure how it got messed up. Sorry
...hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractVectoredRead.java
Outdated
Show resolved
Hide resolved
...hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractVectoredRead.java
Outdated
Show resolved
Hide resolved
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
steveloughran
requested changes
Apr 21, 2022
...mon-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
Show resolved
Hide resolved
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AReadOpContext.java
Outdated
Show resolved
Hide resolved
| The S3A FileSystem supports implementation of vectored read api using which | ||
| a client can provide a list of file ranges to read returning a future read | ||
| object associated with each range. For full api specification please see | ||
| [FSDataInputStream](hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md). |
Contributor
There was a problem hiding this comment.
use a .html suffix
use mvn site:site to generate the site to check links
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md
Outdated
Show resolved
Hide resolved
| S3ATestUtils.removeBaseAndBucketOverrides(conf, | ||
| Constants.AWS_S3_VECTOR_READS_MIN_SEEK_SIZE, | ||
| Constants.AWS_S3_VECTOR_READS_MAX_MERGED_READ_SIZE); | ||
| S3ATestUtils.disableFilesystemCaching(conf); |
| */ | ||
| public static <T> void assertEqual(T actual, T expected, String message) { | ||
| Assertions.assertThat(actual) | ||
| .describedAs("Mismatch in " + message) |
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
Outdated
Show resolved
Hide resolved
|
🎊 +1 overall
This message was automatically generated. |
mehakmeet
reviewed
Apr 25, 2022
Contributor
mehakmeet
left a comment
mehakmeet
left a comment
There was a problem hiding this comment.
+1, pending one small doubt of how we are populating the vectored IO configs.
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
Outdated
Show resolved
Hide resolved
|
🎊 +1 overall
This message was automatically generated. |
Contributor
|
+1 merge at your leisure. you will also need to rebase the branch. you don't need to do a pr for that, just rebase/fix/retest locally and then do a force push with the update. that is allowed on this branch (anything which doesn't match the pattern branch-*) |
mukund-thakur
added a commit
that referenced
this pull request
May 2, 2022
… maxReadSizeForVectorReads (#3964) Part of HADOOP-18103. Introducing fs.s3a.vectored.read.min.seek.size and fs.s3a.vectored.read.max.merged.size to configure min seek and max read during a vectored IO operation in S3A connector. These properties actually define how the ranges will be merged. To completely disable merging set fs.s3a.max.readsize.vectored.read to 0. Contributed By: Mukund Thakur
mukund-thakur
added a commit
that referenced
this pull request
Jun 15, 2022
… maxReadSizeForVectorReads (#3964) Part of HADOOP-18103. Introducing fs.s3a.vectored.read.min.seek.size and fs.s3a.vectored.read.max.merged.size to configure min seek and max read during a vectored IO operation in S3A connector. These properties actually define how the ranges will be merged. To completely disable merging set fs.s3a.max.readsize.vectored.read to 0. Contributed By: Mukund Thakur
mukund-thakur
added a commit
that referenced
this pull request
Jun 21, 2022
… maxReadSizeForVectorReads (#3964) Part of HADOOP-18103. Introducing fs.s3a.vectored.read.min.seek.size and fs.s3a.vectored.read.max.merged.size to configure min seek and max read during a vectored IO operation in S3A connector. These properties actually define how the ranges will be merged. To completely disable merging set fs.s3a.max.readsize.vectored.read to 0. Contributed By: Mukund Thakur
asfgit
pushed a commit
that referenced
this pull request
Jun 22, 2022
… maxReadSizeForVectorReads (#3964) Part of HADOOP-18103. Introducing fs.s3a.vectored.read.min.seek.size and fs.s3a.vectored.read.max.merged.size to configure min seek and max read during a vectored IO operation in S3A connector. These properties actually define how the ranges will be merged. To completely disable merging set fs.s3a.max.readsize.vectored.read to 0. Contributed By: Mukund Thakur
mukund-thakur
added a commit
that referenced
this pull request
Jun 27, 2022
… maxReadSizeForVectorReads (#3964) Part of HADOOP-18103. Introducing fs.s3a.vectored.read.min.seek.size and fs.s3a.vectored.read.max.merged.size to configure min seek and max read during a vectored IO operation in S3A connector. These properties actually define how the ranges will be merged. To completely disable merging set fs.s3a.max.readsize.vectored.read to 0. Contributed By: Mukund Thakur
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
… maxReadSizeForVectorReads (apache#3964) Part of HADOOP-18103. Introducing fs.s3a.vectored.read.min.seek.size and fs.s3a.vectored.read.max.merged.size to configure min seek and max read during a vectored IO operation in S3A connector. These properties actually define how the ranges will be merged. To completely disable merging set fs.s3a.max.readsize.vectored.read to 0. Contributed By: Mukund Thakur
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Part of HADOOP-18103.
Introducing fs.s3a.min.seek.vectored.read and fs.s3a.max.readsize.vectored.read
to configure min seek and max read during a vectored IO operation in S3A connector.
These propeties actually define how the ranges will be merged. To completely
disable merging set fs.s3a.max.readsize.vectored.read to 0.
How was this patch tested?
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?