Merged
Conversation
…ros login user. (apache#4424). Contributed by Xiping Zhang.
…ist.txt (apache#4444) Bump cos_api-bundle to 5.6.69 All copies of httpclient, including shaded ones in libraries used by the s3a, gs and cos cloud connectors, turn out to load their TLD list from the same resource mozilla/public-suffix-list.txt Updating the hadoop-cos dependency ensures that its version of public-suffix-list.txt is up to date -and so the s3a connector able to talk to s3 resources if the cos-api-bundle JAR is where the resource is loaded from. Contributed by André Fonseca
Reviewed-by: Viraj Jasani <vjasani@apache.org> Signed-off-by: Tao Li <tomscut@apache.org>
Co-authored-by: slfan1989 <louj1988@@>
…ode. (apache#4367). Contributed by ZanderXu. Reviewed-by: Mingxiang Li <liaiphag0@gmail.com> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Speed up the magic committer with key changes being * Writes under __magic always retain directory markers * File creation under __magic skips all overwrite checks, including the LIST call intended to stop files being created over dirs. * mkdirs under __magic probes the path for existence but does not look any further. Extra parallelism in task and job commit directory scanning Use of createFile and openFile with parameters which all for HEAD checks to be skipped. The committer can write the summary _SUCCESS file to the path `fs.s3a.committer.summary.report.directory`, which can be in a different file system/bucket if desired, using the job id as the filename. Also: HADOOP-15460. S3A FS to add `fs.s3a.create.performance` Application code can set the createFile() option fs.s3a.create.performance to true to disable the same safety checks when writing under magic directories. Use with care. The createFile option prefix `fs.s3a.create.header.` can be used to add custom headers to S3 objects when created. Contributed by Steve Loughran.
…HDFS-16563. (apache#4408) Regression caused by HDFS-16563; the hdfs exception text was changed, but because it was a YARN test doing the check, Yetus didn't notice. Contributed by zhengchenyu
… servers (apache#4431) Reviewed-by: Steve Loughran <stevel@apache.org> Signed-off-by: Tao Li <tomscut@apache.org>
…pache#4448) Signed-off-by: Tao Li <tomscut@apache.org>
…mber of usable replicas (apache#4410) Co-authored-by: Kevin Wikant <wikak@amazon.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
…nch-3.3 (apache#4378) Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
…pache#4365) * HADOOP-18266. Using HashSet/ TreeSet Constructor for hadoop-common Co-authored-by: Deb <dbsamrat@3c22fba1b03f.ant.amazon.com>
…r due to HDFS-16563 (apache#4428). Contributed by fanshilun. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
…e on MultipleOutputs#close (apache#4247) Contributed by Ravuri Sushma sree. Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
…tributed by Szilard Nemeth.
…ches its max-parallel-apps limit. Contributed by Andras Gyori
…ontributed by Viraj Jasani. Reviewed-by: Tao Li <tomscut@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
…ntributed by Christian Bartolomaus.
…ges (apache#4436) * YARN-9971.YARN Native Service HttpProbe logs THIS_HOST in error messages Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Co-Authored by: Samrat Deb
…-16202 (apache#4472) Fixing a mockito-based test which broke when HADOOP-16202 changed the methods being invoked. Contributed by Steve Loughran
…d even if multiple log aggregation file controllers are configured. Contributed by Szilard Nemeth.
part of HADOOP-18103. Add support for multiple ranged vectored read api in PositionedReadable. The default iterates through the ranges to read each synchronously, but the intent is that FSDataInputStream subclasses can make more efficient readers especially in object stores implementation. Also added implementation in S3A where smaller ranges are merged and sliced byte buffers are returned to the readers. All the merged ranged are fetched from S3 asynchronously. Contributed By: Owen O'Malley and Mukund Thakur
… maxReadSizeForVectorReads (apache#3964) Part of HADOOP-18103. Introducing fs.s3a.vectored.read.min.seek.size and fs.s3a.vectored.read.max.merged.size to configure min seek and max read during a vectored IO operation in S3A connector. These properties actually define how the ranges will be merged. To completely disable merging set fs.s3a.max.readsize.vectored.read to 0. Contributed By: Mukund Thakur
…che#4273) part of HADOOP-18103. Contributed By: Mukund Thakur
part of HADOOP-18103. Required for vectored IO feature. None of current buffer pool implementation is complete. ElasticByteBufferPool doesn't use weak references and could lead to memory leak errors and DirectBufferPool doesn't support caller preferences of direct and heap buffers and has only fixed length buffer implementation. Contributed By: Mukund Thakur
…#4445) part of HADOOP-18103. Handling memory fragmentation in S3A vectored IO implementation by allocating smaller user range requested size buffers and directly filling them from the remote S3 stream and skipping undesired data in between ranges. This patch also adds aborting active vectored reads when stream is closed or unbuffer() is called. Contributed By: Mukund Thakur
This feature adds methods for ranged vectored read operations in PositionedReadable. All stream which implement that interface support the new API. The default implementation reads each range in the vector sequentially. However, specific implementations may provide higher performance versions. This is done in two places * Local FileSystem/Checksum FileSystem * The S3A client. The S3A client first coalesces adjacent and "nearby" ranges together, then fetches each range in separate HTTP GET requests, executed in parallel. As such it delivers significant speedups to applications reading separate blocks of data from the same file, columnar data format libraries in particular. This is the merge commit of the feature branch; the work is in HADOOP-11867. Add a high-performance vectored read API. HADOOP-18104. S3A: Add configs to configure minSeekForVectorReads and maxReadSizeForVectorReads. HADOOP-18107. Adding scale test for vectored reads for large file HADOOP-18105. Implement buffer pooling with weak references. HADOOP-18106. Handle memory fragmentation in S3A Vectored IO. Contributed By: Owen O'Malley and Mukund Thakur
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Chris Nauroth <cnauroth@apache.org>
…e#4484). Contributed by fanshilun. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
…gation (apache#4486) * YARN-10320.Replace FSDataInputStream#read with readFully in Log Aggregation Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
…onStore#confirmMutation (apache#4487) Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
…n some cases (apache#4452) * HDFS-16633.Reserved Space For Replicas is not released on some cases Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
18 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
How was this patch tested?
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?