HADOOP-18112: Rename operation fails during multi object delete of size more than 1000. #4045

mukund-thakur · 2022-03-03T09:43:57Z

Description of PR

Implementing batching of requests during bulk delete operation based on page size.

How was this patch tested?

Added new test. and re-ran the existing integration test.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…ze more than 1000.

mukund-thakur · 2022-03-03T09:45:15Z

I had to implement Lists.partition() method in hadoop-common as some enforcer rule is failing if I try to import guava.

mukund-thakur · 2022-03-03T09:50:56Z

CC @steveloughran @mehakmeet

mehakmeet

looks good, some minor comments.

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/DeleteOperation.java

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Lists.java

hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLists.java

mukund-thakur · 2022-03-04T09:30:08Z

Why these many java doc errors in PR ? I haven't changed the MarkerTool.
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:150: warning: empty <p> tag [ERROR] * <p></p> [ERROR] ^ [ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:964: warning: no @param for source [ERROR] public ScanArgsBuilder withSourceFS(final FileSystem source) { [ERROR] ^ [ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:964: warning: no @return [ERROR] public ScanArgsBuilder withSourceFS(final FileSystem source) { [ERROR] ^ [ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4045@2/ubuntu-focal/src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerTool.java:970: warning: no @param for p [ERROR] public ScanArgsBuilder withPath(final Path p) { [ERROR] ^

steveloughran · 2022-03-07T22:29:42Z

javadoc. hmm. we've had so many problems over javadoc
 versions about tags, where <p/> was blocked, open <p>
was an error and where <p></p> without content an error. 
i don't think there is a good answer for a tool written in the 1990s
which probably doesn't complain about the <blink> tag. ignore

steveloughran

looks ok...just make sure that there's no list rebuilding when lists in range are passed in

hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLists.java

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/tools/MarkerToolOperations.java

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFailureHandling.java

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Lists.java

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFailureHandling.java

hadoop-yetus · 2022-03-09T11:54:44Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 48s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 7 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 37s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 28s		trunk passed
+1 💚	compile	23m 16s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	19m 53s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	3m 47s		trunk passed
+1 💚	mvnsite	2m 41s		trunk passed
+1 💚	javadoc	1m 55s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 28s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 48s		trunk passed
+1 💚	shadedclient	21m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 33s		the patch passed
+1 💚	compile	22m 4s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	22m 4s		the patch passed
+1 💚	compile	19m 50s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	19m 50s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	3m 38s		the patch passed
+1 💚	mvnsite	2m 40s		the patch passed
+1 💚	javadoc	1m 53s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 35s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 6s		the patch passed
+1 💚	shadedclient	21m 23s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 47s		hadoop-common in the patch passed.
+1 💚	unit	2m 28s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 1s		The patch does not generate ASF License warnings.
		222m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4045/5/artifact/out/Dockerfile
GITHUB PR	#4045
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 917309fb88fa 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `64f9319`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4045/5/testReport/
Max. process+thread count	1603 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4045/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran

+1
lovely

Multi object delete of size more than 1000 is not supported by S3 and fails with MalformedXML error. So implementing paging of requests to reduce the number of keys in a single request. Page size can be configured using "fs.s3a.bulk.delete.page.size" Contributed By: Mukund Thakur

mukund-thakur · 2022-03-11T08:28:48Z

merged to branch-3.3

Multi object delete of size more than 1000 is not supported by S3 and fails with MalformedXML error. So implementing paging of requests to reduce the number of keys in a single request. Page size can be configured using "fs.s3a.bulk.delete.page.size" Contributed By: Mukund Thakur

HADOOP-18112: Rename operation fails during multi object delete of si…

317caef

…ze more than 1000.

mukund-thakur requested a review from steveloughran March 3, 2022 09:50

Missed adding the list partition impl

89f4e2f

mehakmeet reviewed Mar 4, 2022

View reviewed changes

apache deleted a comment from hadoop-yetus Mar 7, 2022

steveloughran requested changes Mar 7, 2022

View reviewed changes

mukund-thakur added 3 commits March 8, 2022 17:12

Review comments by Steve and Mehakmeet

4fc41b3

Again missed the update of hadoop-common

24c854a

checkstyle

64f9319

apache deleted a comment from hadoop-yetus Mar 9, 2022

steveloughran approved these changes Mar 9, 2022

View reviewed changes

mukund-thakur merged commit 672e380 into apache:trunk Mar 11, 2022

HADOOP-18112: Rename operation fails during multi object delete of size more than 1000. #4045

HADOOP-18112: Rename operation fails during multi object delete of size more than 1000. #4045

Uh oh!

Conversation

mukund-thakur commented Mar 3, 2022

Description of PR

How was this patch tested?

For code changes:

Uh oh!

mukund-thakur commented Mar 3, 2022

Uh oh!

mukund-thakur commented Mar 3, 2022

Uh oh!

mehakmeet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mukund-thakur commented Mar 4, 2022

Uh oh!

steveloughran commented Mar 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Mar 9, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

mukund-thakur commented Mar 11, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

steveloughran commented Mar 7, 2022 •

edited

Loading