HADOOP-17620. DistCp: Use Iterator for listing target directory as well. #2861

ayushtkn · 2021-04-04T10:19:04Z

https://issues.apache.org/jira/browse/HADOOP-17620

ayushtkn · 2021-04-04T10:20:14Z

Ran the S3A test:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 654.864 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0

AWS_REGION
ap-south-1

hadoop-yetus · 2021-04-04T18:55:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 23s		trunk passed
+1 💚	compile	0m 38s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 28s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	0m 23s		trunk passed
+1 💚	mvnsite	0m 33s		trunk passed
+1 💚	javadoc	0m 28s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 27s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	0m 50s		trunk passed
+1 💚	shadedclient	15m 38s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 26s		the patch passed
+1 💚	compile	0m 25s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	0m 25s		the patch passed
+1 💚	compile	0m 21s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	0m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 16s		hadoop-tools/hadoop-distcp: The patch generated 0 new + 40 unchanged - 1 fixed = 40 total (was 41)
+1 💚	mvnsite	0m 24s		the patch passed
+1 💚	javadoc	0m 20s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	0m 53s		the patch passed
+1 💚	shadedclient	15m 27s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	25m 45s		hadoop-distcp in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		102m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2861/2/artifact/out/Dockerfile
GITHUB PR	#2861
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux cb5fbb8d9a37 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `468ea21`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2861/2/testReport/
Max. process+thread count	544 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2861/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran

worried about using log capture for testing, especially with a move to log4j2 due this year.

What would happen if source is hdfs (we want the listIterator) and test is S3? you may want different settings on each

steveloughran · 2021-04-12T12:51:05Z

...hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java

    GenericTestUtils
        .createFiles(remoteFS, source, getDepth(), getWidth(), getWidth());

+    GenericTestUtils.LogCapturer log =


Not over-enamoured of this test strategy. As well as being brittle, every use of this is going to complicate our move to Log4J2.

Is there any other way we could do this?

If the distcp test collected the IOStatistics of iteration, then those stores which returned it from iterators (s3a currently, abfs is still WiP) could count it and actually assert on real invocations. HDFS Doesn't do this.

Thanx @steveloughran, do you suggest that we should have two options like -useiteratorforsource and -useiteratorfortarget. Do you think in that case we would be able to save out on memory? since the target list is being build as part of CopyCommitter, so even if one takes the normal path, We would get OOM, just when will differ?

Regarding the log stuff, That was the only thing I could think of, to confirm if iterator was used. And during migration to Log4J2, Will moving to something like this will be of any help instead:
https://github.com/apache/hive/blob/master/ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java#L285

hive code looks great!

So, Are you Ok with the other changes? This logger change we can not do now itself in the code, need to do during the migration itself, Was just trying to find a solution to the problem you told, or if there isn't any

...hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java

hadoop-yetus · 2021-04-12T17:32:22Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	33m 47s		trunk passed
+1 💚	compile	0m 32s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 30s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	0m 26s		trunk passed
+1 💚	mvnsite	0m 34s		trunk passed
+1 💚	javadoc	0m 28s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 27s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	0m 49s		trunk passed
+1 💚	shadedclient	13m 51s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 25s		the patch passed
+1 💚	compile	0m 26s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	0m 26s		the patch passed
+1 💚	compile	0m 22s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	0m 22s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 16s		hadoop-tools/hadoop-distcp: The patch generated 0 new + 40 unchanged - 1 fixed = 40 total (was 41)
+1 💚	mvnsite	0m 24s		the patch passed
+1 💚	javadoc	0m 19s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	0m 47s		the patch passed
+1 💚	shadedclient	13m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	23m 5s		hadoop-distcp in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		94m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2861/3/artifact/out/Dockerfile
GITHUB PR	#2861
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux e5c03cfce65e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e2fabb9`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2861/3/testReport/
Max. process+thread count	677 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2861/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ayushtkn · 2021-04-17T12:53:11Z

Circling back:
@steveloughran let me know, if that makes sense, if not I can add an extra option for target as well. or if you have any other suggestions or concerns, Let me know I can try something there as well. :-)

vinayakumarb · 2021-04-23T04:44:50Z

LGTM.

…ll. (#2861). Contributed by Ayush Saxena. Signed-off-by: Vinayakumar B <[email protected]>

…ll. (apache#2861). Contributed by Ayush Saxena. Signed-off-by: Vinayakumar B <[email protected]>

HADOOP-17620. DistCp: Use Iterator for listing target directory as well.

e60036b

Fix checkstyle.

468ea21

ayushtkn requested a review from steveloughran April 11, 2021 11:15

apache deleted a comment from hadoop-yetus Apr 12, 2021

steveloughran requested changes Apr 12, 2021

View reviewed changes

Fix.

e2fabb9

ayushtkn requested review from ferhui and steveloughran April 21, 2021 13:21

ayushtkn merged commit 6800b21 into apache:trunk Apr 23, 2021

asfgit pushed a commit that referenced this pull request Apr 23, 2021

HADOOP-17620. DistCp: Use Iterator for listing target directory as we…

b743d56

…ll. (#2861). Contributed by Ayush Saxena. Signed-off-by: Vinayakumar B <[email protected]>

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HADOOP-17620. DistCp: Use Iterator for listing target directory as we…

378aa3f

…ll. (apache#2861). Contributed by Ayush Saxena. Signed-off-by: Vinayakumar B <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-17620. DistCp: Use Iterator for listing target directory as well. #2861

HADOOP-17620. DistCp: Use Iterator for listing target directory as well. #2861

Uh oh!

ayushtkn commented Apr 4, 2021

Uh oh!

ayushtkn commented Apr 4, 2021

Uh oh!

hadoop-yetus commented Apr 4, 2021

Uh oh!

steveloughran left a comment

Uh oh!

steveloughran Apr 12, 2021

Uh oh!

ayushtkn Apr 12, 2021 •

edited

Loading

Uh oh!

steveloughran Apr 18, 2021

Uh oh!

ayushtkn Apr 18, 2021

Uh oh!

Uh oh!

hadoop-yetus commented Apr 12, 2021

Uh oh!

ayushtkn commented Apr 17, 2021

Uh oh!

vinayakumarb commented Apr 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HADOOP-17620. DistCp: Use Iterator for listing target directory as well. #2861

HADOOP-17620. DistCp: Use Iterator for listing target directory as well. #2861

Uh oh!

Conversation

ayushtkn commented Apr 4, 2021

Uh oh!

ayushtkn commented Apr 4, 2021

Uh oh!

hadoop-yetus commented Apr 4, 2021

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran Apr 12, 2021

Choose a reason for hiding this comment

Uh oh!

ayushtkn Apr 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steveloughran Apr 18, 2021

Choose a reason for hiding this comment

Uh oh!

ayushtkn Apr 18, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hadoop-yetus commented Apr 12, 2021

Uh oh!

ayushtkn commented Apr 17, 2021

Uh oh!

vinayakumarb commented Apr 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ayushtkn Apr 12, 2021 •

edited

Loading