HDFS-16267. Make hdfs_df tool cross platform by GauthamBanasandra · Pull Request #3542 · apache/hadoop

GauthamBanasandra · 2021-10-11T18:16:51Z

Description of PR

The source files for hdfs_df uses getopt for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. We need to replace getopt with boost::program_options to make this cross platform.

How was this patch tested?

Ran hdfs_df locally and verified that it's giving the right output -

Also, the unit tests ran fine -

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2021-10-11T19:09:43Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	44m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
-1 ❌	mvninstall	0m 26s	/branch-mvninstall-root.txt	root in trunk failed.
-1 ❌	compile	0m 23s	/branch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt	hadoop-hdfs-native-client in trunk failed.
-1 ❌	mvnsite	0m 24s	/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-native-client.txt	hadoop-hdfs-native-client in trunk failed.
+1 💚	shadedclient	2m 5s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
-1 ❌	mvninstall	0m 23s	/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-native-client.txt	hadoop-hdfs-native-client in the patch failed.
-1 ❌	compile	0m 23s	/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt	hadoop-hdfs-native-client in the patch failed.
-1 ❌	cc	0m 23s	/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt	hadoop-hdfs-native-client in the patch failed.
-1 ❌	golang	0m 23s	/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt	hadoop-hdfs-native-client in the patch failed.
-1 ❌	javac	0m 23s	/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt	hadoop-hdfs-native-client in the patch failed.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-1 ❌	mvnsite	0m 22s	/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-native-client.txt	hadoop-hdfs-native-client in the patch failed.
+1 💚	shadedclient	1m 12s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 23s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt	hadoop-hdfs-native-client in the patch failed.
+0 🆗	asflicense	0m 23s		ASF License check generated no output?
		51m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/1/artifact/out/Dockerfile
GITHUB PR	#3542
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 332c4f22354c 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 463c7b93cabc9b10286d8dd2bd66b9d378b3bd8c
Default Java	Red Hat, Inc.-1.8.0_302-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/1/testReport/
Max. process+thread count	46 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/1/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-10-12T04:14:35Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 19s		#3542 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
GITHUB PR	#3542
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/2/console
versions	git=2.17.1
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

* The source files for hdfs_df uses getopt for parsing the command line arguments. * getopt is available only on Linux and thus, isn't cross platform. * We need to replace getopt with boost::program_options to make this cross platform.

This reverts commit 2de1aa063d9deeae156929b80a5b4988e1ac41a8.

hadoop-yetus · 2021-10-12T06:51:34Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 2s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	33m 17s		trunk passed
+1 💚	compile	2m 41s		trunk passed
+1 💚	mvnsite	0m 30s		trunk passed
+1 💚	shadedclient	55m 3s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 17s		the patch passed
+1 💚	compile	2m 44s		the patch passed
+1 💚	cc	2m 44s		the patch passed
+1 💚	golang	2m 44s		the patch passed
+1 💚	javac	2m 44s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 21s		the patch passed
+1 💚	shadedclient	19m 13s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	34m 2s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		115m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/artifact/out/Dockerfile
GITHUB PR	#3542
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 8d72fbdcca7b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `1569657`
Default Java	Red Hat, Inc.-1.8.0_302-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/testReport/
Max. process+thread count	543 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-10-12T08:58:41Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	25m 59s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	21m 35s		trunk passed
+1 💚	compile	3m 1s		trunk passed
+1 💚	mvnsite	0m 45s		trunk passed
+1 💚	shadedclient	43m 45s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 24s		the patch passed
+1 💚	compile	2m 41s		the patch passed
+1 💚	cc	2m 41s		the patch passed
+1 💚	golang	2m 41s		the patch passed
+1 💚	javac	2m 41s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 26s		the patch passed
+1 💚	shadedclient	18m 23s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	32m 2s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 48s		The patch does not generate ASF License warnings.
		126m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/artifact/out/Dockerfile
GITHUB PR	#3542
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 23df25a51f88 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `1569657`
Default Java	Red Hat, Inc.-1.8.0_302-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/testReport/
Max. process+thread count	737 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-10-12T11:05:09Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 9s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	24m 57s		trunk passed
+1 💚	compile	2m 38s		trunk passed
+1 💚	mvnsite	0m 27s		trunk passed
+1 💚	shadedclient	52m 57s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 17s		the patch passed
+1 💚	compile	2m 28s		the patch passed
+1 💚	cc	2m 28s		the patch passed
+1 💚	golang	2m 28s		the patch passed
+1 💚	javac	2m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 19s		the patch passed
+1 💚	shadedclient	24m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	31m 27s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		126m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/artifact/out/Dockerfile
GITHUB PR	#3542
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 2190aed5f900 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `1569657`
Default Java	Debian-11.0.12+7-post-Debian-2deb10u1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/testReport/
Max. process+thread count	749 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/console
versions	git=2.20.1 maven=3.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-10-12T12:50:27Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 46s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	20m 34s		trunk passed
+1 💚	compile	2m 56s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	2m 52s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	mvnsite	0m 27s		trunk passed
+1 💚	shadedclient	45m 14s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 16s		the patch passed
+1 💚	compile	2m 42s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	cc	2m 42s		the patch passed
+1 💚	golang	2m 42s		the patch passed
+1 💚	javac	2m 42s		the patch passed
+1 💚	compile	2m 46s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	cc	2m 46s		the patch passed
+1 💚	golang	2m 46s		the patch passed
+1 💚	javac	2m 46s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 20s		the patch passed
+1 💚	shadedclient	18m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	31m 53s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		105m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/artifact/out/Dockerfile
GITHUB PR	#3542
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 64636831de79 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `1569657`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/testReport/
Max. process+thread count	750 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3542/3/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

GauthamBanasandra added 3 commits October 12, 2021 10:23

HDFS-16267. Make hdfs_df tool cross platform

bf5ae23

* The source files for hdfs_df uses getopt for parsing the command line arguments. * getopt is available only on Linux and thus, isn't cross platform. * We need to replace getopt with boost::program_options to make this cross platform.

Trigger CI

6f4caee

Revert "Trigger CI"

1569657

This reverts commit 2de1aa063d9deeae156929b80a5b4988e1ac41a8.

GauthamBanasandra force-pushed the df-x-platform branch from 096410a to 1569657 Compare October 12, 2021 04:54

goiri approved these changes Oct 12, 2021

View reviewed changes

goiri merged commit 107fe22 into apache:trunk Oct 13, 2021

GauthamBanasandra deleted the df-x-platform branch October 13, 2021 16:46

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HDFS-16267. Make hdfs_df tool cross platform (apache#3542)

186b4ff

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16267. Make hdfs_df tool cross platform#3542

HDFS-16267. Make hdfs_df tool cross platform#3542
goiri merged 3 commits intoapache:trunkfrom
GauthamBanasandra:df-x-platform

GauthamBanasandra commented Oct 11, 2021

Uh oh!

hadoop-yetus commented Oct 11, 2021

Uh oh!

hadoop-yetus commented Oct 12, 2021

Uh oh!

hadoop-yetus commented Oct 12, 2021

Uh oh!

hadoop-yetus commented Oct 12, 2021

Uh oh!

hadoop-yetus commented Oct 12, 2021

Uh oh!

hadoop-yetus commented Oct 12, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

GauthamBanasandra commented Oct 11, 2021

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Oct 11, 2021

Uh oh!

hadoop-yetus commented Oct 12, 2021

Uh oh!

hadoop-yetus commented Oct 12, 2021

Uh oh!

hadoop-yetus commented Oct 12, 2021

Uh oh!

hadoop-yetus commented Oct 12, 2021

Uh oh!

hadoop-yetus commented Oct 12, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants