HDFS-16260. Make hdfs_deleteSnapshot tool cross platform by GauthamBanasandra · Pull Request #3532 · apache/hadoop

GauthamBanasandra · 2021-10-08T11:19:50Z

Description of PR

The source files for hdfs_deleteSnapshot uses getopt for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. We need to replace getopt with boost::program_options to make this cross platform.

How was this patch tested?

Tested it by running locally.

I created a snapshot first by running -

$ ./hdfs_createSnapshot -n my-snapshot /mydir

Then deleted the snapshot by running -

$ ./hdfs_deleteSnapshot /mydir my-snapshot

Commands -

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2021-10-08T13:44:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	34m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 15s		trunk passed
+1 💚	compile	2m 41s		trunk passed
+1 💚	mvnsite	0m 29s		trunk passed
+1 💚	shadedclient	53m 11s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 17s		the patch passed
+1 💚	compile	2m 30s		the patch passed
+1 💚	cc	2m 30s		the patch passed
+1 💚	golang	2m 30s		the patch passed
+1 💚	javac	2m 30s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 19s		the patch passed
+1 💚	shadedclient	17m 42s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	31m 41s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		143m 27s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/1/artifact/out/Dockerfile
GITHUB PR	#3532
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 90a7305dd333 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 3712b52094e150f4f8747ebb1a2d03dbf5300fd1
Default Java	Red Hat, Inc.-1.8.0_302-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/1/testReport/
Max. process+thread count	624 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/1/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-10-08T15:48:55Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	20m 40s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	21m 9s		trunk passed
+1 💚	compile	2m 55s		trunk passed
+1 💚	mvnsite	0m 45s		trunk passed
+1 💚	shadedclient	46m 5s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 23s		the patch passed
+1 💚	compile	2m 44s		the patch passed
+1 💚	cc	2m 44s		the patch passed
+1 💚	golang	2m 44s		the patch passed
+1 💚	javac	2m 44s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 25s		the patch passed
+1 💚	shadedclient	18m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	32m 4s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		124m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/1/artifact/out/Dockerfile
GITHUB PR	#3532
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux cdd577de93b1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 3712b52094e150f4f8747ebb1a2d03dbf5300fd1
Default Java	Red Hat, Inc.-1.8.0_302-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/1/testReport/
Max. process+thread count	727 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/1/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-10-08T15:50:40Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 22s		#3532 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
GITHUB PR	#3532
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/1/console
versions	git=2.17.1
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

goiri · 2021-10-08T15:51:04Z

...-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/tools/CMakeLists.txt

If we are splitting per line, let's split main.cc too

goiri · 2021-10-08T15:53:06Z

...ct/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/tools/hdfs-tool-test-fixtures.h

We are doing a lot of refactoring here that is not directly delete snapshot.
Let's do a separate JIRA for this refactor.

* The source files for hdfs_deleteSnapshot uses getopt for parsing the command line arguments. * getopt is available only on Linux and thus, isn't cross platform. We need to replace getopt with boost::program_options to make this cross platform.

* Needed to allow the help option as the only parameter that can be passed. * This fixes a test failure.

hadoop-yetus · 2021-10-10T09:04:54Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 40s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 42s		trunk passed
+1 💚	compile	2m 38s		trunk passed
+1 💚	mvnsite	0m 29s		trunk passed
+1 💚	shadedclient	53m 40s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 16s		the patch passed
+1 💚	compile	2m 33s		the patch passed
+1 💚	cc	2m 33s		the patch passed
+1 💚	golang	2m 33s		the patch passed
+1 💚	javac	2m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	shadedclient	17m 36s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	32m 0s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		109m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/2/artifact/out/Dockerfile
GITHUB PR	#3532
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 5b6c629aa6f0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `27a0542`
Default Java	Red Hat, Inc.-1.8.0_302-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/2/testReport/
Max. process+thread count	724 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/2/console
versions	git=2.9.5 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-10-10T10:45:54Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 41s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	21m 9s		trunk passed
+1 💚	compile	2m 55s		trunk passed
+1 💚	mvnsite	0m 45s		trunk passed
+1 💚	shadedclient	43m 3s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 25s		the patch passed
+1 💚	compile	2m 39s		the patch passed
+1 💚	cc	2m 39s		the patch passed
+1 💚	golang	2m 39s		the patch passed
+1 💚	javac	2m 39s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 27s		the patch passed
+1 💚	shadedclient	17m 59s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	32m 19s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		100m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/2/artifact/out/Dockerfile
GITHUB PR	#3532
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux d99b19640345 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `27a0542`
Default Java	Red Hat, Inc.-1.8.0_302-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/2/testReport/
Max. process+thread count	719 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/2/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-10-10T12:52:09Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 8s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	25m 3s		trunk passed
+1 💚	compile	2m 33s		trunk passed
+1 💚	mvnsite	0m 25s		trunk passed
+1 💚	shadedclient	52m 39s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 16s		the patch passed
+1 💚	compile	2m 26s		the patch passed
+1 💚	cc	2m 26s		the patch passed
+1 💚	golang	2m 26s		the patch passed
+1 💚	javac	2m 26s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 19s		the patch passed
+1 💚	shadedclient	24m 31s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	31m 41s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 33s		The patch does not generate ASF License warnings.
		125m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/2/artifact/out/Dockerfile
GITHUB PR	#3532
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux 431de59b1c4f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `27a0542`
Default Java	Debian-11.0.12+7-post-Debian-2deb10u1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/2/testReport/
Max. process+thread count	571 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/2/console
versions	git=2.20.1 maven=3.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-10-10T14:37:39Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 42s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	20m 27s		trunk passed
+1 💚	compile	2m 50s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	2m 50s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	mvnsite	0m 28s		trunk passed
+1 💚	shadedclient	45m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 18s		the patch passed
+1 💚	compile	2m 40s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	cc	2m 40s		the patch passed
+1 💚	golang	2m 40s		the patch passed
+1 💚	javac	2m 40s		the patch passed
+1 💚	compile	2m 45s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	cc	2m 45s		the patch passed
+1 💚	golang	2m 45s		the patch passed
+1 💚	javac	2m 45s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 20s		the patch passed
+1 💚	shadedclient	18m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	32m 13s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 33s		The patch does not generate ASF License warnings.
		105m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/2/artifact/out/Dockerfile
GITHUB PR	#3532
Optional Tests	dupname asflicense compile cc mvnsite javac unit codespell golang
uname	Linux d66501f5124f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `27a0542`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/2/testReport/
Max. process+thread count	719 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3532/2/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

...dfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_deleteSnapshot.cc

...s-native-client/src/main/native/libhdfspp/tools/hdfs-delete-snapshot/hdfs-delete-snapshot.cc

goiri reviewed Oct 8, 2021

View reviewed changes

GauthamBanasandra added 11 commits October 10, 2021 11:29

Add tests

6fd7070

Refactor tests for reuse

eb2cde3

Allow help as the only one argument

a366819

* Needed to allow the help option as the only parameter that can be passed. * This fixes a test failure.

Add documentation

9f73bfd

Add documentation

cbe7641

Fix typo

cc42a72

Fix documentation

1397f2f

Delete hdfs_deleteSnapshot.cc

66f05fd

Resolve merge conflicts

e7a1ee9

Resolve merge conflicts

27a0542

GauthamBanasandra force-pushed the del-snapshot-x-platform branch from 3712b52 to 27a0542 Compare October 10, 2021 07:13

goiri approved these changes Oct 11, 2021

View reviewed changes

...dfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_deleteSnapshot.cc Show resolved Hide resolved

...s-native-client/src/main/native/libhdfspp/tools/hdfs-delete-snapshot/hdfs-delete-snapshot.cc Show resolved Hide resolved

goiri merged commit a05eb23 into apache:trunk Oct 12, 2021

GauthamBanasandra deleted the del-snapshot-x-platform branch October 12, 2021 04:23

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HDFS-16260. Make hdfs_deleteSnapshot tool cross platform (apache#3532)

fffa443

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16260. Make hdfs_deleteSnapshot tool cross platform#3532

HDFS-16260. Make hdfs_deleteSnapshot tool cross platform#3532
goiri merged 11 commits intoapache:trunkfrom
GauthamBanasandra:del-snapshot-x-platform

GauthamBanasandra commented Oct 8, 2021

Uh oh!

hadoop-yetus commented Oct 8, 2021

Uh oh!

hadoop-yetus commented Oct 8, 2021

Uh oh!

hadoop-yetus commented Oct 8, 2021

Uh oh!

goiri Oct 8, 2021

Uh oh!

goiri Oct 8, 2021

Uh oh!

hadoop-yetus commented Oct 10, 2021

Uh oh!

hadoop-yetus commented Oct 10, 2021

Uh oh!

hadoop-yetus commented Oct 10, 2021

Uh oh!

hadoop-yetus commented Oct 10, 2021

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

GauthamBanasandra commented Oct 8, 2021

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Oct 8, 2021

Uh oh!

hadoop-yetus commented Oct 8, 2021

Uh oh!

hadoop-yetus commented Oct 8, 2021

Uh oh!

goiri Oct 8, 2021

Choose a reason for hiding this comment

Uh oh!

goiri Oct 8, 2021

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Oct 10, 2021

Uh oh!

hadoop-yetus commented Oct 10, 2021

Uh oh!

hadoop-yetus commented Oct 10, 2021

Uh oh!

hadoop-yetus commented Oct 10, 2021

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants