Skip to content

HDFS-16178. Make recursive rmdir in libhdfs++ cross platform #3311

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
goiri merged 2 commits into from
Aug 20, 2021
Merged

HDFS-16178. Make recursive rmdir in libhdfs++ cross platform #3311

goiri merged 2 commits into from
Aug 20, 2021

Conversation

@GauthamBanasandra
Copy link
Member

@GauthamBanasandra GauthamBanasandra commented Aug 18, 2021 •
edited
Loading

Description of PR

  • The TempDir class in libhdfs++ is currently
    using nftw API provided by ftw.h, which is
    only present in Linux and not on Windows.
  • This PR uses the remove_all API from C++17
    std::filesystem to make this cross platform
    in an equivalent manner.

How was this patch tested?

The API used here was tested with standalone
program -
GauthamBanasandra/x-platform@87e68a6

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@GauthamBanasandra
HDFS-16178. Make recursive rmdir in libhdfs++ cross platform
93d8ed5 
* The TempDir class in libhdfs++ is currently
  using nftw API provided by ftw.h, which is
  only present in Linux and not on Windows.
* This PR uses the APIs from C++17
  std::filesystem to make this cross platform.
@hadoop-yetus
Copy link

hadoop-yetus commented Aug 19, 2021

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 83m 26s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 47m 6s trunk passed
+1 💚 compile 5m 1s trunk passed
+1 💚 mvnsite 0m 34s trunk passed
+1 💚 shadedclient 74m 15s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 16s the patch passed
+1 💚 compile 4m 57s the patch passed
+1 💚 cc 4m 57s the patch passed
+1 💚 golang 4m 57s the patch passed
+1 💚 javac 4m 57s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 54s the patch passed
+1 💚 shadedclient 23m 33s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 210m 44s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt hadoop-hdfs-native-client in the patch failed.
+1 💚 asflicense 0m 38s The patch does not generate ASF License warnings.
401m 29s
Reason Tests
Failed CTEST tests test_libhdfs_threaded_hdfspp_test_shim_static
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/1/artifact/out/Dockerfile
GITHUB PR #3311
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell golang
uname Linux 0518f36ad580 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 93d8ed5
Default Java Red Hat, Inc.-1.8.0_302-b08
CTEST https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/1/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/1/testReport/
Max. process+thread count 596 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/1/console
versions git=2.9.5 maven=3.6.3
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Aug 19, 2021

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 56s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 34m 2s trunk passed
+1 💚 compile 2m 34s trunk passed
+1 💚 mvnsite 0m 23s trunk passed
+1 💚 shadedclient 53m 13s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 15s the patch passed
+1 💚 compile 2m 25s the patch passed
+1 💚 cc 2m 25s the patch passed
+1 💚 golang 2m 25s the patch passed
+1 💚 javac 2m 25s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 15s the patch passed
+1 💚 shadedclient 16m 6s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 86m 4s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 0m 30s The patch does not generate ASF License warnings.
161m 50s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/artifact/out/Dockerfile
GITHUB PR #3311
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell golang
uname Linux b714f584b549 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 0ae3c13
Default Java Red Hat, Inc.-1.8.0_302-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/testReport/
Max. process+thread count 517 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/console
versions git=2.9.5 maven=3.6.3
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Aug 20, 2021

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 25m 34s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 23m 10s trunk passed
+1 💚 compile 2m 45s trunk passed
+1 💚 mvnsite 0m 35s trunk passed
+1 💚 shadedclient 42m 47s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 19s the patch passed
+1 💚 compile 2m 34s the patch passed
+1 💚 cc 2m 34s the patch passed
+1 💚 golang 2m 34s the patch passed
+1 💚 javac 2m 34s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 22s the patch passed
+1 💚 shadedclient 15m 51s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 86m 45s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 0m 38s The patch does not generate ASF License warnings.
177m 1s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/artifact/out/Dockerfile
GITHUB PR #3311
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell golang
uname Linux 13a7f62566ec 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 0ae3c13
Default Java Red Hat, Inc.-1.8.0_302-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/testReport/
Max. process+thread count 516 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/console
versions git=2.27.0 maven=3.6.3
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Aug 20, 2021

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 15m 3s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 30m 3s trunk passed
+1 💚 compile 2m 25s trunk passed
+1 💚 mvnsite 0m 22s trunk passed
+1 💚 shadedclient 53m 16s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 14s the patch passed
+1 💚 compile 2m 19s the patch passed
+1 💚 cc 2m 19s the patch passed
+1 💚 golang 2m 19s the patch passed
+1 💚 javac 2m 19s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 15s the patch passed
+1 💚 shadedclient 20m 15s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 90m 48s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 0m 30s The patch does not generate ASF License warnings.
184m 50s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/artifact/out/Dockerfile
GITHUB PR #3311
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell golang
uname Linux 6a46954d984e 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 0ae3c13
Default Java Debian-11.0.12+7-post-Debian-2
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/testReport/
Max. process+thread count 532 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/console
versions git=2.32.0 maven=3.6.3
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Aug 20, 2021

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 0s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 23m 7s trunk passed
+1 💚 compile 2m 54s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 2m 52s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 mvnsite 0m 21s trunk passed
+1 💚 shadedclient 45m 4s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 13s the patch passed
+1 💚 compile 2m 41s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 cc 2m 41s the patch passed
+1 💚 golang 2m 41s the patch passed
+1 💚 javac 2m 41s the patch passed
+1 💚 compile 2m 48s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 cc 2m 48s the patch passed
+1 💚 golang 2m 48s the patch passed
+1 💚 javac 2m 48s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 16s the patch passed
+1 💚 shadedclient 15m 58s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 86m 19s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 0m 28s The patch does not generate ASF License warnings.
156m 53s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/artifact/out/Dockerfile
GITHUB PR #3311
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell golang
uname Linux 7699f5deb741 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 0ae3c13
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/testReport/
Max. process+thread count 520 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3311/2/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@goiri goiri merged commit b6d1971 into apache:trunk Aug 20, 2021
@GauthamBanasandra GauthamBanasandra deleted the rmdir-x-platform branch August 20, 2021 18:31
kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021
@GauthamBanasandra
HDFS-16178. Make recursive rmdir in libhdfs++ cross platform (apache#…
9c59c12 
…3311)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@goiri goiri goiri approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@GauthamBanasandra @hadoop-yetus @goiri