Skip to content

Backport "HBASE-28686 MapReduceBackupCopyJob should support custom DistCp options (#6017)" to branch-2 #6093

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ndimiduk wants to merge 1 commit into from
Closed

Backport "HBASE-28686 MapReduceBackupCopyJob should support custom DistCp options (#6017)" to branch-2 #6093

ndimiduk wants to merge 1 commit into from

Conversation

@ndimiduk
Copy link
Member

@ndimiduk ndimiduk commented Jul 18, 2024

No description provided.

@rmdmattingly @ndimiduk
HBASE-28686 MapReduceBackupCopyJob should support custom DistCp optio…
099d48f 
…ns (apache#6017)

Co-authored-by: Ray Mattingly <[email protected]>
Signed-off-by: Duo Zhang <[email protected]>
Signed-off-by: Nick Dimiduk <[email protected]>
@ndimiduk ndimiduk added the backport This PR is a back port of some issue or issues already committed to master label Jul 18, 2024
@Apache-HBase
Copy link

Apache-HBase commented Jul 18, 2024

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 39s Docker mode activated.
-0 ⚠️ yetus 0m 5s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚 mvninstall 2m 32s branch-2 passed
+1 💚 compile 0m 16s branch-2 passed
+1 💚 javadoc 0m 13s branch-2 passed
+1 💚 shadedjars 5m 17s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
-1 ❌ mvninstall 2m 24s /patch-mvninstall-root.txt root in the patch failed.
-1 ❌ compile 0m 15s /patch-compile-hbase-backup.txt hbase-backup in the patch failed.
-0 ⚠️ javac 0m 15s /patch-compile-hbase-backup.txt hbase-backup in the patch failed.
+1 💚 javadoc 0m 12s the patch passed
+1 💚 shadedjars 5m 19s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
-1 ❌ unit 0m 15s /patch-unit-hbase-backup.txt hbase-backup in the patch failed.
18m 27s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR #6093
Optional Tests javac javadoc unit compile shadedjars
uname Linux f7adbd582070 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2 / 099d48f
Default Java Temurin-1.8.0_412-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/testReport/
Max. process+thread count 67 (vs. ulimit of 30000)
modules C: hbase-backup U: hbase-backup
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented Jul 18, 2024

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 4s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
_ branch-2 Compile Tests _
+1 💚 mvninstall 4m 29s branch-2 passed
+1 💚 compile 0m 36s branch-2 passed
+1 💚 checkstyle 0m 16s branch-2 passed
+1 💚 spotbugs 0m 43s branch-2 passed
+1 💚 spotless 0m 57s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 💚 mvninstall 3m 27s the patch passed
+1 💚 compile 0m 30s the patch passed
+1 💚 javac 0m 30s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 10s the patch passed
+1 💚 spotbugs 0m 45s the patch passed
-1 ❌ hadoopcheck 3m 9s The patch causes 20 errors with Hadoop v2.10.2.
+1 💚 spotless 1m 6s patch has no errors when running spotless:check.
_ Other Tests _
+1 💚 asflicense 0m 15s The patch does not generate ASF License warnings.
33m 18s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #6093
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname Linux 32accbcdce6e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2 / 099d48f
Default Java Eclipse Adoptium-11.0.23+9
hadoopcheck https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/artifact/yetus-general-check/output/patch-javac-2.10.2.txt
Max. process+thread count 79 (vs. ulimit of 30000)
modules C: hbase-backup U: hbase-backup
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented Jul 18, 2024

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 51s Docker mode activated.
-0 ⚠️ yetus 0m 6s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚 mvninstall 4m 35s branch-2 passed
+1 💚 compile 0m 44s branch-2 passed
+1 💚 javadoc 0m 23s branch-2 passed
+1 💚 shadedjars 6m 44s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 3m 47s the patch passed
+1 💚 compile 0m 23s the patch passed
+1 💚 javac 0m 23s the patch passed
+1 💚 javadoc 0m 16s the patch passed
+1 💚 shadedjars 6m 9s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚 unit 11m 37s hbase-backup in the patch passed.
36m 48s
Subsystem Report/Notes
Docker ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR #6093
Optional Tests javac javadoc unit compile shadedjars
uname Linux 7a570803db05 5.4.0-186-generic #206-Ubuntu SMP Fri Apr 26 12:31:10 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2 / 099d48f
Default Java Eclipse Adoptium-11.0.23+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/testReport/
Max. process+thread count 3824 (vs. ulimit of 30000)
modules C: hbase-backup U: hbase-backup
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented Jul 18, 2024

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 53s Docker mode activated.
-0 ⚠️ yetus 0m 4s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚 mvninstall 4m 50s branch-2 passed
+1 💚 compile 0m 34s branch-2 passed
+1 💚 javadoc 0m 21s branch-2 passed
+1 💚 shadedjars 6m 41s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 4m 31s the patch passed
+1 💚 compile 0m 22s the patch passed
+1 💚 javac 0m 22s the patch passed
+1 💚 javadoc 0m 21s the patch passed
+1 💚 shadedjars 6m 38s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚 unit 14m 34s hbase-backup in the patch passed.
40m 54s
Subsystem Report/Notes
Docker ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR #6093
Optional Tests javac javadoc unit compile shadedjars
uname Linux 37c6f3b538e5 5.4.0-177-generic #197-Ubuntu SMP Thu Mar 28 22:45:47 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision branch-2 / 099d48f
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/testReport/
Max. process+thread count 2993 (vs. ulimit of 30000)
modules C: hbase-backup U: hbase-backup
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6093/1/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@ndimiduk
Copy link
Member Author

ndimiduk commented Jul 18, 2024

@rmdmattingly it looks like we need an addendum for Hadoop2. That version of DistCP does not have the constant DistCpConstants#CONF_LABEL_DIRECT_WRITE and indeed may not support the feature.

@rmdmattingly
Copy link
Contributor

rmdmattingly commented Jul 19, 2024

I opened up this replacement that should build: #6102

@ndimiduk
Copy link
Member Author

ndimiduk commented Jul 22, 2024

Superseded by #6102.

@ndimiduk ndimiduk closed this Jul 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backport This PR is a back port of some issue or issues already committed to master

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ndimiduk @Apache-HBase @rmdmattingly