HDDS-1226. ozone-filesystem jar missing in hadoop classpath #560

vivekratnavel · 2019-03-06T01:41:11Z

hadoop-ozone-filesystem-lib-*.jar is missing in hadoop classpath.

This PR aims at having a shaded hadoop-ozone-filesystem-lib-*.jar file in ${OZONE_HOME}/share/hadoop/ozonefs/ path

vivekratnavel · 2019-03-06T01:41:58Z

@bharatviswa504 @elek @avijayanhwx @swagle

hadoop-yetus · 2019-03-06T02:12:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	20	Docker mode activated.
		_ Prechecks _
+1	@author	0	The patch does not contain any @author tags.
		_ trunk Compile Tests _
+1	shadedclient	824	branch has no errors when building and testing our client artifacts.
		_ Patch Compile Tests _
+1	shellcheck	1	There were no new shellcheck issues.
+1	shelldocs	14	There were no new shelldocs issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	795	patch has no errors when building and testing our client artifacts.
		_ Other Tests _
+1	asflicense	30	The patch does not generate ASF License warnings.
		1784

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-560/1/artifact/out/Dockerfile
GITHUB PR	#560
Optional Tests	dupname asflicense shellcheck shelldocs
uname	Linux c2280c5ecb6c 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `945b504`
maven	version: Apache Maven 3.3.9
shellcheck	v0.4.6
Max. process+thread count	340 (vs. ulimit of 5500)
modules	C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-560/1/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

avijayanhwx · 2019-03-06T04:32:13Z

LGTM

elek

Hadoop file system jars are already there:

ls share/ozone/lib/hadoop-ozone-filesystem-*
share/ozone/lib/hadoop-ozone-filesystem-0.4.0-SNAPSHOT.jar
share/ozone/lib/hadoop-ozone-filesystem-lib-0.4.0-SNAPSHOT.jar
share/ozone/lib/hadoop-ozone-filesystem-lib-legacy-0.4.0-SNAPSHOT.jar

And because they are huge (100M) I wouldn't duplicate them. ;-)

But you are right, the share/hadoop/ozone directory is confusing as most of the stuff are already moved to share/ozone/...

If you prefer to move the datanodeplugin to share/ozone I would agree...

elek · 2019-03-06T17:19:03Z

Discussed with @vivekratnavel offline:

The big question here is that how can we put the jar files to the classpath in a version independent mode.

We can use HADOOP_CLASSPATH=share/ozone/lib/hadoop-ozone-filesystem-lib-legacy* but can't use any wildcard for the normal lib as the legacy jar file also will be matched.

One possible solution is to rename the ozonefs-lib project to ozonefs-lib-current.

bharatviswa504 · 2019-03-07T02:01:01Z

hadoop-ozone/dist/dev-support/bin/dist-layout-stitching

 #shaded datanode service
 run mkdir -p "./share/hadoop/ozoneplugin"
 run cp "${ROOT}/hadoop-ozone/objectstore-service/target/hadoop-ozone-objectstore-service-${HDDS_VERSION}-plugin.jar" "./share/hadoop/ozoneplugin/hadoop-ozone-datanode-plugin-${HDDS_VERSION}.jar"



We can open a new Jira to remove share/hadoop/ozoneplugin also?

Yes, we can open one. But rename instead of remove. ozoneplugin is the directory of the ozone datanode plugin for legacy hdfs clusters. We need the jar file but can be moved to share/ozone/datanodeplugin

hadoop-yetus · 2019-03-07T03:41:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	24	Docker mode activated.
		_ Prechecks _
0	yamllint	1	yamllint was not available.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
0	mvndep	69	Maven dependency ordering for branch
+1	mvninstall	1033	trunk passed
+1	compile	961	trunk passed
-1	mvnsite	112	hadoop-ozone in trunk failed.
+1	shadedclient	654	branch has no errors when building and testing our client artifacts.
+1	javadoc	162	trunk passed
		_ Patch Compile Tests _
0	mvndep	26	Maven dependency ordering for patch
-1	mvninstall	108	hadoop-ozone in the patch failed.
-1	mvninstall	14	dist in the patch failed.
-1	mvninstall	11	ozonefs-lib in the patch failed.
+1	compile	905	the patch passed
+1	javac	905	the patch passed
-1	mvnsite	94	hadoop-ozone in the patch failed.
-1	mvnsite	30	ozonefs-lib in the patch failed.
+1	shellcheck	0	There were no new shellcheck issues.
+1	shelldocs	38	There were no new shelldocs issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	4	The patch has no ill-formed XML file.
+1	shadedclient	615	patch has no errors when building and testing our client artifacts.
-1	javadoc	27	ozonefs-lib in the patch failed.
		_ Other Tests _
+1	unit	33	docs in the patch passed.
-1	unit	110	hadoop-ozone in the patch failed.
+1	unit	35	dist in the patch passed.
-1	unit	28	ozonefs-lib in the patch failed.
+1	unit	32	ozonefs-lib-current in the patch passed.
+1	asflicense	46	The patch does not generate ASF License warnings.
		5847

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-560/2/artifact/out/Dockerfile
GITHUB PR	#560
Optional Tests	dupname asflicense mvnsite shellcheck shelldocs compile javac javadoc mvninstall unit shadedclient xml yamllint
uname	Linux 320e3c4e668a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `a55fc36`
maven	version: Apache Maven 3.3.9
Default Java	1.8.0_191
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-560/2/artifact/out/branch-mvnsite-hadoop-ozone.txt
shellcheck	v0.4.6
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-560/2/artifact/out/patch-mvninstall-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-560/2/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-560/2/artifact/out/patch-mvninstall-hadoop-ozone_ozonefs-lib.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-560/2/artifact/out/patch-mvnsite-hadoop-ozone.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-560/2/artifact/out/patch-mvnsite-hadoop-ozone_ozonefs-lib.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-560/2/artifact/out/patch-javadoc-hadoop-ozone_ozonefs-lib.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-560/2/artifact/out/patch-unit-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-560/2/artifact/out/patch-unit-hadoop-ozone_ozonefs-lib.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-560/2/testReport/
Max. process+thread count	469 (vs. ulimit of 5500)
modules	C: hadoop-hdds/docs hadoop-ozone hadoop-ozone/dist hadoop-ozone/ozonefs-lib hadoop-ozone/ozonefs-lib-current U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-560/2/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

elek · 2019-03-07T13:29:29Z

+1. Merged it to the trunk.Thanks the contribution @vivekratnavel

As this is a small risk patch and it would be better to finalize the name earlier, I cherry-pick it to branch-0.4 to be part of the next release.

Closes #560 (cherry picked from commit 475011b)

hadoop-yetus · 2019-03-07T16:58:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	7	#560 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#560
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-560/3/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

elek · 2019-03-13T11:08:26Z

The last commit (typo fix) added as an addendum commit.

Closes #560

Closes #560 (cherry picked from commit c1649bf)

This is due to the recent refactoring of table metrics, for some reason running build locally didn't catch these failed tests. Author: Wei Song <[email protected]> Author: Cameron Lee <[email protected]> Author: Jagadish <[email protected]> Author: Ray Matharu <[email protected]> Reviewers: Jagadish<[email protected]>, Cameron L<[email protected]> Closes apache#560 from weisong44/table-metrics-fix

HDDS-1226. ozone-filesystem jar missing in hadoop classpath

417bb1c

elek added the ozone label Mar 6, 2019

elek requested changes Mar 6, 2019

View reviewed changes

vivekratnavel added 2 commits March 6, 2019 12:59

refactor ozonefs-lib

211a5dd

Merge branch 'trunk' into HDDS-1226

4d4cedd

bharatviswa504 reviewed Mar 7, 2019

View reviewed changes

fix typos

9dda5cf

elek closed this in 475011b Mar 7, 2019

elek pushed a commit that referenced this pull request Mar 7, 2019

HDDS-1226. ozone-filesystem jar missing in hadoop classpath

62ab342

Closes #560 (cherry picked from commit 475011b)

elek reopened this Mar 7, 2019

elek closed this Mar 7, 2019

elek reopened this Mar 7, 2019

Merge branch 'trunk' into HDDS-1226

4307516

elek closed this Mar 13, 2019

elek pushed a commit that referenced this pull request Mar 13, 2019

HDDS-1226. Addendum. ozone-filesystem jar missing in hadoop classpath

c1649bf

Closes #560

elek pushed a commit that referenced this pull request Mar 13, 2019

HDDS-1226. Addendum. ozone-filesystem jar missing in hadoop classpath

32477ed

Closes #560 (cherry picked from commit c1649bf)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-1226. ozone-filesystem jar missing in hadoop classpath #560

HDDS-1226. ozone-filesystem jar missing in hadoop classpath #560

Uh oh!

vivekratnavel commented Mar 6, 2019

Uh oh!

vivekratnavel commented Mar 6, 2019

Uh oh!

hadoop-yetus commented Mar 6, 2019

Uh oh!

avijayanhwx commented Mar 6, 2019

Uh oh!

elek left a comment

Uh oh!

elek commented Mar 6, 2019

Uh oh!

bharatviswa504 Mar 7, 2019

Uh oh!

elek Mar 7, 2019

Uh oh!

hadoop-yetus commented Mar 7, 2019

Uh oh!

elek commented Mar 7, 2019

Uh oh!

hadoop-yetus commented Mar 7, 2019

Uh oh!

elek commented Mar 13, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

HDDS-1226. ozone-filesystem jar missing in hadoop classpath #560

HDDS-1226. ozone-filesystem jar missing in hadoop classpath #560

Uh oh!

Conversation

vivekratnavel commented Mar 6, 2019

Uh oh!

vivekratnavel commented Mar 6, 2019

Uh oh!

hadoop-yetus commented Mar 6, 2019

Uh oh!

avijayanhwx commented Mar 6, 2019

Uh oh!

elek left a comment

Choose a reason for hiding this comment

Uh oh!

elek commented Mar 6, 2019

Uh oh!

bharatviswa504 Mar 7, 2019

Choose a reason for hiding this comment

Uh oh!

elek Mar 7, 2019

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Mar 7, 2019

Uh oh!

elek commented Mar 7, 2019

Uh oh!

hadoop-yetus commented Mar 7, 2019

Uh oh!

elek commented Mar 13, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants