HDDS-2217. Removed redundant LOG4J lines from docker configurations #1582

christeoh · 2019-10-03T13:36:42Z

Removed potentially confusing LOG4J lines from docker configuration

christeoh · 2019-10-03T13:50:53Z

/label ozone

christeoh · 2019-10-03T13:52:34Z

/retest

hadoop-yetus · 2019-10-03T14:51:12Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	429	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
0	shelldocs	0	Shelldocs was not available.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
-1	mvninstall	52	hadoop-hdds in trunk failed.
-1	mvninstall	45	hadoop-ozone in trunk failed.
-1	compile	20	hadoop-hdds in trunk failed.
-1	compile	16	hadoop-ozone in trunk failed.
+1	mvnsite	0	trunk passed
+1	shadedclient	890	branch has no errors when building and testing our client artifacts.
-1	javadoc	22	hadoop-hdds in trunk failed.
-1	javadoc	19	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
-1	mvninstall	35	hadoop-hdds in the patch failed.
-1	mvninstall	38	hadoop-ozone in the patch failed.
-1	compile	24	hadoop-hdds in the patch failed.
-1	compile	19	hadoop-ozone in the patch failed.
-1	javac	24	hadoop-hdds in the patch failed.
-1	javac	19	hadoop-ozone in the patch failed.
+1	mvnsite	0	the patch passed
+1	shellcheck	0	There were no new shellcheck issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	800	patch has no errors when building and testing our client artifacts.
-1	javadoc	21	hadoop-hdds in the patch failed.
-1	javadoc	19	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	27	hadoop-hdds in the patch failed.
-1	unit	25	hadoop-ozone in the patch failed.
+1	asflicense	34	The patch does not generate ASF License warnings.
		2706

Subsystem	Report/Notes
Docker	Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/Dockerfile
GITHUB PR	#1582
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient shellcheck shelldocs
uname	Linux f888af8c88fa 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `5a7483c`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/branch-javadoc-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/patch-javadoc-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/testReport/
Max. process+thread count	306 (vs. ulimit of 5500)
modules	C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/1/console
versions	git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

christeoh · 2019-10-04T04:40:48Z

ci/acceptance appears to be no failing tests.
ci/integration appears to be failing on seemingly unrelated ratis issues?

elek

Thank you very much the patch @christeoh This part looks good to me, but can you please remove the LOG4J2_.... lines as well.

Just as a background:

Audit logging uses a different logger (log4j2) than the normal logging (log4j1). But we don't use the generated log4j2 files at all (=generated based on these environment variables) as the ./bin/ozone shell script configure specific audit log files:

For example:

       HDFS_OM_OPTS="${HDFS_OM_OPTS} -Dlog4j.configurationFile=${HADOOP_CONF_DIR}/om-audit-log4j2.properties"

So we can just remove them.

hadoop-yetus · 2019-10-08T00:47:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	11	#1582 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#1582
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/2/console
versions	git=2.17.1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

christeoh · 2019-10-08T02:37:26Z

/retest

hadoop-yetus · 2019-10-08T02:47:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	10	#1582 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#1582
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/3/console
versions	git=2.17.1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

christeoh · 2019-10-08T05:45:14Z

Looks like there's some issues with Ratis? Is this related to the patch?

elek · 2019-10-08T06:52:42Z

If you think about the integration tests, they are independent (I am working to remove that noise).

I don't know what has been happened but I think you did some merge instead of rebase. Behind this link:

https://github.com/apache/hadoop/pull/1582/files

I see hundred of files which are changed but ideally only docker-config files should be there.

Try:

git checkout HDDS-2217
git rebase origin/trunk
git rebase -i trunk
# chance all the picks to d before the lines which are not part of your patch
git push -f

(Or ping me offline if you have problems...)

christeoh · 2019-10-08T10:44:52Z

/retest

hadoop-yetus · 2019-10-08T12:05:26Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	1191	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
0	shelldocs	0	Shelldocs was not available.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
-1	mvninstall	61	hadoop-hdds in trunk failed.
-1	mvninstall	52	hadoop-ozone in trunk failed.
-1	compile	26	hadoop-hdds in trunk failed.
-1	compile	17	hadoop-ozone in trunk failed.
+1	mvnsite	0	trunk passed
+1	shadedclient	943	branch has no errors when building and testing our client artifacts.
-1	javadoc	22	hadoop-hdds in trunk failed.
-1	javadoc	19	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
-1	mvninstall	35	hadoop-hdds in the patch failed.
-1	mvninstall	38	hadoop-ozone in the patch failed.
-1	compile	25	hadoop-hdds in the patch failed.
-1	compile	18	hadoop-ozone in the patch failed.
-1	javac	25	hadoop-hdds in the patch failed.
-1	javac	18	hadoop-ozone in the patch failed.
+1	mvnsite	0	the patch passed
+1	shellcheck	0	There were no new shellcheck issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	797	patch has no errors when building and testing our client artifacts.
-1	javadoc	22	hadoop-hdds in the patch failed.
-1	javadoc	19	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	27	hadoop-hdds in the patch failed.
-1	unit	25	hadoop-ozone in the patch failed.
+1	asflicense	32	The patch does not generate ASF License warnings.
		3540

Subsystem	Report/Notes
Docker	Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/Dockerfile
GITHUB PR	#1582
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient shellcheck shelldocs
uname	Linux 8625d7f60dca 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `91320b4`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/branch-javadoc-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/patch-javadoc-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/testReport/
Max. process+thread count	306 (vs. ulimit of 5500)
modules	C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1582/4/console
versions	git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

christeoh · 2019-10-08T13:05:21Z

It looks like the yarn resource manager was killed:

https://github.com/elek/ozone-ci-q4/blob/9cd3522fa8b44bf9e20fe4d22106521768b4c7a0/pr/pr-hdds-2217-cfvmz/acceptance/docker-hadoop27-hadoop27-mapreduce-rm.log#L1984

rm_1 | /opt/launcher/plugins/800_launch/launch.sh: line 5: 70 Killed yarn resourcemanager

christeoh · 2019-10-08T20:28:48Z

/retest

elek

rm_1 | /opt/launcher/plugins/800_launch/launch.sh: line 5: 70 Killed yarn resourcemanager

Can be some environment specific issue. I run the hadoop27 m3 multiple times locally and worked well.

1 for the patch. Thank you very much the contribution. Will commit it to the trunk, soon.

(During the commit I also removed three additional, remaining LOG4J from ozonesecure. They can be removed and replaced with insight log level deifnition...)

…g files Closes #1582

christeoh · 2019-10-09T21:12:22Z

Thanks very much for your help @elek and @anuengineer. Onward and upward.

…g files Closes apache#1582

elek added the ozone label Oct 3, 2019

elek reviewed Oct 4, 2019

View reviewed changes

Chris Teoh added 11 commits October 8, 2019 21:37

Removed redundant LOG4J lines from docker configurations

3801f88

Removed redundant LOG4J2 lines

ec6804b

Removed redundant LOG4J2 lines

d7f3b49

Removed redundant LOG4J2 lines

55f6348

Removed redundant LOG4J2 lines

20a5c53

Removed redundant LOG4J2 lines

56d513c

Removed redundant LOG4J2 lines

e7ac2da

Removed redundant LOG4J2 lines

fd25a23

Removed redundant LOG4J2 lines

1baaf8b

Removed redundant LOG4J2 lines

ef6055d

Removed redundant LOG4J2 lines

a57b990

christeoh force-pushed the HDDS-2217 branch from f8a2d73 to a57b990 Compare October 8, 2019 10:42

elek approved these changes Oct 9, 2019

View reviewed changes

elek closed this in 1f954e6 Oct 9, 2019

elek pushed a commit that referenced this pull request Oct 9, 2019

HDDS-2217. Remove log4j and audit configuration from the docker-confi…

4b0a5bc

…g files Closes #1582

amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019

HDDS-2217. Remove log4j and audit configuration from the docker-confi…

cd72670

…g files Closes apache#1582

amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019

HDDS-2217. Remove log4j and audit configuration from the docker-confi…

672ca97

…g files Closes apache#1582

RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020

HDDS-2217. Remove log4j and audit configuration from the docker-confi…

95c8cc2

…g files Closes apache#1582

RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020

HDDS-2217. Remove log4j and audit configuration from the docker-confi…

cab4a46

…g files Closes apache#1582

HDDS-2217. Removed redundant LOG4J lines from docker configurations #1582

HDDS-2217. Removed redundant LOG4J lines from docker configurations #1582

Uh oh!

Conversation

christeoh commented Oct 3, 2019

Uh oh!

christeoh commented Oct 3, 2019

Uh oh!

christeoh commented Oct 3, 2019

Uh oh!

hadoop-yetus commented Oct 3, 2019

Uh oh!

christeoh commented Oct 4, 2019

Uh oh!

elek left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Oct 8, 2019

Uh oh!

christeoh commented Oct 8, 2019

Uh oh!

hadoop-yetus commented Oct 8, 2019

Uh oh!

christeoh commented Oct 8, 2019

Uh oh!

elek commented Oct 8, 2019

Uh oh!

christeoh commented Oct 8, 2019

Uh oh!

hadoop-yetus commented Oct 8, 2019

Uh oh!

christeoh commented Oct 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

christeoh commented Oct 8, 2019

Uh oh!

elek left a comment

Choose a reason for hiding this comment

Uh oh!

christeoh commented Oct 9, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

christeoh commented Oct 8, 2019 •

edited

Loading