Skip to content

HDDS-2225. SCM fails to start in most unsecure environments due to leftover secure config #1568

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
adoroszlai wants to merge 3 commits into from
Closed

HDDS-2225. SCM fails to start in most unsecure environments due to leftover secure config #1568

adoroszlai wants to merge 3 commits into from

Conversation

@adoroszlai
Copy link
Contributor

@adoroszlai adoroszlai commented Oct 1, 2019 •
edited
Loading

What changes were proposed in this pull request?

Remove unnecessary spark container from ozonesecure-mr test, since it generates configs into the wrong directory (a mounted one). These configs are then picked up by other tests and cause failures.

https://issues.apache.org/jira/browse/HDDS-2225

How was this patch tested?

Ran ozonesecure-mr and ozone-recon tests in this order.

@adoroszlai
Copy link
Contributor Author

adoroszlai commented Oct 1, 2019

/label ozone

@elek elek added the ozone label Oct 1, 2019
@hadoop-yetus
Copy link

hadoop-yetus commented Oct 1, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 33 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
0 shelldocs 0 Shelldocs was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
-1 mvninstall 32 hadoop-hdds in trunk failed.
-1 mvninstall 33 hadoop-ozone in trunk failed.
-1 compile 20 hadoop-hdds in trunk failed.
-1 compile 15 hadoop-ozone in trunk failed.
+1 mvnsite 0 trunk passed
+1 shadedclient 872 branch has no errors when building and testing our client artifacts.
-1 javadoc 21 hadoop-hdds in trunk failed.
-1 javadoc 19 hadoop-ozone in trunk failed.
_ Patch Compile Tests _
-1 mvninstall 34 hadoop-hdds in the patch failed.
-1 mvninstall 37 hadoop-ozone in the patch failed.
-1 compile 24 hadoop-hdds in the patch failed.
-1 compile 18 hadoop-ozone in the patch failed.
-1 javac 24 hadoop-hdds in the patch failed.
-1 javac 18 hadoop-ozone in the patch failed.
+1 mvnsite 0 the patch passed
+1 shellcheck 0 There were no new shellcheck issues.
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 788 patch has no errors when building and testing our client artifacts.
-1 javadoc 20 hadoop-hdds in the patch failed.
-1 javadoc 19 hadoop-ozone in the patch failed.
_ Other Tests _
-1 unit 26 hadoop-hdds in the patch failed.
-1 unit 25 hadoop-ozone in the patch failed.
+1 asflicense 30 The patch does not generate ASF License warnings.
2234
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/Dockerfile
GITHUB PR #1568
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient shellcheck shelldocs
uname Linux 1b8876ba4ee1 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / d947ded
Default Java 1.8.0_222
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/branch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/branch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/branch-javadoc-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/patch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/patch-compile-hadoop-hdds.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/patch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/patch-javadoc-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/testReport/
Max. process+thread count 305 (vs. ulimit of 5500)
modules C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/1/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

arp7
arp7 approved these changes Oct 1, 2019
View reviewed changes
Copy link
Contributor

@arp7 arp7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 pending CI

@adoroszlai
Copy link
Contributor Author

adoroszlai commented Oct 2, 2019

Thanks @arp7 for the review.

ozones3 test was run after ozonesecure, but was successfully started and executed, which is the goal of this fix. The only acceptance test failures are in MR createmrenv, which is being fixed in #1537.

@elek
Copy link
Member

elek commented Oct 2, 2019

Thanks @adoroszlai to fix this issue.

Can you please help me to understand how is it possible. My assumption is that all the core-site.xml files are deleted after one test is executed. docker-compose down should delete all the previous state.

@adoroszlai
Copy link
Contributor Author

adoroszlai commented Oct 2, 2019

Can you please help me to understand how is it possible. My assumption is that all the core-site.xml files are deleted after one test is executed. docker-compose down should delete all the previous state.

Thanks @elek for the question. After some more research it turns out that only ozonesecure-mr is flawed: its spark container mounts ../..:/opt/hadoop, and apparently it generates configs into /opt/hadoop/etc/hadoop. These configs are not removed by docker-compose down since it's a mount. It seems they are picked up by subsequent tests, although they generate config into /etc/hadoop.

@elek
Copy link
Member

elek commented Oct 2, 2019

Somewhat drastic but effective solution ;-) I am fine with removing it for now, but we need to create a proper ozone-spark later (with smoketest).

@hadoop-yetus
Copy link

hadoop-yetus commented Oct 2, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 3348 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
0 shelldocs 1 Shelldocs was not available.
0 yamllint 1 yamllint was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
-1 mvninstall 53 hadoop-hdds in trunk failed.
-1 mvninstall 52 hadoop-ozone in trunk failed.
-1 compile 25 hadoop-hdds in trunk failed.
-1 compile 18 hadoop-ozone in trunk failed.
+1 mvnsite 0 trunk passed
+1 shadedclient 996 branch has no errors when building and testing our client artifacts.
-1 javadoc 28 hadoop-hdds in trunk failed.
-1 javadoc 24 hadoop-ozone in trunk failed.
_ Patch Compile Tests _
-1 mvninstall 39 hadoop-hdds in the patch failed.
-1 mvninstall 43 hadoop-ozone in the patch failed.
-1 compile 28 hadoop-hdds in the patch failed.
-1 compile 22 hadoop-ozone in the patch failed.
-1 javac 28 hadoop-hdds in the patch failed.
-1 javac 22 hadoop-ozone in the patch failed.
+1 mvnsite 0 the patch passed
+1 shellcheck 1 There were no new shellcheck issues.
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 851 patch has no errors when building and testing our client artifacts.
-1 javadoc 27 hadoop-hdds in the patch failed.
-1 javadoc 20 hadoop-ozone in the patch failed.
_ Other Tests _
-1 unit 32 hadoop-hdds in the patch failed.
-1 unit 29 hadoop-ozone in the patch failed.
+1 asflicense 38 The patch does not generate ASF License warnings.
5864
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/Dockerfile
GITHUB PR #1568
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient shellcheck shelldocs yamllint
uname Linux 1cf1d4db42fc 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / f1ba9bf
Default Java 1.8.0_222
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/branch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/branch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/branch-javadoc-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/patch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/patch-compile-hadoop-hdds.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/patch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/patch-javadoc-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/testReport/
Max. process+thread count 415 (vs. ulimit of 5500)
modules C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/2/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Oct 3, 2019

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 305 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
0 shelldocs 0 Shelldocs was not available.
0 yamllint 0 yamllint was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
-1 mvninstall 47 hadoop-hdds in trunk failed.
-1 mvninstall 43 hadoop-ozone in trunk failed.
-1 compile 24 hadoop-hdds in trunk failed.
-1 compile 19 hadoop-ozone in trunk failed.
+1 mvnsite 0 trunk passed
+1 shadedclient 794 branch has no errors when building and testing our client artifacts.
-1 javadoc 26 hadoop-hdds in trunk failed.
-1 javadoc 25 hadoop-ozone in trunk failed.
_ Patch Compile Tests _
-1 mvninstall 39 hadoop-hdds in the patch failed.
-1 mvninstall 41 hadoop-ozone in the patch failed.
-1 compile 29 hadoop-hdds in the patch failed.
-1 compile 24 hadoop-ozone in the patch failed.
-1 javac 29 hadoop-hdds in the patch failed.
-1 javac 25 hadoop-ozone in the patch failed.
+1 mvnsite 0 the patch passed
+1 shellcheck 0 There were no new shellcheck issues.
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 711 patch has no errors when building and testing our client artifacts.
-1 javadoc 27 hadoop-hdds in the patch failed.
-1 javadoc 24 hadoop-ozone in the patch failed.
_ Other Tests _
-1 unit 32 hadoop-hdds in the patch failed.
-1 unit 30 hadoop-ozone in the patch failed.
+1 asflicense 39 The patch does not generate ASF License warnings.
2492
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/Dockerfile
GITHUB PR #1568
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient shellcheck shelldocs yamllint
uname Linux ba433dd9aeed 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 51eaeca
Default Java 1.8.0_222
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/branch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/branch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/branch-javadoc-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/patch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/patch-compile-hadoop-hdds.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/patch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/patch-javadoc-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/testReport/
Max. process+thread count 448 (vs. ulimit of 5500)
modules C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1568/3/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@anuengineer
Copy link
Contributor

anuengineer commented Oct 3, 2019

@adoroszlai So now we have removed the mount and gen config? I am presuming that +1s were given for the earlier solution, but with force push I am not able to see the older changes.

@adoroszlai
Copy link
Contributor Author

adoroszlai commented Oct 4, 2019

@anuengineer Thanks for taking a look at this.

So now we have removed the mount and gen config?

The new solution only removes the offending container (spark), which is not required by the test at all. Volume mount and config generation for the other containers is not changed.

I am presuming that +1s were given for the earlier solution, but with force push I am not able to see the older changes.

Both earlier solution (ff36710) and new one (5caa23a) are visible in the list of commits. The second commit just reverts the first attempt. The force push was needed only to rebase on current trunk. I agree, it makes understanding the conversation harder. I'll try a plain merge next time.

Earlier +1s were for the first solution, while @elek's latest comment is for the second one.

@elek elek closed this in ec8f691 Oct 4, 2019
@elek
Copy link
Member

elek commented Oct 4, 2019

Thanks the patch @adoroszlai I am committing it to the trunk. Let's fix Spark later.

@adoroszlai adoroszlai deleted the HDDS-2225 branch October 4, 2019 06:57
@adoroszlai
Copy link
Contributor Author

adoroszlai commented Oct 4, 2019

Thanks @anuengineer, @arp7 and @elek for the reviews.

amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019
@adoroszlai
HDDS-2225. SCM fails to start in most unsecure environments due to le…
fd73222 
…ftover secure config

Closes apache#1568
RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020
@adoroszlai @RogPodge
HDDS-2225. SCM fails to start in most unsecure environments due to le…
f9b8f0b 
…ftover secure config

Closes apache#1568
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@arp7 arp7 arp7 approved these changes

Assignees

No one assigned

Labels

ozone

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@adoroszlai @hadoop-yetus @elek @anuengineer @arp7