Skip to content

HDDS-5123. Use the pre-created apache/ozone-testkrb5 image during secure acceptance tests #2165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
elek merged 32 commits into from
May 31, 2021
Merged

HDDS-5123. Use the pre-created apache/ozone-testkrb5 image during secure acceptance tests #2165

Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
29e48e1
secure docker
elek Jan 15, 2021
34bc3c1
fix ozonesecure-mr and ozonesecure-om-ha
elek Feb 19, 2021
8e2ab42
keytab refresh
elek Feb 19, 2021
52df597
Merge remote-tracking branch 'origin/master' into docker-secure
elek Feb 19, 2021
6fe7538
license and HTTP.keytab fix
elek Feb 25, 2021
f6e4541
keytab update
elek Feb 25, 2021
2a872b8
keytab regeneration
elek Feb 26, 2021
3ecce67
retrigger build
elek Feb 28, 2021
9117315
do not filter pre-generated keytabs
elek Feb 28, 2021
f778bc0
keytab and script update
elek Mar 1, 2021
7b858c3
new parametrized build for generating the missing om/testuser
elek Mar 1, 2021
da51447
retrigger build
elek Mar 1, 2021
e1314d8
update keytabs with hadoop/rm
elek Mar 1, 2021
58fbd04
fix keytabs of nm/rm/jhs
elek Mar 2, 2021
a8728e7
adding HTTP/ozone keytab
elek Mar 2, 2021
ab1c6dc
fix krb5.conf for kms
elek Mar 2, 2021
41f9a1d
keytab update
elek Mar 2, 2021
b54c313
retrigger build
elek Mar 2, 2021
25134e1
restore jhs container name
elek Mar 8, 2021
b8d3fb7
switch to the latest secure image
elek Apr 19, 2021
40791c0
Merge remote-tracking branch 'origin/master' into docker-secure
elek Apr 19, 2021
100aa8e
update keytabs
elek Apr 19, 2021
694a27a
fix the new ozonesecure-ha
elek Apr 21, 2021
a670db6
Merge remote-tracking branch 'origin/master' into docker-secure-work
elek Apr 26, 2021
044183d
address review comments
elek May 11, 2021
323941d
restore WAITFOR lines
elek May 11, 2021
fa3fd36
Merge remote-tracking branch 'origin/master' into docker-secure-work
elek May 11, 2021
0f298cc
fix host name fix
elek May 13, 2021
9e6fdaf
Merge remote-tracking branch 'origin/master' into docker-secure-work
elek May 17, 2021
ffd06c2
use scm kerberos instance name
elek May 19, 2021
b56a846
Merge remote-tracking branch 'origin/master' into docker-secure-work
elek May 19, 2021
73057b5
additional fixes related to the scm instance name
elek May 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions hadoop-ozone/dist/dev-support/bin/dist-layout-stitching
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,12 @@ run cp -p -r "${ROOT}/hadoop-ozone/dist/src/main/smoketest" .
run cp -p -r "${ROOT}/hadoop-ozone/dist/target/k8s" kubernetes
run cp -p -r "${ROOT}/hadoop-ozone/dist/target/Dockerfile" .

#Copy pre-generated keytabs
run cp -p -R "${ROOT}/hadoop-ozone/dist/src/main/keytabs" compose/ozonesecure
run cp -p -R "${ROOT}/hadoop-ozone/dist/src/main/keytabs" compose/ozonesecure-mr
run cp -p -R "${ROOT}/hadoop-ozone/dist/src/main/keytabs" compose/ozonesecure-om-ha
run cp -p -R "${ROOT}/hadoop-ozone/dist/src/main/keytabs" compose/ozonesecure-ha
Comment thread
adoroszlai marked this conversation as resolved.
Outdated

#workaround for https://issues.apache.org/jira/browse/MRESOURCES-236
find ./compose -name "*.sh" -exec chmod 755 {} \;
find ./kubernetes -name "*.sh" -exec chmod 755 {} \;
1 change: 1 addition & 0 deletions hadoop-ozone/dist/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
<file.encoding>UTF-8</file.encoding>
<downloadSources>true</downloadSources>
<docker.ozone-runner.version>20210329-1</docker.ozone-runner.version>
<docker.ozone-testkr5b.image>apache/ozone-testkrb5:20210419-1</docker.ozone-testkr5b.image>
</properties>

<build>
Expand Down
35 changes: 0 additions & 35 deletions hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/Dockerfile-krb5
View file
Open in desktop

This file was deleted.

34 changes: 0 additions & 34 deletions hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/README.md
View file
Open in desktop

This file was deleted.

20 changes: 0 additions & 20 deletions hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/kadm5.acl
View file
Open in desktop

This file was deleted.

25 changes: 0 additions & 25 deletions hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/launcher.sh
View file
Open in desktop

This file was deleted.

1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure-ha/.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,5 @@
HDDS_VERSION=${hdds.version}
HADOOP_VERSION=3
OZONE_RUNNER_VERSION=${docker.ozone-runner.version}
OZONE_TESTKRB5_IMAGE=${docker.ozone-testkr5b.image}
OZONE_OPTS=
47 changes: 27 additions & 20 deletions hadoop-ozone/dist/src/main/compose/ozonesecure-ha/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,12 @@
version: "3"
services:
kdc:
build:
context: ../common/docker-image/docker-krb5
dockerfile: Dockerfile-krb5
image: ozone-insecure-krb5
image: ${OZONE_TESTKRB5_IMAGE}
hostname: kdc
volumes:
- ../..:/opt/hadoop
- ./keytabs:/etc/security/keytabs
command: ["krb5kdc","-n"]
networks:
ozone_net:
ipv4_address: 172.25.0.100
Expand All @@ -33,10 +32,12 @@ services:
- 9600:9600
env_file:
- ./docker-config
environment:
HADOOP_CONF_DIR: /opt/hadoop/etc/hadoop
volumes:
- ./keytabs:/etc/security/keytabs
- ./krb5.conf:/etc/krb5.conf
- ../../libexec/transformation.py:/opt/transformation.py
environment:
HADOOP_CONF_DIR: /opt/hadoop/etc/hadoop
command: ["hadoop", "kms"]
networks:
ozone_net:
Expand All @@ -45,6 +46,8 @@ services:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
volumes:
- ../..:/opt/hadoop
- ./keytabs:/etc/security/keytabs
- ./krb5.conf:/etc/krb5.conf
ports:
- 9864:9999
command: ["/opt/hadoop/bin/ozone","datanode"]
Expand All @@ -55,8 +58,6 @@ services:
env_file:
- docker-config
environment:
WAITFOR: scm3.org:9865
Comment thread
adoroszlai marked this conversation as resolved.
Outdated
KERBEROS_KEYTABS: dn HTTP
OZONE_OPTS:
networks:
ozone_net:
Expand All @@ -65,6 +66,8 @@ services:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
volumes:
- ../..:/opt/hadoop
- ./keytabs:/etc/security/keytabs
- ./krb5.conf:/etc/krb5.conf
ports:
- 9866:9999
command: ["/opt/hadoop/bin/ozone","datanode"]
Expand All @@ -75,8 +78,6 @@ services:
env_file:
- docker-config
environment:
WAITFOR: scm3.org:9865
KERBEROS_KEYTABS: dn HTTP
OZONE_OPTS:
networks:
ozone_net:
Expand All @@ -85,6 +86,8 @@ services:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
volumes:
- ../..:/opt/hadoop
- ./keytabs:/etc/security/keytabs
- ./krb5.conf:/etc/krb5.conf
ports:
- 9868:9999
command: ["/opt/hadoop/bin/ozone","datanode"]
Expand All @@ -95,8 +98,6 @@ services:
env_file:
- docker-config
environment:
WAITFOR: scm3.org:9865
KERBEROS_KEYTABS: dn HTTP
OZONE_OPTS:
networks:
ozone_net:
Expand All @@ -106,14 +107,15 @@ services:
hostname: om1
volumes:
- ../..:/opt/hadoop
- ./keytabs:/etc/security/keytabs
- ./krb5.conf:/etc/krb5.conf
ports:
- 9880:9874
- 9890:9872
#- 18001:18001
environment:
WAITFOR: scm3.org:9865
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
KERBEROS_KEYTABS: om HTTP
OZONE_OPTS:
env_file:
- ./docker-config
Expand All @@ -130,14 +132,15 @@ services:
hostname: om2
volumes:
- ../..:/opt/hadoop
- ./keytabs:/etc/security/keytabs
- ./krb5.conf:/etc/krb5.conf
ports:
- 9882:9874
- 9892:9872
#- 18002:18002
environment:
WAITFOR: scm3.org:9865
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
KERBEROS_KEYTABS: om HTTP
OZONE_OPTS:
env_file:
- ./docker-config
Expand All @@ -154,14 +157,15 @@ services:
hostname: om3
volumes:
- ../..:/opt/hadoop
- ./keytabs:/etc/security/keytabs
- ./krb5.conf:/etc/krb5.conf
ports:
- 9884:9874
- 9894:9872
#- 18003:18003
environment:
WAITFOR: scm3.org:9865
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
KERBEROS_KEYTABS: om HTTP
OZONE_OPTS:
env_file:
- ./docker-config
Expand All @@ -178,13 +182,14 @@ services:
hostname: s3g
volumes:
- ../..:/opt/hadoop
- ./keytabs:/etc/security/keytabs
- ./krb5.conf:/etc/krb5.conf
ports:
- 9878:9878
env_file:
- ./docker-config
command: ["/opt/hadoop/bin/ozone","s3g"]
environment:
KERBEROS_KEYTABS: s3g HTTP testuser
OZONE_OPTS:
networks:
ozone_net:
Expand All @@ -194,13 +199,14 @@ services:
hostname: scm1.org
volumes:
- ../..:/opt/hadoop
- ./keytabs:/etc/security/keytabs
- ./krb5.conf:/etc/krb5.conf
ports:
- 9990:9876
- 9992:9860
env_file:
- docker-config
environment:
KERBEROS_KEYTABS: scm HTTP testuser testuser2
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-3}"
OZONE_OPTS:
Expand All @@ -220,14 +226,14 @@ services:
hostname: scm2.org
volumes:
- ../..:/opt/hadoop
- ./keytabs:/etc/security/keytabs
- ./krb5.conf:/etc/krb5.conf
ports:
- 9994:9876
- 9996:9860
env_file:
- docker-config
environment:
WAITFOR: scm1.org:9865
KERBEROS_KEYTABS: scm HTTP testuser testuser2
ENSURE_SCM_BOOTSTRAPPED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-3}"
OZONE_OPTS:
Expand All @@ -247,14 +253,15 @@ services:
hostname: scm3.org
volumes:
- ../..:/opt/hadoop
- ./keytabs:/etc/security/keytabs
- ./krb5.conf:/etc/krb5.conf
ports:
- 9998:9876
- 10002:9860
env_file:
- docker-config
environment:
WAITFOR: scm2.org:9865
KERBEROS_KEYTABS: scm HTTP testuser testuser2
ENSURE_SCM_BOOTSTRAPPED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-3}"
OZONE_OPTS:
Expand Down
Loading