Skip to content

HDDS-3990. Test Kubernetes examples with acceptance tests #1223

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adoroszlai merged 13 commits into from
Jul 30, 2020
Merged

HDDS-3990. Test Kubernetes examples with acceptance tests #1223

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
9c596bc
HDDS-3990. Test Kubernetes examples with acceptance tests
elek Jul 20, 2020
9144ea9
retrigger build with empty commit
elek Jul 21, 2020
81c68f5
remove parameter of combine_reports
elek Jul 27, 2020
17afb8d
move bats to the tests (and fix test name)
elek Jul 27, 2020
eb308b0
typo in post-commit (insstall)
elek Jul 27, 2020
4bd1e91
Merge remote-tracking branch 'origin/master' into HDDS-3990
elek Jul 27, 2020
473a02e
Merge remote-tracking branch 'elek/HDDS-3990' into HDDS-3990
elek Jul 27, 2020
ddce73e
Merge remote-tracking branch 'origin/master' into HDDS-3990
elek Jul 29, 2020
6f15bb4
adjust debug log and wrap final generation
elek Jul 30, 2020
8fe4e59
Use updated checkout logic
elek Jul 30, 2020
28e150f
Add names for each new build step
elek Jul 30, 2020
118d26b
Merge remote-tracking branch 'origin/master' into HDDS-3990
elek Jul 30, 2020
04fd344
fix bats
elek Jul 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions .github/workflows/post-commit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,3 +238,61 @@ jobs:
name: coverage
path: target/coverage
continue-on-error: true
kubernetes:
name: kubernetes
runs-on: ubuntu-18.04
steps:
- uses: actions/cache@v2
with:
path: ~/.m2/repository
key: maven-repo-${{ hashFiles('**/pom.xml') }}
- uses: actions/cache@v2
with:
path: |
~/.pnpm-store
**/node_modules
key: ${{ runner.os }}-pnpm-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-
- name: checkout to /mnt/ozone
run: |
sudo chmod 777 /mnt
git clone https://github.com/${GITHUB_REPOSITORY}.git /mnt/ozone
cd /mnt/ozone
git fetch origin "${GITHUB_REF}"
git checkout FETCH_HEAD
git reset --hard
- name: install robotframework
run: sudo pip install robotframework
- name: install k3s
run: curl -sfL https://get.k3s.io | sh -
- name: copy kubeconfig
run: |
sudo mkdir ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u) ~/.kube/config
- name: install flekszible
run: |
cd /tmp
wget https://github.com/elek/flekszible/releases/download/v1.8.1/flekszible_1.8.1_Linux_x86_64.tar.gz -O - | tar -zx
chmod +x flekszible
sudo mv flekszible /usr/bin/flekszible
- name: run a full build
run: |
cd /mnt/ozone
hadoop-ozone/dev-support/checks/build.sh -Pcoverage
- name: execute test
run: |
cd /mnt/ozone/hadoop-ozone/dist/target/ozone-* && sudo mkdir .aws && sudo chmod 777 .aws && sudo chown 1000 .aws
cd /mnt/ozone && hadoop-ozone/dev-support/checks/kubernetes.sh
- uses: actions/upload-artifact@master
if: always()
with:
name: kubernetes
path: /mnt/ozone/target/kubernetes
continue-on-error: true
- run: |
#Never cache local artifacts
rm -rf ~/.m2/repository/org/apache/hadoop/hdds
rm -rf ~/.m2/repository/org/apache/hadoop/ozone
if: always()
36 changes: 36 additions & 0 deletions hadoop-ozone/dev-support/checks/kubernetes.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd "$DIR/../../.." || exit 1

REPORT_DIR=${OUTPUT_DIR:-"$DIR/../../../target/kubernetes"}

OZONE_VERSION=$(grep "<ozone.version>" "pom.xml" | sed 's/<[^>]*>//g'| sed 's/^[ \t]*//')
DIST_DIR="$DIR/../../dist/target/ozone-$OZONE_VERSION"

if [ ! -d "$DIST_DIR" ]; then
echo "Distribution dir is missing. Doing a full build"
"$DIR/build.sh" -Pcoverage
fi

mkdir -p "$REPORT_DIR"

cd "$DIST_DIR/kubernetes/examples" || exit 1
./test-all.sh
RES=$?
cp result/* "$REPORT_DIR/"
cp "$REPORT_DIR/log.html" "$REPORT_DIR/summary.html"
exit $RES
1 change: 1 addition & 0 deletions hadoop-ozone/dist/dev-support/bin/dist-layout-stitching
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,3 +131,4 @@ run cp -p -r "${ROOT}/hadoop-ozone/dist/target/Dockerfile" .

#workaround for https://issues.apache.org/jira/browse/MRESOURCES-236
find ./compose -name "*.sh" -exec chmod 755 {} \;
find ./kubernetes -name "*.sh" -exec chmod 755 {} \;
15 changes: 15 additions & 0 deletions hadoop-ozone/dist/src/main/k8s/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
result
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/k8s/definitions/ozone/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ data:
OZONE-SITE.XML_ozone.om.address: "om-0.om"
OZONE-SITE.XML_ozone.scm.client.address: "scm-0.scm"
OZONE-SITE.XML_ozone.scm.names: "scm-0.scm"
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "3"
LOG4J.PROPERTIES_log4j.rootLogger: "INFO, stdout"
LOG4J.PROPERTIES_log4j.appender.stdout: "org.apache.log4j.ConsoleAppender"
LOG4J.PROPERTIES_log4j.appender.stdout.layout: "org.apache.log4j.PatternLayout"
Expand Down
2 changes: 1 addition & 1 deletion hadoop-ozone/dist/src/main/k8s/definitions/ozone/definitions/onenode.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ description: remove scheduling rules to make it possible to run multiple datanod
- type: Remove
trigger:
metadata:
name: ozone-datanode
name: datanode
path:
- spec
- template
Expand Down
0 ...n/k8s/definitions/pv-test/flekszible.yaml → ...efinitions/test-webserver/flekszible.yaml
View file
Open in desktop
File renamed without changes.
0 ...nitions/pv-test/webserver-deployment.yaml → .../test-webserver/webserver-deployment.yaml
View file
Open in desktop
File renamed without changes.
0 ...efinitions/pv-test/webserver-service.yaml → ...ons/test-webserver/webserver-service.yaml
View file
Open in desktop
File renamed without changes.
0 ...definitions/pv-test/webserver-volume.yaml → ...ions/test-webserver/webserver-volume.yaml
View file
Open in desktop
File renamed without changes.
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/k8s/examples/getting-started/config-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ data:
OZONE-SITE.XML_ozone.om.address: om-0.om
OZONE-SITE.XML_ozone.scm.client.address: scm-0.scm
OZONE-SITE.XML_ozone.scm.names: scm-0.scm
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "3"
LOG4J.PROPERTIES_log4j.rootLogger: INFO, stdout
LOG4J.PROPERTIES_log4j.appender.stdout: org.apache.log4j.ConsoleAppender
LOG4J.PROPERTIES_log4j.appender.stdout.layout: org.apache.log4j.PatternLayout
Expand Down
10 changes: 0 additions & 10 deletions hadoop-ozone/dist/src/main/k8s/examples/getting-started/datanode-statefulset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,16 +37,6 @@ spec:
prometheus.io/port: "9882"
prometheus.io/path: /prom
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: component
operator: In
values:
- datanode
topologyKey: kubernetes.io/hostname
securityContext:
fsGroup: 1000
containers:
Expand Down
38 changes: 38 additions & 0 deletions hadoop-ozone/dist/src/main/k8s/examples/getting-started/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

export K8S_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

cd "$K8S_DIR"

# shellcheck source=/dev/null
source "../testlib.sh"

rm -rf result

regenerate_resources

start_k8s_env

execute_robot_test scm-0 smoketest/basic/basic.robot

combine_reports

stop_k8s_env

flekszible generate
Copy link
Contributor

@adoroszlai adoroszlai Jul 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, this flekszible generate call is only needed when executed from source dir (hadoop-ozone/dist/src/main/k8s) to restore source files. I propose it to be executed as part of stop_k8s_env:

  1. I ran tests from target/..., had some errors and tried to find out what's wrong. It was confusing to see resource files referencing non-existent docker image apache/ozone:0.6.0-SNAPSHOT (plus other differences compared to the files actually used for the test).
  2. Avoid possible omission in new scripts.
  3. Reduce code duplication.
  4. Save some very minimal runtime cost.

Copy link
Member Author

@elek elek Jul 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we have a main problem: some of the Kubernetes examples couldn't be started in the test environment. For example when the example is configured to schedule one datanode per real node, but we have only one node.

Therefore, before the tests the resource files are heavily modified:

  1. instead of using latest image, the current build is mounted to the /opt/ozone (similar to the docker tests)
  2. anti-affinity rules are removed (enable to run multiple datanode on the same node)
  3. real persistence is removed

These are executed by the regenerate_resources step at the beginning of the test. Which is something like this:

flekszible generate -t mount:hostPath="$OZONE_ROOT",path=/opt/hadoop -t image:image=apache/ozone-runner:20200420-1 -t ozone/onenode

As you can see we define three new transformations on the fly:

  1. -t mount:hostPath="$OZONE_ROOT",path=/opt/hadoop --> use the files from the current build
  2. -t image:image=apache/ozone-runner:20200420-1, use standard runner
  3. -t ozone/onenode enable to schedule more datanode to the same node

The line which is commented by you:

flekszible generate

Is 100% optional. After the test, it restores the original state of the files. It can be added to the stop_k8s_env (as it can always be added).

But based on the experience with the docker test.sh files, I would prefer to use more explicit lines in the test.sh.

As test.sh files are read frequently but modified only by a few times, I tend to make it slightly more verbose, but easier to understand (you can assume that stop_k8s_env stops kubernetes pods, but I wouldn't like to hide any hidden functionlaty there.

But theses are just my thought, as it's a beginning of new test, I am really open to modify to any direction.

Copy link
Contributor

@adoroszlai adoroszlai Jul 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming my guess.

I'm fine with making test.sh more verbose at the cost of some duplication. However, I propose extracting this optional flekszible generate to a function, and let it check if it's being run from src dir, to address my first point.

Copy link
Member Author

@elek elek Jul 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and let it check if it's being run from src dir, to address my first point

I ran tests from target/..., had some errors and tried to find out what's wrong.

I am not sure what was the problem in your case. It supposed to work from the target directory. In fact hadoop-ozone/dev-support/checks/kubernetes.sh executes all the tests from target by default.

apache/ozone:0.6.0-SNAPSHOT is used because during the release it becomes apache/ozone:0.6.0 which is the distributed container.

./test.sh (with the first flekszible generation) replaces all the images with a dev version (ozone-runner + mount).

It seems to be working for me and passed on the github CI, but if you see any error, please let me know as it should work everywhere (do you have the latest released flekszible?)

Copy link
Contributor

@adoroszlai adoroszlai Jul 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me clarify: I think the script would have worked fine, but I tried to use k3d (k3s in docker) instead of plain k3s. Since the local dir was not available in the container, it failed.

The confusing part was that the resource files did not mention volume mounts, since they were converted back at the end. So only by looking at regenerate_resources in testlib did I realize what's wrong.

Copy link
Member Author

@elek elek Jul 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's what came into my mind, too.

Today, it's not possible to run test on remote clusters (k3d is a remote cluster) as local mount doesn't work.

I can introduce some options in the future (would be great to execute tests on real environments, too).

For now, I will add a log message to the regenerate_resource to print a warning to the stdout.


1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/k8s/examples/minikube/config-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ data:
OZONE-SITE.XML_ozone.om.address: om-0.om
OZONE-SITE.XML_ozone.scm.client.address: scm-0.scm
OZONE-SITE.XML_ozone.scm.names: scm-0.scm
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "3"
LOG4J.PROPERTIES_log4j.rootLogger: INFO, stdout
LOG4J.PROPERTIES_log4j.appender.stdout: org.apache.log4j.ConsoleAppender
LOG4J.PROPERTIES_log4j.appender.stdout.layout: org.apache.log4j.PatternLayout
Expand Down
38 changes: 38 additions & 0 deletions hadoop-ozone/dist/src/main/k8s/examples/minikube/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

export K8S_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

cd "$K8S_DIR"

# shellcheck source=/dev/null
source "../testlib.sh"

rm -rf result

regenerate_resources

start_k8s_env

execute_robot_test scm-0 smoketest/basic/basic.robot

combine_reports

stop_k8s_env

flekszible generate

4 changes: 2 additions & 2 deletions hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/Flekszible
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,11 @@ import:
- type: Image
image: "@docker.image@"
- type: ozone/tracing
- path: pv-test
- path: test-webserver
destination: pv-test
- path: ozone-csi
destination: csi
- path: pv-test
- path: test-webserver
destination: pv-test
transformations:
- type: Namespace
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/config-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ data:
OZONE-SITE.XML_ozone.om.address: om-0.om
OZONE-SITE.XML_ozone.scm.client.address: scm-0.scm
OZONE-SITE.XML_ozone.scm.names: scm-0.scm
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "3"
LOG4J.PROPERTIES_log4j.rootLogger: INFO, stdout
LOG4J.PROPERTIES_log4j.appender.stdout: org.apache.log4j.ConsoleAppender
LOG4J.PROPERTIES_log4j.appender.stdout.layout: org.apache.log4j.PatternLayout
Expand Down
10 changes: 0 additions & 10 deletions hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/datanode-statefulset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,16 +37,6 @@ spec:
prometheus.io/port: "9882"
prometheus.io/path: /prom
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: component
operator: In
values:
- datanode
topologyKey: kubernetes.io/hostname
securityContext:
fsGroup: 1000
containers:
Expand Down
38 changes: 38 additions & 0 deletions hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

export K8S_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

cd "$K8S_DIR"

# shellcheck source=/dev/null
source "../testlib.sh"

rm -rf result

regenerate_resources

start_k8s_env

execute_robot_test scm-0 smoketest/basic/basic.robot

combine_reports

stop_k8s_env

flekszible generate

2 changes: 1 addition & 1 deletion hadoop-ozone/dist/src/main/k8s/examples/ozone/Flekszible
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ import:
destination: pv-test
- path: ozone-csi
destination: csi
- path: pv-test
- path: test-webserver
destination: pv-test
transformations:
- type: Namespace
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/k8s/examples/ozone/config-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ data:
OZONE-SITE.XML_ozone.om.address: om-0.om
OZONE-SITE.XML_ozone.scm.client.address: scm-0.scm
OZONE-SITE.XML_ozone.scm.names: scm-0.scm
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "3"
LOG4J.PROPERTIES_log4j.rootLogger: INFO, stdout
LOG4J.PROPERTIES_log4j.appender.stdout: org.apache.log4j.ConsoleAppender
LOG4J.PROPERTIES_log4j.appender.stdout.layout: org.apache.log4j.PatternLayout
Expand Down
38 changes: 38 additions & 0 deletions hadoop-ozone/dist/src/main/k8s/examples/ozone/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

export K8S_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

cd "$K8S_DIR"

# shellcheck source=/dev/null
source "../testlib.sh"

rm -rf result

regenerate_resources

start_k8s_env

execute_robot_test scm-0 smoketest/basic/basic.robot

combine_reports

stop_k8s_env

flekszible generate

Loading