Skip to content

Add proper tests for Spark images #154

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hashhar merged 1 commit into from
Nov 2, 2022
Merged

Add proper tests for Spark images #154

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion bin/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,16 @@ function check_gpdb() {
environment_compose exec gpdb su gpadmin -l -c "pg_isready"
}

function check_spark() {
environment_compose exec spark curl -f http://localhost:10213
}

function run_spark_tests() {
environment_compose exec spark beeline -u jdbc:hive2://localhost:10213 -e 'SELECT 1;' &&
environment_compose exec spark beeline -u jdbc:hive2://localhost:10213 -e 'SHOW DATABASES;' &&
true
}

function check_health() {
local service=$1
test "$(environment_compose ps --format json | jq -er --arg name "$service" '.[] | select(.Service == $name) | .Health')" == "healthy"
Expand Down Expand Up @@ -200,7 +210,14 @@ for ARCH in "${platforms[@]}"; do
elif [[ ${ENVIRONMENT} == *"openldap"* ]]; then
retry check_openldap
elif [[ ${ENVIRONMENT} == *"spark"* ]]; then
retry check_health spark
# wait until Spark is started
retry check_spark

# run tests
set -x
set +e
sleep 10
Copy link
Copy Markdown
Member

@nineinchnick nineinchnick Nov 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is sleep necessary? Add a comment? Also what about +e? Do we want to restore it after the test?

Copy link
Copy Markdown
Member Author

@hashhar hashhar Nov 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sleep is probably not needed - I just copied surrounding code.

set -e is done outside loop in single place.

Copy link
Copy Markdown
Member

@nineinchnick nineinchnick Nov 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about other iterations in the loop?

Copy link
Copy Markdown
Member Author

@hashhar hashhar Nov 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's set inside loop but after all branches. Sorry I said "outside loop".

run_spark_tests
else
echo >&2 "ERROR: no test defined for ${ENVIRONMENT}"
cleanup
Expand Down
2 changes: 0 additions & 2 deletions testing/spark3-delta/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,6 @@ ENV PATH="${SPARK_HOME}/bin:${PATH}"

EXPOSE 10213

HEALTHCHECK --interval=10s --timeout=5s --start-period=10s \
CMD curl -f http://127.0.0.1:10213/
CMD spark-submit \
--master "local[*]" \
--class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
Expand Down
2 changes: 0 additions & 2 deletions testing/spark3-hudi/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,6 @@ ENV PATH="${SPARK_HOME}/bin:${PATH}"

EXPOSE 10213

HEALTHCHECK --interval=10s --timeout=5s --start-period=10s \
CMD curl -f http://127.0.0.1:10213/
CMD spark-submit \
--master "local[*]" \
--class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
Expand Down
2 changes: 0 additions & 2 deletions testing/spark3-iceberg/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,6 @@ ENV PATH="${SPARK_HOME}/bin:${PATH}"
# add hive user needed in interactions with the Apache Hive environment
RUN useradd -ms /bin/bash hive

HEALTHCHECK --interval=10s --timeout=5s --start-period=10s \
CMD curl -f http://127.0.0.1:10213/
CMD spark-submit \
--master "local[*]" \
--class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 \
Expand Down