Skip to content

HDDS-12771. xcompat fails if run in itself due to undefined OZONE_CURRENT_VERSION #8236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adoroszlai merged 5 commits into from
Apr 15, 2025
Merged

HDDS-12771. xcompat fails if run in itself due to undefined OZONE_CURRENT_VERSION #8236

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
dcc8a36
treat unset variable as error
adoroszlai Apr 6, 2025
a9dfab6
HDDS-12771. xcompat fails if run in itself due to undefined OZONE_CUR…
adoroszlai Apr 6, 2025
399f045
undefined variables in testlib.sh
adoroszlai Apr 6, 2025
15d087f
undefined variable in test-all.sh
adoroszlai Apr 6, 2025
221de46
revert whitespace change
adoroszlai Apr 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/compatibility/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:misc

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-balancer/test-ec.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:balancer

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE0}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR
export OM_SERVICE_ID="om"
Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-balancer/test-ratis.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:balancer

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR
export OM_SERVICE_ID="om"
Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-csi/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.

set -u -o pipefail

COMPOSE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-ha/test-hadoop.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:MR

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-ha/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:HA-unsecure

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-om-prepare/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-topology/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/test-ec.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:EC

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/test-failures1.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:failing

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/test-failures2.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:failing

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/test-hadoop.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:MR

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/test-legacy-bucket.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:unsecure

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/test-s3a.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:s3a

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:unsecure

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozonescripts/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:misc

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test-leadership.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:leadership

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test-om-bootstrap.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@
# * the keys are read from the snapshot and not the active file system
# * the contents of each key should be the same as the key name

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test-s3a.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:s3a

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test-s3g-virtual-host.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:misc

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test-scm-decommission.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:leadership

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:HA-secure

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure-mr/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:MR

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure/test-certificate-rotation.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:cert-rotation

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure/test-ec.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:EC

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure/test-fcq.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:secure

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure/test-haproxy-s3g.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:secure

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure/test-root-ca-rotation.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:cert-rotation

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure/test-vault.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:misc

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
4 changes: 3 additions & 1 deletion hadoop-ozone/dist/src/main/compose/ozonesecure/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

#suite:secure

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down Expand Up @@ -66,4 +68,4 @@ docker-compose up -d --scale datanode=3
execute_robot_test scm -v container:1 -v count:3 replication/wait.robot

#test public key and certificate recovery
source "$COMPOSE_DIR/public-key-cert-recovery-test.sh"
source "$COMPOSE_DIR/public-key-cert-recovery-test.sh"
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/restart/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.

set -u -o pipefail

COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

Expand Down
3 changes: 2 additions & 1 deletion hadoop-ozone/dist/src/main/compose/test-all.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

set -u -o pipefail

#
# Test executor to test all the compose/*/test.sh test scripts.
Expand Down Expand Up @@ -49,7 +50,7 @@ fi

if [[ "${OZONE_ACCEPTANCE_TEST_TYPE}" == "robot" ]]; then
# does not apply to JUnit tests run via Maven
generate_report "acceptance" "${ALL_RESULT_DIR}" "${XUNIT_RESULT_DIR}"
generate_report "acceptance" "${ALL_RESULT_DIR}" "${XUNIT_RESULT_DIR:-}"
fi

exit $RESULT
31 changes: 18 additions & 13 deletions hadoop-ozone/dist/src/main/compose/testlib.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,23 +14,21 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -e -o pipefail

set -e -u -o pipefail

_testlib_this="${BASH_SOURCE[0]}"
_testlib_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

: "${COMPOSE_DIR:=$_testlib_dir}"
COMPOSE_ENV_NAME=$(basename "$COMPOSE_DIR")
RESULT_DIR=${RESULT_DIR:-"$COMPOSE_DIR/result"}
RESULT_DIR_INSIDE="/tmp/smoketest/$(basename "$COMPOSE_ENV_NAME")/result"

OM_HA_PARAM=""
if [[ -n "${OM_SERVICE_ID}" ]] && [[ "${OM_SERVICE_ID}" != "om" ]]; then
OM_HA_PARAM="--om-service-id=${OM_SERVICE_ID}"
fi

source ${_testlib_dir}/compose_v2_compatibility.sh
source "${_testlib_dir}/../smoketest/testlib.sh"

: ${OZONE_COMPOSE_RUNNING:=false}
: ${SECURITY_ENABLED:=false}
: ${SCM:=scm}

# version is used in bucket name, which does not allow uppercase
Expand Down Expand Up @@ -71,7 +69,7 @@ all_tests_in_immediate_child_dirs() {
## @description Find all test*.sh scripts in immediate child dirs,
## @description applying OZONE_ACCEPTANCE_SUITE or OZONE_TEST_SELECTOR filter.
find_tests(){
if [[ -n "${OZONE_ACCEPTANCE_SUITE}" ]]; then
if [[ -n "${OZONE_ACCEPTANCE_SUITE:-}" ]]; then
tests=$(all_tests_in_immediate_child_dirs | xargs grep -l "^#suite:${OZONE_ACCEPTANCE_SUITE}$" || echo "")

# 'misc' is default suite, add untagged tests, too
Expand All @@ -86,7 +84,7 @@ find_tests(){
echo "No tests found for suite ${OZONE_ACCEPTANCE_SUITE}"
exit 1
fi
elif [[ -n "${OZONE_TEST_SELECTOR}" ]]; then
elif [[ -n "${OZONE_TEST_SELECTOR:-}" ]]; then
tests=$(all_tests_in_immediate_child_dirs | grep "${OZONE_TEST_SELECTOR}" || echo "")

if [[ -z "${tests}" ]]; then
Expand Down Expand Up @@ -215,22 +213,29 @@ execute_robot_test(){
fi
done

SMOKETEST_DIR_INSIDE="${OZONE_DIR:-/opt/hadoop}/smoketest"
: ${OZONE_DIR:=/opt/hadoop}
SMOKETEST_DIR_INSIDE="$OZONE_DIR/smoketest"

RESULT_DIR_INSIDE="/tmp/smoketest/$COMPOSE_ENV_NAME/result"
OUTPUT_PATH="$RESULT_DIR_INSIDE/${OUTPUT_FILE}"

OM_HA_PARAM=""
if [[ -n "${OM_SERVICE_ID:-}" ]] && [[ "${OM_SERVICE_ID}" != "om" ]]; then
OM_HA_PARAM="--om-service-id=${OM_SERVICE_ID}"
fi

set +e

# shellcheck disable=SC2068
docker-compose exec -T "$CONTAINER" mkdir -p "$RESULT_DIR_INSIDE" \
&& docker-compose exec -T "$CONTAINER" robot \
-v ENCRYPTION_KEY:"${OZONE_BUCKET_KEY_NAME}" \
-v ENCRYPTION_KEY:"${OZONE_BUCKET_KEY_NAME:-}" \
-v OM_HA_PARAM:"${OM_HA_PARAM}" \
-v OM_SERVICE_ID:"${OM_SERVICE_ID:-om}" \
-v OZONE_DIR:"${OZONE_DIR}" \
-v SECURITY_ENABLED:"${SECURITY_ENABLED}" \
-v SCM:"${SCM}" \
${ARGUMENTS[@]} --log NONE --report NONE "${OZONE_ROBOT_OPTS[@]}" --output "$OUTPUT_PATH" \
Copy link
Contributor

@sarvekshayr sarvekshayr Apr 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wanted to check - was there a specific reason for dropping ${OZONE_ROBOT_OPTS[@]}?
I noticed the variable doesn't seem to be set anywhere, so wanted to confirm if it was unused or had no real effect?

Copy link
Contributor Author

@adoroszlai adoroszlai Apr 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was there a specific reason for dropping ${OZONE_ROBOT_OPTS[@]}?

Being unused.

According to git history, it was introduced in HDDS-616, but was unused even in that version of the code.

Copy link
Contributor

@sarvekshayr sarvekshayr Apr 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for clarifying.

${ARGUMENTS[@]} --log NONE --report NONE --output "$OUTPUT_PATH" \
"$SMOKETEST_DIR_INSIDE/$TEST"
local -i rc=$?

Expand Down Expand Up @@ -319,7 +324,7 @@ create_containers() {
}

get_output_name() {
if [[ -n "${OUTPUT_NAME}" ]]; then
if [[ -n "${OUTPUT_NAME:-}" ]]; then
echo "${OUTPUT_NAME}-"
fi
}
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/upgrade/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@

#suite:upgrade

set -u -o pipefail

TEST_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )
source "$TEST_DIR/testlib.sh"
Expand Down
Loading