Skip to content

HDDS-11134. Create compatibility test for FSO bucket usage #7350

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adoroszlai merged 9 commits into from
Oct 28, 2024
Merged

HDDS-11134. Create compatibility test for FSO bucket usage #7350

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
8664d79
HDDS-11134. Create compatibility test for FSO bucket usage
adoroszlai Jul 10, 2024
7b9dcfb
Skip If - not available with old Robot Framework
adoroszlai Jul 10, 2024
8cf3cee
Merge remote-tracking branch 'origin/master' into HDDS-11134
adoroszlai Oct 23, 2024
82e979c
bucket name cannot have uppercase chars
adoroszlai Oct 23, 2024
6e250c0
log cluster version
adoroszlai Oct 23, 2024
6117126
remove -SNAPSHOT for brevity
adoroszlai Oct 24, 2024
877f91d
Merge remote-tracking branch 'origin/master' into HDDS-11134
adoroszlai Oct 25, 2024
9abc78a
fix variable name
adoroszlai Oct 25, 2024
c131aa2
explain need for sed/tr
adoroszlai Oct 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions hadoop-ozone/dist/src/main/compose/xcompat/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR
basename=$(basename ${COMPOSE_DIR})

current_version="${ozone.version}"
# version is used in bucket name, which does not allow uppercase
current_version="$(echo "${ozone.version}" | sed -e 's/-SNAPSHOT//' | tr '[:upper:]' '[:lower:]')"
# TODO: debug acceptance test failures for client versions 1.0.0 on secure clusters
old_versions="1.1.0 1.2.1 1.3.0 1.4.0" # container is needed for each version in clients.yaml

Expand Down Expand Up @@ -55,13 +56,22 @@ _init() {

_write() {
_kinit
execute_robot_test ${container} -N "xcompat-cluster-${cluster_version}-client-${client_version}-write" -v SUFFIX:${client_version} compatibility/write.robot
execute_robot_test ${container} -N "xcompat-cluster-${cluster_version}-client-${client_version}-write" \
-v CLIENT_VERSION:${client_version} \
-v CLUSTER_VERSION:${cluster_version} \
-v SUFFIX:${client_version} \
compatibility/write.robot
}

_read() {
_kinit
local data_version="$1"
execute_robot_test ${container} -N "xcompat-cluster-${cluster_version}-client-${client_version}-read-${data_version}" -v SUFFIX:${data_version} compatibility/read.robot
execute_robot_test ${container} -N "xcompat-cluster-${cluster_version}-client-${client_version}-read-${data_version}" \
-v CLIENT_VERSION:${client_version} \
-v CLUSTER_VERSION:${cluster_version} \
-v DATA_VERSION:${data_version} \
-v SUFFIX:${data_version} \
compatibility/read.robot
}

test_bucket_encryption() {
Expand All @@ -70,7 +80,7 @@ test_bucket_encryption() {
}

test_cross_compatibility() {
echo "Starting cluster with COMPOSE_FILE=${COMPOSE_FILE}"
echo "Starting ${cluster_version} cluster with COMPOSE_FILE=${COMPOSE_FILE}"

OZONE_KEEP_RESULTS=true start_docker_env

Expand Down
10 changes: 9 additions & 1 deletion hadoop-ozone/dist/src/main/smoketest/compatibility/read.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,4 +37,12 @@ Dir Can Be Listed Using Shell
File Can Be Get
Execute ozone fs -get o3fs://bucket1.vol1/dir-${SUFFIX}/file-${SUFFIX} /tmp/
Execute diff -q ${TESTFILE} /tmp/file-${SUFFIX}
[teardown] Execute rm /tmp/file-${SUFFIX}
[teardown] Execute rm -f /tmp/file-${SUFFIX}

FSO Bucket Can Be Read
Copy link
Contributor

@ivanzlenko ivanzlenko Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't feel right that we are using this approach with relying on test results from another test here. I know it is the same for a common bucket, but could it be avoided for a new test?

Copy link
Contributor Author

@adoroszlai adoroszlai Oct 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ivanzlenko Which "another test" do you mean?

Copy link
Contributor Author

@adoroszlai adoroszlai Oct 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ivanzlenko can you please clarify your comment?

Copy link
Contributor

@ivanzlenko ivanzlenko Oct 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adoroszlai maybe I'm wrong but it looks like we are consuming here the result of test from write.robot.
If it is not I'm confused why have teardown in one case and not in the other.
More or less want just to clarify things.

Copy link
Contributor Author

@adoroszlai adoroszlai Oct 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

read.robot relies on the content created by write.robot by design, because we need to cross-check between versions. E.g. running a 2.0.0 cluster write data with both 1.4.0 and 2.0.0 clients, then verify that they can read content written by the other. This cannot be done in a single self-contained robot test, since that is run in a specific container with a specific Ozone version.

upgrade test is somewhat similar (write with one version, upgrade, write with new version, read both with new version).

Copy link
Contributor

@ivanzlenko ivanzlenko Oct 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification! Will keep this in mind!

Pass Execution If '${DATA_VERSION}' < '${FSO_VERSION}' Skipped write test case
Pass Execution If '${CLIENT_VERSION}' < '${FSO_VERSION}' Client does not support FSO
Pass Execution If '${CLUSTER_VERSION}' < '${FSO_VERSION}' Cluster does not support FSO
Execute ozone fs -get ofs://om/vol1/fso-bucket-${SUFFIX}/dir/subdir/file ${TEMP_DIR}/
Execute diff -q ${TESTFILE} ${TEMP_DIR}/file
[teardown] Execute rm -f ${TEMP_DIR}/file
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/smoketest/compatibility/setup.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Resource ../ozone-lib/shell.robot

*** Variables ***
${SUFFIX} ${EMPTY}
${FSO_VERSION} 1.3.0


*** Keywords ***
Expand Down
7 changes: 7 additions & 0 deletions hadoop-ozone/dist/src/main/smoketest/compatibility/write.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,10 @@ Dir Can Be Created

File Can Be Put
Execute ozone fs -put ${TESTFILE} o3fs://bucket1.vol1/dir-${SUFFIX}/file-${SUFFIX}

FSO Bucket Can Be Created and Used
Pass Execution If '${CLIENT_VERSION}' < '${FSO_VERSION}' Client does not support FSO
Pass Execution If '${CLUSTER_VERSION}' < '${FSO_VERSION}' Cluster does not support FSO
Execute ozone sh bucket create --layout FILE_SYSTEM_OPTIMIZED /vol1/fso-bucket-${SUFFIX}
Execute ozone fs -mkdir -p ofs://om/vol1/fso-bucket-${SUFFIX}/dir/subdir
Execute ozone fs -put ${TESTFILE} ofs://om/vol1/fso-bucket-${SUFFIX}/dir/subdir/file