Skip to content

HDDS-6084. [Multi-Tenant] Handle upgrades to version supporting S3 multi-tenancy #3018

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
smengcl merged 18 commits into from
Feb 10, 2022
Merged

HDDS-6084. [Multi-Tenant] Handle upgrades to version supporting S3 multi-tenancy #3018

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
d4d6912
HDDS-6084. Handle upgrades to version supporting S3 multi-tenancy
smengcl Jan 24, 2022
d6effa7
Weave aspect; Add layout feature limit to tenant read-only requests, …
smengcl Jan 24, 2022
7fcc689
Add annotation to the rest of the tenant write requests.
smengcl Jan 25, 2022
10c3bac
ElementType.CONSTRUCTOR doesn't work; cleanup.
smengcl Jan 25, 2022
32dcfe2
checkLayoutFeature should work with all methods with OM as the first …
smengcl Jan 25, 2022
18ab2aa
Add test; checkstyle.
smengcl Jan 25, 2022
977f1f9
Address test failures.
smengcl Jan 26, 2022
59859ea
Add TODO, auto-generate weaver XML in the future.
smengcl Jan 28, 2022
074dada
Integration test: trigger finalization from the client rather than di…
smengcl Feb 2, 2022
c8466ad
Integration check: preFinalizationCheck / cleanup.
smengcl Feb 2, 2022
b4010ac
Cleanup addressed comment.
smengcl Feb 2, 2022
57af2cf
More restrictive checking of JoinPoint in checkLayoutFeature.
smengcl Feb 2, 2022
62269f1
Acceptance test: add aws s3api testing before and after upgrade final…
smengcl Feb 4, 2022
3ce0326
QoL: Fix comment syntax in docker-config leading to annoying `WARNING…
smengcl Feb 4, 2022
e3baa68
QoL: Fix docker deprecate warning: `--no-ansi` -> `--ansi never`
smengcl Feb 4, 2022
993f1c2
Integration test: Move pre-finalization checks and finalization to cl…
smengcl Feb 4, 2022
6c37edb
Move all create operations to generate; Remove HDDS-6261 repro.
smengcl Feb 8, 2022
47c104c
Create custom key instead of relying on /opt/hadoop/ files.
smengcl Feb 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions hadoop-ozone/dist/src/main/compose/testlib.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,8 +147,8 @@ start_docker_env(){
create_results_dir
export OZONE_SAFEMODE_MIN_DATANODES="${datanode_count}"

docker-compose --no-ansi down
if ! { docker-compose --no-ansi up -d --scale datanode="${datanode_count}" \
docker-compose --ansi never down
if ! { docker-compose --ansi never up -d --scale datanode="${datanode_count}" \
&& wait_for_safemode_exit \
&& wait_for_om_leader ; }; then
[[ -n "$OUTPUT_NAME" ]] || OUTPUT_NAME="$COMPOSE_ENV_NAME"
Expand Down Expand Up @@ -235,7 +235,7 @@ execute_command_in_container(){
## @param List of container names, eg datanode_1 datanode_2
stop_containers() {
set -e
docker-compose --no-ansi stop $@
docker-compose --ansi never stop $@
set +e
}

Expand All @@ -244,7 +244,7 @@ stop_containers() {
## @param List of container names, eg datanode_1 datanode_2
start_containers() {
set -e
docker-compose --no-ansi start $@
docker-compose --ansi never start $@
set +e
}

Expand Down Expand Up @@ -280,9 +280,9 @@ wait_for_port(){

## @description Stops a docker-compose based test environment (with saving the logs)
stop_docker_env(){
docker-compose --no-ansi logs > "$RESULT_DIR/docker-$OUTPUT_NAME.log"
docker-compose --ansi never logs > "$RESULT_DIR/docker-$OUTPUT_NAME.log"
if [ "${KEEP_RUNNING:-false}" = false ]; then
docker-compose --no-ansi down
docker-compose --ansi never down
fi
}

Expand Down
4 changes: 2 additions & 2 deletions hadoop-ozone/dist/src/main/compose/upgrade/compose/ha/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ OZONE-SITE.XML_ozone.om.address.omservice.om1=om1
OZONE-SITE.XML_ozone.om.address.omservice.om2=om2
OZONE-SITE.XML_ozone.om.address.omservice.om3=om3
OZONE-SITE.XML_ozone.om.ratis.enable=true
// setting ozone.scm.ratis.enable to false for now, as scm ha upgrade is
// not supported yet. This is supposed to work without SCM HA configuration
# setting ozone.scm.ratis.enable to false for now, as scm ha upgrade is
# not supported yet. This is supposed to work without SCM HA configuration
OZONE-SITE.XML_ozone.scm.ratis.enable=false
OZONE-SITE.XML_ozone.scm.pipeline.creation.interval=30s
OZONE-SITE.XML_ozone.scm.pipeline.owner.container.count=1
Expand Down
3 changes: 2 additions & 1 deletion hadoop-ozone/dist/src/main/compose/upgrade/testlib.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,8 +106,9 @@ generate() {

## @description Validates that data exists on the cluster.
## @param The prefix of the data to be validated.
## @param S3PREFIX is a workaround for HDDS-6261. Can remove (or be repurposed) once HDDS-6261 is resolved.
validate() {
execute_robot_test scm -v PREFIX:"$1" upgrade/validate.robot
execute_robot_test scm -v PREFIX:"$1" -v S3PREFIX:"$2" upgrade/validate.robot
}

## @description Checks that the metadata layout version of the provided node matches what is expected.
Expand Down
24 changes: 12 additions & 12 deletions ...-ozone/dist/src/main/compose/upgrade/upgrades/non-rolling-upgrade/1.1.0-1.2.0/callback.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,36 +41,36 @@ setup() {

with_old_version() {
generate old1
validate old1
validate old1 old1
}

with_new_version_pre_finalized() {
_check_hdds_mlvs 0
_check_om_mlvs 0

validate old1
validate old1 old1

generate new1
validate new1
validate new1 new1
}

with_old_version_downgraded() {
validate old1
validate new1
# TODO: FIXME: HDDS-6261: OM will crash when overwriting key previously created by newer OM if S3PREFIX=old1
validate old1 old1-workaround
validate new1 new1

generate old2
validate old2
validate old2 old2
}

with_new_version_finalized() {
_check_hdds_mlvs 2
# OM currently only has one layout version.
_check_om_mlvs 0
Comment on lines -67 to -68

@adoroszlai adoroszlai May 26, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smengcl @errose28 maybe I'm missing something, but wouldn't this fail if we actually ran upgrade path test?

https://github.com/adoroszlai/hadoop-ozone/runs/6613025379#step:5:738

I think actual OM MLV for 1.2.0 and 1.2.1 is still 0, regardless of this change.

@smengcl smengcl May 26, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right.

Though in the latest feature branch it is still 0 (probably fixed when I was resolving conflicts when doing the merge master 3 days ago or some time ago):

https://github.com/smengcl/hadoop-ozone/blob/HDDS-4944/hadoop-ozone/dist/src/main/compose/upgrade/upgrades/non-rolling-upgrade/1.1.0-1.2.0/callback.sh#L47-L49

Not sure why I bumped it when doing this PR.

@smengcl smengcl May 26, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway, the om version bump in with_new_version_finalized is intended. As after the finalization it should be bumped to the latest version. But in pre_finalized it should still be zero.

@adoroszlai adoroszlai May 26, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this callback applies when new version is 1.2.0.

@smengcl smengcl May 26, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was going to say when I was merging master to the feature branch just 3 days ago, I had to bump the _check_om_mlvs further to 3 to pass acceptance (misc):

https://github.com/apache/ozone/blame/HDDS-4944/hadoop-ozone/dist/src/main/compose/upgrade/upgrades/non-rolling-upgrade/1.2.1-1.3.0/callback.sh#L74

But then I realized this is the one from 1.1.0 to 1.2.0.

Yup so in this case it should still be zero (in 1.2.0).

@smengcl smengcl May 26, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is due to the upgrade test.sh script was actually testing the latest compiled version rather than the actual 1.2.0 at the time, and didn't have 1.2.1-1.3.0 yet. And yes it should be reverted back to 0 now (if I haven't done that yet).

Thanks for catching this @adoroszlai !

@smengcl smengcl May 26, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed this in commit: 00a5007 (PR #3450)

_check_om_mlvs 1

validate old1
validate new1
validate old2
validate old1 old1
validate new1 new1
validate old2 old2

generate new2
validate new2
validate new2 news
}
2 changes: 1 addition & 1 deletion hadoop-ozone/dist/src/main/smoketest/upgrade/finalize.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Resource ../commonlib.robot
Test Timeout 5 minutes
Test Setup Run Keyword if '${SECURITY_ENABLED}' == 'true' Kinit test user testuser testuser.keytab

** Test Cases ***
*** Test Cases ***
Finalize SCM
${result} = Execute ozone admin scm finalizeupgrade
#Wait Until Keyword Succeeds 3min 10sec Should contain ${result} OM Preparation successful!
Expand Down
6 changes: 6 additions & 0 deletions hadoop-ozone/dist/src/main/smoketest/upgrade/generate.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,9 @@ Create a volume, bucket and key
Should not contain ${output} Failed
${output} = Execute ozone sh key put /${PREFIX}-volume/${PREFIX}-bucket/${PREFIX}-key /opt/hadoop/NOTICE.txt
Should not contain ${output} Failed

Create a bucket and key in volume s3v
${output} = Execute ozone sh bucket create /s3v/${PREFIX}-bucket
Should not contain ${output} Failed
${output} = Execute ozone sh key put /s3v/${PREFIX}-bucket/key1 /opt/hadoop/NOTICE.txt
Should not contain ${output} Failed
2 changes: 1 addition & 1 deletion hadoop-ozone/dist/src/main/smoketest/upgrade/prepare.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Resource ../commonlib.robot
Test Timeout 5 minutes
Test Setup Run Keyword if '${SECURITY_ENABLED}' == 'true' Kinit test user testuser testuser.keytab

** Test Cases ***
*** Test Cases ***
Prepare Ozone Manager
${result} = Execute ozone admin om prepare -id %{OM_SERVICE_ID}
Wait Until Keyword Succeeds 3min 10sec Should contain ${result} OM Preparation successful!
Expand Down
23 changes: 23 additions & 0 deletions hadoop-ozone/dist/src/main/smoketest/upgrade/validate.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ Documentation Smoketest ozone cluster startup
Library OperatingSystem
Library BuiltIn
Resource ../commonlib.robot
Resource ../s3/commonawslib.robot
Test Timeout 5 minutes

*** Variables ***
Expand All @@ -28,3 +29,25 @@ Read data from previously created key
${random} = Generate Random String 5 [NUMBERS]
${output} = Execute ozone sh key get /${PREFIX}-volume/${PREFIX}-bucket/${PREFIX}-key /tmp/key-${random}
Should not contain ${output} Failed

Setup credentials for S3
# TODO: Run "Setup secure v4 headers" instead when security is enabled
Run Keyword Setup dummy credentials for S3

Read data from previously created key using S3 API
${result} = Execute AWSS3APICli and checkrc get-object --bucket ${PREFIX}-bucket --key key1 /tmp/key1.result 0
# Note: "Compare files" doesn't work on NOTICE.txt as it is updated in new Ozone versions.
Comment thread
smengcl marked this conversation as resolved.
Outdated
# Compare files /opt/hadoop/NOTICE.txt /tmp/key1.result
Execute and checkrc rm /tmp/key1.result 0

Create bucket using S3 API if the bucket doesn't exist
Comment thread
smengcl marked this conversation as resolved.
Outdated
# Note: s3api doesn't return error if the bucket already exists
${result} = Create bucket with name ${S3PREFIX}-bucket

Write key using S3 API
${result} = Execute AWSS3APICli and checkrc put-object --bucket ${S3PREFIX}-bucket --key key2 --body /opt/hadoop/NOTICE.txt 0

Read key using S3 API
${result} = Execute AWSS3APICli and checkrc get-object --bucket ${S3PREFIX}-bucket --key key2 /tmp/key2.result 0
# Compare files /opt/hadoop/NOTICE.txt /tmp/key2.result
Execute and checkrc rm /tmp/key2.result 0
9 changes: 5 additions & 4 deletions ...-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/upgrade/TestHDDSUpgrade.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@
import static org.apache.hadoop.hdds.scm.ScmConfigKeys.OZONE_DATANODE_PIPELINE_LIMIT;
import static org.apache.hadoop.hdds.scm.ScmConfigKeys.OZONE_SCM_HA_ENABLE_KEY;
import static org.apache.hadoop.hdds.scm.pipeline.Pipeline.PipelineState.OPEN;
import static org.apache.hadoop.hdds.upgrade.HDDSLayoutFeature.INITIAL_VERSION;
import static org.apache.hadoop.ozone.upgrade.InjectedUpgradeFinalizationExecutor.UpgradeTestInjectionPoints.AFTER_COMPLETE_FINALIZATION;
import static org.apache.hadoop.ozone.upgrade.InjectedUpgradeFinalizationExecutor.UpgradeTestInjectionPoints.AFTER_POST_FINALIZE_UPGRADE;
import static org.apache.hadoop.ozone.upgrade.InjectedUpgradeFinalizationExecutor.UpgradeTestInjectionPoints.AFTER_PRE_FINALIZE_UPGRADE;
Expand Down Expand Up @@ -78,6 +77,7 @@
import org.apache.hadoop.ozone.client.io.OzoneOutputStream;
import org.apache.hadoop.ozone.container.common.interfaces.Container;
import org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine;
import org.apache.hadoop.ozone.om.upgrade.OMLayoutFeature;
import org.apache.hadoop.ozone.upgrade.BasicUpgradeFinalizer;
import org.apache.hadoop.ozone.upgrade.InjectedUpgradeFinalizationExecutor;
import org.apache.hadoop.ozone.upgrade.InjectedUpgradeFinalizationExecutor.UpgradeTestInjectionPoints;
Expand Down Expand Up @@ -155,8 +155,9 @@ public static void initClass() {
.setTotalPipelineNumLimit(NUM_DATA_NODES + 1)
.setHbInterval(500)
.setHbProcessorInterval(500)
.setScmLayoutVersion(INITIAL_VERSION.layoutVersion())
.setDnLayoutVersion(INITIAL_VERSION.layoutVersion());
.setOmLayoutVersion(OMLayoutFeature.INITIAL_VERSION.layoutVersion())
.setScmLayoutVersion(HDDSLayoutFeature.INITIAL_VERSION.layoutVersion())
.setDnLayoutVersion(HDDSLayoutFeature.INITIAL_VERSION.layoutVersion());

// Setting the provider to a max of 100 clusters. Some of the tests here
// use multiple clusters, so its hard to know exactly how many will be
Expand Down Expand Up @@ -219,7 +220,7 @@ private void createKey() throws IOException {
* Helper function to test Pre-Upgrade conditions on the SCM
*/
private void testPreUpgradeConditionsSCM() {
Assert.assertEquals(INITIAL_VERSION.layoutVersion(),
Assert.assertEquals(HDDSLayoutFeature.INITIAL_VERSION.layoutVersion(),
scmVersionManager.getMetadataLayoutVersion());
for (ContainerInfo ci : scmContainerManager.getContainers()) {
Assert.assertEquals(HddsProtos.LifeCycleState.OPEN, ci.getState());
Expand Down
Loading