HDDS-6084. [Multi-Tenant] Handle upgrades to version supporting S3 multi-tenancy #3018

smengcl · 2022-01-25T22:17:56Z

https://issues.apache.org/jira/browse/HDDS-6084

To allow upgrades and downgrades (from pre-finalized state) after multi-tenancy is introduced, the following items must be handled:

Add OM layout feature for multi-tenancy
Block requests to new tenancy APIs while pre-finalized.
Integration and acceptance tests to verify old S3 users are not affected.
- Integration test addition in TestMultiTenantVolume.
- Add acceptance test cases using S3 API in hadoop-ozone/dist/src/main/compose/upgrade/upgrades/non-rolling-upgrade/1.1.0-1.2.0/callback.sh and friends.

Change-Id: I165122fb0e324073d260aed847e4aa3c9b41930c

…and TenantCreate. Change-Id: I3b25525fe301a9d6378ca18be7f8dce4ff1c9867

Change-Id: I453f532062b8f43bf9809d9db55b8a6e4f0fe876

Change-Id: Id96a666efa13423b55454d88a761aae5c8ee2ebd

…arg now. Change-Id: If0558ac9e83b751215ccffd6f0822f9d0d124608

Change-Id: I3471aa2ea3353117775253a3c258f9967d2857a1

Change-Id: Icdb73fb568a5c5dc1c490e26f3e988c6e5a7c9bf

errose28

Thanks for working on this @smengcl and adding one of the first OM layout features. Some minor questions/comments in line.

Also, can we add tests for reading/writing s3 keys from an aws client to the upgrade acceptance test to prove we did not break existing s3 functionality? EDIT: I see that's marked as a todo item in the description.

...op-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/OMLayoutFeature.java

...ne/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/OMLayoutFeatureAspect.java

hadoop-ozone/ozone-manager/src/main/resources/META-INF/aop.xml

...gration-test/src/test/java/org/apache/hadoop/ozone/om/multitenant/TestMultiTenantVolume.java

…rectly on the OM. Change-Id: I84ac02452df899ba6f3b9afa5f2a33390ea8bae5

Change-Id: I3150a4df8a11d8ab97f1ead5e11e9076755b4937

Change-Id: I2bc0b8dbf0c29c904c053e901868150ffa3e3424

Change-Id: Ifed62b7a2ae80c82f15930ceb20149b155835b19

...gration-test/src/test/java/org/apache/hadoop/ozone/om/multitenant/TestMultiTenantVolume.java

…ization. Change-Id: I7fb48bd6086044f00985a8ec820356c7d6575044

…: Python-dotenv could not parse statement starting` Change-Id: Iea465756226bfacc81f328f4082dac1ece95d37a

Change-Id: I08c0156b4468f899822afcb2a73aede7a8ca7e8f

…uster init. Change-Id: I3c017a47f62195e0f921b9f6fa13e42116db51b0

smengcl · 2022-02-04T03:51:40Z

Note: the Acceptance (MR) test failure is resolved by HDDS-6239. That fix will be backported during the next merge of master branch into this feature branch.

I have triggered a CI in my fork on a branch that has HDDS-6239 applied:
https://github.com/smengcl/hadoop-ozone/commits/HDDS-6084-CI

errose28

Thanks for adding the acceptance tests @smengcl just a few comments on those, everything else looks good.

...gration-test/src/test/java/org/apache/hadoop/ozone/om/multitenant/TestMultiTenantVolume.java

hadoop-ozone/dist/src/main/smoketest/upgrade/validate.robot

Change-Id: If126ab0e6a9d4a5e9a57498a662e462d9f59a771

Change-Id: Iacc9a438124a4504a6b78fd7eaa66ab4f20e3b33

smengcl · 2022-02-09T00:02:26Z

Triggered CI: https://github.com/smengcl/hadoop-ozone/commits/HDDS-6084-CI-2

smengcl · 2022-02-10T22:35:47Z

After 9 (!) retriggers on my CI-2 branch I finally got a green run.

Thanks @avijayanhwx for the +1. I will merge this shortly.

errose28

Thanks for the updates @smengcl lgtm as well.

smengcl · 2022-02-10T23:16:16Z

Thanks @errose28 and @avijayanhwx for the review and +1's.

* HDDS-4944: (268 commits) HDDS-6366. [Multi-Tenant] Disallow specifying custom accessId in OzoneManager (apache#3166) HDDS-6275. [Multi-Tenant] Add feature documentation and CLI quick start guide (apache#3095) HDDS-6063. [Multi-Tenant] Use VOLUME_LOCK in read and write requests, and some minor refactoring (apache#3051) HDDS-6214. [Multi-Tenant] Fix KMS Encryption/Decryption (apache#3010) HDDS-6322. Fix Recon getting inccorrect sequenceNumber from OM (apache#3090) HDDS-5913. Avoid integer overflow when setting dfs.container.ratis.lo… (apache#2785) HDDS-6313. Remove replicas in ContainerStateMap when a container is deleted (apache#3086) HDDS-6186. Selective checks: skip integration check for unit test changes (apache#3061) HDDS-6310. Update json-smart to 2.4.7. (apache#3080) HDDS-6190. Cleanup unnecessary datanode id path checks. (apache#2993) HDDS-6304. Add enforcer to make sure ozone.version equals project.version (apache#3075) HDDS-6309. Update ozone-runner version to 20220212-1 (apache#3079) HDDS-6293. Allow using custom ozone-runner image (apache#3072) HDDS-4126. Freon key generator should support >2GB files. (apache#3054) HDDS-6088. Implement O3FS/OFS getFileChecksum() using file checksum helpers - addendum: fix checkstyle HDDS-6088. Implement O3FS/OFS getFileChecksum() using file checksum helpers. (apache#2935) HDDS-6084. [Multi-Tenant] Handle upgrades to version supporting S3 multi-tenancy (apache#3018) HDDS-6257. Wrong stack trace for S3 errors (apache#3066) HDDS-6278 Improve memory profile for listStatus API call. (apache#3053) HDDS-6285. ozonesecure-mr intermittently failing with timeout downloading packages (apache#3057) ...

adoroszlai · 2022-05-26T18:09:00Z

hadoop-ozone/dist/src/main/compose/upgrade/upgrades/non-rolling-upgrade/1.1.0-1.2.0/callback.sh

-  # OM currently only has one layout version.
-  _check_om_mlvs 0


@smengcl @errose28 maybe I'm missing something, but wouldn't this fail if we actually ran upgrade path test?

https://github.com/adoroszlai/hadoop-ozone/runs/6613025379#step:5:738

I think actual OM MLV for 1.2.0 and 1.2.1 is still 0, regardless of this change.

You are right.

Though in the latest feature branch it is still 0 (probably fixed when I was resolving conflicts when doing the merge master 3 days ago or some time ago):

https://github.com/smengcl/hadoop-ozone/blob/HDDS-4944/hadoop-ozone/dist/src/main/compose/upgrade/upgrades/non-rolling-upgrade/1.1.0-1.2.0/callback.sh#L47-L49

Not sure why I bumped it when doing this PR.

Anyway, the om version bump in with_new_version_finalized is intended. As after the finalization it should be bumped to the latest version. But in pre_finalized it should still be zero.

But this callback applies when new version is 1.2.0.

I was going to say when I was merging master to the feature branch just 3 days ago, I had to bump the _check_om_mlvs further to 3 to pass acceptance (misc):

https://github.com/apache/ozone/blame/HDDS-4944/hadoop-ozone/dist/src/main/compose/upgrade/upgrades/non-rolling-upgrade/1.2.1-1.3.0/callback.sh#L74

But then I realized this is the one from 1.1.0 to 1.2.0.

Yup so in this case it should still be zero (in 1.2.0).

This is due to the upgrade test.sh script was actually testing the latest compiled version rather than the actual 1.2.0 at the time, and didn't have 1.2.1-1.3.0 yet. And yes it should be reverted back to 0 now (if I haven't done that yet).

Thanks for catching this @adoroszlai !

Addressed this in commit: 00a5007 (PR #3450)

smengcl added 5 commits January 24, 2022 12:20

HDDS-6084. Handle upgrades to version supporting S3 multi-tenancy

d4d6912

Change-Id: I165122fb0e324073d260aed847e4aa3c9b41930c

Weave aspect; Add layout feature limit to tenant read-only requests, …

d6effa7

…and TenantCreate. Change-Id: I3b25525fe301a9d6378ca18be7f8dce4ff1c9867

Add annotation to the rest of the tenant write requests.

7fcc689

Change-Id: I453f532062b8f43bf9809d9db55b8a6e4f0fe876

ElementType.CONSTRUCTOR doesn't work; cleanup.

10c3bac

Change-Id: Id96a666efa13423b55454d88a761aae5c8ee2ebd

checkLayoutFeature should work with all methods with OM as the first …

32dcfe2

…arg now. Change-Id: If0558ac9e83b751215ccffd6f0822f9d0d124608

smengcl requested a review from errose28 January 25, 2022 22:17

smengcl added 2 commits January 25, 2022 14:51

Add test; checkstyle.

18ab2aa

Change-Id: I3471aa2ea3353117775253a3c258f9967d2857a1

Address test failures.

977f1f9

Change-Id: Icdb73fb568a5c5dc1c490e26f3e988c6e5a7c9bf

smengcl requested a review from prashantpogde January 27, 2022 21:34

errose28 reviewed Feb 1, 2022

View reviewed changes

smengcl added 5 commits February 2, 2022 10:10

Add TODO, auto-generate weaver XML in the future.

59859ea

Integration test: trigger finalization from the client rather than di…

074dada

…rectly on the OM. Change-Id: I84ac02452df899ba6f3b9afa5f2a33390ea8bae5

Integration check: preFinalizationCheck / cleanup.

c8466ad

Change-Id: I3150a4df8a11d8ab97f1ead5e11e9076755b4937

Cleanup addressed comment.

b4010ac

Change-Id: I2bc0b8dbf0c29c904c053e901868150ffa3e3424

More restrictive checking of JoinPoint in checkLayoutFeature.

57af2cf

Change-Id: Ifed62b7a2ae80c82f15930ceb20149b155835b19

errose28 reviewed Feb 2, 2022

View reviewed changes

...gration-test/src/test/java/org/apache/hadoop/ozone/om/multitenant/TestMultiTenantVolume.java Outdated Show resolved Hide resolved

smengcl added 4 commits February 3, 2022 17:36

Acceptance test: add aws s3api testing before and after upgrade final…

62269f1

…ization. Change-Id: I7fb48bd6086044f00985a8ec820356c7d6575044

QoL: Fix comment syntax in docker-config leading to annoying `WARNING…

3ce0326

…: Python-dotenv could not parse statement starting` Change-Id: Iea465756226bfacc81f328f4082dac1ece95d37a

QoL: Fix docker deprecate warning: --no-ansi -> --ansi never

e3baa68

Change-Id: I08c0156b4468f899822afcb2a73aede7a8ca7e8f

Integration test: Move pre-finalization checks and finalization to cl…

993f1c2

…uster init. Change-Id: I3c017a47f62195e0f921b9f6fa13e42116db51b0

errose28 reviewed Feb 8, 2022

View reviewed changes

smengcl added 2 commits February 8, 2022 15:00

Move all create operations to generate; Remove HDDS-6261 repro.

6c37edb

Change-Id: If126ab0e6a9d4a5e9a57498a662e462d9f59a771

Create custom key instead of relying on /opt/hadoop/ files.

47c104c

Change-Id: Iacc9a438124a4504a6b78fd7eaa66ab4f20e3b33

adoroszlai mentioned this pull request Feb 9, 2022

HDDS-6289. Upgrade acceptance test log flooded with parse error #3063

Merged

avijayanhwx approved these changes Feb 9, 2022

View reviewed changes

errose28 approved these changes Feb 10, 2022

View reviewed changes

smengcl merged commit 83a8c5f into apache:HDDS-4944 Feb 10, 2022

adoroszlai reviewed May 26, 2022

View reviewed changes

HDDS-6084. [Multi-Tenant] Handle upgrades to version supporting S3 multi-tenancy #3018

HDDS-6084. [Multi-Tenant] Handle upgrades to version supporting S3 multi-tenancy #3018

Uh oh!

Conversation

smengcl commented Jan 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

errose28 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smengcl commented Feb 4, 2022

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smengcl commented Feb 9, 2022

Uh oh!

smengcl commented Feb 10, 2022

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

smengcl commented Feb 10, 2022

Uh oh!

adoroszlai May 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smengcl May 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smengcl May 26, 2022

Choose a reason for hiding this comment

Uh oh!

adoroszlai May 26, 2022

Choose a reason for hiding this comment

Uh oh!

smengcl May 26, 2022

Choose a reason for hiding this comment

Uh oh!

smengcl May 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smengcl May 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

smengcl commented Jan 25, 2022 •

edited

Loading

errose28 left a comment •

edited

Loading

adoroszlai May 26, 2022 •

edited

Loading

smengcl May 26, 2022 •

edited

Loading

smengcl May 26, 2022 •

edited

Loading

smengcl May 26, 2022 •

edited

Loading