HDDS-7922. [FSO] S3G folder support fso layout filestatus s3A compatibility #4448

mladjan-gadzic · 2023-03-22T12:10:27Z

What changes were proposed in this pull request?

s3g currently says:

/volume/bucket/dir/ exists.
/volume/bucket/dir exists.

s3a is expecting:

/volume/bucket/dir/ exists.
/volume/bucket/dir does not exist.

Proposed is an implementation to resolve the issue and to cover corner cases to support fso folders over s3g by modifying the s3g ObjectEndpoint to check whether the requested path is a directory, if so, to only return ok if path has trailing '/'. This would support s3 Head requests and Get requests. The type of object, where file or directory is determined in the head-object and get-object request through checking the keyInfo (modifying the OzoneKey and OzoneKeyDetails) isFile().

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7922

How was this patch tested?

manual
unit
smoke

This reverts commit 7f09e13.

…" tests" This reverts commit e92dd34.

This reverts commit a495d0d.

This reverts commit 9afb488.

kerneltime · 2023-03-27T16:17:16Z

cc @SaketaChalamchala @hemantk-12 @tanvipenumudy

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java

hemantk-12

Thanks for the patch @mladjan-gadzic

Overall looks good to me. Left couple of minor comments.

hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java

mladjan-gadzic · 2023-03-30T09:24:57Z

@duongkame @hemantk-12 thank you for the review!

hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectHead.java

tanvipenumudy

nit: Could we consider including a test case that verifies the behavior when attempting to retrieve metadata for a file using a key path that ends with a slash ("/")? The expected behavior is that the request should fail with a 404 status code since the specified key path does not correspond to a directory.

mladjan-gadzic · 2023-04-03T10:59:34Z

@tanvipenumudy thanks for the review! Adding one more test for above-mentioned case makes perfect sense.

hadoop-ozone/dist/src/main/smoketest/s3/objectputget.robot

whbing · 2023-06-05T05:17:40Z

hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectGet.java

+    // WHEN
+    final OS3Exception ex =
+        Assertions.assertThrows(OS3Exception.class,
+            () -> rest.get(bucketName, keyPath, null, 0, null, body));


variable body not found in my env (pull from master branch and merged this pr). I'm not sure where is body defined.

Thanks for looking into it!

It is connected with ac42bf0#diff-fa46c2b691ace246572f975dc2119ab738a19d283ac7848d35e78e8a2709af96L73 change. I merged upstream/master and resolved conflicts. It should be fine now.

mladjan-gadzic · 2023-06-28T20:02:04Z

@kerneltime any chance we can get a motion on this PR? It's been a while and all the comments have been addressed.

DaveTeng0 · 2023-06-29T05:52:12Z

@mladjan-gadzic I saw an integration test (filesystem) failure, could you help take a look? Thanks.

mladjan-gadzic · 2023-06-29T08:38:57Z

@mladjan-gadzic I saw an integration test (filesystem) failure, could you help take a look? Thanks.

@DaveTeng0 thanks for having a look! Failing tests are part of already known issue which has its own Jira https://issues.apache.org/jira/browse/HDDS-8526.

smengcl

+1 pending CI and conflict resolution.

@mladjan-gadzic Would you resolve the merge conflicts by merging latest master branch into your PR dev branch? Thanks

# Conflicts: # hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKeyDetails.java # hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java

mladjan-gadzic · 2023-06-30T13:45:40Z

Thank you @smengcl for the review. I merged upstream/master and resolved conflicts. CI on my fork passed. Unfortunatelly I do not have a permission to run CI on upstream. If you could do so, that would be great!

smengcl · 2023-06-30T20:51:43Z

Thanks @duongkame @hemantk-12 @tanvipenumudy @whbing @DaveTeng0 for reviewing this.

…bility (apache#4448)

* master: (36 commits) HDDS-8990. Intermittent timeout waiting on datanode4 9856 to become available (apache#5039) Revert "HDDS-7750. Incorrect WRITE ACL check. (apache#4992)" HDDS-7750. Incorrect WRITE ACL check. (apache#4992) HDDS-8985. Intermittent timeout exiting safe mode in HA secure tests (apache#5033) HDDS-8593. Add RootCARotationPoller to CertClient (apache#5030) HDDS-7645. Kubernetes check should fail fast if cluster cannot start (apache#5028) HDDS-8981. TestRootedOzoneFileSystem runs out of disk space (apache#5029) HDDS-8592. Fetch and save all root certificates during service's certificate rotation. (apache#5025) HDDS-8981. Disable TestRootedOzoneFileSystem#testSafeMode HDDS-8591. Create scheduler to check for new root ca certificates (apache#4961) HDDS-8979. error validating kustomization.yaml (apache#5024) HDDS-8973. Ozone SCM HA should not allocates duplicate IDs when transferring leadership (apache#5018) HDDS-8970. Snapshot Diff should return path relative to bucket root (apache#5015) HDDS-8975. Clarify SCM HA auto-bootstrap doc (apache#5021) HDDS-8689. Rotate Root CA and Sub CA in SCM. (apache#4943) HDDS-8436. Support setSafeMode(), isFileClosed() FileSystem API (apache#4825) HDDS-8880. Intermittent fork timeout in TestOMRatisSnapshots (apache#5022) HDDS-8962. Ensure docker env is stopped (apache#5011) HDDS-7794. [snapshot] SnapshotDiff should throw better error messages for exception handling (apache#5007) HDDS-7922. [FSO] S3G folder support fso layout filestatus s3A compatibility (apache#4448) ...

mladjan-gadzic added 17 commits February 8, 2023 14:30

Fail request for bucket with FSO layout where key is a directory

3a0d8c4

Fail request when key is not a file and path does not end with '/'

637b95f

Address George's comments

20cb8f4

Merge branch 'master' into HDDS-7922

7b82e8f

Address Neil's comments

84d7b23

Make isFile required in constructors and cleanup not used code

e04d3a3

Fix smoketest to be inline with implementation

7f09e13

Increase readability and maintainability for unit tests

03ee03f

Change object name because it interferes with "Zero byte file" tests

e92dd34

Revert "Fix smoketest to be inline with implementation"

6f467ba

This reverts commit 7f09e13.

Revert "Change object name because it interferes with "Zero byte file…

00306f7

…" tests" This reverts commit e92dd34.

Remove GET patch

9afb488

Remove test for GET patch

a495d0d

Temp fix for Zero byte file acceptance test

bbb746c

Revert "Remove test for GET patch"

e001dcf

This reverts commit a495d0d.

Revert "Remove GET patch"

821c591

This reverts commit 9afb488.

Refactor for readability

0ceb8f0

neils-dev added the gr label Mar 22, 2023

mladjan-gadzic marked this pull request as ready for review March 24, 2023 10:52

kerneltime requested review from duongkame, kerneltime and neils-dev March 27, 2023 16:17

duongkame reviewed Mar 29, 2023

View reviewed changes

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java Outdated Show resolved Hide resolved

hemantk-12 reviewed Mar 29, 2023

View reviewed changes

hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java Outdated Show resolved Hide resolved

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java Outdated Show resolved Hide resolved

Improve readability

204a4d9

tanvipenumudy reviewed Apr 3, 2023

View reviewed changes

hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestObjectHead.java Show resolved Hide resolved

tanvipenumudy reviewed Apr 3, 2023

View reviewed changes

Add test when key is a file and key path ends with a slash(/)

8232ed8

smengcl reviewed Apr 4, 2023

View reviewed changes

hadoop-ozone/dist/src/main/smoketest/s3/objectputget.robot Show resolved Hide resolved

kerneltime mentioned this pull request Apr 17, 2023

HDDS-8424. Restore legacy bucket behavior for directory objects. #4576

Merged

whbing reviewed Jun 5, 2023

View reviewed changes

mladjan-gadzic added 2 commits June 5, 2023 09:08

Merge remote-tracking branch 'upstream/master' into HDDS-7922

95b4c72

Resolve conflicts with upstream/master

286eb92

smengcl approved these changes Jun 29, 2023

View reviewed changes

mladjan-gadzic added 2 commits June 30, 2023 11:51

Merge remote-tracking branch 'upstream/master' into HDDS-7922

9366bd2

# Conflicts: # hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKeyDetails.java # hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java

Cleanup after resolving merge conflicts.

9e218fb

smengcl merged commit 68ae858 into apache:master Jun 30, 2023

vtutrinov pushed a commit to Cyrill/ozone that referenced this pull request Jul 3, 2023

HDDS-7922. [FSO] S3G folder support fso layout filestatus s3A compati…

31f6e72

…bility (apache#4448)

vtutrinov pushed a commit to Cyrill/ozone that referenced this pull request Jul 3, 2023

HDDS-7922. [FSO] S3G folder support fso layout filestatus s3A compati…

f14fd9b

…bility (apache#4448)

hemantk-12 mentioned this pull request Jul 24, 2023

HDDS-9072. Set isFile when creating OmKeyInfo for createFile code path #5108

Merged

mladjan-gadzic mentioned this pull request Aug 15, 2023

HDDS-8437. [FSO] S3A compatibility - dfs -mkdir creates a zero byte file instead of a directory #5124

Merged

HDDS-7922. [FSO] S3G folder support fso layout filestatus s3A compatibility #4448

HDDS-7922. [FSO] S3G folder support fso layout filestatus s3A compatibility #4448

Uh oh!

Conversation

mladjan-gadzic commented Mar 22, 2023

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

kerneltime commented Mar 27, 2023

Uh oh!

Uh oh!

hemantk-12 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mladjan-gadzic commented Mar 30, 2023

Uh oh!

Uh oh!

tanvipenumudy left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mladjan-gadzic commented Apr 3, 2023

Uh oh!

Uh oh!

whbing Jun 5, 2023

Choose a reason for hiding this comment

Uh oh!

mladjan-gadzic Jun 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mladjan-gadzic commented Jun 28, 2023

Uh oh!

DaveTeng0 commented Jun 29, 2023

Uh oh!

mladjan-gadzic commented Jun 29, 2023

Uh oh!

smengcl left a comment

Choose a reason for hiding this comment

Uh oh!

mladjan-gadzic commented Jun 30, 2023

Uh oh!

smengcl commented Jun 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

tanvipenumudy left a comment •

edited

Loading

mladjan-gadzic Jun 5, 2023 •

edited

Loading