HDDS-4209. S3A Filesystem does not work with Ozone S3.

[bharatviswa504]

What changes were proposed in this pull request?

When Key is 0 bytes, and ending with "/" call createDirectory API. In this way below scenario can work from S3A.

hdfs dfs -mkdir s3a://sept11-1/dir1/dir2/dir3
hdfs dfs -put /etc/hadoop/conf/ozone-site.xml s3a://sept11-1/dir1/dir2/dir3/key1

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-4209

How was this patch tested?

Tested it on a cluster

[root@bvoz-1 ~]# oapi create-bucket --bucket sept11-1
{
"Location": "https://bvoz-1.bvoz.root.hwx.site:9879/sept11-1"
}

[root@bvoz-1 ~]# hdfs dfs -put /etc/hadoop/conf/ozone-site.xml s3a://sept11-1/dir1/dir2/dir3/key1

[root@bvoz-1 ~]# hdfs dfs -ls -R s3a://sept11-1/
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
drwxrwxrwx - root root 0 2020-09-11 23:28 s3a://sept11-1/dir1
drwxrwxrwx - root root 0 2020-09-11 23:28 s3a://sept11-1/dir1/dir2
drwxrwxrwx - root root 0 2020-09-11 23:28 s3a://sept11-1/dir1/dir2/dir3
-rw-rw-rw- 1 root root 2271 2020-09-11 23:28 s3a://sept11-1/dir1/dir2/dir3/key1

[elek]

Thanks to identify problem and to work on this @bharatviswa504

It seems to be a workaround which fixes s3a but I have two comments:

1. In general I would prefer to keep the fs/object-store separation. I think it makes the abstraction harder to follow/maintain if we start to use fs related om calls from s3, which supposed to use the object-store endpoints.

For example in case of an fs related change, s3 behavior will be changed.

2. In this specific case, intermediate directories will be created even if OZONE_OM_ENABLE_FILESYSTEM_PATHS is not enabled. I created HDDS-4238 to make it more visible:

Without this patch it passes, but with the patch it fails.

3. I think it's a safer approach to fix the normalization (in case of OZONE_OM_ENABLE_FILESYSTEM_PATHS enabled), to avoid the removal of / from the end if the file size is zero.

[bharatviswa504]

In this specific case, intermediate directories will be created even if OZONE_OM_ENABLE_FILESYSTEM_PATHS is not >enabled. I created HDDS-4238 to make it more visible:

Good point. Might be we need to have a flag/param in createDirectory to say create intermediate directories or not. In this way, we can use this flag from the client in proto along with the normalization flag(current config in code) to create intermediate directories or not.

Example:
boolean zerobyteFile default=false

It Will be set to true only from S3G, and when create directory comes to OM, it uses a normalization flag to create intermediate directories, else just create an entry without any intermediate directories.

I think it's a safer approach to fix the normalization (in case of OZONE_OM_ENABLE_FILESYSTEM_PATHS enabled), to >avoid the removal of / from the end if the file size is zero.

My reasoning to take this approach is once HDDS-2939 comes in Ozone directory and key are not distinguished with trailing "/". So, using putObject when length is zero might not be a correct solution in OM, as the entries will be still created in keyTable. For this, if we want to go this route, then might be if ending with "/" and size is zero, in putObject we should create an entry in the directory table. So, instead of doing these changes in OM, I thought it could be safer to do in S3G. As in S3G when someones tries to create a zero byte file with trailing "/" might be they want to simulate like a directory in Object Store.

Let me know your thoughts on how to proceed?

[avijayanhwx]

1. bject-store endpoints.

Thanks for working on this @bharatviswa504 and reviewing it @elek.

My 2 cents.

Overall, I am OK with this approach.

In my mind, the s3g is a component whose principal responsibility is to support s3 use cases on Ozone as much as possible (hopefully 100%). The s3g belongs to Ozone world, and would be the right place to make changes for anything that needs knowledge of Ozone API internals for supporting s3 related APIs.

If we retain the trailing "/" for files that are of "0" byte, we would need to think about how to support the use case of creating a zero byte file, and appending to it.

I don't think we should add more configs to solve this problem. I would rather have 95% support of s3 in the short term than add more complexity and "states" OM APIs can be in based on config values.

[elek]

My reasoning to take this approach is once HDDS-2939 comes in Ozone directory and key are not distinguished with trailing "/". So, using putObject when length is zero might not be a correct solution in OM, as the entries will be still created in keyTable. For this, if we want to go this route, then might be if ending with "/" and size is zero, in putObject we should create an entry in the directory table.

Thanks to explain it @bharatviswa504 I re-read the HDDS-2939 spec, and it's not clear how the 100% compatibility (in case OZONE_OM_ENABLE_FILESYSTEM_PATHS=false) can be achieved with prefixes if we don't create key entries for all the keys.

Can I create both /a/b/c/d and /a/b/c/d/ keys with HDDS-2939 (OZONE_OM_ENABLE_FILESYSTEM_PATHS=false)?

(cc @rakeshadr, @linyiqun)

In this specific case, intermediate directories will be created even if OZONE_OM_ENABLE_FILESYSTEM_PATHS is not enabled. I created HDDS-4238 to make it more visible.

This is still a problem. I think it's easier to fix with adjusting normalization to handle this case. HDDS-2939 seems to have bigger problems which should be solved before the merge, until that we can quickly fix this problem.

But if you have any other proposed solution for the mentioned problem (intermediate directories will be created even if OZONE_OM_ENABLE_FILESYSTEM_PATHS is not enabled), please let me know.

[elek]

/pending In this specific case, intermediate directories will be created even if OZONE_OM_ENABLE_FILESYSTEM_PATHS

[github-actions]

Marking this issue as un-mergeable as requested.

Please use /ready comment when it's resolved.

Please note that the PR will be closed after 21 days of inactivity from now. (But can be re-opened anytime later...)

In this specific case, intermediate directories will be created even if OZONE_OM_ENABLE_FILESYSTEM_PATHS

[bharatviswa504]

Closing this for now, as we donot support S3A FileSystem in filesystem path enabled mode.
Need to document this.