HDDS-8076. Use container cache in Key listing API.

[duongkame]

What changes were proposed in this pull request?

Use container cache in Key listing API to improve Key/file listing performance.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-8076

How was this patch tested?

Unit test.

[smengcl]

nice.

need another +1 as I might not have the full context.

[kerneltime]

We need to check if the client is caching or exposing the pipeline-refreshed results in any way. One alternative without breaking behavior would be adding a lighter-weight list keys API (something discussed in the context of ls command and list bucket API context).

[duongkame]

We need to check if the client is caching or exposing the pipeline-refreshed results in any way. One alternative without breaking behavior would be adding a lighter-weight list keys API (something discussed in the context of ls command and list bucket API context).

In the ozone-client, I don't see any code using block location out of a listStatus result.

lock location info is used in RpcClient getKey or getFile API that read key metadata individually from OM and creates OzoneInputStream that allows reading data from datanodes given the block location information.
OzoneInputStream will retry reading individual key metadata if it finds the metatadata doesn't work.

OFS listStatus convert the files information and block location to a structure of org.apache.hadoop.fs.LocatedFileStatus. I'm not sure how this is used by external dependency. But I guess any data read needs to be done by OzoneInputStream eventually.

There's one problem left, newer clients will have OzoneInputStream retry getting block location with getKeyInfo(cacheRefresh=true) and that will force the container location cache to refresh in OM. Yet, older clients will call lookupKey which calls SCM directly to grab container location. There will be a recurring performance degradation for older clients if OM caches an outdated container location.
But yet, I don't see a usage of block information from the result of listStatus yet.

[kerneltime]

I will file a separate jira to work on lightweight list keys. For now, this looks good. Will try out the code and review.

[sumitagrawl]

LGTM, cache TTL is 6 hours as default, as currently observed that location information is not used.
I can see below JIRA where block location is populated for integration with Hive as per code comment...

HDDS-2188. Implement LocatedFileStatus & getFileBlockLocations to provide node/localization information to Yarn/Mapreduce
HDDS-2914. Certain Hive queries started to fail on generating splits (#563)
HDDS-2914. Certain Hive queries started to fail on generating splits

Plz check if this integration have some imact

[duongkame]

LGTM, cache TTL is 6 hours as default, as currently observed that location information is not used. I can see below JIRA where block location is populated for integration with Hive as per code comment...

HDDS-2188. Implement LocatedFileStatus & getFileBlockLocations to provide node/localization information to Yarn/Mapreduce HDDS-2914. Certain Hive queries started to fail on generating splits (#563) HDDS-2914. Certain Hive queries started to fail on generating splits

Plz check if this integration have some imact

Thanks for the deep-dive, @sumitagrawl. As discussed in the community meeting, listFileStatus doesn't calculate block tokens for the return BlockLocationInformation. Thus, clients of the listFileStatus API can't use the result to read data. So, whatever the usage of LocatedFileStatus & getFileBlockLocations from the list API is not on the critical path of reading data and therefore no guarantee of strong block location info consistency.

I also got through the JIRAs you found. They helped me understand how the LocatedFileStatus is brought to OFS API. Also, the JIRAs only concerns with getFileStatus API which provide individual file information, and not listFileStatus. My best guess is that the LocatedFileStatus is reused for both OFS APIs for programming consistency (and not actually intended to listFileStatus clients).