HDDS-9347. Fix Ozone FS listStatus() cache-table inconsistencies.

[sadanand48]

What changes were proposed in this pull request?

Few bugs found and fixed while testing with cache unflushed

• Fake dir not created when fsPaths is disabled
• CacheIterators should always skip null values as when we delete an object we mark it as null in cache first and later delete it from the RawTable
• StartKey means the starting key from which listStatus should display in order not that keyName should start with the key. This is fixed. The below current code has this bug.

if (cacheOmKeyInfo != null
          && cacheKey.startsWith(startCacheKey)
          && cacheKey.compareTo(startCacheKey) >= 0) {

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9347

How was this patch tested?

Added an extra combination in TestRootedOzoneFileSystem to test when the OM Table cache is not flushed, This is useful in catching hold of issues that is caused by cache-table inconsistencies

Ran the entire test with extra combinations too : 15x10 times here : https://github.com/sadanand48/hadoop-ozone/actions/runs/6415907619

[adoroszlai]

@sadanand48 thanks for the patch. Can you please trigger repeated runs for a few more FS test classes (TestOzoneFileSystem, etc.) on this branch in your fork?

[adoroszlai]

TestOzoneFileSystem 10x15 passed: https://github.com/sadanand48/hadoop-ozone/actions/runs/6493771280

[arp7]

@swamirishi can you review this change?

[kerneltime]

@SaketaChalamchala @tanvipenumudy can you take a look at this as well?

[jojochuang]

Looks very similar to the bug described in #5093 / HDDS-8877

[swamirishi]

Thanks for the patch @sadanand48. Overall the patch looks good to me. Have a minor nitpicky comment you can take care of.

[swamirishi]

from what i understand this list would be for only one bucket. This can be outside the loop, it would be a constant string for this function.

[adoroszlai]

Can we move this to a follow-up task? It would be nice to avoid blocking the fix for a single nitpicky comment.

[sadanand48]

Thanks @swamirishi for the review. Addressed the comment.

[kerneltime]

Is there a test that will fail without this fix?

[adoroszlai]

Yes, e.g. testListStatusIteratorOnPageSize fails intermittently with small page size.

https://github.com/apache/ozone/actions/runs/6263989725/job/17010674585#step:5:3786

[sadanand48]

Also the new test changes in this PR will fail without code changes

[adoroszlai]

Thanks @sadanand48 for the patch, @jojochuang, @kerneltime, @swamirishi for the review.