HDDS-10399. Fixed error in shallow listing an empty directory LEGACY/OBS buckets.

[SaketaChalamchala]

What changes were proposed in this pull request?

When shallow listing an empty directory in a LEGACY/OBS bucket we see the error mentioned below. The fix includes adding a condition to check if list status returns any statues before removing the startKey from the previous round in OzoneBucket.java

ozone sh bucket create /s3v/testbucket -l OBJECT_STORE
hadoop fs -mkdir -p s3a://testbucket/data/test/
hadoop fs -mkdir -p s3a://testbucket/data/test/mr_job/input
24/02/20 23:56:17 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
24/02/20 23:56:17 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
24/02/20 23:56:17 INFO impl.MetricsSystemImpl: s3a-file-system metrics system started
24/02/20 23:56:18 INFO Configuration.deprecation: No unit for fs.s3a.connection.request.timeout(0) assuming SECONDS
24/02/20 23:56:18 INFO impl.DirectoryPolicyImpl: Directory markers will be kept
mkdir: getFileStatus on s3a://testbucket/data/test: com.amazonaws.services.s3.model.AmazonS3Exception: Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Server Error; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null:500 Server Error: Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Server Error; Request ID: null; S3 Extended Request ID: null; Proxy: null)


In S3 gateway log:
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
        at java.base/java.util.Objects.checkIndex(Objects.java:372)
        at java.base/java.util.ArrayList.remove(ArrayList.java:535)
        at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.getNextShallowListOfKeys(OzoneBucket.java:1234)
        at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.getNextListOfKeys(OzoneBucket.java:1136)
        at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.hasNext(OzoneBucket.java:1110)
        at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.get(BucketEndpoint.java:208)
        at jdk.internal.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
        ... 55 more 

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10399

How was this patch tested?

Added Unit Tests.

[SaketaChalamchala]

@whbing @captainzmc @ivandika3 Can you please take a look?

[whbing]

@SaketaChalamchala Thanks for this pr !
Adding statuses.size() > 0 logic looks good to me. +1.

BTW, I'm a little confused in what scenario need mkdir in an OBS bucket. It's not allowed to mkdir in an OBS bucket when using ofs://. Anyway, the above changes are fine.

[SaketaChalamchala]

@SaketaChalamchala Thanks for this pr ! Adding statuses.size() > 0 logic looks good to me. +1.

BTW, I'm a little confused in what scenario need mkdir in an OBS bucket. It's not allowed to mkdir in an OBS bucket when using ofs://. Anyway, the above changes are fine.

You're right mkdir on OBS bucket using ofs:// is not allowed but, it is allowed using s3a://. That's the operation that is failing.

[ivandika3]

Thanks for the patch. LGTM +1.

[adoroszlai]

Thanks @SaketaChalamchala for the patch, @ivandika3, @whbing for the review.