Implement PrestoS3FileSystem#listFiles for direct recursive listings

[pettyjamesm]

Cross contribution of prestodb/presto#15024

Implements FileSystem#listFiles(Path, boolean recursive) for PrestoS3FileSystem which in theory adds support for directly listing S3 files underneath a prefix without recursive calls through each "directory". This direct traversal requires much fewer requests when dealing with nested directories but may violate some PathFilter implementation's expectation of being called at each directory level, and may perform worse when a large number objects are contained within hidden paths (since filtering would be performed after the fact). I'm open to suggestions about how to balance this trade-off and integrate this with the DirectoryLister.

Incidentally, a straightforward improvement to getFileStatus fell out of the implementation allowing the isDir check for a path prefix (with no object present) to be done by limiting the listing result size to 1 instead of listing the full default 1000.

[raunaqmorarka]

Related PR #3221

[electrum]

Thanks!

[findepi]

Without a change in BackgroundHiveSplitLoader are we benefitting from this already?

[pettyjamesm]

Without a change in BackgroundHiveSplitLoader are we benefitting from this already?

At the moment, no. I didn’t currently have the bandwidth to adapt the code we have to work in a world with caches and / or hudi integration so I contributed just this piece for now. I’ve talked with @electrum about the basic shape of what you would have to do in order to leverage this from split loading so it’s possible someone could pick that up in the mean time.