HADOOP-11867. Add a high performance vectored read API to file system.

[mukund-thakur]

Rebased work on top of #1830

Conflicts:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BufferedFSInputStream.java
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
pom.xml

Description of PR

Adding support for multiple ranged read async api in PositionedReadable. The default iterates through the ranges to read each synchronously, but the intent is that FSDataInputStream subclasses can make more efficient readers especially object stores implementation.

How was this patch tested?

Added benchmarks.
Added UT's
Added new contract tests for new API spec.

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[mukund-thakur]

CC @steveloughran @mehakmeet @omalley

[steveloughran]

Quick review.

• needs filesystem spec

• and a contract test

• test to include
  -overlapping ranges
  -beyond file length
  -negative values
  -on heap and off heap buffers

• be good to have a PoC of the s3a one, so that we can happy the API xfers to the store

• and some ORC/parquet branch which shows it can be used

• is there any API we'd want, e.g an fs command which does a single async ranged get

hadoop fs ranged -start 0 -end 345  -out output.txt s3a://file

would be good for QE tests

• and we should have a path capability. Anything which supports seek will also do this (so ftp won't)

[bogthe]

Did an initial review and looks promising.

I'm super curious to see what an async read will bring in terms of performance improvements for a job (even if it's a quick scrappy implementation in the FS).

[steveloughran]

I just realized I had some pending comments on one of the updates; maybe out of date now and they were all minor. Let me re-review.

[steveloughran]

here's some comments I had outstanding since 2021 .. sorry. will do a bigger review now

[steveloughran]

These are all needless and making the patch a bit bigger than it should be. was yetus complaining?

[mukund-thakur]

I don't know how it is showing up here. In intellJ diff I don't see this.

[steveloughran]

can you use openFile here? just to make sure that codepath is happy

[steveloughran]

OK, I've had a look at the recent changes, and had some shared screen reviews over zoom

here's what I propose

• create a feature branch in the asf repo where we can rebase/reorder commits
• get what is done in as the base commit
• then we can split work on the final changes up into new PRs, each with their own JIRA.

where those changes can be

• buffer pooling (API to these calls and better weak reference handling in the elastic one or nearby)
• s3a split handling
• huge file support (new tests in the s3a huge file suite)

then, once happy we can do a rebase followed by merge commit into trunk/branch-3.

key point: this patch has been going and is ready to stabilize, which can be done with incremental changes in a feature branch.

[mehakmeet]

Did an initial Review, looks really good.
I think we should add an integration test around the ordered disjoint list of ranges as well to showcase them not merging and reading or having a few ranges where some are merged and some aren't by validating them.
Also, are both minSeek and maxReadSize going to be configurable? I assume they are hardcoded just so you can test them out..

[mukund-thakur]

Did an initial Review, looks really good. I think we should add an integration test around the ordered disjoint list of ranges as well to showcase them not merging and reading or having a few ranges where some are merged and some aren't by validating them.

Yeah already in my TODO. Thanks.

Also, are both minSeek and maxReadSize going to be configurable? I assume they are hardcoded just so you can test them out..
Yes I will be making them configurable in consecutive PR's

Also Please refer to the new PR now. #3904

[steveloughran]

now this is in the feature branch, this can be closed