HDDS-7117. Consider reading chunk files using MappedByteBuffer.

[szetszwo]

What changes were proposed in this pull request?

From @jojochuang

Running Impala TPC-DS which stresses Ozone DN read path.
BufferUtils#assignByteBuffers stands out as one of the offender.
We can experiment with MappedByteBuffer and see if it makes performance better.

What is the link to the Apache JIRA

HDDS-7117

How was this patch tested?

New unit test.

[jojochuang]

Thanks @szetszwo for the PR!
The PR makes sense to me. I'll need to verify the performance difference in the cluster.

[jojochuang]

TODO: I hope this patch can better leverage OS cache, get rid of the temporary heap buffer. Will verify and benchmark in a real cluster.

[szetszwo]

@jojochuang , thanks a lot for testing this!

[jojochuang]

Please hold the merge as I am still doing a few more perf benchmarks. Thanks.

[szetszwo]

@jojochuang , sure. Will wait for your test results. Thanks a lot!

[swamirishi]

@jojochuang Can this be merged? Are the test results out?

[ChenSammi]

@szetszwo , thanks for working on this. Do you have time to do a patch rebase?

I did a micro performance test with a 4.5GB file, comparing the on heap buffer, off heap buffer and memory mapped buffer, here is the result.

By default, DN will use 1MB(bytes.per.checksum) as the buffer size. So using memory mapped buffer can increase at least micro level performance a lot.

[szetszwo]

@ChenSammi , thanks a lot for running the benchmark! When the file size is large, the performance is expected to be better for MappedByteBuffer . Your benchmark confirms the expectation.

Let me update this change.

[ChenSammi]

@szetszwo , there is another your PR #3730 which aims to improve the read performance too. I didn't got chance to closely look at #3730 yet. Not sure is there overlap between this one and #3730?

[szetszwo]

@ChenSammi , this PR is to change reading the local files using java MappedByteBuffer while #3730 uses Netty ChunkedNioFile.

[ChenSammi]

The last patch LGTM +1.

[szetszwo]

@ChenSammi , thanks a lot for reviewing this!