HDDS-9843. Ozone client high memory (heap) utilization

[duongkame]

What changes were proposed in this pull request?

Ozone client allocates large chunks of heap memory when writing keys (BlockOutputStream, BufferPool).

Extract client-related buffer improvement from #5497.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9843

How was this patch tested?

CI: https://github.com/duongkame/ozone/actions/runs/7759444701

[jojochuang]

will there be a similar effort to improve input stream memory consumption?

[duongkame]

will there be a similar effort to improve input stream memory consumption?

Yes. https://issues.apache.org/jira/browse/HDDS-10283
Also, this PR is still a WIP, the non-incremental buffer needs changing as well.

[duongkame]

This one is blocked by https://issues.apache.org/jira/browse/HDDS-10288.

[jojochuang]

Looks good. It would be nice to enable the leak detection for some of the tests that use BlockOutputStream (e.g. TestBlockOutputStream)

[duongkame]

Looks good. It would be nice to enable the leak detection for some of the tests that use BlockOutputStream (e.g. TestBlockOutputStream)

Thanks for the review @jojochuang. All the tests with MiniCluster have LeakDetection enabled by default already.
https://github.com/apache/ozone/blob/master/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java#L120

[smengcl]

hmm. leak detected in tests?

Error:  org.apache.hadoop.hdds.scm.storage.TestCommitWatcher.testReleaseBuffers  Time elapsed: 18.44 s  <<< FAILURE!
java.lang.AssertionError: Found 2 leaked objects, check logs
	at org.apache.hadoop.hdds.utils.db.CodecBuffer.assertNoLeaks(CodecBuffer.java:232)
	at org.apache.hadoop.hdds.utils.db.CodecTestUtil.gc(CodecTestUtil.java:54)

Error:  org.apache.hadoop.hdds.scm.storage.TestCommitWatcher.testReleaseBuffersOnException  Time elapsed: 35.839 s  <<< FAILURE!
java.lang.AssertionError: Found 4 leaked objects, check logs
	at org.apache.hadoop.hdds.utils.db.CodecBuffer.assertNoLeaks(CodecBuffer.java:232)

https://github.com/apache/ozone/actions/runs/7823740701/job/21345534308#step:6:178

[smengcl]

Thanks @duongkame . lgtm

btw do we have data from (manual) testing that actually shows the lower client JVM heap usage?

[adoroszlai]

Thanks @duongkame for the improvement.

Since it is extracted from #5497, I think we should add the following when merging this PR:

Co-authored-by: Tsz-Wo Nicholas Sze <[email protected]>

[szetszwo]

@duongkame , thanks a lot for working on this!

+1 the change looks good.

[szetszwo]

@jojochuang , @smengcl and @adoroszlai , thanks a lot for reviewing this!