HADOOP-18105 Implement buffer pooling with weak references

[mukund-thakur]

Description of PR

part of HADOOP-18103.
Required for vectored IO feature. None of current buffer pool
implementation is complete. ElasticByteBufferPool doesn't use
weak refrences and could lead to memory leak errors and
DirectBufferPool doesn't support caller prefrences of direct
and heap buffers and has only fixed length buffer implementation.

How was this patch tested?

Added new unit tests and tested through vectored read api integration test.

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[mukund-thakur]

CC @mehakmeet @steveloughran

[mukund-thakur]

Don't know why javadoc is playing up in all my patches.

[steveloughran]

there are some active patches for javadoc; don't worry too much for now.

[steveloughran]

happy with the revision, but your new test has to use intercept() when it wants to raise exceptions in a test.

[steveloughran]

a few minor suggestions. deciding how to handle an unknown buffer being returned is the key one

+1 pending those changes.

[mukund-thakur]

deciding how to handle an unknown buffer being returned is the key one

What do you mean by this? I don't think there is a way to know whether the buffer being returned currently through putBuffer() was part of this pool or not.
Right now the puBuffer() call succeeds if I create a random buffer directly in my tests.

[steveloughran]

Right now the puBuffer() call succeeds if I create a random buffer directly in my tests.

ok. we treat that as a success. just add a javadoc warning of this and say "may change in future"

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 56s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ feature-vectored-io Compile Tests _
+1 💚	mvninstall	40m 25s		feature-vectored-io passed
+1 💚	compile	25m 10s		feature-vectored-io passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	21m 42s		feature-vectored-io passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 31s		feature-vectored-io passed
+1 💚	mvnsite	1m 59s		feature-vectored-io passed
-1 ❌	javadoc	1m 37s	/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt	hadoop-common in feature-vectored-io failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.
+1 💚	javadoc	2m 1s		feature-vectored-io passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 5s		feature-vectored-io passed
+1 💚	shadedclient	26m 5s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 5s		the patch passed
+1 💚	compile	24m 12s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	24m 12s		the patch passed
+1 💚	compile	21m 33s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	21m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 25s		the patch passed
+1 💚	mvnsite	1m 56s		the patch passed
-1 ❌	javadoc	1m 24s	/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt	hadoop-common in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.
+1 💚	javadoc	1m 58s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 2s		the patch passed
+1 💚	shadedclient	25m 43s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 16s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 17s		The patch does not generate ASF License warnings.
		226m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4263/5/artifact/out/Dockerfile
GITHUB PR	#4263
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux e73fb3e76e1a 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-vectored-io / 24ac304
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4263/5/testReport/
Max. process+thread count	1236 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4263/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[steveloughran]

+1 from me