HADOOP-19073 WASB: Fix connection leak in FolderRenamePending #6534

xuzifu666 · 2024-02-07T10:44:12Z

Description of PR

Currently connection leak in FolderRenamePending in getting bytes. the pr aim to fix it

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

xuzifu666 · 2024-02-07T10:49:49Z

@ayushtkn @slfan1989 PTAL

hadoop-yetus · 2024-02-07T12:49:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	42m 6s		trunk passed
+1 💚	compile	0m 37s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 34s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 31s		trunk passed
+1 💚	mvnsite	0m 39s		trunk passed
+1 💚	javadoc	0m 38s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 4s		trunk passed
+1 💚	shadedclient	33m 57s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 28s		the patch passed
+1 💚	compile	0m 28s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 28s		the patch passed
+1 💚	compile	0m 26s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 26s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 19s		the patch passed
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	javadoc	0m 25s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 2s		the patch passed
+1 💚	shadedclient	32m 14s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 57s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		123m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6534/1/artifact/out/Dockerfile
GITHUB PR	#6534
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux ac3427661d95 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `649c40f`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6534/1/testReport/
Max. process+thread count	700 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6534/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-02-07T13:01:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 56s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	42m 5s		trunk passed
+1 💚	compile	0m 36s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 34s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 32s		trunk passed
+1 💚	mvnsite	0m 38s		trunk passed
+1 💚	javadoc	0m 38s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 33s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 3s		trunk passed
+1 💚	shadedclient	31m 59s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 26s		the patch passed
+1 💚	compile	0m 28s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 28s		the patch passed
+1 💚	compile	0m 25s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 25s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 18s		the patch passed
+1 💚	mvnsite	0m 28s		the patch passed
+1 💚	javadoc	0m 25s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	1m 3s		the patch passed
+1 💚	shadedclient	32m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 15s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		134m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6534/2/artifact/out/Dockerfile
GITHUB PR	#6534
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 5eae444d189b 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `649c40f`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6534/2/testReport/
Max. process+thread count	624 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6534/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

zhangshuyan0

+1.

zhangshuyan0 · 2024-02-08T03:16:54Z

Well, this may not belong to the HDFS module.

slfan1989 · 2024-02-08T03:22:09Z

Well, this may not belong to the HDFS module.

@zhangshuyan0 Thank you for your review! but I think this PR needs Steve’s help to review it.

@xuzifu666 Thanks for the contribution! we need to wait for the results of Steve's review.

xuzifu666 · 2024-02-08T03:35:16Z

Well, this may not belong to the HDFS module.

@zhangshuyan0 Thank you for your review! but I think this PR needs Steve’s help to review it.

@xuzifu666 Thanks for the contribution! we need to wait for the results of Steve's review.

OK， @steveloughran could you give the final review? or @slfan1989 could you help to ping the commiters related to the pr to review it？Thanks ~

steveloughran · 2024-02-09T14:36:22Z

moved to hadoop; azure component.

now, the bad news: as it goes near a cloud store, you have to run the entire hadoop-azure tests for the wasb component, and tell us which region you tested against.

https://hadoop.apache.org/docs/stable/hadoop-azure/testing_azure.html

sorry, but yetus doesn't have any credentials. It doesn't take long and is designed to clean up afterwards

xuzifu666 · 2024-02-09T14:42:51Z

moved to hadoop; azure component.

now, the bad news: as it goes near a cloud store, you have to run the entire hadoop-azure tests for the wasb component, and tell us which region you tested against.

https://hadoop.apache.org/docs/stable/hadoop-azure/testing_azure.html

sorry, but yetus doesn't have any credentials. It doesn't take long and is designed to clean up afterwards

OK，anything else me to do for the pull request？@steveloughran

steveloughran · 2024-02-09T19:38:43Z

@xuzifu666 yes.

Which azure region did you run all the wasb tests against
and what were your maven command line arguments used?
did any tests fail?

xuzifu666 · 2024-02-18T10:51:06Z

@xuzifu666 yes.

Which azure region did you run all the wasb tests against

and what were your maven command line arguments used?

did any tests fail?

@steveloughran Hi，seems network connection available，test is hard for me to execute，but the pr may be a obvious connection leak，could you give a help to confirm it？Thank any way~

hadoop-yetus · 2024-04-25T14:19:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
			_ Prechecks _
+1 💚	dupname	0m 00s		No case conflicting files found.
+0 🆗	spotbugs	0m 00s		spotbugs executables are not available.
+0 🆗	codespell	0m 01s		codespell was not available.
+0 🆗	detsecrets	0m 01s		detect-secrets was not available.
+1 💚	@author	0m 00s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 00s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	86m 45s		trunk passed
+1 💚	compile	4m 35s		trunk passed
+1 💚	checkstyle	4m 20s		trunk passed
+1 💚	mvnsite	4m 45s		trunk passed
+1 💚	javadoc	4m 26s		trunk passed
+1 💚	shadedclient	140m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 33s		the patch passed
+1 💚	compile	2m 06s		the patch passed
+1 💚	javac	2m 06s		the patch passed
+1 💚	blanks	0m 01s		The patch has no blanks issues.
+1 💚	checkstyle	1m 56s		the patch passed
+1 💚	mvnsite	2m 13s		the patch passed
+1 💚	javadoc	2m 01s		the patch passed
+1 💚	shadedclient	150m 23s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	5m 41s		The patch does not generate ASF License warnings.
		399m 51s

Subsystem	Report/Notes
GITHUB PR	#6534
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	MINGW64_NT-10.0-17763 2ea128c33854 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys
Build tool	maven
Personality	/c/hadoop/dev-support/bin/hadoop.sh
git revision	trunk / `649c40f`
Default Java	Azul Systems, Inc.-1.8.0_332-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6534/1/testReport/
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6534/1/console
versions	git=2.44.0.windows.1
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

xuzifu666 · 2024-04-25T22:47:32Z

@xuzifu666 yes.

Which azure region did you run all the wasb tests against

and what were your maven command line arguments used?

did any tests fail?

@steveloughran Hi，seems network connection available，test is hard for me to execute，but the pr may be a obvious connection leak，could you give a help to confirm it？Thank any way~
cc@steveloughran

xuzifu666 · 2024-04-26T06:13:34Z

@steveloughran could you give the final review or close the pr? Thanks

steveloughran · 2024-05-02T19:42:15Z

let me try and test this myself.

steveloughran · 2024-05-02T19:43:15Z

(which I can't do today as both my hadoop source trees are testing my "support parallel tests against the same s3 bucket" right now...

xuzifu666 · 2024-05-06T03:01:44Z

let me try and test this myself.

OK，Thanks，currently it can be work now？ @steveloughran

steveloughran · 2024-05-13T17:31:06Z

I've been away for a week and am catching up, so not tested yet, sorry...

steveloughran

+1 merging anyway as it clearly does what we need and is very simple to review

steveloughran · 2024-05-13T17:34:07Z

@xuzifu666 before I merge, what name do you want to be credited with in the patch...your github account doesn't have one.

steveloughran · 2024-05-15T13:39:42Z

thanks, merged to trunk and updated the JIRA.

If you do a cherrypick of this commit and submit as a PR against branch-3.4 I'll merge it there too, once yetus is happy

…#6534) Contributed by xuyu

HDFS-17373. Fix connection leak in FolderRenamePending

9ca9268

github-actions bot added trunk TOOLS ABFS labels Feb 7, 2024

style fix

649c40f

zhangshuyan0 approved these changes Feb 8, 2024

View reviewed changes

steveloughran changed the title ~~HDFS-17373. Fix connection leak in FolderRenamePending~~ HADOOP-19073 WASB: Fix connection leak in FolderRenamePending Feb 9, 2024

steveloughran approved these changes May 13, 2024

View reviewed changes

steveloughran merged commit cf9559e into apache:trunk May 15, 2024
1 of 4 checks passed

K0K0V0K pushed a commit to K0K0V0K/hadoop that referenced this pull request May 17, 2024

HADOOP-19073 WASB: Fix connection leak in FolderRenamePending (apache…

a4899e3

…#6534) Contributed by xuyu

K0K0V0K pushed a commit to K0K0V0K/hadoop that referenced this pull request May 17, 2024

HADOOP-19073 WASB: Fix connection leak in FolderRenamePending (apache…

05af8d3

…#6534) Contributed by xuyu

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-19073 WASB: Fix connection leak in FolderRenamePending #6534

HADOOP-19073 WASB: Fix connection leak in FolderRenamePending #6534

xuzifu666 commented Feb 7, 2024 •

edited

Loading

xuzifu666 commented Feb 7, 2024

hadoop-yetus commented Feb 7, 2024

hadoop-yetus commented Feb 7, 2024

zhangshuyan0 left a comment

zhangshuyan0 commented Feb 8, 2024

slfan1989 commented Feb 8, 2024 •

edited

Loading

xuzifu666 commented Feb 8, 2024 •

edited

Loading

steveloughran commented Feb 9, 2024

xuzifu666 commented Feb 9, 2024

steveloughran commented Feb 9, 2024

xuzifu666 commented Feb 18, 2024

hadoop-yetus commented Apr 25, 2024

xuzifu666 commented Apr 25, 2024

xuzifu666 commented Apr 26, 2024

steveloughran commented May 2, 2024

steveloughran commented May 2, 2024

xuzifu666 commented May 6, 2024

steveloughran commented May 13, 2024

steveloughran left a comment

steveloughran commented May 13, 2024

steveloughran commented May 15, 2024

HADOOP-19073 WASB: Fix connection leak in FolderRenamePending #6534

HADOOP-19073 WASB: Fix connection leak in FolderRenamePending #6534

Conversation

xuzifu666 commented Feb 7, 2024 • edited Loading

Description of PR

How was this patch tested?

For code changes:

xuzifu666 commented Feb 7, 2024

hadoop-yetus commented Feb 7, 2024

hadoop-yetus commented Feb 7, 2024

zhangshuyan0 left a comment

Choose a reason for hiding this comment

zhangshuyan0 commented Feb 8, 2024

slfan1989 commented Feb 8, 2024 • edited Loading

xuzifu666 commented Feb 8, 2024 • edited Loading

steveloughran commented Feb 9, 2024

xuzifu666 commented Feb 9, 2024

steveloughran commented Feb 9, 2024

xuzifu666 commented Feb 18, 2024

hadoop-yetus commented Apr 25, 2024

xuzifu666 commented Apr 25, 2024

xuzifu666 commented Apr 26, 2024

steveloughran commented May 2, 2024

steveloughran commented May 2, 2024

xuzifu666 commented May 6, 2024

steveloughran commented May 13, 2024

steveloughran left a comment

Choose a reason for hiding this comment

steveloughran commented May 13, 2024

steveloughran commented May 15, 2024

xuzifu666 commented Feb 7, 2024 •

edited

Loading

slfan1989 commented Feb 8, 2024 •

edited

Loading

xuzifu666 commented Feb 8, 2024 •

edited

Loading