Cherry-pick necessary mirror PRs from trunk branch to Branch-3.4 branch #8105

slfan1989 · 2025-11-27T00:49:15Z

Description of PR

The purpose of this PR is to maintain consistency in compilation between the trunk and branch-3.4. It mainly involves cherry-picking compilation-related PRs from the trunk branch to address the EOL (End of Life) issues of CentOS 7, CentOS 8, and Debian 10.

How was this patch tested?

CI.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…7 as default (apache#7608) 1. Added "ubuntu:noble" to supported docker platforms. 2. Modified `start-build-env.sh` to use expected Dockerfiles dynamically. ex: `bash start-build-env.sh ubuntu_24` where `ubuntu_24` is the extra suffix in the name of the Dockerfile. 3. Added `jdk17++` profile activated when jdk17 is available. Successfully built the entire Hadoop tar with native support on env created using `bash start-build-env.sh ubuntu_24`

* The current version of libxxhash - 0.8.1 isn't available on the msys repo. * This is causing the Hadoop Jenkins CI for Windows to fail while building the docker image for Windows 10. * This PR upgrades libxxhash to 0.8.3 to fix this issue.

apache#7822) * HADOOP-19636. [JDK17] Remove CentOS 7 Support and Clean Up Dockerfile. Co-authored-by: Cheng Pan <[email protected]> Co-authored-by: Istvan Toth <[email protected]> Reviewed-by: Xiaoqiao He <[email protected]> Reviewed-by: Steve Loughran <[email protected]> Reviewed-by: Chris Nauroth <[email protected]> Reviewed-by: Cheng Pan <[email protected]> Reviewed-by: Istvan Toth <[email protected]> Signed-off-by: Shilun Fan <[email protected]>

Closes apache#7900 Signed-off-by: Shilun Fan <[email protected]> Signed-off-by: Chris Nauroth <[email protected]>

…ckylinux_8 (apache#7917) HADOOP-19661. (Follow-up) Rename Dockerfile_centos_8 to Dockerfile_rockylinux_8. Signed-off-by: Shilun Fan <[email protected]>

…#7898) Contributed by Cheng Pan. * HADOOP-19659. Upgrade Debian 10 to 11 in build env Dockerfile Signed-off-by: Shilun Fan <[email protected]>

…er image (apache#7931) Contributed by Cheng Pan. * HADOOP-19661. (Follow-up) Install missing deps for Rocky Linux 8 docker image Reviewed-by: Shilun Fan <[email protected]> Signed-off-by: Shilun Fan <[email protected]>

… image (apache#7932) Contributed by Cheng Pan. * HADOOP-19659. (Follow-up) Install missing procps for Debian 11 docker image. Signed-off-by: Shilun Fan <[email protected]>

hadoop-yetus · 2025-11-27T00:53:36Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8105/1/console in case of problems.

hadoop-yetus · 2025-11-27T18:02:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	14m 24s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+0 🆗	jsonlint	0m 0s		jsonlint was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ branch-3.4 Compile Tests _
+0 🆗	mvndep	2m 45s		Maven dependency ordering for branch
+1 💚	mvninstall	35m 12s		branch-3.4 passed
+1 💚	compile	16m 59s		branch-3.4 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	15m 32s		branch-3.4 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	mvnsite	21m 22s		branch-3.4 passed
+1 💚	javadoc	9m 14s		branch-3.4 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	7m 35s		branch-3.4 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	48m 14s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for patch
+1 💚	mvninstall	29m 5s		the patch passed
+1 💚	compile	16m 26s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	16m 26s		the patch passed
+1 💚	compile	15m 14s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	15m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	hadolint	0m 7s		No new issues.
+1 💚	mvnsite	18m 2s		the patch passed
-1 ❌	shellcheck	0m 2s	/results-shellcheck.txt	The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
+1 💚	javadoc	9m 2s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	7m 36s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	shadedclient	50m 11s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	740m 57s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 50s		The patch does not generate ASF License warnings.
		1031m 32s

Reason	Tests
Failed junit tests	hadoop.mapred.gridmix.TestGridmixSubmission
	hadoop.mapred.gridmix.TestLoadJob
	hadoop.fs.s3a.TestS3AUnbuffer
	hadoop.fs.s3a.impl.streams.TestStreamFactories

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8105/1/artifact/out/Dockerfile
GITHUB PR	#8105
Optional Tests	dupname asflicense codespell detsecrets hadolint shellcheck shelldocs mvnsite unit jsonlint compile javac javadoc mvninstall shadedclient xmllint
uname	Linux aa7bf47c230c 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / `820bc02`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8105/1/testReport/
Max. process+thread count	3710 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8105/1/console
versions	git=2.25.1 maven=3.6.3 hadolint=1.11.1-0-g0e692dd shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2025-11-29T01:55:49Z

@steveloughran Could you please help review this PR? Thank you very much!

ahmarsuhail · 2025-12-01T14:50:32Z

there are some unit test failures here which are related to my last change, and that i had missed before merging. This PR fixes those: #8109

steveloughran

+1

slfan1989 · 2025-12-03T20:57:20Z

@steveloughran @ahmarsuhail Thank you very much for helping to review the code. I will directly merge this PR into the branch-3.4 branch.

slfan1989 · 2025-12-03T21:05:36Z

@steveloughran Apologies, I used the web git merge directly and noticed that the commit history got merged as well. I need to force refresh the branch-3.4 and then push the changes locally.

I have submitted a request to temporarily enable force push (https://issues.apache.org/jira/projects/INFRA/issues/INFRA-27460) in order to restore the commit to #8115.

slfan1989 · 2025-12-03T21:16:45Z

there are some unit test failures here which are related to my last change, and that i had missed before merging. This PR fixes those: #8109

@ahmarsuhail Thank you for the clarification. I've seen your submission for #8117 and have already given it a +1.

stoty and others added 9 commits November 27, 2025 08:39

HADOOP-19495. Add JDK 24 to Ubuntu 20.04 docker development images.

8ef2aef

HADOOP-19661. Migrate CentOS 8 to Rocky Linux 8 in build env Dockerfile

e0cc64c

Closes apache#7900 Signed-off-by: Shilun Fan <[email protected]> Signed-off-by: Chris Nauroth <[email protected]>

HADOOP-19661. (Follow-up) Rename Dockerfile_centos_8 to Dockerfile_ro…

bd71ba2

…ckylinux_8 (apache#7917) HADOOP-19661. (Follow-up) Rename Dockerfile_centos_8 to Dockerfile_rockylinux_8. Signed-off-by: Shilun Fan <[email protected]>

HADOOP-19659. Upgrade Debian 10 to 11 in build env Dockerfile (apache…

c6fd02b

…#7898) Contributed by Cheng Pan. * HADOOP-19659. Upgrade Debian 10 to 11 in build env Dockerfile Signed-off-by: Shilun Fan <[email protected]>

HADOOP-19659. (Follow-up) Install missing procps for Debian 11 docker…

820bc02

… image (apache#7932) Contributed by Cheng Pan. * HADOOP-19659. (Follow-up) Install missing procps for Debian 11 docker image. Signed-off-by: Shilun Fan <[email protected]>

slfan1989 marked this pull request as draft November 27, 2025 00:50

slfan1989 marked this pull request as ready for review November 29, 2025 01:21

slfan1989 requested a review from steveloughran November 29, 2025 01:32

slfan1989 mentioned this pull request Nov 29, 2025

HADOOP-19605. (3.4) Upgrade Protobuf 3.25.5 for docker images (#7780) #8078

Open

steveloughran mentioned this pull request Nov 30, 2025

HADOOP-19559. Backport of Updates S3A default stream to AAL, IoStats. #8099

Merged

4 tasks

steveloughran approved these changes Dec 1, 2025

View reviewed changes

slfan1989 merged commit adb0b00 into apache:branch-3.4 Dec 3, 2025
0 of 4 checks passed

slfan1989 mentioned this pull request Dec 3, 2025

Revert "Cherry-pick necessary mirror PRs from trunk branch to Branch-3.4 branch" #8119

Merged

slfan1989 mentioned this pull request Dec 3, 2025

HADOOP-19559. [Backport] Fixes failing unit tests. #8117

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Cherry-pick necessary mirror PRs from trunk branch to Branch-3.4 branch #8105

Cherry-pick necessary mirror PRs from trunk branch to Branch-3.4 branch #8105

Uh oh!

slfan1989 commented Nov 27, 2025 •

edited

Loading

Uh oh!

hadoop-yetus commented Nov 27, 2025

Uh oh!

hadoop-yetus commented Nov 27, 2025

Uh oh!

slfan1989 commented Nov 29, 2025

Uh oh!

ahmarsuhail commented Dec 1, 2025

Uh oh!

steveloughran left a comment

Uh oh!

slfan1989 commented Dec 3, 2025

Uh oh!

Uh oh!

slfan1989 commented Dec 3, 2025 •

edited

Loading

Uh oh!

slfan1989 commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Cherry-pick necessary mirror PRs from trunk branch to Branch-3.4 branch #8105

Cherry-pick necessary mirror PRs from trunk branch to Branch-3.4 branch #8105

Uh oh!

Conversation

slfan1989 commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Nov 27, 2025

Uh oh!

hadoop-yetus commented Nov 27, 2025

Uh oh!

slfan1989 commented Nov 29, 2025

Uh oh!

ahmarsuhail commented Dec 1, 2025

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

slfan1989 commented Dec 3, 2025

Uh oh!

Uh oh!

slfan1989 commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slfan1989 commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

slfan1989 commented Nov 27, 2025 •

edited

Loading

slfan1989 commented Dec 3, 2025 •

edited

Loading