HDDS-8911. Update Hadoop to 3.3.6 #4985

jojochuang · 2023-06-26T21:42:45Z

What changes were proposed in this pull request?

Update build hadoop.version dependency to 3.3.6
Update tests to use Apache Hadoop 3.3.6 docker image.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-8911

How was this patch tested?

Existing tests. https://github.com/jojochuang/ozone/actions/runs/5382664472

Change-Id: I934094457b8d98ace07cf4c3f43abdd59d49e929 Update Change-Id: Icc5e52f3c0f6a42c65b0b1817e4d26555320a5b3 Fix Change-Id: I5a8c10896172ed9ac2346824aa2d9141c31520dc Revert "Fix" This reverts commit 4a1ceeb4e3f6290feccb15d378f5b9ab8ad507d3. Revert "Update" This reverts commit 83dce5c2cd6baad6b92f981d4758cb0e24932667. Update Change-Id: I37ad5774d0b88f436693348b0f785fa0f293ac65 Old Hadoop versions, use flokkr/hadoop image. Hadoop 3.3 line, use apache/hadoop image. Change-Id: Id70d316c3672efdc7f7452e19d9ab887b62a3ec7

Change-Id: Iad17f2776c003ea3d049aa8e4ce412d23e84bbf0

Change-Id: Ic1665c2e99304602d0f2964b31858f585a6a2b62

Change-Id: I6f0429e83c8b0dddcb59fd45f7e18fc691eaf3ea

Change-Id: I7689a84e01ff69b6978b79245fd8a58ecf66e34a (cherry picked from commit 4835638)

Change-Id: Ieccbc62f0c3b454a6654419582ba87600678537a

Change-Id: I44b49d761bddc195cf75889722ef70aab4837e6f

Change-Id: Ie71a16c738fa2d7118d4a8966fccb736d3087424

adoroszlai

Thanks @jojochuang for working on this (and Hadoop release itself).

adoroszlai · 2023-06-27T05:16:57Z

hadoop-ozone/dist/src/main/compose/ozonesecure-ha/.env


 HDDS_VERSION=${hdds.version}
-HADOOP_VERSION=3
+HADOOP_VERSION=3.3.6


Now that the Docker image has the same version as Hadoop itself, we can change this to avoid the need for future updates:

Suggested change

HADOOP_VERSION=3.3.6

HADOOP_VERSION=${hadoop.version}

adoroszlai · 2023-06-27T05:17:04Z

hadoop-ozone/dist/src/main/compose/ozonesecure-mr/.env

 HDDS_VERSION=${hdds.version}
 HADOOP_IMAGE=apache/hadoop
-HADOOP_VERSION=3.3.5
+HADOOP_VERSION=3.3.6


Suggested change

HADOOP_VERSION=3.3.6

HADOOP_VERSION=${hadoop.version}

adoroszlai · 2023-06-27T05:17:09Z

hadoop-ozone/dist/src/main/compose/ozonesecure/.env


 HDDS_VERSION=${hdds.version}
-HADOOP_VERSION=3
+HADOOP_VERSION=3.3.6


Suggested change

HADOOP_VERSION=3.3.6

HADOOP_VERSION=${hadoop.version}

adoroszlai · 2023-06-27T05:38:51Z

hadoop-ozone/dist/src/main/license/jar-report.txt

@@ -1,11 +1,10 @@
-share/ozone/lib/accessors-smart.jar
+share/ozone/lib/FastInfoset.jar


Nit: Three kinds of changes to jar-report.txt are mixed in this PR.

real dependency changes (see below)

fixing strictly out of alphabetical order items (e.g. hdds-rocks-native)

reordering due to different sort behavior (lowercase/uppercase, chars other than alphabetical) across implementations

I would prefer to stick only to the first kind of change in this PR:

@@ -1,11 +1,9 @@ -share/ozone/lib/accessors-smart.jar share/ozone/lib/activation.jar share/ozone/lib/animal-sniffer-annotations.jar share/ozone/lib/annotations.jar share/ozone/lib/annotations.jar share/ozone/lib/aopalliance.jar share/ozone/lib/aopalliance-repackaged.jar -share/ozone/lib/asm.jar share/ozone/lib/aspectjrt.jar share/ozone/lib/aspectjweaver.jar share/ozone/lib/awaitility.jar @@ -156,7 +154,6 @@ share/ozone/lib/jooq-meta.jar share/ozone/lib/jsch.jar share/ozone/lib/json-simple.jar -share/ozone/lib/json-smart.jar share/ozone/lib/jsp-api.jar share/ozone/lib/jsr305.jar share/ozone/lib/jsr311-api.jar

Rationale: The changes are in transitive dependencies, visible only in this file (no POM change). Reordering lines hides the real difference.

Unfortunately dependency check instructs to update the file mention using update-jar-report.sh:

Please update the hadoop-ozone/dist/src/main/license/bin/LICENSE.txt file with the modification AND execute hadoop-ozone/dist/src/main/license/update-jar-report.sh when you are ready (after a full build)

which re-generates from scratch, causing such mixed changes. To avoid those, one can simply manually remove the unnecessary entries.

Yeah... I wasn't sure how the jar report is generated. Let's see if it is happy with this.

It sorts both old (reference) and new lists exactly to avoid false positives due to difference in sort:

ozone/hadoop-ozone/dev-support/checks/dependency.sh

Lines 31 to 32 in 4559ef4

#implementation of sort cli is not exactly the same everywhere. It's better to sort with the same command locally

(diff -uw <(sort hadoop-ozone/dist/src/main/license/jar-report.txt) <(sort hadoop-ozone/dist/src/main/license/current.txt) || true ) > "$REPORT_FILE"

So any order is fine as far as dependency check is concerned.

adoroszlai · 2023-06-27T05:58:47Z

hadoop-ozone/dist/src/main/smoketest/.env

 # limitations under the License.

-HADOOP_VERSION=3
+HADOOP_VERSION=3.3.6


Suggested change

HADOOP_VERSION=3.3.6

HADOOP_VERSION=${hadoop.version}

…nvironment variables. Change-Id: Iab6db08da7e99a9e2799b916bbd2fb3a51010aa6

adoroszlai

Thanks @jojochuang for updating the patch.

* master: HDDS-8555. [Snapshot] When snapshot feature is disabled, block OM startup if there are still snapshots in the system (apache#4994) HDDS-8782. Improve Volume Scanner Health checks. (apache#4867) HDDS-8447. Datanodes should not process container deletes for failed volumes. (apache#4901) HDDS-5869. Added support for stream on S3Gateway write path (apache#4970) HDDS-8859. [Snapshot] Return failure message to client for a failed snapshot diff jobs (apache#4993) HDDS-8939. [Snapshot] isBlockLocationSame check should be skipped if object is not OmKeyInfo. (apache#4991) HDDS-8923. Expose XceiverClient cache stats as metrics (apache#4979) HDDS-8913. ContainerManagerImpl: reduce processing while locked (apache#4967) HDDS-8935. [Snapshot] Fallback to full diff if getDetlaFiles from compaction DAG fails (apache#4986) HDDS-8911. Update Hadoop to 3.3.6 (apache#4985) HDDS-8931. Allow EC PipelineChoosingPolicy to be defined separately from Ratis (apache#4983) HDDS-8895. Support dynamic change of ozone.readonly.administrators in SCM (apache#4977) HDDS-6814. Make OM service ID optional for `ozone s3` commands if only one is defined in config (apache#4953) HDDS-8925. BaseFreonGenerator may not complete if last attempts fail (apache#4975) HDDS-7100. Container scanner incorrectly marks containers unhealthy when DN is shutdown (apache#4951) HDDS-8919. Allow EC pipelines to be created and then added to PipelineManager in two steps (apache#4968) HDDS-8901. Enable mTLS for InterSCMGrpcProtocol. (apache#4964) Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Container.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/ContainerTestUtils.java

jojochuang added 8 commits June 26, 2023 12:49

Use apache/hadoop image if Hadoop is 3.3.x; otherwise use flokkr/hadoop.

1e8469e

Change-Id: Iad17f2776c003ea3d049aa8e4ce412d23e84bbf0

Hack

96d8f3d

Change-Id: Ic1665c2e99304602d0f2964b31858f585a6a2b62

Remove the hack. apache/hadoop has a 3.3.5 tagged image

4c60699

Change-Id: I6f0429e83c8b0dddcb59fd45f7e18fc691eaf3ea

Update to Hadoop 3.3.6 RC0

31caf5f

Change-Id: I7689a84e01ff69b6978b79245fd8a58ecf66e34a (cherry picked from commit 4835638)

Remove staging repository.

87d5022

Change-Id: Ieccbc62f0c3b454a6654419582ba87600678537a

Use apache/hadoop:3.3.6 image

21b09c1

Change-Id: I44b49d761bddc195cf75889722ef70aab4837e6f

Update more HADOOP_VERSIOn

82a536f

Change-Id: Ie71a16c738fa2d7118d4a8966fccb736d3087424

jojochuang requested review from adoroszlai and smengcl June 26, 2023 21:42

adoroszlai reviewed Jun 27, 2023

View reviewed changes

Manuall recorded jar-report and harmonized HADOOP_VERSION in docker e…

8ae7bcc

…nvironment variables. Change-Id: Iab6db08da7e99a9e2799b916bbd2fb3a51010aa6

adoroszlai approved these changes Jun 27, 2023

View reviewed changes

adoroszlai merged commit 6cfd7dc into apache:master Jun 27, 2023

vtutrinov pushed a commit to Cyrill/ozone that referenced this pull request Jul 3, 2023

HDDS-8911. Update Hadoop to 3.3.6 (apache#4985)

00ff8bb

vtutrinov pushed a commit to Cyrill/ozone that referenced this pull request Jul 3, 2023

HDDS-8911. Update Hadoop to 3.3.6 (apache#4985)

ec1a311

adoroszlai mentioned this pull request Jul 15, 2023

HDDS-8961. Write a standalone tool to create a visual graph from compaction log files. #5061

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-8911. Update Hadoop to 3.3.6 #4985

HDDS-8911. Update Hadoop to 3.3.6 #4985

Uh oh!

jojochuang commented Jun 26, 2023

Uh oh!

adoroszlai left a comment

Uh oh!

adoroszlai Jun 27, 2023

Uh oh!

adoroszlai Jun 27, 2023

Uh oh!

adoroszlai Jun 27, 2023

Uh oh!

adoroszlai Jun 27, 2023

Uh oh!

jojochuang Jun 27, 2023

Uh oh!

adoroszlai Jun 27, 2023 •

edited

Loading

Uh oh!

adoroszlai Jun 27, 2023

Uh oh!

adoroszlai left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -1,11 +1,10 @@
		share/ozone/lib/accessors-smart.jar
		share/ozone/lib/FastInfoset.jar

	#implementation of sort cli is not exactly the same everywhere. It's better to sort with the same command locally
	(diff -uw <(sort hadoop-ozone/dist/src/main/license/jar-report.txt) <(sort hadoop-ozone/dist/src/main/license/current.txt) \|\| true ) > "$REPORT_FILE"

HDDS-8911. Update Hadoop to 3.3.6 #4985

HDDS-8911. Update Hadoop to 3.3.6 #4985

Uh oh!

Conversation

jojochuang commented Jun 26, 2023

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

adoroszlai Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

adoroszlai Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

adoroszlai Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

jojochuang Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

adoroszlai Jun 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adoroszlai Jun 27, 2023

Choose a reason for hiding this comment

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

adoroszlai Jun 27, 2023 •

edited

Loading