HADOOP-18098. Basic verification for the release candidate vote #3944

virajjasani · 2022-01-29T06:59:04Z

Description of PR

We should provide script for the basic sanity of Hadoop release candidates. It should include:

Signature
Checksum
Rat check
Build from src
Build tarball from src

Although we can include unit test as well, but overall unit test run is going to be significantly higher, and precommit Jenkins builds provide better view of UT sanity.

How was this patch tested?

Locally

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?

virajjasani · 2022-01-29T07:16:55Z

For 3.3.2RC3, here is the sample command:

./dev-support/hadoop-vote.sh -s https://people.apache.org/~sunchao/hadoop-3.3.2-RC3/ -k D8F1A97BE51ECA98 -o /tmp/hadoop3.3.2RC3/

hadoop-yetus · 2022-01-29T08:25:14Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	13m 10s		Maven dependency ordering for branch
+1 💚	mvninstall	26m 26s		trunk passed
+1 💚	mvnsite	0m 0s		trunk passed
+1 💚	shadedclient	21m 32s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 0s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 0s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	19m 25s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		84m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3944/1/artifact/out/Dockerfile
GITHUB PR	#3944
Optional Tests	dupname asflicense mvnsite unit codespell shellcheck shelldocs
uname	Linux 85b8e4564dce 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e89a84f`
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3944/1/testReport/
Max. process+thread count	676 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3944/1/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

jojochuang · 2022-02-01T07:42:49Z

dev-support/hadoop-vote.sh

+function build_from_source() {
+    rm -f "${OUTPUT_PATH_PREFIX}"_build_from_source
+    # No unit test run.
+    mvn clean install "${MVN_PROPERTIES[@]}" -DskipTests 2>&1 | tee "${OUTPUT_PATH_PREFIX}"_build_from_source && BUILD_FROM_SOURCE_PASSED=1


it would be great to build with additional profiles:
-Pyarn-ui as well as -Pnative to build native code (ok, maybe optional)

Sounds good, added an option to add list of profiles to use. Hence, now the command can be:
./dev-support/hadoop-vote.sh -s https://people.apache.org/~sunchao/hadoop-3.3.2-RC2/ -k D8F1A97BE51ECA98 -o /tmp/hadoop3.3.2RC20/ -P native -P yarn-ui

hadoop-yetus · 2022-02-01T13:42:04Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 58s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 58s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 56s		trunk passed
+1 💚	mvnsite	0m 0s		trunk passed
+1 💚	shadedclient	19m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 0s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 0s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	19m 30s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 33s		The patch does not generate ASF License warnings.
		79m 3s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3944/2/artifact/out/Dockerfile
GITHUB PR	#3944
Optional Tests	dupname asflicense mvnsite unit codespell shellcheck shelldocs
uname	Linux f232d752fb5a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3e7228d`
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3944/2/testReport/
Max. process+thread count	683 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3944/2/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

virajjasani · 2022-02-04T07:06:05Z

Some sample results for the latest RCs:

After cleaning my local .m2 cache, I saw build failing due to the known issue as discussed on the VOTE thread.

./dev-support/hadoop-vote.sh -s https://people.apache.org/~sunchao/hadoop-3.3.2-RC2/ -k D8F1A97BE51ECA98 -o /tmp/hadoop3.3.2RC20/ -P native -P yarn-ui

        * Signature: ok
        * Checksum : ok
        * Rat check (1.8.0_301): ok
         - mvn clean apache-rat:check 
        * Built from source (1.8.0_301): failed
         - mvn clean install  -DskipTests -P native -P yarn-ui
        * Built tar from source (1.8.0_301): failed
         - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

./dev-support/hadoop-vote.sh -s https://people.apache.org/~sunchao/hadoop-3.3.2-RC3/ -k D8F1A97BE51ECA98 -o /tmp/hadoop3.3.2RC3/

        * Signature: ok
        * Checksum : ok
        * Rat check (1.8.0_301): ok
         - mvn clean apache-rat:check 
        * Built from source (1.8.0_301): failed
         - mvn clean install  -DskipTests
        * Built tar from source (1.8.0_301): failed
         - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

Once we have the new RC, I will use this script to provide the basic verification of the RC.

@jojochuang @aajisaka

jojochuang · 2022-02-04T10:13:47Z

I think this is good... Thanks Viraj. Do we have to wait for the next RC before merging this?

virajjasani · 2022-02-04T10:15:56Z

I believe we are good to merge this and we can directly test for successful result when new RC lands. Just in case some minor bug is encountered later, I will create addendum for sure.
Thanks

…he#3944)

HADOOP-18098. Basic verification of release candidates

e89a84f

virajjasani changed the title ~~HADOOP-18098. Basic verification of release candidates~~ HADOOP-18098. Basic verification for the release candidate vote Jan 29, 2022

jojochuang reviewed Feb 1, 2022

View reviewed changes

allow additional profiles to be used while building

3e7228d

virajjasani requested a review from jojochuang February 2, 2022 12:54

jojochuang approved these changes Feb 7, 2022

View reviewed changes

jojochuang merged commit b39b334 into apache:trunk Feb 7, 2022

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HADOOP-18098. Basic verification for the release candidate vote (apac…

3ffe712

…he#3944)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-18098. Basic verification for the release candidate vote #3944

HADOOP-18098. Basic verification for the release candidate vote #3944

Uh oh!

virajjasani commented Jan 29, 2022

Uh oh!

virajjasani commented Jan 29, 2022 •

edited

Loading

Uh oh!

hadoop-yetus commented Jan 29, 2022

Uh oh!

jojochuang Feb 1, 2022

Uh oh!

virajjasani Feb 1, 2022

Uh oh!

hadoop-yetus commented Feb 1, 2022

Uh oh!

virajjasani commented Feb 4, 2022 •

edited

Loading

Uh oh!

jojochuang commented Feb 4, 2022

Uh oh!

virajjasani commented Feb 4, 2022 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HADOOP-18098. Basic verification for the release candidate vote #3944

HADOOP-18098. Basic verification for the release candidate vote #3944

Uh oh!

Conversation

virajjasani commented Jan 29, 2022

Description of PR

How was this patch tested?

For code changes:

Uh oh!

virajjasani commented Jan 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented Jan 29, 2022

Uh oh!

jojochuang Feb 1, 2022

Choose a reason for hiding this comment

Uh oh!

virajjasani Feb 1, 2022

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Feb 1, 2022

Uh oh!

virajjasani commented Feb 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jojochuang commented Feb 4, 2022

Uh oh!

virajjasani commented Feb 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

virajjasani commented Jan 29, 2022 •

edited

Loading

virajjasani commented Feb 4, 2022 •

edited

Loading

virajjasani commented Feb 4, 2022 •

edited

Loading