HDFS-16336. De-flake TestRollingUpgrade#testRollback #3686

virajjasani · 2021-11-19T18:12:32Z

Description of PR

Test TestRollingUpgrade#testRollback is flaky. It keeps failing intermittently.

How was this patch tested?

The rolling upgrade related metrics are sometimes not null because of other tests, however having null metrics mean is prerequisite for this test, hence we need to introduce wait.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?

hadoop-yetus · 2021-11-19T23:47:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 38s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 4s		trunk passed
+1 💚	compile	1m 25s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 22s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 0s		trunk passed
+1 💚	mvnsite	1m 28s		trunk passed
+1 💚	javadoc	1m 1s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 36s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 9s		trunk passed
+1 💚	shadedclient	22m 25s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 17s		the patch passed
+1 💚	compile	1m 22s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 22s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 53s		the patch passed
+1 💚	mvnsite	1m 23s		the patch passed
+1 💚	javadoc	0m 54s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 27s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 22s		the patch passed
+1 💚	shadedclient	22m 3s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	234m 47s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 44s		The patch does not generate ASF License warnings.
		333m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3686/1/artifact/out/Dockerfile
GITHUB PR	#3686
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux e809b4e5f2d4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `dc2ac90`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3686/1/testReport/
Max. process+thread count	3063 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3686/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-11-20T00:33:38Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	31m 57s		trunk passed
+1 💚	compile	1m 28s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 21s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 1s		trunk passed
+1 💚	mvnsite	1m 28s		trunk passed
+1 💚	javadoc	1m 1s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 30s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 14s		trunk passed
+1 💚	shadedclient	22m 18s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 17s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 17s		the patch passed
+1 💚	compile	1m 13s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 53s		the patch passed
+1 💚	mvnsite	1m 17s		the patch passed
+1 💚	javadoc	0m 51s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 25s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 14s		the patch passed
+1 💚	shadedclient	21m 54s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	225m 17s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		323m 12s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3686/4/artifact/out/Dockerfile
GITHUB PR	#3686
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 7b2b09ade6cb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `dc2ac90`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3686/4/testReport/
Max. process+thread count	3829 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3686/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-11-20T00:53:45Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 53s		trunk passed
+1 💚	compile	1m 30s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 23s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 58s		trunk passed
+1 💚	mvnsite	1m 27s		trunk passed
+1 💚	javadoc	1m 5s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 36s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 33s		trunk passed
+1 💚	shadedclient	26m 29s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 25s		the patch passed
+1 💚	compile	1m 25s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 25s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 55s		the patch passed
+1 💚	mvnsite	1m 25s		the patch passed
+1 💚	javadoc	0m 54s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 30s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 36s		the patch passed
+1 💚	shadedclient	25m 9s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	236m 21s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 48s		The patch does not generate ASF License warnings.
		343m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3686/2/artifact/out/Dockerfile
GITHUB PR	#3686
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 2eae79065e9c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `dc2ac90`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3686/2/testReport/
Max. process+thread count	3500 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3686/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-11-20T00:57:49Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 5s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 33s		trunk passed
+1 💚	compile	1m 31s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 19s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 59s		trunk passed
+1 💚	mvnsite	1m 29s		trunk passed
+1 💚	javadoc	1m 4s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 26s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 26s		trunk passed
+1 💚	shadedclient	24m 11s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 19s		the patch passed
+1 💚	compile	1m 26s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 26s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 52s		the patch passed
+1 💚	mvnsite	1m 19s		the patch passed
+1 💚	javadoc	0m 55s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 24s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 32s		the patch passed
+1 💚	shadedclient	23m 9s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	243m 15s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 47s		The patch does not generate ASF License warnings.
		347m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3686/3/artifact/out/Dockerfile
GITHUB PR	#3686
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux bfdeaa0e164b 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `dc2ac90`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3686/3/testReport/
Max. process+thread count	2943 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3686/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

virajjasani · 2021-11-20T08:35:19Z

@aajisaka @tasanuma Could you please take a look?
Thanks

aajisaka · 2021-11-21T22:26:57Z

@virajjasani
I think this failure is due to the remaining metrics created by another test. The fix is to wait for the metrics to be removed. Is that correct?

virajjasani · 2021-11-22T04:52:05Z

@virajjasani I think this failure is due to the remaining metrics created by another test. The fix is to wait for the metrics to be removed. Is that correct?

That's correct @aajisaka. After some considerations, thought of keeping bit higher wait time of 8s.

virajjasani · 2021-11-22T07:37:31Z

Updated the description of the PR.

virajjasani · 2021-11-23T08:31:43Z

Does this introduction of wait look good @aajisaka @tasanuma?

virajjasani · 2021-11-24T07:10:03Z

FYI @ferhui if you would also like to take a look. Thanks

aajisaka

+1. Thank you @virajjasani

Signed-off-by: Akira Ajisaka <[email protected]> (cherry picked from commit e8566b3)

…. Contributed by Viraj Jasani..

#3686)" This reverts commit 3b79e0f.

#3686). Contributed by Viraj Jasani.." This reverts commit 4231067.

GuoPhilipse · 2021-12-09T02:24:35Z

@aajisaka @virajjasani the fix may fit for all checkMxBeanIsNull in TestRollingUpgrade ，it appears in other callers.
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3724/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt, and TestRollingUpgrade .testRollBackImage seems need high timeout, Would you please take a look ?

Signed-off-by: Akira Ajisaka <[email protected]> (cherry picked from commit e8566b3)

#3686)" This reverts commit 3b79e0f.

Signed-off-by: Akira Ajisaka <[email protected]>

HDFS-16336. De-flake TestRollingUpgrade#testRollback

dc2ac90

aajisaka self-requested a review November 21, 2021 22:27

aajisaka approved these changes Nov 24, 2021

View reviewed changes

aajisaka merged commit e8566b3 into apache:trunk Nov 24, 2021

aajisaka pushed a commit that referenced this pull request Nov 24, 2021

HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686)

450efae

Signed-off-by: Akira Ajisaka <[email protected]> (cherry picked from commit e8566b3)

aajisaka pushed a commit that referenced this pull request Nov 24, 2021

HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686)

af452d2

Signed-off-by: Akira Ajisaka <[email protected]> (cherry picked from commit e8566b3)

asfgit pushed a commit that referenced this pull request Nov 24, 2021

HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686)

3b79e0f

asfgit pushed a commit that referenced this pull request Nov 24, 2021

HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686)…

4231067

…. Contributed by Viraj Jasani..

aajisaka added a commit that referenced this pull request Nov 25, 2021

Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (

4a032bc

#3686)" This reverts commit 3b79e0f.

aajisaka added a commit that referenced this pull request Nov 25, 2021

Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (

6a28496

#3686). Contributed by Viraj Jasani.." This reverts commit 4231067.

sunchao pushed a commit that referenced this pull request Jan 4, 2022

HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686)

3ae8d4c

Signed-off-by: Akira Ajisaka <[email protected]> (cherry picked from commit e8566b3)

sunchao pushed a commit that referenced this pull request Jan 4, 2022

HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686)

328d3b8

sunchao pushed a commit that referenced this pull request Jan 4, 2022

Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (

5c18979

#3686)" This reverts commit 3b79e0f.

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HDFS-16336. De-flake TestRollingUpgrade#testRollback (apache#3686)

6561c86

Signed-off-by: Akira Ajisaka <[email protected]>

HDFS-16336. De-flake TestRollingUpgrade#testRollback #3686

HDFS-16336. De-flake TestRollingUpgrade#testRollback #3686

Uh oh!

Conversation

virajjasani commented Nov 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Nov 19, 2021

Uh oh!

hadoop-yetus commented Nov 20, 2021

Uh oh!

hadoop-yetus commented Nov 20, 2021

Uh oh!

hadoop-yetus commented Nov 20, 2021

Uh oh!

virajjasani commented Nov 20, 2021

Uh oh!

aajisaka commented Nov 21, 2021

Uh oh!

virajjasani commented Nov 22, 2021

Uh oh!

virajjasani commented Nov 22, 2021

Uh oh!

virajjasani commented Nov 23, 2021

Uh oh!

virajjasani commented Nov 24, 2021

Uh oh!

aajisaka left a comment

Choose a reason for hiding this comment

Uh oh!

GuoPhilipse commented Dec 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

virajjasani commented Nov 19, 2021 •

edited

Loading