Skip to content

TEZ-4420 : Upgrade Hadoop to 3.3.4 #272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
abstractdog merged 7 commits into from
Feb 24, 2023
Merged

TEZ-4420 : Upgrade Hadoop to 3.3.4 #272

abstractdog merged 7 commits into from
Feb 24, 2023

Conversation

@amanraj2520
Copy link
Contributor

@amanraj2520 amanraj2520 commented Feb 21, 2023

Upgraded Hadoop to 3.3.4 and some dependency imports for tests.
Have successfully tested in my local running all the test cases.

@amanraj2520
Copy link
Contributor Author

amanraj2520 commented Feb 21, 2023 •
edited
Loading

@abstractdog I have tested all tests in my local. The only test failure was TestLocalMode previously which I see you have fixed recently in master. Can you please review this.

@tez-yetus

This comment was marked as outdated.

Sign in to view
abstractdog
abstractdog reviewed Feb 22, 2023
View reviewed changes
@amanraj2520
Copy link
Contributor Author

amanraj2520 commented Feb 22, 2023

Just FYI why I made this change in tez-tests
This issue comes when PowerMock library conflicts with Mockito as described here https://stackoverflow.com/questions/71973762/java-lang-nosuchmethoderror-org-mockito-answers-getlorg-mockito-stubbing-answ

@amanraj2520
Copy link
Contributor Author

amanraj2520 commented Feb 22, 2023

@abstractdog Can you please review this now

steveloughran
steveloughran reviewed Feb 22, 2023
View reviewed changes
@tez-yetus

This comment was marked as outdated.

Sign in to view
@tez-yetus

This comment was marked as outdated.

Sign in to view
@tez-yetus

This comment was marked as outdated.

Sign in to view
@tez-yetus

This comment was marked as outdated.

Sign in to view
@amanraj2520
Copy link
Contributor Author

amanraj2520 commented Feb 23, 2023

@abstractdog Can you please review this.

@abstractdog
Copy link
Contributor

abstractdog commented Feb 23, 2023

Just FYI why I made this change in tez-tests This issue comes when PowerMock library conflicts with Mockito as described here https://stackoverflow.com/questions/71973762/java-lang-nosuchmethoderror-org-mockito-answers-getlorg-mockito-stubbing-answ

I cannot see powermock used by tez, so assuming that the powermock used by hadoop conflicts with the mockito used by tez? is that right? if so, theoretically, this conflict can be resolved in the future in case of a mockito version change in tez or rather a powermock version change in hadoop (as mockito is quite new one in tez since TEZ-4383)
please clarify this scenario here by an xml comment: pom.xmls are full of dependencies without any context and hence extremely easy to break, I would like to see the explanation here (any comment that clarifies my points above is welcome :) )

@amanraj2520
Copy link
Contributor Author

amanraj2520 commented Feb 23, 2023

Hi @abstractdog Sorry my bad. I checked the Hadoop 3.3.4 version in my local repo. It had the use of powermock libraries which I was using to debug and mock some functionalities for branch-3 (Hive) upgrade with Hadoop 3.3.4. These were unnecessary changes. I refreshed my local and deleted all the jars. Now it downloaded oss Hadoop 3.3.4 and built correctly. We only need to include mockito-core for the tez-tests that's it. I have pushed my commit. Kindly check it once.

@amanraj2520
Copy link
Contributor Author

amanraj2520 commented Feb 23, 2023

@abstractdog Added a snapshot of a sample successful test which was failing earlier :
image

@amanraj2520
Copy link
Contributor Author

amanraj2520 commented Feb 23, 2023

Just for everyone's knowledge, if we do not include mockito-core, the tests will fail with the following errors:
`[INFO] Running org.apache.tez.mapreduce.TestMRRJobsDAGApi
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.92 s <<< FAILURE! - in org.apache.tez.mapreduce.TestMRRJobsDAGApi
[ERROR] org.apache.tez.mapreduce.TestMRRJobsDAGApi Time elapsed: 2.914 s <<< ERROR!
java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
at org.apache.tez.mapreduce.TestMRRJobsDAGApi.setup(TestMRRJobsDAGApi.java:156)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
at org.apache.tez.mapreduce.TestMRRJobsDAGApi.setup(TestMRRJobsDAGApi.java:156)

[INFO] Running org.apache.tez.mapreduce.TestMRRJobs
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.416 s <<< FAILURE! - in org.apache.tez.mapreduce.TestMRRJobs
[ERROR] org.apache.tez.mapreduce.TestMRRJobs Time elapsed: 3.412 s <<< ERROR!
java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
at org.apache.tez.mapreduce.TestMRRJobs.setup(TestMRRJobs.java:71)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
at org.apache.tez.mapreduce.TestMRRJobs.setup(TestMRRJobs.java:71)

[INFO] Running org.apache.tez.test.TestAM
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.019 s <<< FAILURE! - in org.apache.tez.test.TestAM
[ERROR] org.apache.tez.test.TestAM Time elapsed: 3.015 s <<< ERROR!
java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
at org.apache.tez.test.TestAM.setup(TestAM.java:66)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
at org.apache.tez.test.TestAM.setup(TestAM.java:66)

[INFO] Running org.apache.tez.test.TestAMRecovery
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.463 s <<< FAILURE! - in org.apache.tez.test.TestAMRecovery
[ERROR] org.apache.tez.test.TestAMRecovery Time elapsed: 3.46 s <<< ERROR!
java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
at org.apache.tez.test.TestAMRecovery.beforeClass(TestAMRecovery.java:109)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
at org.apache.tez.test.TestAMRecovery.beforeClass(TestAMRecovery.java:109)`

@tez-yetus
Copy link

tez-yetus commented Feb 23, 2023

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 37s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+0 🆗 mvndep 6m 36s Maven dependency ordering for branch
+1 💚 mvninstall 10m 9s master passed
+1 💚 compile 2m 47s master passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu222.04
+1 💚 compile 2m 32s master passed with JDK Private Build-1.8.0_352-8u352-ga-1~22.04-b08
+1 💚 javadoc 2m 35s master passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu222.04
+1 💚 javadoc 2m 6s master passed with JDK Private Build-1.8.0_352-8u352-ga-1~22.04-b08
_ Patch Compile Tests _
+0 🆗 mvndep 1m 15s Maven dependency ordering for patch
+1 💚 mvninstall 4m 31s the patch passed
+1 💚 compile 2m 47s the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu222.04
+1 💚 javac 2m 47s the patch passed
+1 💚 compile 2m 31s the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~22.04-b08
+1 💚 javac 2m 31s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
+1 💚 javadoc 2m 24s the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu222.04
+1 💚 javadoc 2m 2s the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~22.04-b08
_ Other Tests _
+1 💚 unit 39m 47s tez-tests in the patch passed.
+1 💚 unit 67m 27s root in the patch passed.
+1 💚 asflicense 0m 50s The patch does not generate ASF License warnings.
152m 2s
Subsystem Report/Notes
Docker ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-272/6/artifact/out/Dockerfile
GITHUB PR #272
Optional Tests dupname asflicense javac javadoc unit xml compile
uname Linux 5d791a3b716f 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/tez.sh
git revision master / be99489
Default Java Private Build-1.8.0_352-8u352-ga-1~22.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu222.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~22.04-b08
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-272/6/testReport/
Max. process+thread count 2090 (vs. ulimit of 5500)
modules C: tez-tests . U: .
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-272/6/console
versions git=2.34.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@abstractdog abstractdog self-requested a review February 24, 2023 07:37
abstractdog
abstractdog approved these changes Feb 24, 2023
View reviewed changes
Copy link
Contributor

@abstractdog abstractdog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks much cleaner, thanks @amanraj2520
+1

@abstractdog abstractdog merged commit db9eb1e into apache:master Feb 24, 2023
@abstractdog abstractdog mentioned this pull request Feb 24, 2023
Closed
@steveloughran
Copy link

steveloughran commented Feb 24, 2023

nice. well done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@abstractdog abstractdog abstractdog approved these changes

+1 more reviewer

@steveloughran steveloughran steveloughran left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@amanraj2520 @tez-yetus @abstractdog @steveloughran