Skip to content

Upgrade Hadoop to 3.3.4 #3578

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zhangbutao wants to merge 1 commit into from
Closed

Upgrade Hadoop to 3.3.4 #3578

zhangbutao wants to merge 1 commit into from

Conversation

@zhangbutao
Copy link
Contributor

@zhangbutao zhangbutao commented Sep 7, 2022

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

@zhangbutao zhangbutao marked this pull request as draft September 7, 2022 09:20
ayushtkn
ayushtkn reviewed Sep 7, 2022
View reviewed changes
Copy link
Member

@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the build will pass, I tried till 3.3.3 when trying to upgrade, hive doesn't pose any issues there, but Tez doesn't work with Hive in actual cluster that is why we settled to 3.3.1. You need to upgrade Tez as well to 3.3.4 and then upgrade Hive to that Tez version.
Can check more details on the previous 3.3.1 upgrade PR...

@sonarqubecloud
Copy link

sonarqubecloud bot commented Sep 7, 2022

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 1 Code Smell

No Coverage information No Coverage information
No Duplication information No Duplication information

@zhangbutao
Copy link
Contributor Author

zhangbutao commented Sep 7, 2022

the build will pass, I tried till 3.3.3 when trying to upgrade, hive doesn't pose any issues there, but Tez doesn't work with Hive in actual cluster that is why we settled to 3.3.1. You need to upgrade Tez as well to 3.3.4 and then upgrade Hive to that Tez version. Can check more details on the previous 3.3.1 upgrade PR...

@ayushtkn Thanks for the info. Maybe i should test this PR with actual cluster and i will check this. I also foud the Tez ticket about upgrading hadoop to 3.3.3 apache/tez#213. If I understand you correctly, we should deal with both tickets at the same time. As hadoop-3.3.4 is a latest stable version, updating hadoop from 3.3.1 to 3.3.4 is a good choice.

@ayushtkn
Copy link
Member

ayushtkn commented Sep 7, 2022

Yes. you got me right, we should do both the tickets

@zhangbutao
Copy link
Contributor Author

zhangbutao commented Sep 8, 2022 •
edited
Loading

hs2_start_warn

Tested this Pr with Hadoop3.3.4 & Tez0.10.2, and HS2 started failed with above WARN. This is because Tez0.10.2 added related hadoop3.3.1 jars(e.g. hadoop-hdfs-client-3.3.1.jar) to binary package and this caused some conflict in hive's classpath.

If i change the hadoop version to 3.3.4 in Tez0.10.2, hs2 will run successfully. :)

@zhangbutao zhangbutao mentioned this pull request Sep 8, 2022
Closed
@github-actions
Copy link

github-actions bot commented Nov 8, 2022

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the [email protected] list if the patch is in need of reviews.

@github-actions github-actions bot added the stale label Nov 8, 2022
@github-actions github-actions bot closed this Nov 16, 2022
@ayushtkn ayushtkn mentioned this pull request Feb 8, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ayushtkn ayushtkn ayushtkn left review comments

Assignees

No one assigned

Labels

stale tests passed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zhangbutao @ayushtkn @kgyrtkirk