Skip to content

[backport] Upgrade Hadoop 2 to Hadoop 3 #161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
KeerthanaSrikanth wants to merge 3 commits into from
Closed

[backport] Upgrade Hadoop 2 to Hadoop 3 #161

KeerthanaSrikanth wants to merge 3 commits into from

Conversation

@KeerthanaSrikanth
Copy link

@KeerthanaSrikanth KeerthanaSrikanth commented Nov 6, 2023

@tejaswini-imply @KeerthanaSrikanth
Update gha & travis checks (apache#13412)
ec46d7b 
* update static-checks GHA to run sequentially
remove static-checks from travis.yml
move docs, web-console, packaging checks from travis to GHA

* nit

* nit

* groups all checks, runs on 8, 11, 17 jdks

* nit

* adds license info

* update permissions on scripts folder

* nit

* nit

* fix packaging check

* changes naming, cleans repo before license checks

* simulate failure

* bump up license checks

* test license checks failure

* test license checks failure

* test license checks failure

* verify gha script run exit code

* fail fast in case of shell script

* verified fail fast in case of shell script

(cherry picked from commit 30498c1)
@tejaswini-imply @KeerthanaSrikanth
Update Hadoop3 as default build version (apache#14005)
a1ea791 
Hadoop 2 often causes red security scans on Druid distribution because of the dependencies it brings. We want to move away from Hadoop 2 and provide Hadoop 3 distribution available. Switch druid to building with Hadoop 3 by default. Druid will still be compatible with Hadoop 2 and users can build hadoop-2 compatible distribution using hadoop2 profile.

(cherry picked from commit 774073b)
@tejaswini-imply @KeerthanaSrikanth
update hadoop version to 3.3.6 (apache#14489)
d025d2e 
(cherry picked from commit baa64e6)
@KeerthanaSrikanth KeerthanaSrikanth requested review from a team as code owners November 6, 2023 06:38
@cla-assistant
Copy link

cla-assistant bot commented Nov 6, 2023

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

1 similar comment
@cla-assistant
Copy link

cla-assistant bot commented Nov 6, 2023

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@xvrl
Copy link
Member

xvrl commented Nov 14, 2023

do you mind clarifying why you had to cherry-pick apache#13412 ?

@KeerthanaSrikanth
Copy link
Author

KeerthanaSrikanth commented Nov 14, 2023

do you mind clarifying why you had to cherry-pick apache#13412 ?

Without cherry-picking this, the static-checks.yml had too many conflicts to resolve directly.

@KeerthanaSrikanth KeerthanaSrikanth marked this pull request as draft November 14, 2023 05:02
@KeerthanaSrikanth
Copy link
Author

KeerthanaSrikanth commented Dec 12, 2023

Hadoop 3 upgrade will be done when we upgrade to Druid 28

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@KeerthanaSrikanth @xvrl @tejaswini-imply