Skip to content

Conversation

@myskov
Copy link
Contributor

@myskov myskov commented Mar 24, 2024

What changes were proposed in this pull request?

HDDS-10578. Clean up Hadoop dependencies

There are a lot of Hadoop dependencies, and a significant portion of them are now obsolete.
I removed unused dependencies.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10578

How was this patch tested?

Existing tests

Copy link
Contributor

@adoroszlai adoroszlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @myskov for the cleanup. I think some more hadoop dependencies can be removed, but otherwise LGTM.

pom.xml Outdated
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-api</artifactId>
<version>${hadoop.version}</version>
</dependency>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this one is also unused.

Copy link
Contributor Author

@myskov myskov Mar 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the patch, removed this one

<artifactId>hadoop-mapreduce-client-jobclient</artifactId>
<version>${hadoop.version}</version>
<type>test-jar</type>
</dependency>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one, too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hadoop-mapreduce-client-jobclient is used by integration-test module

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Root POM defines two similar dependencies:

ozone/pom.xml

Lines 376 to 386 in 0a405d7

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-jobclient</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-jobclient</artifactId>
<version>${hadoop.version}</version>
<type>test-jar</type>
</dependency>

but only one of them is used (the one without <type>test-jar):

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-jobclient</artifactId>
<scope>test</scope>
<exclusions>

Copy link
Contributor

@adoroszlai adoroszlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @myskov for updating the patch.

@myskov myskov merged commit 80bafd0 into apache:master Mar 26, 2024
@myskov myskov deleted the HDDS-10578_cleanup_hadoop_dependencies branch March 26, 2024 13:39
@myskov
Copy link
Contributor Author

myskov commented Mar 26, 2024

Thanks @kerneltime and @kerneltime for reviewing the patch

@adoroszlai adoroszlai added the dependencies Pull requests that update a dependency file label Mar 26, 2024
jojochuang pushed a commit to jojochuang/ozone that referenced this pull request May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants