Skip to content

HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom#3555

Merged
tasanuma merged 2 commits intoapache:trunkfrom
virajjasani:HADOOP-17967-trunk
Oct 21, 2021
Merged

HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom#3555
tasanuma merged 2 commits intoapache:trunkfrom
virajjasani:HADOOP-17967-trunk

Conversation

@virajjasani
Copy link
Copy Markdown
Contributor

No description provided.

@hadoop-yetus
Copy link
Copy Markdown

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 47s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+0 🆗 mvndep 13m 12s Maven dependency ordering for branch
+1 💚 mvninstall 22m 12s trunk passed
+1 💚 compile 23m 10s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 20m 36s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 mvnsite 25m 58s trunk passed
+1 💚 javadoc 8m 14s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 8m 7s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 shadedclient 140m 33s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 39s Maven dependency ordering for patch
+1 💚 mvninstall 42m 7s the patch passed
+1 💚 compile 22m 52s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 22m 52s the patch passed
+1 💚 compile 20m 41s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 20m 41s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 22m 9s the patch passed
+1 💚 xml 0m 37s The patch has no ill-formed XML file.
+1 💚 javadoc 8m 17s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 8m 41s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 shadedclient 37m 41s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 809m 19s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 1m 33s The patch does not generate ASF License warnings.
1100m 35s
Reason Tests
Failed junit tests hadoop.yarn.csi.client.TestCsiClient
hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure
hadoop.tools.dynamometer.TestDynamometerInfra
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3555/1/artifact/out/Dockerfile
GITHUB PR #3555
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml
uname Linux 3d871369f6da 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 10bea6e
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3555/1/testReport/
Max. process+thread count 3226 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-common-project/hadoop-kms hadoop-common-project/hadoop-registry hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-native-client hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-tools/hadoop-distcp hadoop-tools/hadoop-federation-balance hadoop-hdfs-project/hadoop-hdfs-rbf hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure hadoop-tools/hadoop-sls hadoop-tools/hadoop-azure-datalake hadoop-tools/hadoop-fs2img hadoop-cloud-storage-project/hadoop-cos hadoop-cloud-storage-project/hadoop-huaweicloud . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3555/1/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link
Copy Markdown

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 49s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+0 🆗 mvndep 12m 58s Maven dependency ordering for branch
+1 💚 mvninstall 22m 3s trunk passed
+1 💚 compile 23m 47s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 20m 11s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 mvnsite 26m 16s trunk passed
+1 💚 javadoc 8m 10s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 7m 42s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 shadedclient 139m 54s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 38s Maven dependency ordering for patch
+1 💚 mvninstall 41m 22s the patch passed
+1 💚 compile 23m 17s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 23m 17s the patch passed
+1 💚 compile 20m 26s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 20m 26s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 22m 18s the patch passed
+1 💚 xml 0m 36s The patch has no ill-formed XML file.
+1 💚 javadoc 8m 36s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 8m 1s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 shadedclient 36m 55s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 812m 13s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 1m 33s The patch does not generate ASF License warnings.
1101m 45s
Reason Tests
Failed junit tests hadoop.yarn.csi.client.TestCsiClient
hadoop.hdfs.TestRollingUpgrade
hadoop.tools.dynamometer.TestDynamometerInfra
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3555/2/artifact/out/Dockerfile
GITHUB PR #3555
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml
uname Linux 57645d308ac7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 10bea6e
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3555/2/testReport/
Max. process+thread count 3090 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-common-project/hadoop-kms hadoop-common-project/hadoop-registry hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-native-client hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-tools/hadoop-distcp hadoop-tools/hadoop-federation-balance hadoop-hdfs-project/hadoop-hdfs-rbf hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure hadoop-tools/hadoop-sls hadoop-tools/hadoop-azure-datalake hadoop-tools/hadoop-fs2img hadoop-cloud-storage-project/hadoop-cos hadoop-cloud-storage-project/hadoop-huaweicloud . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3555/2/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@amahussein
Copy link
Copy Markdown
Contributor

Hi @virajjasani ,
Can we hold-on this change for a little bit?
We are going to need the per-module enforcer to remove other Guava classes. At least the Preconditions.
I cannot remember top of my head which other classes need to be incrementally replaced. Maybe we need to generate a list of all guava classes that are widely used across the modules.

If we get that PR merged then we will reintroduce the same lines again very soon which is going to be redundant work.

@virajjasani
Copy link
Copy Markdown
Contributor Author

I am fine either way but I think we might have to make this change soon for sure, because not every single pom of all sub-modules are covered, only the ones where VFT is being used are covered. Hence, someone can still technically use Guava provided VFT annotation in any of those modules, it is only after this change we can restrict in every single module just because of the hadoop-main pom, which is directly or indirectly parent pom for every module/sub-module.

@virajjasani
Copy link
Copy Markdown
Contributor Author

virajjasani commented Oct 18, 2021

We will still have commits landed in the meanwhile and I believe we could consider checking-in this PR. In the end, I think this is the only PR pending for removal of VFT's dependency on Guava, and for Preconditions and any other dependencies on Guava, I think it's still worth adding this enforcer as required to cover particular set of modules/sub-modules so that no two or more enforcer rules becomes mutually dependent on each other, because once we define such inter-dependency, we will again have to be more careful with backports, so it becomes even more complicated.

Thoughts @tasanuma @amahussein ?

@tasanuma
Copy link
Copy Markdown
Member

I wish there was a better way, but I can't think of one. I don't think we can wait too long to merge this PR, because as Viraj said, not every single pom of all sub-modules are covered.

Copy link
Copy Markdown
Member

@tasanuma tasanuma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I will merge it tomorrow if there is no objection.

@ayushtkn
Copy link
Copy Markdown
Member

Just casually exploring, got curious on how does this work. Checked in the PR branch, got one import

https://github.com/virajjasani/hadoop/blob/HADOOP-17967-trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java#L655

  @org.apache.hadoop.thirdparty.com.google.common.annotations.VisibleForTesting
  int getPriorityLevel(Schedulable e) {

Why it is not blocking this & we would need an Addendum somewhere if I am not looking at some old code or have messed up.

@virajjasani
Copy link
Copy Markdown
Contributor Author

Thanks for catching this @ayushtkn. Both checkstyle rules as well as restrict-imports-enforcer maven plugin, focus on imports of specific pattern and produces checkstyle error or build failure based on what we have used from the two of them. However, now I have realized that none of them can detect usage without imports e.g similar to the one you have provided above:

  @org.apache.hadoop.thirdparty.com.google.common.annotations.VisibleForTesting
  int getPriorityLevel(Schedulable e) {

I will fix this anyways in the next commit on this PR but how we can restrict such direct usage that don't use imported version of annotation or class or interface might require some brainstorming, no direct technique comes to mind as of now, let me get back if I can find something.

@amahussein
Copy link
Copy Markdown
Contributor

I will fix this anyways in the next commit on this PR but how we can restrict such direct usage that don't use imported version of annotation or class or interface might require some brainstorming, no direct technique comes to mind as of now, let me get back if I can find something.

Thanks @virajjasani , @tasanuma and @ayushtkn
Yes, of course, this means that we may have to do some other cycles in case someone used direct guava calls without import, but do we need to worry too much about that?
Once all usage of guava is removed, it should be removed from the maven dependency.
This will take care of someone trying to directly use a specific guava class.
IMHO, I prefer we invest the time getting rid of the guava calls rather than trying to defend against possible backdoors.

@virajjasani
Copy link
Copy Markdown
Contributor Author

IMHO, I prefer we invest the time getting rid of the guava calls rather than trying to defend against possible backdoors.

Sounds good, given that once all usage of guava is removed, it should be anyways removed from the maven dependency.

@ayushtkn
Copy link
Copy Markdown
Member

Shaded guava you can remove, Actual guava I think No. There are transitive dependencies which require guava.

@virajjasani
Copy link
Copy Markdown
Contributor Author

Shaded guava you can remove, Actual guava I think No. There are transitive dependencies which require guava.

You are right, have seen this transitive dependency issues with Yarn timelineservice as well as some other modules using Curator dependency. But I think when we get to the point where our codebase is no longer using Guava libraries, we can consider whether provided dependencies of guava helps (just thoughts for now, at least we might require runtime scope), and in the meantime shaded guava can be removed.

@hadoop-yetus
Copy link
Copy Markdown

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 49s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+0 🆗 mvndep 12m 51s Maven dependency ordering for branch
+1 💚 mvninstall 21m 13s trunk passed
+1 💚 compile 22m 19s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 18m 38s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 3m 39s trunk passed
+1 💚 mvnsite 24m 28s trunk passed
+1 💚 javadoc 7m 32s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 7m 36s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗 spotbugs 0m 23s branch/hadoop-hdfs-project/hadoop-hdfs-native-client no spotbugs output file (spotbugsXml.xml)
+1 💚 shadedclient 50m 48s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 37s Maven dependency ordering for patch
+1 💚 mvninstall 48m 29s the patch passed
+1 💚 compile 27m 31s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 27m 31s the patch passed
+1 💚 compile 25m 20s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 25m 20s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 4m 42s the patch passed
+1 💚 mvnsite 26m 30s the patch passed
+1 💚 xml 0m 42s The patch has no ill-formed XML file.
+1 💚 javadoc 10m 11s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 9m 37s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+0 🆗 spotbugs 0m 22s hadoop-hdfs-project/hadoop-hdfs-native-client has no data from spotbugs
+1 💚 shadedclient 58m 51s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 777m 27s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 1m 34s The patch does not generate ASF License warnings.
1256m 1s
Reason Tests
Failed junit tests hadoop.yarn.csi.client.TestCsiClient
hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate
hadoop.tools.dynamometer.TestDynamometerInfra
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3555/4/artifact/out/Dockerfile
GITHUB PR #3555
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle
uname Linux 7bb8409b8e03 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 917fec2
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3555/4/testReport/
Max. process+thread count 3291 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-common-project/hadoop-kms hadoop-common-project/hadoop-registry hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-native-client hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-yarn-project/hadoop-yarn hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-tools/hadoop-distcp hadoop-tools/hadoop-federation-balance hadoop-hdfs-project/hadoop-hdfs-rbf hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure hadoop-tools/hadoop-sls hadoop-tools/hadoop-azure-datalake hadoop-tools/hadoop-fs2img hadoop-cloud-storage-project/hadoop-cos hadoop-cloud-storage-project/hadoop-huaweicloud . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3555/4/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Copy link
Copy Markdown
Member

@tasanuma tasanuma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The failed tests are not related. +1. It seems there are no objections to this change. I will merge it.

@tasanuma tasanuma merged commit 516f36c into apache:trunk Oct 21, 2021
@tasanuma
Copy link
Copy Markdown
Member

Thanks for your contribution, @virajjasani. Thanks for your review and discussion, @amahussein and @ayushtkn.

If possible, using guava calls directly should be banned, but since it is a rare case, I don't think it will be much of a problem without it. Maybe we can discuss it in HADOOP-17968.

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022
prabhjyotsingh pushed a commit to acceldata-io/hadoop that referenced this pull request Mar 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants