TEZ-4642: Introduce spotless plugin and enforce basic import styles #423

abstractdog · 2025-07-30T11:29:35Z

Huge patch that reorganize imports. Easy to implement and port:

insert the spotless plugin to root pom.xml
run "mvn spotless:apply"

Import Ordering Enforcement
Configured to match a reasonable standard: java,javax,org.apache,com,net,io
Automatically removes unused imports
Separates import groups with blank lines

Build Integration
Fails builds when import violations are found (during validate phase)
Works with all Maven goals: install, package, compile, etc.

abstractdog · 2025-07-30T11:33:30Z

LOL, yetus thinks that the patch is not applicable, I'm afraid we need to merge and test afterwards with a dummy precommit
no functional/behavioral change is expected

maheshrajus · 2025-07-30T13:23:36Z

@abstractdog we can merge it and test after raising dummy PR to master branch. Samething we followed for spotbugs related PRs aswell :)

ayushtkn

LGTM, minor suggestion.

This gonna make backports a big pain, if this iteself isn't backported

ayushtkn · 2025-07-30T20:37:57Z

pom.xml

+        <plugin>
+          <groupId>com.diffplug.spotless</groupId>
+          <artifactId>spotless-maven-plugin</artifactId>
+          <version>2.43.0</version>


Can you define a variable for the version and use it here, like other dependencies. Moreover why not use the latest version:
https://mvnrepository.com/artifact/com.diffplug.spotless/spotless-maven-plugin/2.46.1

tez-yetus · 2025-07-31T05:10:46Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 5s		#423 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

Subsystem	Report/Notes
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-423/4/console
versions	git=2.34.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

ayushtkn · 2025-07-31T18:19:50Z

Ohh, I remember this Yetus issue. The size of the PR is too big that is why it is failing to apply. If you do locally on your mac

curl --fail -H 'Accept: application/vnd.github.v3.patch' --output /private/tmp/yetus-10894.25287/input.patch --location https://api.github.com/repos/apache/tez/pulls/423

This will fail locally as well

ayushsaxena@ayushsaxena tez %  curl --fail -H 'Accept: application/vnd.github.v3.patch' --output /private/tmp/yetus-10894.25287/input.patch --location https://api.github.com/repos/apache/tez/pulls/423 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   392    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (56) The requested URL returned error: 406

More details here:
https://issues.apache.org/jira/browse/YETUS-719?focusedCommentId=17874835&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17874835

Maybe we can hack it here as well, get the result and then revert that hacky commit then merge this PR

Can you apply this diff and trigger the build please

diff --git a/Jenkinsfile b/Jenkinsfile
index 79c2a3e3c..e9d6c4368 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -35,7 +35,7 @@ pipeline {
         DOCKERFILE = "${SOURCEDIR}/build-tools/docker/Dockerfile"
         YETUS='yetus'
         // Branch or tag name.  Yetus release tags are 'rel/X.Y.Z'
-        YETUS_VERSION='rel/0.15.1'
+        YETUS_VERSION='a7d29a6a72750a0c5c39512f33945e773e69303e'
 
     }
 
@@ -52,7 +52,7 @@ pipeline {
                     checkout([
                         $class: 'GitSCM',
                         branches: [[name: "${env.YETUS_VERSION}"]],
-                        userRemoteConfigs: [[ url: 'https://github.com/apache/yetus']]]
+                        userRemoteConfigs: [[ url: 'https://github.com/ayushtkn/yetus']]]
                     )
                 }
             }

abstractdog · 2025-07-31T18:25:37Z

Ohh, I remember this Yetus issue. The size of the PR is too big that is why it is failing to apply. If you do locally on your mac

curl --fail -H 'Accept: application/vnd.github.v3.patch' --output /private/tmp/yetus-10894.25287/input.patch --location https://api.github.com/repos/apache/tez/pulls/423

This will fail locally as well

ayushsaxena@ayushsaxena tez %  curl --fail -H 'Accept: application/vnd.github.v3.patch' --output /private/tmp/yetus-10894.25287/input.patch --location https://api.github.com/repos/apache/tez/pulls/423 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   392    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (56) The requested URL returned error: 406

More details here: https://issues.apache.org/jira/browse/YETUS-719?focusedCommentId=17874835&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17874835

Maybe we can hack it here as well, get the result and then revert that hacky commit then merge this PR

Can you apply this diff and trigger the build please

diff --git a/Jenkinsfile b/Jenkinsfile
index 79c2a3e3c..e9d6c4368 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -35,7 +35,7 @@ pipeline {
         DOCKERFILE = "${SOURCEDIR}/build-tools/docker/Dockerfile"
         YETUS='yetus'
         // Branch or tag name.  Yetus release tags are 'rel/X.Y.Z'
-        YETUS_VERSION='rel/0.15.1'
+        YETUS_VERSION='a7d29a6a72750a0c5c39512f33945e773e69303e'
 
     }
 
@@ -52,7 +52,7 @@ pipeline {
                     checkout([
                         $class: 'GitSCM',
                         branches: [[name: "${env.YETUS_VERSION}"]],
-                        userRemoteConfigs: [[ url: 'https://github.com/apache/yetus']]]
+                        userRemoteConfigs: [[ url: 'https://github.com/ayushtkn/yetus']]]
                     )
                 }
             }

nice! let me try

abstractdog · 2025-07-31T19:49:39Z

seems to be working, in progress:
https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-423/5/console

tez-yetus · 2025-08-01T09:13:30Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 36s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 5s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 248 new or modified test files.
			_ master Compile Tests _
+0 🆗	mvndep	2m 28s		Maven dependency ordering for branch
+1 💚	mvninstall	7m 34s		master passed
+1 💚	compile	8m 0s		master passed
+1 💚	checkstyle	7m 3s		master passed
+1 💚	javadoc	6m 12s		master passed
+0 🆗	spotbugs	1m 11s		hadoop-shim in master has 3 extant spotbugs warnings.
+0 🆗	spotbugs	0m 47s		tez-api in master has 610 extant spotbugs warnings.
+0 🆗	spotbugs	0m 25s		tez-common in master has 13 extant spotbugs warnings.
+0 🆗	spotbugs	0m 32s		tez-runtime-internals in master has 114 extant spotbugs warnings.
+0 🆗	spotbugs	0m 44s		tez-runtime-library in master has 241 extant spotbugs warnings.
+0 🆗	spotbugs	0m 32s		tez-mapreduce in master has 129 extant spotbugs warnings.
+0 🆗	spotbugs	0m 29s		tez-examples in master has 2 extant spotbugs warnings.
+0 🆗	spotbugs	0m 56s		tez-dag in master has 785 extant spotbugs warnings.
+0 🆗	spotbugs	0m 34s		tez-tests in master has 6 extant spotbugs warnings.
+0 🆗	spotbugs	0m 32s		tez-ext-service-tests in master has 73 extant spotbugs warnings.
+0 🆗	spotbugs	0m 30s		tez-plugins/tez-protobuf-history-plugin in master has 31 extant spotbugs warnings.
+0 🆗	spotbugs	0m 28s		tez-plugins/tez-yarn-timeline-history in master has 2 extant spotbugs warnings.
+0 🆗	spotbugs	0m 26s		tez-plugins/tez-yarn-timeline-history-with-acls in master has 2 extant spotbugs warnings.
+0 🆗	spotbugs	0m 27s		tez-plugins/tez-yarn-timeline-cache-plugin in master has 1 extant spotbugs warnings.
+0 🆗	spotbugs	0m 29s		tez-plugins/tez-yarn-timeline-history-with-fs in master has 5 extant spotbugs warnings.
+0 🆗	spotbugs	0m 33s		tez-plugins/tez-history-parser in master has 21 extant spotbugs warnings.
+0 🆗	spotbugs	0m 32s		tez-plugins/tez-aux-services in master has 5 extant spotbugs warnings.
+0 🆗	spotbugs	4m 0s		root in master has 2066 extant spotbugs warnings.
+0 🆗	spotbugs	0m 31s		tez-tools/analyzers/job-analyzer in master has 20 extant spotbugs warnings.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for patch
+1 💚	mvninstall	7m 49s		the patch passed
+1 💚	codespell	0m 27s		No new issues.
+1 💚	compile	8m 12s		the patch passed
+1 💚	javac	8m 12s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 15s		The patch passed checkstyle in hadoop-shim
+1 💚	checkstyle	0m 19s		tez-api: The patch generated 0 new + 708 unchanged - 15 fixed = 708 total (was 723)
+1 💚	checkstyle	0m 17s		tez-common: The patch generated 0 new + 112 unchanged - 4 fixed = 112 total (was 116)
+1 💚	checkstyle	0m 18s		tez-runtime-internals: The patch generated 0 new + 237 unchanged - 1 fixed = 237 total (was 238)
+1 💚	checkstyle	0m 26s		tez-runtime-library: The patch generated 0 new + 1717 unchanged - 9 fixed = 1717 total (was 1726)
+1 💚	checkstyle	0m 20s		tez-mapreduce: The patch generated 0 new + 768 unchanged - 2 fixed = 768 total (was 770)
+1 💚	checkstyle	0m 17s		tez-examples: The patch generated 0 new + 60 unchanged - 3 fixed = 60 total (was 63)
+1 💚	checkstyle	0m 32s		tez-dag: The patch generated 0 new + 2974 unchanged - 37 fixed = 2974 total (was 3011)
+1 💚	checkstyle	0m 19s		tez-tests: The patch generated 0 new + 481 unchanged - 9 fixed = 481 total (was 490)
+1 💚	checkstyle	0m 17s		tez-ext-service-tests: The patch generated 0 new + 111 unchanged - 6 fixed = 111 total (was 117)
+1 💚	checkstyle	0m 17s		The patch passed checkstyle in tez-protobuf-history-plugin
+1 💚	checkstyle	0m 17s		tez-plugins/tez-yarn-timeline-history: The patch generated 0 new + 175 unchanged - 3 fixed = 175 total (was 178)
+1 💚	checkstyle	0m 15s		The patch passed checkstyle in tez-yarn-timeline-history-with-acls
+1 💚	checkstyle	0m 15s		The patch passed checkstyle in tez-yarn-timeline-cache-plugin
+1 💚	checkstyle	0m 17s		tez-plugins/tez-yarn-timeline-history-with-fs: The patch generated 0 new + 83 unchanged - 1 fixed = 83 total (was 84)
+1 💚	checkstyle	0m 18s		The patch passed checkstyle in tez-history-parser
+1 💚	checkstyle	0m 17s		The patch passed checkstyle in tez-aux-services
+1 💚	checkstyle	0m 16s		The patch passed checkstyle in job-analyzer
+1 💚	checkstyle	0m 15s		The patch passed checkstyle in hadoop-shim-2.8
+1 💚	checkstyle	0m 59s		root: The patch generated 0 new + 7877 unchanged - 90 fixed = 7877 total (was 7967)
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	6m 27s		the patch passed
+1 💚	spotbugs	17m 45s		the patch passed
			_ Other Tests _
+1 💚	unit	0m 26s		hadoop-shim in the patch passed.
+1 💚	unit	1m 56s		tez-api in the patch passed.
+1 💚	unit	0m 30s		tez-common in the patch passed.
+1 💚	unit	0m 46s		tez-runtime-internals in the patch passed.
+1 💚	unit	4m 41s		tez-runtime-library in the patch passed.
+1 💚	unit	1m 6s		tez-mapreduce in the patch passed.
+1 💚	unit	0m 22s		tez-examples in the patch passed.
+1 💚	unit	4m 11s		tez-dag in the patch passed.
+1 💚	unit	33m 51s		tez-tests in the patch passed.
+1 💚	unit	3m 46s		tez-ext-service-tests in the patch passed.
+1 💚	unit	0m 27s		tez-protobuf-history-plugin in the patch passed.
+1 💚	unit	1m 26s		tez-yarn-timeline-history in the patch passed.
+1 💚	unit	1m 34s		tez-yarn-timeline-history-with-acls in the patch passed.
+1 💚	unit	0m 18s		tez-yarn-timeline-cache-plugin in the patch passed.
+1 💚	unit	1m 23s		tez-yarn-timeline-history-with-fs in the patch passed.
+1 💚	unit	2m 3s		tez-history-parser in the patch passed.
+1 💚	unit	2m 31s		tez-aux-services in the patch passed.
+1 💚	unit	2m 14s		job-analyzer in the patch passed.
+1 💚	unit	0m 21s		hadoop-shim-2.8 in the patch passed.
+1 💚	unit	59m 27s		root in the patch passed.
+1 💚	asflicense	5m 50s		The patch does not generate ASF License warnings.
		245m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-423/6/artifact/out/Dockerfile
GITHUB PR	#423
Optional Tests	dupname asflicense javac javadoc unit spotbugs checkstyle codespell detsecrets compile shellcheck shelldocs xmllint
uname	Linux 3bbec5c12f12 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-423/src/.yetus/personality.sh
git revision	master / `3a7c01d`
Default Java	Ubuntu-21.0.8+9-Ubuntu-0ubuntu124.04.1
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-423/6/testReport/
Max. process+thread count	1540 (vs. ulimit of 5500)
modules	C: hadoop-shim tez-api tez-common tez-runtime-internals tez-runtime-library tez-mapreduce tez-examples tez-dag tez-tests tez-ext-service-tests tez-plugins/tez-protobuf-history-plugin tez-plugins/tez-yarn-timeline-history tez-plugins/tez-yarn-timeline-history-with-acls tez-plugins/tez-yarn-timeline-cache-plugin tez-plugins/tez-yarn-timeline-history-with-fs tez-plugins/tez-history-parser tez-plugins/tez-aux-services tez-tools/analyzers/job-analyzer hadoop-shim-impls/hadoop-shim-2.8 . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-423/6/console
versions	git=2.43.0 maven=3.8.7 spotbugs=4.9.3 codespell=2.0.0 shellcheck=0.7.1
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

abstractdog · 2025-08-01T09:59:48Z

looks clear, thanks @ayushtkn! reverted the workaround commit

tez-yetus · 2025-08-01T10:00:12Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 6s		#423 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

Subsystem	Report/Notes
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-423/7/console
versions	git=2.34.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

abstractdog · 2025-08-18T07:47:03Z

@ayushtkn: let me know if we can merge this

ayushtkn

LGTM

This comment was marked as outdated.

Sign in to view

TEZ-4642: Introduce spotless plugin and enforce basic import styles

e188238

abstractdog force-pushed the TEZ-4642 branch from 287fe50 to e188238 Compare July 30, 2025 11:32

This comment was marked as outdated.

Sign in to view

abstractdog requested a review from ayushtkn July 30, 2025 11:48

ayushtkn reviewed Jul 30, 2025

View reviewed changes

This comment was marked as outdated.

Sign in to view

spotless-maven-plugin.version 2.46.1

fde0a09

abstractdog force-pushed the TEZ-4642 branch from 16b767a to fde0a09 Compare July 31, 2025 05:10

This comment was marked as outdated.

Sign in to view

abstractdog force-pushed the TEZ-4642 branch from 3a7c01d to fde0a09 Compare August 1, 2025 09:59

abstractdog requested a review from ayushtkn August 18, 2025 07:46

ayushtkn approved these changes Aug 18, 2025

View reviewed changes

abstractdog merged commit c981a94 into apache:master Aug 18, 2025
8 checks passed

TEZ-4642: Introduce spotless plugin and enforce basic import styles #423

TEZ-4642: Introduce spotless plugin and enforce basic import styles #423

Conversation

abstractdog commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

abstractdog commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maheshrajus commented Jul 30, 2025

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

ayushtkn Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

abstractdog Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

tez-yetus commented Jul 31, 2025

Uh oh!

ayushtkn commented Jul 31, 2025

Uh oh!

abstractdog commented Jul 31, 2025

Uh oh!

abstractdog commented Jul 31, 2025

Uh oh!

This comment was marked as outdated.

tez-yetus commented Aug 1, 2025

Uh oh!

abstractdog commented Aug 1, 2025

Uh oh!

tez-yetus commented Aug 1, 2025

Uh oh!

abstractdog commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

abstractdog commented Jul 30, 2025 •

edited

Loading

abstractdog commented Jul 30, 2025 •

edited

Loading

abstractdog commented Aug 18, 2025 •

edited

Loading