[SPARK-40799] [BUILD] [CONNECT] Enforce scalafmt for Spark Connect module. #38258

grundprinzip · 2022-10-14T13:56:33Z

What changes were proposed in this pull request?

This changes enables enforcing scalafmt for the Spark Connect module to have a consistent enforced coding style and reduce unnecessary code-review cycles. This patch specifically only enforces this on the Spark Connect module to avoid impact on the other areas of the Spark codebase.

To pass the pre-commit, this change applies scalafmt on the existing code-base.

Why are the changes needed?

Faster, focussed code reviews.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Manual testing / CI

AmplabJenkins · 2022-10-15T12:04:05Z

Can one of the admins verify this patch?

hvanhovell

LGTM

HyukjinKwon · 2022-10-16T14:30:16Z

Will merge this in a few days.

amaliujia · 2022-10-16T18:26:21Z

Will the github precommit fail if scala formatter is not applied yet in a PR?

grundprinzip · 2022-10-16T18:41:52Z

Will the github precommit fail if scala formatter is not applied yet in a PR?

Yes, this integrates in the check that is run as part of the dev/lint-scala script.

amaliujia · 2022-10-16T18:42:48Z

Will the github precommit fail if scala formatter is not applied yet in a PR?

Yes, this integrates in the check that is run as part of the dev/lint-scala script.

Nice! LGTM!

amaliujia · 2022-10-16T18:46:23Z

dev/lint-scala

+if [[ $? -ne 0 ]]; then
+  echo "The scalafmt check failed on connector/connect."
+  echo "Before submitting your change, please make sure to format your code using the following command:"
+  echo "./build/mvn -Pscala-2.12 scalafmt:format -Dscalafmt.skip=fase -Dscalafmt.validateOnly=false -pl connector/connect"


:) No one loves SBT

I am not familiar with SBT though maybe this works with SBT.

This piece here is taken from the existing dev/scalafmt script and slightly adjusted to only reformat the Spark Connect module. I don't think we've ported the usage of scalafmt to SBT.

grundprinzip · 2022-10-16T18:55:41Z

@HyukjinKwon let me know when you're ready to merge and I will address the merge conflicts by running scalafmt again to fix the existing issues.

grundprinzip · 2022-10-17T07:55:29Z

Checking that the build will fail because the lint fails

HyukjinKwon · 2022-10-17T11:44:27Z

Merged to master.

HyukjinKwon · 2022-10-17T11:53:41Z

ah, I am sorry it was my bad. I didn't see the last changes made. Let me revert and reopen.

grundprinzip · 2022-10-17T13:01:15Z

Verified that the pre-commit is failing appropriately.

HyukjinKwon · 2022-10-18T00:28:31Z

Merged to master.

…istent of lint-scala script as was ### What changes were proposed in this pull request? This PR proposes to keep `dev/lint-scala` quiet as was. ### Why are the changes needed? To remove noisy output from the `dev/lint-scala` script. **Before** Success ``` Scalastyle checks passed. Using `mvn` from path: /.../spark/build/apache-maven-3.8.6/bin/mvn [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Detecting the operating system and CPU architecture [INFO] ------------------------------------------------------------------------ [INFO] os.detected.name: osx [INFO] os.detected.arch: x86_64 [INFO] os.detected.version: 10.16 [INFO] os.detected.version.major: 10 [INFO] os.detected.version.minor: 16 [INFO] os.detected.classifier: osx-x86_64 [INFO] [INFO] ----------------< org.apache.spark:spark-connect_2.12 >----------------- [INFO] Building Spark Project Connect 3.4.0-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli) spark-connect_2.12 --- [INFO] parsed config (v3.5.9): dev/.scalafmt.conf [INFO] Scalafmt results: 0 of 11 were unformatted Details: Formatted: Connect.scala Formatted: DataTypeProtoConverter.scala Formatted: SparkConnectPlanner.scala Formatted: SparkConnectPlugin.scala Formatted: SparkConnectCommandPlanner.scala Formatted: SparkConnectStreamHandler.scala Formatted: SparkConnectService.scala Formatted: package.scala Formatted: SparkConnectProtoSuite.scala Formatted: SparkConnectPlannerSuite.scala Formatted: SparkConnectCommandPlannerSuite.scala [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.257 s [INFO] Finished at: 2022-10-21T11:18:19+09:00 [INFO] ------------------------------------------------------------------------ ``` Failure ``` Scalastyle checks passed. Using `mvn` from path: /Users/hyukjin.kwon/workspace/forked/spark/build/apache-maven-3.8.6/bin/mvn [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Detecting the operating system and CPU architecture [INFO] ------------------------------------------------------------------------ [INFO] os.detected.name: osx [INFO] os.detected.arch: x86_64 [INFO] os.detected.version: 10.16 [INFO] os.detected.version.major: 10 [INFO] os.detected.version.minor: 16 [INFO] os.detected.classifier: osx-x86_64 [INFO] [INFO] ----------------< org.apache.spark:spark-connect_2.12 >----------------- [INFO] Building Spark Project Connect 3.4.0-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli) spark-connect_2.12 --- [INFO] parsed config (v3.5.9): dev/.scalafmt.conf [INFO] Scalafmt results: 0 of 11 were unformatted Details: Formatted: Connect.scala Formatted: DataTypeProtoConverter.scala Formatted: SparkConnectPlanner.scala Formatted: SparkConnectPlugin.scala Formatted: SparkConnectCommandPlanner.scala Formatted: SparkConnectStreamHandler.scala Formatted: SparkConnectService.scala Formatted: package.scala Formatted: SparkConnectProtoSuite.scala Formatted: SparkConnectPlannerSuite.scala Formatted: SparkConnectCommandPlannerSuite.scala [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.257 s [INFO] Finished at: 2022-10-21T11:18:19+09:00 [INFO] ------------------------------------------------------------------------ (python3.9) ➜ spark git:(master) ./dev/lint-scala Scalastyle checks passed. Using `mvn` from path: /Users/hyukjin.kwon/workspace/forked/spark/build/apache-maven-3.8.6/bin/mvn [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Detecting the operating system and CPU architecture [INFO] ------------------------------------------------------------------------ [INFO] os.detected.name: osx [INFO] os.detected.arch: x86_64 [INFO] os.detected.version: 10.16 [INFO] os.detected.version.major: 10 [INFO] os.detected.version.minor: 16 [INFO] os.detected.classifier: osx-x86_64 [INFO] [INFO] ----------------< org.apache.spark:spark-connect_2.12 >----------------- [INFO] Building Spark Project Connect 3.4.0-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli) spark-connect_2.12 --- [INFO] parsed config (v3.5.9): dev/.scalafmt.conf [ERROR] unformatted file at: /Users/hyukjin.kwon/workspace/forked/spark/connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala [INFO] Scalafmt results: 1 of 11 were unformatted Details: Formatted: Connect.scala Formatted: DataTypeProtoConverter.scala Requires formatting: SparkConnectPlanner.scala Formatted: SparkConnectPlugin.scala Formatted: SparkConnectCommandPlanner.scala Formatted: SparkConnectStreamHandler.scala Formatted: SparkConnectService.scala Formatted: package.scala Formatted: SparkConnectProtoSuite.scala Formatted: SparkConnectPlannerSuite.scala Formatted: SparkConnectCommandPlannerSuite.scala [ERROR] org.apache.maven.plugin.MojoExecutionException: Scalafmt: Unformatted files found at org.antipathy.mvn_scalafmt.FormatMojo.execute (FormatMojo.java:91) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137) at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370) at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293) at org.apache.maven.cli.MavenCli.main (MavenCli.java:196) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.223 s [INFO] Finished at: 2022-10-21T11:19:58+09:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.antipathy:mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli) on project spark-connect_2.12: Error formatting Scala files: Scalafmt: Unformatted files found -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException The scalafmt check failed on connector/connect. Before submitting your change, please make sure to format your code using the following command: ./build/mvn -Pscala-2.12 scalafmt:format -Dscalafmt.skip=fase -Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl connector/connect ``` **After** Success ``` Scalastyle checks passed. Scalafmt checks passed. ``` Failure ``` Scalastyle checks passed. The scalafmt check failed on connector/connect at following occurrences: Requires formatting: SparkConnectPlanner.scala Before submitting your change, please make sure to format your code using the following command: ./build/mvn -Pscala-2.12 scalafmt:format -Dscalafmt.skip=fase -Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl connector/connect ``` In this way, this is consistent before #38258. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Manually tested as described above. Closes #38326 from HyukjinKwon/SPARK-40799-followup. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

### What changes were proposed in this pull request? This changes enables enforcing `scalafmt` for the Spark Connect module to have a consistent enforced coding style and reduce unnecessary code-review cycles. This patch specifically only enforces this on the Spark Connect module to avoid impact on the other areas of the Spark codebase. To pass the pre-commit, this change applies `scalafmt` on the existing code-base. ### Why are the changes needed? Faster, focussed code reviews. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manual testing / CI Closes apache#38258 from grundprinzip/spark-40799. Authored-by: Martin Grund <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

…istent of lint-scala script as was ### What changes were proposed in this pull request? This PR proposes to keep `dev/lint-scala` quiet as was. ### Why are the changes needed? To remove noisy output from the `dev/lint-scala` script. **Before** Success ``` Scalastyle checks passed. Using `mvn` from path: /.../spark/build/apache-maven-3.8.6/bin/mvn [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Detecting the operating system and CPU architecture [INFO] ------------------------------------------------------------------------ [INFO] os.detected.name: osx [INFO] os.detected.arch: x86_64 [INFO] os.detected.version: 10.16 [INFO] os.detected.version.major: 10 [INFO] os.detected.version.minor: 16 [INFO] os.detected.classifier: osx-x86_64 [INFO] [INFO] ----------------< org.apache.spark:spark-connect_2.12 >----------------- [INFO] Building Spark Project Connect 3.4.0-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli) spark-connect_2.12 --- [INFO] parsed config (v3.5.9): dev/.scalafmt.conf [INFO] Scalafmt results: 0 of 11 were unformatted Details: Formatted: Connect.scala Formatted: DataTypeProtoConverter.scala Formatted: SparkConnectPlanner.scala Formatted: SparkConnectPlugin.scala Formatted: SparkConnectCommandPlanner.scala Formatted: SparkConnectStreamHandler.scala Formatted: SparkConnectService.scala Formatted: package.scala Formatted: SparkConnectProtoSuite.scala Formatted: SparkConnectPlannerSuite.scala Formatted: SparkConnectCommandPlannerSuite.scala [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.257 s [INFO] Finished at: 2022-10-21T11:18:19+09:00 [INFO] ------------------------------------------------------------------------ ``` Failure ``` Scalastyle checks passed. Using `mvn` from path: /Users/hyukjin.kwon/workspace/forked/spark/build/apache-maven-3.8.6/bin/mvn [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Detecting the operating system and CPU architecture [INFO] ------------------------------------------------------------------------ [INFO] os.detected.name: osx [INFO] os.detected.arch: x86_64 [INFO] os.detected.version: 10.16 [INFO] os.detected.version.major: 10 [INFO] os.detected.version.minor: 16 [INFO] os.detected.classifier: osx-x86_64 [INFO] [INFO] ----------------< org.apache.spark:spark-connect_2.12 >----------------- [INFO] Building Spark Project Connect 3.4.0-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli) spark-connect_2.12 --- [INFO] parsed config (v3.5.9): dev/.scalafmt.conf [INFO] Scalafmt results: 0 of 11 were unformatted Details: Formatted: Connect.scala Formatted: DataTypeProtoConverter.scala Formatted: SparkConnectPlanner.scala Formatted: SparkConnectPlugin.scala Formatted: SparkConnectCommandPlanner.scala Formatted: SparkConnectStreamHandler.scala Formatted: SparkConnectService.scala Formatted: package.scala Formatted: SparkConnectProtoSuite.scala Formatted: SparkConnectPlannerSuite.scala Formatted: SparkConnectCommandPlannerSuite.scala [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.257 s [INFO] Finished at: 2022-10-21T11:18:19+09:00 [INFO] ------------------------------------------------------------------------ (python3.9) ➜ spark git:(master) ./dev/lint-scala Scalastyle checks passed. Using `mvn` from path: /Users/hyukjin.kwon/workspace/forked/spark/build/apache-maven-3.8.6/bin/mvn [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Detecting the operating system and CPU architecture [INFO] ------------------------------------------------------------------------ [INFO] os.detected.name: osx [INFO] os.detected.arch: x86_64 [INFO] os.detected.version: 10.16 [INFO] os.detected.version.major: 10 [INFO] os.detected.version.minor: 16 [INFO] os.detected.classifier: osx-x86_64 [INFO] [INFO] ----------------< org.apache.spark:spark-connect_2.12 >----------------- [INFO] Building Spark Project Connect 3.4.0-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli) spark-connect_2.12 --- [INFO] parsed config (v3.5.9): dev/.scalafmt.conf [ERROR] unformatted file at: /Users/hyukjin.kwon/workspace/forked/spark/connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala [INFO] Scalafmt results: 1 of 11 were unformatted Details: Formatted: Connect.scala Formatted: DataTypeProtoConverter.scala Requires formatting: SparkConnectPlanner.scala Formatted: SparkConnectPlugin.scala Formatted: SparkConnectCommandPlanner.scala Formatted: SparkConnectStreamHandler.scala Formatted: SparkConnectService.scala Formatted: package.scala Formatted: SparkConnectProtoSuite.scala Formatted: SparkConnectPlannerSuite.scala Formatted: SparkConnectCommandPlannerSuite.scala [ERROR] org.apache.maven.plugin.MojoExecutionException: Scalafmt: Unformatted files found at org.antipathy.mvn_scalafmt.FormatMojo.execute (FormatMojo.java:91) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137) at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370) at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293) at org.apache.maven.cli.MavenCli.main (MavenCli.java:196) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.223 s [INFO] Finished at: 2022-10-21T11:19:58+09:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.antipathy:mvn-scalafmt_2.12:1.1.1640084764.9f463a9:format (default-cli) on project spark-connect_2.12: Error formatting Scala files: Scalafmt: Unformatted files found -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException The scalafmt check failed on connector/connect. Before submitting your change, please make sure to format your code using the following command: ./build/mvn -Pscala-2.12 scalafmt:format -Dscalafmt.skip=fase -Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl connector/connect ``` **After** Success ``` Scalastyle checks passed. Scalafmt checks passed. ``` Failure ``` Scalastyle checks passed. The scalafmt check failed on connector/connect at following occurrences: Requires formatting: SparkConnectPlanner.scala Before submitting your change, please make sure to format your code using the following command: ./build/mvn -Pscala-2.12 scalafmt:format -Dscalafmt.skip=fase -Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl connector/connect ``` In this way, this is consistent before apache#38258. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Manually tested as described above. Closes apache#38326 from HyukjinKwon/SPARK-40799-followup. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

dongjoon-hyun · 2025-01-10T15:30:15Z

dev/lint-scala

+  echo "Before submitting your change, please make sure to format your code using the following command:"
+  echo "./build/mvn -Pscala-2.12 scalafmt:format -Dscalafmt.skip=fase -Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl connector/connect"
+  exit 1
+fi


Hi, @grundprinzip and @HyukjinKwon .

This PR makes lint-scala ignore the exit code of scalastyle.

if [[ $? -ne 0 ]]; then

This checks explicitly for a non-zero exit code. Why would this not cover your case?

dongjoon-hyun · 2025-01-10T15:30:32Z

I made a fix here.

[SPARK-50784][TESTS] Fix lint-scala not to ignore scalastyle errors #49443

### What changes were proposed in this pull request? This PR aims to fix `lint-scala` script not to ignore `scalastyle` errors. ### Why are the changes needed? This bug was introduced via the following PR at Apache Spark 3.4.0. - #38258 After the above PR, `lint-scala` ignores `scalastyle` error and only considers the exit code of `scalafmt` like the following CI result. - #49428 (comment) ![Screenshot 2025-01-10 at 07 14 31](https://github.com/user-attachments/assets/bdaa3be3-5daf-401b-a46f-7c02b7610158) ### Does this PR introduce _any_ user-facing change? No, this is a dev-only tool change. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #49443 from dongjoon-hyun/SPARK-50784. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? This PR aims to fix `lint-scala` script not to ignore `scalastyle` errors. ### Why are the changes needed? This bug was introduced via the following PR at Apache Spark 3.4.0. - #38258 After the above PR, `lint-scala` ignores `scalastyle` error and only considers the exit code of `scalafmt` like the following CI result. - #49428 (comment) ![Screenshot 2025-01-10 at 07 14 31](https://github.com/user-attachments/assets/bdaa3be3-5daf-401b-a46f-7c02b7610158) ### Does this PR introduce _any_ user-facing change? No, this is a dev-only tool change. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #49443 from dongjoon-hyun/SPARK-50784. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 9d4b7a5) Signed-off-by: Dongjoon Hyun <[email protected]>

enforce scalafmt

2f08a56

github-actions bot added BUILD CONNECT SQL labels Oct 14, 2022

hvanhovell approved these changes Oct 15, 2022

View reviewed changes

HyukjinKwon approved these changes Oct 16, 2022

View reviewed changes

amaliujia reviewed Oct 16, 2022

View reviewed changes

merge

a98ea62

grundprinzip added 2 commits October 17, 2022 10:12

fix

c9e725c

triggering the check

50486ed

HyukjinKwon closed this in 2341525 Oct 17, 2022

HyukjinKwon reopened this Oct 17, 2022

grundprinzip added 2 commits October 17, 2022 14:02

fixing changed Only

deb890b

format changes

6bc0689

HyukjinKwon closed this in cd95044 Oct 18, 2022

HyukjinKwon mentioned this pull request Oct 21, 2022

[SPARK-40799][BUILD][CONNECT][FOLLOW-UP] Keep the console output consistent of lint-scala script as was #38326

Closed

dongjoon-hyun mentioned this pull request Jan 10, 2025

[SPARK-50784][TESTS] Fix lint-scala not to ignore scalastyle errors #49443

Closed

dongjoon-hyun reviewed Jan 10, 2025

View reviewed changes

[SPARK-40799] [BUILD] [CONNECT] Enforce scalafmt for Spark Connect module. #38258

[SPARK-40799] [BUILD] [CONNECT] Enforce scalafmt for Spark Connect module. #38258

Uh oh!

Conversation

grundprinzip commented Oct 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

AmplabJenkins commented Oct 15, 2022

Uh oh!

hvanhovell left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Oct 16, 2022

Uh oh!

amaliujia commented Oct 16, 2022

Uh oh!

grundprinzip commented Oct 16, 2022

Uh oh!

amaliujia commented Oct 16, 2022

Uh oh!

amaliujia Oct 16, 2022

Choose a reason for hiding this comment

Uh oh!

amaliujia Oct 16, 2022

Choose a reason for hiding this comment

Uh oh!

grundprinzip Oct 16, 2022

Choose a reason for hiding this comment

Uh oh!

grundprinzip commented Oct 16, 2022

Uh oh!

grundprinzip commented Oct 17, 2022

Uh oh!

HyukjinKwon commented Oct 17, 2022

Uh oh!

HyukjinKwon commented Oct 17, 2022

Uh oh!

grundprinzip commented Oct 17, 2022

Uh oh!

HyukjinKwon commented Oct 18, 2022

Uh oh!

dongjoon-hyun Jan 10, 2025

Choose a reason for hiding this comment

Uh oh!

grundprinzip Jan 10, 2025

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jan 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

grundprinzip commented Oct 14, 2022 •

edited

Loading