[SPARK-44798][BUILD] Fix Scala 2.13 mima check after SPARK-44705 merged #42479
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zhengruifeng
approved these changes
Aug 14, 2023
Contributor
Author
|
I make this as draft first, let's wait for the conclusion of the issue I described in #42385 (comment) |
Contributor
Author
|
Merged into master |
Contributor
Author
|
Thanks @zhengruifeng |
valentinp17
pushed a commit
to valentinp17/spark
that referenced
this pull request
Aug 24, 2023
### What changes were proposed in this pull request? This pr aims add a new `ProblemFilters` to `MimaExcludes.scala` to fix mima check for Scala 2.13 after SPARK-44705 merged. ### Why are the changes needed? Scala 2.13's daily tests have been failing the mima check for several days: - https://github.com/apache/spark/actions/runs/5849050388/job/15856979487 <img width="1315" alt="image" src="https://github.com/apache/spark/assets/1475305/fa4f859a-293b-429b-a030-a332e6a587d3"> ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions - Manual verification： 1. The mima check was passing before SPARK-44705 ``` // [SPARK-44765][CONNECT] Simplify retries of ReleaseExecute git reset --hard 9bde882 dev/change-scala-version.sh 2.13 dev/mima -Pscala-2.13 ``` ``` [success] Total time: 48 s, completed 2023-8-14 13:02:11 ``` 2. The mima check failed after SPARK-44705 was merged ``` // [SPARK-44705][PYTHON] Make PythonRunner single-threaded git reset --hard 8aaff55 dev/change-scala-version.sh 2.13 dev/mima -Pscala-2.13 ``` ``` [error] spark-core: Failed binary compatibility check against org.apache.spark:spark-core_2.13:3.4.0! Found 1 potential problems (filtered 4019) [error] * method this(org.apache.spark.api.python.BasePythonRunner,java.io.DataInputStream,org.apache.spark.api.python.BasePythonRunner#WriterThread,Long,org.apache.spark.SparkEnv,java.net.Socket,scala.Option,java.util.concurrent.atomic.AtomicBoolean,org.apache.spark.TaskContext)Unit in class org.apache.spark.api.python.BasePythonRunner#ReaderIterator's type is different in current version, where it is (org.apache.spark.api.python.BasePythonRunner,java.io.DataInputStream,org.apache.spark.api.python.BasePythonRunner#Writer,Long,org.apache.spark.SparkEnv,org.apache.spark.api.python.PythonWorker,scala.Option,java.util.concurrent.atomic.AtomicBoolean,org.apache.spark.TaskContext)Unit instead of (org.apache.spark.api.python.BasePythonRunner,java.io.DataInputStream,org.apache.spark.api.python.BasePythonRunner#WriterThread,Long,org.apache.spark.SparkEnv,java.net.Socket,scala.Option,java.util.concurrent.atomic.AtomicBoolean,org.apache.spark.TaskContext)Unit [error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.api.python.BasePythonRunner#ReaderIterator.this") [error] java.lang.RuntimeException: Failed binary compatibility check against org.apache.spark:spark-core_2.13:3.4.0! Found 1 potential problems (filtered 4019) [error] at scala.sys.package$.error(package.scala:30) [error] at com.typesafe.tools.mima.plugin.SbtMima$.reportModuleErrors(SbtMima.scala:89) [error] at com.typesafe.tools.mima.plugin.MimaPlugin$.$anonfun$projectSettings$2(MimaPlugin.scala:36) [error] at com.typesafe.tools.mima.plugin.MimaPlugin$.$anonfun$projectSettings$2$adapted(MimaPlugin.scala:26) [error] at scala.collection.Iterator.foreach(Iterator.scala:943) [error] at scala.collection.Iterator.foreach$(Iterator.scala:943) [error] at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) [error] at com.typesafe.tools.mima.plugin.MimaPlugin$.$anonfun$projectSettings$1(MimaPlugin.scala:26) [error] at com.typesafe.tools.mima.plugin.MimaPlugin$.$anonfun$projectSettings$1$adapted(MimaPlugin.scala:25) [error] at scala.Function1.$anonfun$compose$1(Function1.scala:49) [error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63) [error] at sbt.std.Transform$$anon$4.work(Transform.scala:69) [error] at sbt.Execute.$anonfun$submit$2(Execute.scala:283) [error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24) [error] at sbt.Execute.work(Execute.scala:292) [error] at sbt.Execute.$anonfun$submit$1(Execute.scala:283) [error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265) [error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:65) [error] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [error] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [error] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [error] at java.lang.Thread.run(Thread.java:750) [error] (core / mimaReportBinaryIssues) Failed binary compatibility check against org.apache.spark:spark-core_2.13:3.4.0! Found 1 potential problems (filtered 4019) [error] Total time: 74 s (01:14), completed 2023-8-14 13:05:25 ``` 3. with this pr, mima check pass ``` gh pr checkout 42479 dev/change-scala-version.sh 2.13 dev/mima -Pscala-2.13 ``` ``` [success] Total time: 99 s (01:39), completed 2023-8-14 13:10:13 ``` Closes apache#42479 from LuciferYang/SPARK-44798. Authored-by: yangjie01 <[email protected]> Signed-off-by: yangjie01 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This pr aims add a new
ProblemFilterstoMimaExcludes.scalato fix mima check for Scala 2.13 after SPARK-44705 merged.Why are the changes needed?
Scala 2.13's daily tests have been failing the mima check for several days:
Does this PR introduce any user-facing change?
No
How was this patch tested?