Skip to content

Conversation

@sarutak
Copy link
Member

@sarutak sarutak commented Nov 19, 2021

What changes were proposed in this pull request?

This PR adds more MiMa exclusion rules for Scala 2.13.
#34649 partially resolved the compatibility issue but additional 3 compatibility problems are raised.

$ build/sbt clean
$ dev/change-scala-version.sh 2.13
$ build/sbt -Pscala-2.13 clean
$ dev/mima

...
[error] spark-core: Failed binary compatibility check against org.apache.spark:spark-core_2.13:3.2.0! Found 3 potential problems (filtered 910)
[error]  * synthetic method copy$default$8()scala.collection.mutable.ListBuffer in class org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments has a different result type in current version, where it is scala.Option rather than scala.collection.mutable.ListBuffer
[error]    filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.copy$default$8")
[error]  * synthetic method copy$default$9()scala.Option in class org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments has a different result type in current version, where it is Int rather than scala.Option
[error]    filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.copy$default$9")
[error]  * the type hierarchy of object org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments is different in current version. Missing types {scala.runtime.AbstractFunction10}
[error]    filter with: ProblemFilters.exclude[MissingTypesProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend$Arguments$")
...

Why are the changes needed?

To keep the build stable.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Confirmed MiMa passed.

$ build/sbt clean
$ dev/change-scala-version.sh 2.13
$ build/sbt -Pscala-2.13 clean
$ dev/mima

@github-actions github-actions bot added the BUILD label Nov 19, 2021
@sarutak
Copy link
Member Author

sarutak commented Nov 19, 2021

@dongjoon-hyun Thank you for letting me know this issue! After I cleaned my repository and ran dev/mima, I observed this issue too.

// The followings are necessary for Scala 2.13.
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.*"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.*"),
ProblemFilters.exclude[MissingTypesProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend$Arguments$")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this also come from SPARK-35672?

Copy link
Member Author

@sarutak sarutak Nov 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think so.The parameter list of the case class Arguments was changed (userClassPath was removed) in SPARK-35672, which caused the class hierarchy of the class $Arguments$.

$ javap core/target/scala-2.13/classes/org/apache/spark/executor/CoarseGrainedExecutorBackend\$Arguments\$.class 
Compiled from "CoarseGrainedExecutorBackend.scala"
public class org.apache.spark.executor.CoarseGrainedExecutorBackend$Arguments$ extends scala.runtime.AbstractFunction9<java.lang.String, java.lang.String, java.lang.String, java.lang.String, java.lang.Object, java.lang.String, scala.Option<java.lang.String>, scala.Option<java.lang.String>, java.lang.Object, org.apache.spark.executor.CoarseGrainedExecutorBackend$Arguments> implements java.io.Serializable {

@SparkQA
Copy link

SparkQA commented Nov 19, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49915/

@SparkQA
Copy link

SparkQA commented Nov 19, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49915/

@SparkQA
Copy link

SparkQA commented Nov 19, 2021

Test build #145443 has finished for PR 34664 at commit d20b841.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@sarutak
Copy link
Member Author

sarutak commented Nov 19, 2021

Merging to master. Thank you @dongjoon-hyun for the review !

@sarutak sarutak closed this in 4f20898 Nov 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants