[SPARK-28759][BUILD] Upgrade scala-maven-plugin to 4.2.0 and fix build profile on AppVeyor

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to upgrade scala-maven-plugin from 3.4.4 to 4.2.0.

Upgrade to 4.1.1 was reverted due to unexpected build failure on AppVeyor.

The root cause seems to be an issue specific to AppVeyor - loading the system library 'kernel32.dll' seems being failed.

Suppressed: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.platform.win32.Kernel32
        at sbt.internal.io.WinMilli$.getHandle(Milli.scala:264)
        at sbt.internal.io.WinMilli$.getModifiedTimeNative(Milli.scala:289)
        at sbt.internal.io.WinMilli$.getModifiedTimeNative(Milli.scala:260)
        at sbt.internal.io.MilliNative.getModifiedTime(Milli.scala:61)
        at sbt.internal.io.Milli$.getModifiedTime(Milli.scala:360)
        at sbt.io.IO$.$anonfun$getModifiedTimeOrZero$1(IO.scala:1373)
        at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
        at sbt.internal.io.Retry$.liftedTree2$1(Retry.scala:38)
        at sbt.internal.io.Retry$.impl$1(Retry.scala:38)
        at sbt.internal.io.Retry$.apply(Retry.scala:52)
        at sbt.internal.io.Retry$.apply(Retry.scala:24)
        at sbt.io.IO$.getModifiedTimeOrZero(IO.scala:1373)
        at sbt.internal.inc.caching.ClasspathCache$.fromCacheOrHash$1(ClasspathCache.scala:44)
        at sbt.internal.inc.caching.ClasspathCache$.$anonfun$hashClasspath$1(ClasspathCache.scala:53)
        at scala.collection.parallel.mutable.ParArray$Map.leaf(ParArray.scala:659)
        at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)
        at scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)
        at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)
        at scala.collection.parallel.mutable.ParArray$Map.tryLeaf(ParArray.scala:650)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal(Tasks.scala:170)
        ... 25 more

By setting -Djna.nosys=true, it directly loads the library from the jar instead of system's.

In this way, the build seems working fine.

Why are the changes needed?

It upgrades the plugin to fix bugs and fixes the CI build.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

It was tested at #25497

[HyukjinKwon]

cc @dongjoon-hyun, @srowen and @wangyum

[HyukjinKwon]

4.2.0 has davidB/scala-maven-plugin#358 fix too FWIW.

[srowen]

Oh nice, so this possibly enables cross-compilation from JDK 11 to JDK 8 now? great

[dongjoon-hyun]

+1, LGTM. (Pending Jenkins)
Thank you for the upstream scala-maven-plugin bug fix and updating this, @HyukjinKwon !

[SparkQA]

Test build #109950 has finished for PR 25633 at commit 0fd34b3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Merged to master. Thank you all!

[felixcheung]

do we need to document this as "building spark on windows" then? ;)

[HyukjinKwon]

Oh, do you mean https://github.com/apache/spark/blob/master/R/WINDOWS.md#building-sparkr-on-windows ? Actually this flag fixed an issue specific to Appveyor. So regular Windows build won't need this flag.

Actually, I have some more additional and required information to document there. Building Spark on Windows requires bash (by Windows 11 or Cygwin) due to, at least, here:

spark/core/pom.xml

Line 464 in 9ea37b0

<arg value="${project.basedir}/../build/spark-build-info"/>

[HyukjinKwon]

Let me document this it later in few days.