[SPARK-3437][BUILD] Support crossbuilding in maven. With new scala-install-plugin.

[ScrapCodes]

Since this plugin is not deployed anywhere, for anyone trying this patch has to publish it locally by cloning the following repo https://github.com/ScrapCodes/scala-install-plugin. And then running mvn install.

[SparkQA]

QA tests have started for PR 2357 at commit 609dd98.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2357 at commit 609dd98.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

This sort of conflicts with parts of #2329. Would it be possible to add a flag to the new plugin so that it can automatically skip projects where a certain property is set? That way you just set "scala.install.skip" instead of copying a whole lot of XML for each module you don't want to install.

No idea if that's possible, and I'm aware that's not very maven-approved, but just a suggestion.

[ScrapCodes]

It will not conflict with that, it works the same way as maven install plugin. (accidentally deleted my previous post. :/)

[pwendell]

Hey @ScrapCodes rather than forking the existing install plug-in, could we write our own plug-in that runs before the install plug-in? Forking seems like it will be difficult of e.g. the maven install plug-in updates.

[ScrapCodes]

you are right about that, but we are forking it because we want an install plugin modified. If it was possible to run a plugin just before install and install plugin magically does the job correctly. That would have been nicer. But that isn't possible unless I try to hack things using reflection. Its like maven keeps its copy of objects before letting plugins use it.

[ScrapCodes]

I am anyway trying more options to not need to modify maven-install-plugin.

[srowen]

• If you bind a plugin to install phase and declare it before maven-install-plugin, will it happen to respect the ordering?
• This is arguably something that can happen before install, in package phase? There's a prepare-package phase as well

[ScrapCodes]

Hi @srowen, like I said. We can run a plugin before maven install - no questions about that. But since maven install gets a copy of STATE(somehow via guice). Altering it in another maven plugin does not help much. So as of now, I do not see a way around not modifying maven install plugin.

[srowen]

I see, I thought you mentioned above that running a plugin before install would work. It sounds like there is some internal state of the plugin you need to modify, OK. I don't know, maybe it's useful to elaborate this and see if anyone else can see a workaround.

It is worth keeping track of alternatives when evaluating just how far to hack this. For example: Scala 2.11 support could be bound up with Spark 2.x support. Or it can be in a branch that is maintained over a few minor versions, which is not such a big deal if the delta is just flipping "2.10" -> "2.11" in many places.

[ScrapCodes]

Turns out this is not very useful either.

[SparkQA]

QA tests have started for PR 2357 at commit 609dd98.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2357 at commit 609dd98.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21183/

[AmplabJenkins]

Can one of the admins verify this patch?