[SPARK-3787] Assembly jar name is wrong when we build with sbt omitting -Dhadoop.version

[sarutak]

This PR is another solution for When we build with sbt with profile for hadoop and without property for hadoop version like:

sbt/sbt -Phadoop-2.2 assembly

jar name is always used default version (1.0.4).

When we build with maven with same condition for sbt, default version for each profile is used.
For instance, if we build like:

mvn -Phadoop-2.2 package

jar name is used hadoop2.2.0 as a default version of hadoop-2.2.

[SparkQA]

QA tests have started for PR 2647 at commit eebbb7d.

• This patch merges cleanly.

[jkleckner]

Typo cnahged -> changed

[sarutak]

Thanks, I've fixed.

[SparkQA]

QA tests have finished for PR 2647 at commit eebbb7d.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case e: Exception => logError("Source class " + classPath + " cannot be instantiated", e)
  • case class AddWebUIFilter(filterName:String, filterParams: Map[String, String], proxyBase :String)
  • case class LogicalRDD(output: Seq[Attribute], rdd: RDD[Row])(sqlContext: SQLContext)
  • case class PhysicalRDD(output: Seq[Attribute], rdd: RDD[Row]) extends LeafNode
  • case class ExistingRdd(output: Seq[Attribute], rdd: RDD[Row]) extends LeafNode
  • case class SparkLogicalPlan(alreadyPlanned: SparkPlan)(@transient sqlContext: SQLContext)

[SparkQA]

QA tests have started for PR 2647 at commit 5fc1259.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2647 at commit 5fc1259.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

QA tests have started for PR 2647 at commit b2318eb.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2647 at commit b2318eb.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sarutak]

retest this please.

[SparkQA]

QA tests have started for PR 2647 at commit ad1f96e.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2647 at commit ad1f96e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

QA tests have started for PR 2647 at commit c81806b.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2647 at commit c81806b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

I'd hate to have to hard-code more stuff like this. How about just documenting that hadoop.version is required, or failing in this situation? SBT isn't the build of record at this point anyway, so not sure if its assembly output is so relevant.

[sarutak]

Yeah, enforcing to set hadoop.version is reasonable. I'll try it, thanks!

[liancheng]

Is there a programmatic way to read POM properties from SBT? For example, make-distribution.sh uses mvn help:evaluate -Dexpression=hadoop.versoin.

[sarutak]

@srowen @liancheng Thanks for your advice! I found the systematic way which refers hadoop.version property from pom.xml.

[sarutak]

retest this please.

[sarutak]

Jenkins wouldn't pick this PR up so I opened another PR #3046 for this issue.