Skip to content

Remove hardcoded Spark version string to use SBT defined version instead #600

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pierre-borckmans wants to merge 6 commits into from
Closed

Remove hardcoded Spark version string to use SBT defined version instead #600

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
12769f4
Remove hardcoded Spark version - use the version from the manifest (S…
Apr 30, 2014
f7c2298
Add spark version label in ui (master index page)
Apr 30, 2014
d26bc1e
Merge remote-tracking branch 'upstream/master'
Apr 30, 2014
0f88e95
Changed SparkContext SPARK_VERSION to use Utils.getSparkVersion
Apr 30, 2014
142be89
Modified Spark REPL to display Utils.getSparkVersion in Welcome message
Apr 30, 2014
da8886e
Changed getSparkVersion signature to omit the parentheses
Apr 30, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions core/src/main/scala/org/apache/spark/SparkContext.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1253,7 +1253,7 @@ class SparkContext(config: SparkConf) extends Logging {
*/
object SparkContext extends Logging {

private[spark] val SPARK_VERSION = "1.0.0"
private[spark] val SPARK_VERSION = getClass.getPackage.getImplementationVersion
Copy link
Author

@pierre-borckmans pierre-borckmans Apr 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be be:

private[spark] val SPARK_VERSION = Utils.getSparkVersion

instead, I forgot to update it.


private[spark] val SPARK_JOB_DESCRIPTION = "spark.job.description"

Expand Down Expand Up @@ -1565,4 +1565,3 @@ private[spark] class WritableConverter[T](
val writableClass: ClassTag[T] => Class[_ <: Writable],
val convert: Writable => T)
extends Serializable

1 change: 1 addition & 0 deletions core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ private[spark] class MasterPage(parent: MasterWebUI) extends WebUIPage("") {
<div class="row-fluid">
<div class="span12">
<ul class="unstyled">
<li><strong>Version:</strong> {Utils.getSparkVersion}</li>
<li><strong>URL:</strong> {state.uri}</li>
<li><strong>Workers:</strong> {state.workers.size}</li>
<li><strong>Cores:</strong> {state.workers.map(_.cores).sum} Total,
Expand Down
5 changes: 5 additions & 0 deletions core/src/main/scala/org/apache/spark/util/Utils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1037,6 +1037,11 @@ private[spark] object Utils extends Logging {
obj.getClass.getSimpleName.replace("$", "")
}

/** Return the current version of Spark */
def getSparkVersion(): String = {
Copy link
Contributor

@aarondav aarondav Apr 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you intend it to be called without parentheses, it should also be defined without them.

Copy link
Author

@pierre-borckmans pierre-borckmans Apr 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, i'll fix this

getClass.getPackage.getImplementationVersion
Copy link
Contributor

@pwendell pwendell Apr 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this also work in the maven build?

Copy link
Author

@pierre-borckmans pierre-borckmans Apr 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not check as I was not really aware of the Maven build.
Apparently, I always overlooked this page:
http://spark.apache.org/docs/latest/building-with-maven.html

I'll check this now.

Copy link
Author

@pierre-borckmans pierre-borckmans Apr 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to work as well.
In fact, the build system does not matter:
http://docs.oracle.com/javase/7/docs/api/java/lang/Package.html
The version string is retrieved from the JAR manifest.

Copy link
Contributor

@aarondav aarondav Apr 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this manifest property must be correctly set from the build system, but it's good to know if both systems do so.

Copy link
Author

@pierre-borckmans pierre-borckmans Apr 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed.
I confirm that both sbt assembly and mvn package produce a fat jar in which the manifest contains the version.

unzip -p assembly/target/scala-2.10/spark-assembly-1.0.0-SNAPSHOT-hadoop2.0.0-mr1-cdh4.4.0.jar META-INF/MANIFEST.MF

outputs:

Implementation-Vendor: The Apache Software Foundation
Implementation-Title: Spark Project Core
Implementation-Version: 1.0.0-SNAPSHOT
Implementation-Vendor-Id: org.apache.spark
Built-By: ria
Build-Jdk: 1.6.0_27
Specification-Vendor: The Apache Software Foundation
Specification-Title: Spark Project Core
Created-By: Apache Maven 3.0.4
Specification-Version: 1.0.0-SNAPSHOT
Archiver-Version: Plexus Archiver

in both cases

}

/** Return an option that translates JNothing to None */
def jsonOption(json: JValue): Option[JValue] = {
json match {
Expand Down
6 changes: 4 additions & 2 deletions repl/src/main/scala/org/apache/spark/repl/SparkILoopInit.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ import scala.reflect.internal.util.Position
import scala.util.control.Exception.ignoring
import scala.tools.nsc.util.stackTraceString

import org.apache.spark.util.Utils

/**
* Machinery for the asynchronous initialization of the repl.
*/
Expand All @@ -26,9 +28,9 @@ trait SparkILoopInit {
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.0.0-SNAPSHOT
/___/ .__/\_,_/_/ /_/\_\ version %s
/_/
""")
""").format(Utils.getSparkVersion)
import Properties._
val welcomeMsg = "Using Scala %s (%s, Java %s)".format(
versionString, javaVmName, javaVersion)
Expand Down