Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion docs/interpreter/spark.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,10 +195,21 @@ You can also set other Spark properties which are not listed in the table. For a
(ex: http://{{PORT}}-{{SERVICE_NAME}}.{{SERVICE_DOMAIN}})
</td>
</tr>
<td>spark.webui.yarn.useProxy</td>
<tr>
<td>spark.webui.yarn.useProxy</td>
<td>false</td>
<td>whether use yarn proxy url as spark weburl, e.g. http://localhost:8088/proxy/application_1583396598068_0004</td>
</tr>
<tr>
<td>spark.repl.target</td>
<td>jvm-1.6</td>
<td>
Manually specifying the Java version of Spark Interpreter Scala REPL,Available options:<br/>
scala-compile v2.10.7 to v2.11.12 supports "jvm-1.5, jvm-1.6, jvm-1.7 and jvm-1.8", and the default value is jvm-1.6.<br/>
scala-compile v2.10.1 to v2.10.6 supports "jvm-1.5, jvm-1.6, jvm-1.7", and the default value is jvm-1.6.<br/>
scala-compile v2.12.x defaults to jvm-1.8, and only supports jvm-1.8.
</td>
</tr>
</table>

Without any configuration, Spark interpreter works out of box in local mode. But if you want to connect to your Spark cluster, you'll need to follow below two simple steps.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,13 @@ class SparkScala210Interpreter(override val conf: SparkConf,
sparkHttpServer = server
conf.set("spark.repl.class.uri", uri)
}
val target = conf.get("spark.repl.target", "jvm-1.6")

val settings = new Settings()
settings.embeddedDefaults(sparkInterpreterClassLoader)
settings.usejavacp.value = true
settings.target.value = target

this.userJars = getUserJars()
LOGGER.info("UserJars: " + userJars.mkString(File.pathSeparator))
settings.classpath.value = userJars.mkString(File.pathSeparator)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,12 +66,14 @@ class SparkScala211Interpreter(override val conf: SparkConf,
sparkHttpServer = server
conf.set("spark.repl.class.uri", uri)
}
val target = conf.get("spark.repl.target", "jvm-1.6")

val settings = new Settings()
settings.processArguments(List("-Yrepl-class-based",
"-Yrepl-outdir", s"${outputDir.getAbsolutePath}"), true)
settings.embeddedDefaults(sparkInterpreterClassLoader)
settings.usejavacp.value = true
settings.target.value = target

this.userJars = getUserJars()
LOGGER.info("UserJars: " + userJars.mkString(File.pathSeparator))
Expand Down