Skip to content

ZEPPELIN-262 Use spark-submit to run spark interpreter process #270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Leemoonsoo wants to merge 7 commits into from
Closed

ZEPPELIN-262 Use spark-submit to run spark interpreter process #270

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2d27e9c
use spark-submit to run spark interpreter process when SPARK_HOME is …
Leemoonsoo Sep 2, 2015
5d3154e
Remove clean. otherwise mvn clean package will remove interpreter/spa…
Leemoonsoo Sep 2, 2015
cac2bb8
Take care classpath of SparkIMain
Leemoonsoo Sep 2, 2015
c9418c6
Bring back some entries with more commments
Leemoonsoo Sep 2, 2015
d4acd1b
Add PYTHONPATH
Leemoonsoo Sep 3, 2015
a8a3440
handle spark.files correctly for pyspark when spark-submit is used
Leemoonsoo Sep 4, 2015
4eb0848
export and check SPARK_SUBMIT
Leemoonsoo Sep 5, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,7 @@ If you set `SPARK_HOME`, you should deploy spark binary on the same location to
Yarn

# ./conf/zeppelin-env.sh
export HADOOP_CONF_DIR=/path/to/hadoop_conf_dir

`HADOOP_CONF_DIR` should contains yarn-site.xml and core-site.xml.
export SPARK_HOME=/path/to/spark_dir

### Run
./bin/zeppelin-daemon.sh start
Expand Down
94 changes: 37 additions & 57 deletions bin/interpreter.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,77 +72,57 @@ fi

# set spark related env variables
if [[ "${INTERPRETER_ID}" == "spark" ]]; then
# add Hadoop jars into classpath
if [[ -n "${HADOOP_HOME}" ]]; then
# Apache
addEachJarInDir "${HADOOP_HOME}/share"

# CDH
addJarInDir "${HADOOP_HOME}"
addJarInDir "${HADOOP_HOME}/lib"
fi

# autodetect HADOOP_CONF_HOME by heuristic
if [[ -n "${HADOOP_HOME}" ]] && [[ -z "${HADOOP_CONF_DIR}" ]]; then
if [[ -d "${HADOOP_HOME}/etc/hadoop" ]]; then
export HADOOP_CONF_DIR="${HADOOP_HOME}/etc/hadoop"
elif [[ -d "/etc/hadoop/conf" ]]; then
export HADOOP_CONF_DIR="/etc/hadoop/conf"
fi
fi

if [[ -n "${HADOOP_CONF_DIR}" ]] && [[ -d "${HADOOP_CONF_DIR}" ]]; then
ZEPPELIN_CLASSPATH+=":${HADOOP_CONF_DIR}"
fi

# add Spark jars into classpath
if [[ -n "${SPARK_HOME}" ]]; then
addJarInDir "${SPARK_HOME}/lib"
PYSPARKPATH="${SPARK_HOME}/python:${SPARK_HOME}/python/lib/pyspark.zip:${SPARK_HOME}/python/lib/py4j-0.8.2.1-src.zip"
SPARK_SUBMIT="${SPARK_HOME}/bin/spark-submit"
SPARK_APP_JAR="$(ls ${ZEPPELIN_HOME}/interpreter/spark/zeppelin-spark*.jar)"
# This will evantually passes SPARK_APP_JAR to classpath of SparkIMain
ZEPPELIN_CLASSPATH=${SPARK_APP_JAR}

@weipuz weipuz Mar 9, 2016

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello, when I set SPARK_HOME to my external Spark, I found the zeppelin-interpreter-sparkxxx.log file are gone. I digged further and found if I change this line 79 in interpreter.sh ZEPPELIN_CLASSPATH=${SPARK_APP_JAR} to ZEPPELIN_CLASSPATH+=${SPARK_APP_JAR} I get all the Spark interpreter log back. Is this a bug in the code or I miss understood something?
Regards,
Weipu

@AhyoungRyu AhyoungRyu Mar 9, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@weipuz Thanks for digging it !
I noticed that issue too. ( I set SPARK_HOME and can not get spark log file. )
I also changed ZEPPELIN_CLASSPATH=${SPARK_APP_JAR} to ZEPPELIN_CLASSPATH+=${SPARK_APP_JAR} as you said. Then finally I can get my zeppelin-interpreter-spark-***.log file. As long as it's not an intended implementation, I think we need to fix this.

@Leemoonsoo Leemoonsoo Mar 9, 2016

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it need to be fixed 👍

@AhyoungRyu AhyoungRyu Mar 9, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@weipuz @Leemoonsoo I pushed a patch for this issue with HOT FIX tag at #769 : )
Thanks again @weipuz for reporting this.

else
# add Hadoop jars into classpath
if [[ -n "${HADOOP_HOME}" ]]; then
# Apache
addEachJarInDir "${HADOOP_HOME}/share"

# CDH

@vinayshukla vinayshukla Sep 4, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CDH? Is there hadoop distribution specific path?

@Leemoonsoo Leemoonsoo Sep 5, 2015

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's part of heuristic to search and add hadoop jar files

addJarInDir "${HADOOP_HOME}"
addJarInDir "${HADOOP_HOME}/lib"
fi

addJarInDir "${INTERPRETER_DIR}/dep"
PYSPARKPATH="${ZEPPELIN_HOME}/interpreter/spark/pyspark/pyspark.zip:${ZEPPELIN_HOME}/interpreter/spark/pyspark/py4j-0.8.2.1-src.zip"
fi

# autodetect SPARK_CONF_DIR
if [[ -n "${SPARK_HOME}" ]] && [[ -z "${SPARK_CONF_DIR}" ]]; then
if [[ -d "${SPARK_HOME}/conf" ]]; then
SPARK_CONF_DIR="${SPARK_HOME}/conf"
if [[ -z "${PYTHONPATH}" ]]; then
export PYTHONPATH="${PYSPARKPATH}"
else
export PYTHONPATH="${PYTHONPATH}:${PYSPARKPATH}"
fi
unset PYSPARKPATH

# autodetect HADOOP_CONF_HOME by heuristic
if [[ -n "${HADOOP_HOME}" ]] && [[ -z "${HADOOP_CONF_DIR}" ]]; then
if [[ -d "${HADOOP_HOME}/etc/hadoop" ]]; then
export HADOOP_CONF_DIR="${HADOOP_HOME}/etc/hadoop"
elif [[ -d "/etc/hadoop/conf" ]]; then
export HADOOP_CONF_DIR="/etc/hadoop/conf"
fi
fi
fi

# read spark-*.conf if exists
if [[ -d "${SPARK_CONF_DIR}" ]]; then
ls ${SPARK_CONF_DIR}/spark-*.conf > /dev/null 2>&1
if [[ "$?" -eq 0 ]]; then
for file in ${SPARK_CONF_DIR}/spark-*.conf; do
while read -r line; do
echo "${line}" | grep -e "^spark[.]" > /dev/null
if [ "$?" -ne 0 ]; then
# skip the line not started with 'spark.'
continue;
fi
SPARK_CONF_KEY=`echo "${line}" | sed -e 's/\(^spark[^ ]*\)[ \t]*\(.*\)/\1/g'`
SPARK_CONF_VALUE=`echo "${line}" | sed -e 's/\(^spark[^ ]*\)[ \t]*\(.*\)/\2/g'`
export ZEPPELIN_JAVA_OPTS+=" -D${SPARK_CONF_KEY}=\"${SPARK_CONF_VALUE}\""
done < "${file}"
done
if [[ -n "${HADOOP_CONF_DIR}" ]] && [[ -d "${HADOOP_CONF_DIR}" ]]; then
ZEPPELIN_CLASSPATH+=":${HADOOP_CONF_DIR}"
fi
fi

if [[ -z "${PYTHONPATH}" ]]; then
export PYTHONPATH="${PYSPARKPATH}"
else
export PYTHONPATH="${PYTHONPATH}:${PYSPARKPATH}"
export SPARK_CLASSPATH+=":${ZEPPELIN_CLASSPATH}"
fi

unset PYSPARKPATH
fi

export SPARK_CLASSPATH+=":${ZEPPELIN_CLASSPATH}"
CLASSPATH+=":${ZEPPELIN_CLASSPATH}"

${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} -cp ${CLASSPATH} ${ZEPPELIN_SERVER} ${PORT} &
if [[ -n "${SPARK_SUBMIT}" ]]; then
${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path "${CLASSPATH}" --driver-java-options "${JAVA_INTP_OPTS}" ${SPARK_SUBMIT_OPTIONS} ${SPARK_APP_JAR} ${PORT} &
else
${ZEPPELIN_RUNNER} ${JAVA_INTP_OPTS} -cp ${CLASSPATH} ${ZEPPELIN_SERVER} ${PORT} &
fi

pid=$!
if [[ -z "${pid}" ]]; then
return 1;
Expand Down
16 changes: 5 additions & 11 deletions conf/zeppelin-env.sh.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,8 @@
# export ZEPPELIN_IDENT_STRING # A string representing this instance of zeppelin. $USER by default.
# export ZEPPELIN_NICENESS # The scheduling priority for daemons. Defaults to 0.

# export ZEPPELIN_SPARK_USEHIVECONTEXT # Use HiveContext instead of SQLContext if set true. true by default.
# export ZEPPELIN_SPARK_CONCURRENTSQL # Execute multiple SQL concurrently if set true. false by default.
# export ZEPPELIN_SPARK_MAXRESULT # Max number of SparkSQL result to display. 1000 by default.

# Options read in YARN client mode
# export HADOOP_CONF_DIR # yarn-site.xml is located in configuration directory in HADOOP_CONF_DIR.

# Pyspark (supported with Spark 1.2.1 and above)
# To configure pyspark, you need to set spark distribution's path to 'spark.home' property in Interpreter setting screen in Zeppelin GUI
# export PYSPARK_PYTHON # path to the python command. must be the same path on the driver(Zeppelin) and all workers.
# export PYTHONPATH # extra PYTHONPATH.
## Spark configuration
# export SPARK_HOME # When it is defined, load it instead of Zeppelin embedded Spark libraries
# export SPARK_SUBMIT_OPTIONS # options to pass to spark submit. eg) "--driver-memory 512M --executor-memory 1G".
# export ZEPPELIN_SPARK_CONCURRENTSQL # Execute multiple SQL concurrently if set true. false by default.
# export ZEPPELIN_SPARK_MAXRESULT # Max number of SparkSQL result to display. 1000 by default.
11 changes: 0 additions & 11 deletions spark-dependencies/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -689,17 +689,6 @@
</executions>
</plugin>

<plugin>
<artifactId>maven-clean-plugin</artifactId>
<configuration>
<filesets>
<fileset>
<directory>../interpreter/spark/dep</directory>
</fileset>
</filesets>
</configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
Expand Down
41 changes: 29 additions & 12 deletions spark/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,6 @@
<groupId>${project.groupId}</groupId>
<artifactId>zeppelin-interpreter</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>

<dependency>
Expand Down Expand Up @@ -368,23 +367,41 @@

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.8</version>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
</transformers>
</configuration>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
<goal>shade</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/../../interpreter/spark</outputDirectory>
<overWriteReleases>false</overWriteReleases>
<overWriteSnapshots>false</overWriteSnapshots>
<overWriteIfNewer>true</overWriteIfNewer>
<includeScope>runtime</includeScope>
</configuration>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.8</version>
<executions>
<execution>
<phase>package</phase>
<goals>
Expand Down