Skip to content

Minor update for spark_cluster_mode.md #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
astroshim merged 2 commits into from
Aug 11, 2016
Merged

Minor update for spark_cluster_mode.md #9

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9e9390c
Minor update for spark_cluster_mode.md
AhyoungRyu Aug 11, 2016
cde5f8d
Modify document description so that this docs can be searched
AhyoungRyu Aug 11, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified docs/assets/themes/zeppelin/img/docs-img/zeppelin_yarn_conf.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
29 changes: 14 additions & 15 deletions docs/install/spark_cluster_mode.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
layout: page
title: "Apache Zeppelin on Spark cluster mode"
description: ""
description: "This document will guide you how you can build and configure the environment on 3 types of Spark cluster manager with Apache Zeppelin using docker scripts."
group: install
---
<!--
Expand Down Expand Up @@ -56,12 +56,12 @@ spark_standalone bash;
```

### 3. Configure Spark interpreter in Zeppelin
Set Spark master as `spark://localhost:7077` in Zeppelin **Interpreters** setting page.
Set Spark master as `spark://<hostname>:7077` in Zeppelin **Interpreters** setting page.

<img src="../assets/themes/zeppelin/img/docs-img/standalone_conf.png" />

### 4. Run Zeppelin with Spark interpreter
After running single paragraph with Spark interpreter in Zeppelin, browse `https://localhost:8080` and check whether Spark cluster is running well or not.
After running single paragraph with Spark interpreter in Zeppelin, browse `https://<hostname>:8080` and check whether Spark cluster is running well or not.

<img src="../assets/themes/zeppelin/img/docs-img/spark_ui.png" />

Expand All @@ -73,7 +73,7 @@ ps -ef | grep spark


## Spark on Yarn mode
You can simply set up [Spark on Yarn](http://spark.apache.org/docs/latest/running-on-yarn.html) docker environment with below steps.
You can simply set up [Spark on Yarn](http://spark.apache.org/docs/latest/running-on-yarn.html) environment with below steps using Docker scripts.

> **Note :** Since Apache Zeppelin and Spark use same `8080` port for their web UI, you might need to change `zeppelin.server.port` in `conf/zeppelin-site.xml`.

Expand Down Expand Up @@ -113,31 +113,30 @@ docker run -it \

### 3. Verify running Spark on Yarn.

You can simply verify the processes of Spark and Yarn is running well in Docker with below command.

You can simply verify the processes of Spark and Yarn are running well in Docker with below command.

```
ps -ef
ps -ef | grep spark
```

and also You can see the hdfs web ui on `http://localhost:50070/` and yarn ui on `http://localhost:8088/cluster` and spark ui on `http://localhost:8080/`.

You can also check each application web UI for HDFS on `http://<hostname>:50070/`, YARN on `http://<hostname>:8088/cluster` and Spark on `http://<hostname>:8080/`.

### 4. Configure Spark interpreter in Zeppelin
Set following configurations to the `zeppelin-env.sh`.
Set following configurations to `conf/zeppelin-env.sh`.

```
export MASTER=yarn-client
export HADOOP_CONF_DIR=[PATH OF HADOOP CONF]
export SPARK_HOME=[PATH OF SPARK HOME]
export HADOOP_CONF_DIR=[your_hadoop_conf_path]
export SPARK_HOME=[your_spark_home_path]
```
Hadoop configuration path(HADOOP_CONF_DIR) is the `scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf`.

Please make sure the spark master as `yarn-client` in Zeppelin **Interpreters** setting page.
`HADOOP_CONF_DIR`(Hadoop configuration path) is defined in `/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf`.

Don't forget to set Spark `master` as `yarn-client` in Zeppelin **Interpreters** setting page like below.

<img src="../assets/themes/zeppelin/img/docs-img/zeppelin_yarn_conf.png" />

### 5. Run Zeppelin with Spark interpreter
After running single paragraph with Spark interpreter in Zeppelin, browse `http://localhost:8088/cluster/apps` and check zeppelin application running well or not.
After running a single paragraph with Spark interpreter in Zeppelin, browse `http://<hostname>:8088/cluster/apps` and check Zeppelin application is running well or not.

<img src="../assets/themes/zeppelin/img/docs-img/yarn_applications.png" />