'); + + results.forEach(function(result) { + var item = loaded_data[result.ref]; + var appendString = ''+item.title+'
'+prod_url+base_url+item.url.trim()+'
'+item.excerpt+'
'; + + $search_results.append(appendString); + }); + } else { + $search_results.html('
Your search did not match any documents.
Make sure that all words are spelled correctly or try more general keywords.
+
+### 4. Run Zeppelin with Spark interpreter
+After running single paragraph with Spark interpreter in Zeppelin, browse `https://
+
+You can also simply verify that Spark is running well in Docker with below command.
+
+```
+ps -ef | grep spark
+```
+
+
+## Spark on YARN mode
+You can simply set up [Spark on YARN](http://spark.apache.org/docs/latest/running-on-yarn.html) docker environment with below steps.
+
+> **Note :** Since Apache Zeppelin and Spark use same `8080` port for their web UI, you might need to change `zeppelin.server.port` in `conf/zeppelin-site.xml`.
+
+### 1. Build Docker file
+You can find docker script files under `scripts/docker/spark-cluster-managers`.
+
+```
+cd $ZEPPELIN_HOME/scripts/docker/spark-cluster-managers/spark_yarn
+docker build -t "spark_yarn" .
+```
+
+### 2. Run docker
+
+```
+docker run -it \
+ -p 5000:5000 \
+ -p 9000:9000 \
+ -p 9001:9001 \
+ -p 8088:8088 \
+ -p 8042:8042 \
+ -p 8030:8030 \
+ -p 8031:8031 \
+ -p 8032:8032 \
+ -p 8033:8033 \
+ -p 8080:8080 \
+ -p 7077:7077 \
+ -p 8888:8888 \
+ -p 8081:8081 \
+ -p 50010:50010 \
+ -p 50075:50075 \
+ -p 50020:50020 \
+ -p 50070:50070 \
+ --name spark_yarn \
+ -h sparkmaster \
+ spark_yarn bash;
+```
+
+### 3. Verify running Spark on YARN.
+
+You can simply verify the processes of Spark and YARN are running well in Docker with below command.
+
+```
+ps -ef
+```
+
+You can also check each application web UI for HDFS on `http://
+
+### 5. Run Zeppelin with Spark interpreter
+After running a single paragraph with Spark interpreter in Zeppelin, browse `http://
+
+
+
+## Spark on Mesos mode
+You can simply set up [Spark on Mesos](http://spark.apache.org/docs/latest/running-on-mesos.html) docker environment with below steps.
+
+
+### 1. Build Docker file
+
+```
+cd $ZEPPELIN_HOME/scripts/docker/spark-cluster-managers/spark_mesos
+docker build -t "spark_mesos" .
+```
+
+
+### 2. Run docker
+
+```
+docker run --net=host -it \
+-p 8080:8080 \
+-p 7077:7077 \
+-p 8888:8888 \
+-p 8081:8081 \
+-p 8082:8082 \
+-p 5050:5050 \
+-p 5051:5051 \
+-p 4040:4040 \
+-h sparkmaster \
+--name spark_mesos \
+spark_mesos bash;
+```
+
+### 3. Verify running Spark on Mesos.
+
+You can simply verify the processes of Spark and Mesos are running well in Docker with below command.
+
+```
+ps -ef
+```
+
+You can also check each application web UI for Mesos on `http://
+
+
+### 5. Run Zeppelin with Spark interpreter
+After running a single paragraph with Spark interpreter in Zeppelin, browse `http://
+
diff --git a/docs/install/upgrade.md b/docs/install/upgrade.md
index a831eb98d4e..60a8f885a5b 100644
--- a/docs/install/upgrade.md
+++ b/docs/install/upgrade.md
@@ -1,7 +1,7 @@
---
layout: page
-title: "Manual upgrade procedure for Zeppelin"
-description: ""
+title: "Manual Zeppelin version upgrade procedure"
+description: "This document will guide you through a procedure of manual upgrade your Apache Zeppelin instance to a newer version. Apache Zeppelin keeps backward compatibility for the notebook file format."
group: install
---
{% include JB/setup %}
-# Vagrant Virtual Machine for Apache Zeppelin
+# Apache Zeppelin on Vagrant Virtual Machine
## Overview
-Apache Zeppelin distribution includes a scripts directory
+Apache Zeppelin distribution includes a script directory
`scripts/vagrant/zeppelin-dev`
-This script creates a virtual machine that launches a repeatable, known set of core dependencies required for developing Zeppelin. It can also be used to run an existing Zeppelin build if you don't plan to build from source.
+This script creates a virtual machine that launches a repeatable, known set of core dependencies required for developing Zeppelin. It can also be used to run an existing Zeppelin build if you don't plan to build from source.
For PySpark users, this script includes several helpful [Python Libraries](#python-extras).
For SparkR users, this script includes several helpful [R Libraries](#r-extras).
@@ -88,7 +88,7 @@ By default, Vagrant will share your project directory (the directory with the Va
Running the following commands in the guest machine should display these expected versions:
`node --version` should report *v0.12.7*
-`mvn --version` should report *Apache Maven 3.3.3* and *Java version: 1.7.0_85*
+`mvn --version` should report *Apache Maven 3.3.9* and *Java version: 1.7.0_85*
The virtual machine consists of:
@@ -96,7 +96,7 @@ The virtual machine consists of:
- Node.js 0.12.7
- npm 2.11.3
- ruby 1.9.3 + rake, make and bundler (only required if building jekyll documentation)
- - Maven 3.3.3
+ - Maven 3.3.9
- Git
- Unzip
- libfontconfig to avoid phatomJs missing dependency issues
diff --git a/docs/install/yarn_install.md b/docs/install/yarn_install.md
index 466007d2daa..e2427546a66 100644
--- a/docs/install/yarn_install.md
+++ b/docs/install/yarn_install.md
@@ -1,7 +1,7 @@
---
layout: page
title: "Install Zeppelin to connect with existing YARN cluster"
-description: ""
+description: "This page describes how to pre-configure a bare metal node, configure Apache Zeppelin and connect it to existing YARN cluster running Hortonworks flavour of Hadoop."
group: install
---
{% include JB/setup %}
# Alluxio Interpreter for Apache Zeppelin
diff --git a/docs/interpreter/bigquery.md b/docs/interpreter/bigquery.md
index 1e92aa98757..7ebe2e2fda8 100644
--- a/docs/interpreter/bigquery.md
+++ b/docs/interpreter/bigquery.md
@@ -1,9 +1,23 @@
---
layout: page
-title: "BigQuery Interpreter"
-description: ""
+title: "BigQuery Interpreter for Apache Zeppelin"
+description: "BigQuery is a highly scalable no-ops data warehouse in the Google Cloud Platform."
group: interpreter
---
+
+{% include JB/setup %}
# BigQuery Interpreter for Apache Zeppelin
diff --git a/docs/interpreter/cassandra.md b/docs/interpreter/cassandra.md
index 89b4ea5806f..5d8929bd588 100644
--- a/docs/interpreter/cassandra.md
+++ b/docs/interpreter/cassandra.md
@@ -1,9 +1,22 @@
---
layout: page
-title: "Cassandra Interpreter"
-description: "Cassandra Interpreter"
+title: "Cassandra CQL Interpreter for Apache Zeppelin"
+description: "Apache Cassandra database is the right choice when you need scalability and high availability without compromising performance."
group: interpreter
---
+
{% include JB/setup %}
# Cassandra CQL Interpreter for Apache Zeppelin
diff --git a/docs/interpreter/elasticsearch.md b/docs/interpreter/elasticsearch.md
index 7c6e2cec4de..0e59cae5051 100644
--- a/docs/interpreter/elasticsearch.md
+++ b/docs/interpreter/elasticsearch.md
@@ -1,9 +1,22 @@
---
layout: page
-title: "Elasticsearch Interpreter"
-description: ""
+title: "Elasticsearch Interpreter for Apache Zeppelin"
+description: "Elasticsearch is a highly scalable open-source full-text search and analytics engine."
group: interpreter
---
+
{% include JB/setup %}
# Elasticsearch Interpreter for Apache Zeppelin
diff --git a/docs/interpreter/flink.md b/docs/interpreter/flink.md
index 95ed6410958..3d065465dbd 100644
--- a/docs/interpreter/flink.md
+++ b/docs/interpreter/flink.md
@@ -1,9 +1,22 @@
---
layout: page
-title: "Flink Interpreter"
-description: ""
+title: "Flink Interpreter for Apache Zeppelin"
+description: "Apache Flink is an open source platform for distributed stream and batch data processing."
group: interpreter
---
+
{% include JB/setup %}
# Flink interpreter for Apache Zeppelin
diff --git a/docs/interpreter/geode.md b/docs/interpreter/geode.md
index 24abf2ea561..f833f9d8911 100644
--- a/docs/interpreter/geode.md
+++ b/docs/interpreter/geode.md
@@ -1,9 +1,22 @@
---
layout: page
-title: "Geode OQL Interpreter"
-description: ""
+title: "Geode/Gemfire OQL Interpreter for Apache Zeppelin"
+description: "Apache Geode (incubating) provides a database-like consistency model, reliable transaction processing and a shared-nothing architecture to maintain very low latency performance with high concurrency processing."
group: interpreter
---
+
{% include JB/setup %}
# Geode/Gemfire OQL Interpreter for Apache Zeppelin
diff --git a/docs/interpreter/hbase.md b/docs/interpreter/hbase.md
index 4598f2fe4be..12e05174359 100644
--- a/docs/interpreter/hbase.md
+++ b/docs/interpreter/hbase.md
@@ -1,9 +1,22 @@
---
layout: page
-title: "HBase Shell Interpreter"
-description: ""
+title: "HBase Shell Interpreter for Apache Zeppelin"
+description: "HBase Shell is a JRuby IRB client for Apache HBase. This interpreter provides all capabilities of Apache HBase shell within Apache Zeppelin."
group: interpreter
---
+
{% include JB/setup %}
# HBase Shell Interpreter for Apache Zeppelin
diff --git a/docs/interpreter/hdfs.md b/docs/interpreter/hdfs.md
index 7369ba0feaa..d7b7bf885d8 100644
--- a/docs/interpreter/hdfs.md
+++ b/docs/interpreter/hdfs.md
@@ -1,9 +1,22 @@
---
layout: page
-title: "HDFS File System Interpreter"
-description: ""
+title: "HDFS File System Interpreter for Apache Zeppelin"
+description: "Hadoop File System is a distributed, fault tolerant file system part of the hadoop project and is often used as storage for distributed processing engines like Hadoop MapReduce and Apache Spark or underlying file systems like Alluxio."
group: interpreter
---
+
{% include JB/setup %}
# HDFS File System Interpreter for Apache Zeppelin
diff --git a/docs/interpreter/hive.md b/docs/interpreter/hive.md
index 037055b2654..feeb7a3f536 100644
--- a/docs/interpreter/hive.md
+++ b/docs/interpreter/hive.md
@@ -1,9 +1,22 @@
---
layout: page
-title: "Hive Interpreter"
-description: ""
+title: "Hive Interpreter for Apache Zeppelin"
+description: "Apache Hive data warehouse software facilitates querying and managing large datasets residing in distributed storage. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. At the same time this language also allows traditional map/reduce programmers to plug in their custom mappers and reducers when it is inconvenient or inefficient to express this logic in HiveQL."
group: interpreter
---
+
{% include JB/setup %}
# Hive Interpreter for Apache Zeppelin
diff --git a/docs/interpreter/ignite.md b/docs/interpreter/ignite.md
index d157d3121f2..a085dc1aef2 100644
--- a/docs/interpreter/ignite.md
+++ b/docs/interpreter/ignite.md
@@ -1,9 +1,22 @@
---
layout: page
-title: "Ignite Interpreter"
-description: "Ignite user guide"
+title: "Ignite Interpreter for Apache Zeppelin"
+description: "Apache Ignite in-memory Data Fabric is a high-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time, orders of magnitude faster than possible with traditional disk-based or flash technologies."
group: interpreter
---
+
{% include JB/setup %}
# Ignite Interpreter for Apache Zeppelin
diff --git a/docs/interpreter/jdbc.md b/docs/interpreter/jdbc.md
index f7b60a2b0ba..72963c092b3 100644
--- a/docs/interpreter/jdbc.md
+++ b/docs/interpreter/jdbc.md
@@ -1,13 +1,25 @@
---
layout: page
-title: "Generic JDBC Interpreter"
-description: "JDBC user guide"
+title: "Generic JDBC Interpreter for Apache Zeppelin"
+description: "Generic JDBC Interpreter lets you create a JDBC connection to any data source. You can use Postgres, MySql, MariaDB, Redshift, Apache Hive, Apache Phoenix, Apache Drill and Apache Tajo using JDBC interpreter."
group: interpreter
---
+
{% include JB/setup %}
-
-# Generic JDBC Interpreter for Apache Zeppelin
+# Generic JDBC Interpreter for Apache Zeppelin
diff --git a/docs/interpreter/lens.md b/docs/interpreter/lens.md
index 20b2c0ccb9d..b929220339a 100644
--- a/docs/interpreter/lens.md
+++ b/docs/interpreter/lens.md
@@ -1,9 +1,22 @@
---
layout: page
-title: "Lens Interpreter"
-description: "Lens user guide"
+title: "Lens Interpreter for Apache Zeppelin"
+description: "Apache Lens provides an Unified Analytics interface. Lens aims to cut the Data Analytics silos by providing a single view of data across multiple tiered data stores and optimal execution environment for the analytical query. It seamlessly integrates Hadoop with traditional data warehouses to appear like one."
group: interpreter
---
+
{% include JB/setup %}
# Lens Interpreter for Apache Zeppelin
diff --git a/docs/interpreter/livy.md b/docs/interpreter/livy.md
index 2f47364aae8..1c040b9ff22 100644
--- a/docs/interpreter/livy.md
+++ b/docs/interpreter/livy.md
@@ -1,9 +1,22 @@
---
layout: page
-title: "Livy Interpreter"
-description: ""
+title: "Livy Interpreter for Apache Zeppelin"
+description: "Livy is an open source REST interface for interacting with Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in YARN."
group: interpreter
---
+
{% include JB/setup %}
# Livy Interpreter for Apache Zeppelin
@@ -30,7 +43,7 @@ We added some common configurations for spark, and you can set any configuration
This link contains all spark configurations: http://spark.apache.org/docs/latest/configuration.html#available-properties.
And instead of starting property with `spark.` it should be replaced with `livy.spark.`.
Example: `spark.master` to `livy.spark.master`
-
+
| Property | @@ -50,7 +63,7 @@ Example: `spark.master` to `livy.spark.master`|||
|---|---|---|---|
| zeppelin.livy.spark.maxResult | 1000 | -Max number of SparkSQL result to display. | +Max number of Spark SQL result to display. |
| livy.spark.driver.cores | @@ -102,8 +115,31 @@ Example: `spark.master` to `livy.spark.master`Upper bound for the number of executors. | ||
| livy.spark.jars.packages | ++ | Adding extra libraries to livy interpreter | +
| Property | +Example | +Description | +
|---|---|---|
| livy.spark.jars.packages | +io.spray:spray-json_2.10:1.3.1 | +Adding extra libraries to livy interpreter | +
diff --git a/docs/interpreter/r.md b/docs/interpreter/r.md
index bc94fff9893..2b224917ceb 100644
--- a/docs/interpreter/r.md
+++ b/docs/interpreter/r.md
@@ -1,9 +1,22 @@
---
layout: page
-title: "R Interpreter"
-description: ""
+title: "R Interpreter for Apache Zeppelin"
+description: "R is a free software environment for statistical computing and graphics."
group: interpreter
---
+
{% include JB/setup %}
# R Interpreter for Apache Zeppelin
@@ -104,7 +117,7 @@ And vice versa:
* The `knitr` environment is persistent. If you run a chunk from Zeppelin that changes a variable, then run the same chunk again, the variable has already been changed. Use immutable variables.
-* (Note that `%spark.r` and `$r` are two different ways of calling the same interpreter, as are `%spark.knitr` and `%knitr`. By default, Zeppelin puts the R interpreters in the `%spark.` Interpreter Group.
+* (Note that `%spark.r` and `%r` are two different ways of calling the same interpreter, as are `%spark.knitr` and `%knitr`. By default, Zeppelin puts the R interpreters in the `%spark.` Interpreter Group.
* Using the `%r` interpreter, if you return a data.frame, HTML, or an image, it will dominate the result. So if you execute three commands, and one is `hist()`, all you will see is the histogram, not the results of the other commands. This is a Zeppelin limitation.
diff --git a/docs/interpreter/scalding.md b/docs/interpreter/scalding.md
index d2e86e742d9..22027f22dee 100644
--- a/docs/interpreter/scalding.md
+++ b/docs/interpreter/scalding.md
@@ -1,9 +1,22 @@
---
layout: page
-title: "Scalding Interpreter"
-description: ""
+title: "Scalding Interpreter for Apache Zeppelin"
+description: "Scalding is an open source Scala library for writing MapReduce jobs."
group: interpreter
---
+
{% include JB/setup %}
# Scalding Interpreter for Apache Zeppelin
diff --git a/docs/interpreter/shell.md b/docs/interpreter/shell.md
index abcaf1c99d1..50a8ded6ccb 100644
--- a/docs/interpreter/shell.md
+++ b/docs/interpreter/shell.md
@@ -1,9 +1,22 @@
---
layout: page
-title: "Shell Interpreter"
-description: "Shell Interpreter"
+title: "Shell interpreter for Apache Zeppelin"
+description: "Shell interpreter uses Apache Commons Exec to execute external processes."
group: interpreter
---
+
{% include JB/setup %}
# Shell interpreter for Apache Zeppelin
diff --git a/docs/interpreter/spark.md b/docs/interpreter/spark.md
index 81f48443a93..90ac9b0b948 100644
--- a/docs/interpreter/spark.md
+++ b/docs/interpreter/spark.md
@@ -1,12 +1,24 @@
---
layout: page
-title: "Spark Interpreter Group"
-description: ""
+title: "Apache Spark Interpreter for Apache Zeppelin"
+description: "Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs."
group: interpreter
---
+
{% include JB/setup %}
-
# Spark Interpreter for Apache Zeppelin
@@ -79,7 +91,7 @@ You can also set other Spark properties which are not listed in the table. For a
ex) 512m, 32g
+
+
+
+ + + +
+ + +
diff --git a/docs/search_data.json b/docs/search_data.json
new file mode 100644
index 00000000000..3df19e958d6
--- /dev/null
+++ b/docs/search_data.json
@@ -0,0 +1,17 @@
+---
+layout: null
+---
+{
+ {% for page in site.pages %}{% if page.title != nil %}
+
+ "{{ page.url | slugify }}": {
+ "title": "{{ page.title | xml_escape }}",
+ "content" : "{{page.content | strip_html | strip_newlines | escape | remove: "\"}}",
+ "url": " {{ page.url | xml_escape }}",
+ "group": "{{ page.group }}",
+ "excerpt": {{ page.description | strip_html | truncatewords: 40 | jsonify }}
+ }
+ {% unless forloop.last %},{% endunless %}
+ {% endif %}
+ {% endfor %}
+}
diff --git a/docs/security/authentication.md b/docs/security/authentication.md
index 7ce160aa2b5..60352277883 100644
--- a/docs/security/authentication.md
+++ b/docs/security/authentication.md
@@ -1,7 +1,7 @@
---
layout: page
title: "Authentication for NGINX"
-description: "Authentication for NGINX"
+description: "There are multiple ways to enable authentication in Apache Zeppelin. This page describes HTTP basic auth using NGINX."
group: security
---
+{% include JB/setup %}
+
# Authentication for NGINX
-Authentication is company-specific.
+There are multiple ways to enable authentication in Apache Zeppelin. This page describes HTTP basic auth using NGINX.
One option is to use [Basic Access Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication).
## HTTP Basic Authentication using NGINX
diff --git a/docs/security/datasource_authorization.md b/docs/security/datasource_authorization.md
index 3f86b8ab00d..edf39fb26cf 100644
--- a/docs/security/datasource_authorization.md
+++ b/docs/security/datasource_authorization.md
@@ -1,7 +1,7 @@
---
layout: page
-title: "Data Source Authorization"
-description: "Data Source Authorization"
+title: "Data Source Authorization in Apache Zeppelin"
+description: "Apache Zeppelin supports protected data sources. In case of a MySql database, every users can set up their own credentials to access it."
group: security
---
+{% include JB/setup %}
+
# Data Source Authorization in Apache Zeppelin
diff --git a/docs/security/notebook_authorization.md b/docs/security/notebook_authorization.md
index 87885676ef3..b6511516691 100644
--- a/docs/security/notebook_authorization.md
+++ b/docs/security/notebook_authorization.md
@@ -1,7 +1,7 @@
---
layout: page
-title: "Notebook Authorization"
-description: "Notebook Authorization"
+title: "Notebook Authorization in Apache Zeppelin"
+description: "This page will guide you how you can set the permission for Zeppelin notebooks. This document assumes that Apache Shiro authentication was set up."
group: security
---
+{% include JB/setup %}
+
# Zeppelin Notebook Authorization
diff --git a/docs/security/shiroauthentication.md b/docs/security/shiroauthentication.md
index a5c9c317b36..969e61e5543 100644
--- a/docs/security/shiroauthentication.md
+++ b/docs/security/shiroauthentication.md
@@ -1,7 +1,7 @@
---
layout: page
-title: "Shiro Security for Apache Zeppelin"
-description: ""
+title: "Apache Shiro Authentication for Apache Zeppelin"
+description: "Apache Shiro is a powerful and easy-to-use Java security framework that performs authentication, authorization, cryptography, and session management. This document explains step by step how Shiro can be used for Zeppelin notebook authentication."
group: security
---
{% include JB/setup %}
-# Shiro authentication for Apache Zeppelin
+# Apache Shiro authentication for Apache Zeppelin
@@ -105,6 +105,59 @@ finance = *
group1 = *
```
+## Configure Realm (optional)
+Realms are responsible for authentication and authorization in Apache Zeppelin. By default, Apache Zeppelin uses [IniRealm](https://shiro.apache.org/static/latest/apidocs/org/apache/shiro/realm/text/IniRealm.html) (users and groups are configurable in `conf/shiro.ini` file under `[user]` and `[group]` section). You can also leverage Shiro Realms like [JndiLdapRealm](https://shiro.apache.org/static/latest/apidocs/org/apache/shiro/realm/ldap/JndiLdapRealm.html), [JdbcRealm](https://shiro.apache.org/static/latest/apidocs/org/apache/shiro/realm/jdbc/JdbcRealm.html) or create [our own](https://shiro.apache.org/static/latest/apidocs/org/apache/shiro/realm/AuthorizingRealm.html).
+To learn more about Apache Shiro Realm, please check [this documentation](http://shiro.apache.org/realm.html).
+
+We also provide community custom Realms.
+
+### Active Directory
+
+```
+activeDirectoryRealm = org.apache.zeppelin.server.ActiveDirectoryGroupRealm
+activeDirectoryRealm.systemUsername = userNameA
+activeDirectoryRealm.systemPassword = passwordA
+activeDirectoryRealm.hadoopSecurityCredentialPath = jceks://file/user/zeppelin/conf/zeppelin.jceks
+activeDirectoryRealm.searchBase = CN=Users,DC=SOME_GROUP,DC=COMPANY,DC=COM
+activeDirectoryRealm.url = ldap://ldap.test.com:389
+activeDirectoryRealm.groupRolesMap = "CN=aGroupName,OU=groups,DC=SOME_GROUP,DC=COMPANY,DC=COM":"group1"
+activeDirectoryRealm.authorizationCachingEnabled = false
+```
+
+
+Also instead of specifying systemPassword in clear text in shiro.ini administrator can choose to specify the same in "hadoop credential".
+Create a keystore file using the hadoop credential commandline, for this the hadoop commons should be in the classpath
+`hadoop credential create activeDirectoryRealm.systempassword -provider jceks://file/user/zeppelin/conf/zeppelin.jceks`
+
+Change the following values in the Shiro.ini file, and uncomment the line:
+`activeDirectoryRealm.hadoopSecurityCredentialPath = jceks://file/user/zeppelin/conf/zeppelin.jceks`
+
+### LDAP
+
+```
+ldapRealm = org.apache.zeppelin.server.LdapGroupRealm
+# search base for ldap groups (only relevant for LdapGroupRealm):
+ldapRealm.contextFactory.environment[ldap.searchBase] = dc=COMPANY,dc=COM
+ldapRealm.contextFactory.url = ldap://ldap.test.com:389
+ldapRealm.userDnTemplate = uid={0},ou=Users,dc=COMPANY,dc=COM
+ldapRealm.contextFactory.authenticationMechanism = SIMPLE
+```
+
+### ZeppelinHub
+[ZeppelinHub](https://www.zeppelinhub.com) is a service that synchronize your Apache Zeppelin notebooks and enables you to collaborate easily.
+
+To enable login with your ZeppelinHub credential, apply the following change in `conf/shiro.ini` under `[main]` section.
+
+```
+### A sample for configuring ZeppelinHub Realm
+zeppelinHubRealm = org.apache.zeppelin.realm.ZeppelinHubRealm
+## Url of ZeppelinHub
+zeppelinHubRealm.zeppelinhubUrl = https://www.zeppelinhub.com
+securityManager.realms = $zeppelinHubRealm
+```
+
+> Note: ZeppelinHub is not releated to apache Zeppelin project.
+
## Secure your Zeppelin information (optional)
By default, anyone who defined in `[users]` can share **Interpreter Setting**, **Credential** and **Configuration** information in Apache Zeppelin.
Sometimes you might want to hide these information for your use case.
@@ -123,3 +176,4 @@ If you want to grant this permission to other users, you can change **roles[ ]**
Search Docs
+ ++ + + +
+ + +
> **NOTE :** All of the above configurations are defined in the `conf/shiro.ini` file. This documentation is originally from [SECURITY-README.md](https://github.com/apache/zeppelin/blob/master/SECURITY-README.md). + diff --git a/docs/storage/storage.md b/docs/storage/storage.md index 19ddd95af82..76012fff4b4 100644 --- a/docs/storage/storage.md +++ b/docs/storage/storage.md @@ -1,7 +1,7 @@ --- layout: page -title: "Storage" -description: "Notebook Storage option for Zeppelin" +title: "Notebook Storage for Apache Zeppelin" +description: Apache Zeppelin has a pluggable notebook storage mechanism controlled by zeppelin.notebook.storage configuration option with multiple implementations." group: storage --- +{% include JB/setup %} + # Notebook storage options for Apache Zeppelin diff --git a/flink/pom.xml b/flink/pom.xml index f4c12d53846..628f5427561 100644 --- a/flink/pom.xml +++ b/flink/pom.xml @@ -120,21 +120,18 @@
Python Interpreter help
') - print ('Python 2 & 3 compatibility
') - print ('The interpreter is compatible with Python 2 & 3.
')
- print ('To change Python version, ')
- print ('change in the interpreter configuration the python to the ')
- print ('desired version (example : python=/usr/bin/python3)
Python modules
') - print ('The interpreter can use all modules already installed ') - print ('(with pip, easy_install, etc)
') - print ('Forms
') - print ('You must install py4j in order to use ' - 'the form feature (pip install py4j)') - print ('Input form
') - print ('print (z.input("f1","defaultValue"))')
- print ('Selection form
') - print ('print(z.select("f2", [("o1","1"), ("o2","2")],2))')
- print ('Checkbox form
') - print (' print("".join(z.checkbox("f3", [("o1","1"), '
- '("o2","2")],["1"])))')
- print ('Matplotlib graph
') - print ('The interpreter can display matplotlib graph with ')
- print ('the function z.show()
')
- print (' You need to already have matplotlib module installed ')
- print ('to use this functionality !
') - print ('''
import matplotlib.pyplot as plt -plt.figure() -(.. ..) -z.show(plt) -plt.close() -''') - print ('
z.show function can take optional parameters ') - print ('to adapt graph width and height
example :")
- print ('''
''')
- print ('z.show(plt,width='50px') -z.show(plt,height='150px')
Pandas DataFrame
') - print (' You need to have Pandas module installed ')
- print ('to use this functionality (pip install pandas) !
') - print (""" -
The interpreter can visualize Pandas DataFrame
-with the function z.show()
-
-""")
- print ('
-import pandas as pd
-df = pd.read_csv("bank.csv", sep=";")
-z.show(df)
-SQL over Pandas DataFrame
') - print (' You need to have Pandas&Pandasql modules installed ')
- print ('to use this functionality (pip install pandas pandasql) !
') - print (""" -
Python interpreter group includes %sql interpreter that can query
-Pandas DataFrames using SQL and visualize results using Zeppelin Table Display System
-
-
- -
-""")
+ print("""%html
+
-%python
-import pandas as pd
-df = pd.read_csv("bank.csv", sep=";")
-
-- -
-%python.sql -%sql -SELECT * from df LIMIT 5 -
Python Interpreter help
+ +Python 2 & 3 compatibility
+The interpreter is compatible with Python 2 & 3.
+ To change Python version,
+ change in the interpreter configuration the python to the
+ desired version (example : python=/usr/bin/python3)
Python modules
+The interpreter can use all modules already installed + (with pip, easy_install, etc)
+ +Forms
+ You must install py4j in order to use + the form feature (pip install py4j) +Input form
+print (z.input("f1","defaultValue"))
+ Selection form
+print(z.select("f2", [("o1","1"), ("o2","2")],2))
+ Checkbox form
+ print("".join(z.checkbox("f3", [("o1","1"), ("o2","2")],["1"])))')
+
+ Matplotlib graph
+The interpreter can display matplotlib graph with
+ the function z.show()
+ You need to already have matplotlib module installed
+ to use this functionality !
+
import matplotlib.pyplot as plt + plt.figure() + (.. ..) + z.show(plt) + plt.close() ++
z.show function can take optional parameters + to adapt graph dimensions (width and height) and format (png or svg)
example :
+
+
+ z.show(plt,width='50px + z.show(plt,height='150px', fmt='svg')
Pandas DataFrame
+ You need to have Pandas module installed
+ to use this functionality (pip install pandas) !
+
The interpreter can visualize Pandas DataFrame
+ with the function z.show()
+
+
+
+ import pandas as pd
+ df = pd.read_csv("bank.csv", sep=";")
+ z.show(df)
+ SQL over Pandas DataFrame
+ You need to have Pandas&Pandasql modules installed
+ to use this functionality (pip install pandas pandasql) !
+ +
Python interpreter group includes %sql interpreter that can query
+ Pandas DataFrames using SQL and visualize results using Zeppelin Table Display System
+
+
+
+ """)
class PyZeppelinContext(object):
@@ -112,18 +115,17 @@ class PyZeppelinContext(object):
errorMsg = "You must install py4j Python module " \
"(pip install py4j) to use Zeppelin dynamic forms features"
- def __init__(self, zc):
- self.z = zc
+ def __init__(self):
self.max_result = 1000
def input(self, name, defaultValue=""):
- print (self.errorMsg)
+ print(self.errorMsg)
def select(self, name, options, defaultValue=""):
- print (self.errorMsg)
+ print(self.errorMsg)
def checkbox(self, name, options, defaultChecked=[]):
- print (self.errorMsg)
+ print(self.errorMsg)
def show(self, p, **kwargs):
if hasattr(p, '__name__') and p.__name__ == "matplotlib.pyplot":
@@ -139,20 +141,20 @@ def show_dataframe(self, df, **kwargs):
"""Pretty prints DF using Table Display System
"""
limit = len(df) > self.max_result
- header_buf = io.StringIO("")
- header_buf.write(df.columns[0])
+ header_buf = StringIO("")
+ header_buf.write(str(df.columns[0]))
for col in df.columns[1:]:
header_buf.write("\t")
- header_buf.write(col)
+ header_buf.write(str(col))
header_buf.write("\n")
- body_buf = io.StringIO("")
+ body_buf = StringIO("")
rows = df.head(self.max_result).values if limit else df.values
for row in rows:
- body_buf.write(row[0])
+ body_buf.write(str(row[0]))
for cell in row[1:]:
body_buf.write("\t")
- body_buf.write(cell)
+ body_buf.write(str(cell))
body_buf.write("\n")
body_buf.seek(0); header_buf.seek(0)
#TODO(bzz): fix it, so it shows red notice, as in Spark
@@ -162,21 +164,29 @@ def show_dataframe(self, df, **kwargs):
#)
body_buf.close(); header_buf.close()
- def show_matplotlib(self, p, width="0", height="0", **kwargs):
+ def show_matplotlib(self, p, fmt="png", width="auto", height="auto",
+ **kwargs):
"""Matplotlib show function
"""
- img = io.StringIO()
- p.savefig(img, format='svg')
- img.seek(0)
- style = ""
- if (width != "0"):
- style += 'width:' + width
- if (height != "0"):
- if (len(style) != 0):
- style += ","
- style += 'height:' + height
- print("%html
+ %python
+ import pandas as pd
+ df = pd.read_csv("bank.csv", sep=";")
+
+ +
+ %python.sql + %sql + SELECT * from df LIMIT 5 ++
" + img.read() + "
")
+ if fmt == "png":
+ img = BytesIO()
+ p.savefig(img, format=fmt)
+ img_str = b"data:image/png;base64,"
+ img_str += base64.b64encode(img.getvalue().strip())
+ img_tag = "
"
+ # Decoding is necessary for Python 3 compability
+ img_str = img_str.decode("ascii")
+ img_str = img_tag.format(img=img_str, width=width, height=height)
+ elif fmt == "svg":
+ img = StringIO()
+ p.savefig(img, format=fmt)
+ img_str = img.getvalue()
+ else:
+ raise ValueError("fmt must be 'png' or 'svg'")
+
+ html = "%html
{img}
"
+ print(html.format(width=width, height=height, img=img_str))
img.close()
-z = PyZeppelinContext("")
+z = PyZeppelinContext()
diff --git a/python/src/main/resources/bootstrap_input.py b/python/src/main/resources/bootstrap_input.py
index d15b93ad8c7..e006816256c 100644
--- a/python/src/main/resources/bootstrap_input.py
+++ b/python/src/main/resources/bootstrap_input.py
@@ -25,11 +25,11 @@
class Py4jZeppelinContext(PyZeppelinContext):
"""A context impl that uses Py4j to communicate to JVM
"""
- def __init__(self, zc):
- super(Py4jZeppelinContext, self).__init__(zc)
+ def __init__(self, z):
+ self.z = z
self.paramOption = gateway.jvm.org.apache.zeppelin.display.Input.ParamOption
self.javaList = gateway.jvm.java.util.ArrayList
- self.max_result = 1000 #TODO(bzz): read `zeppelin.python.maxResult` from JVM
+ self.max_result = self.z.getMaxResult()
def input(self, name, defaultValue=""):
return self.z.getGui().input(name, defaultValue)
diff --git a/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterPandasSqlTest.java b/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterPandasSqlTest.java
index 5f26adb2413..f9538562c22 100644
--- a/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterPandasSqlTest.java
+++ b/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterPandasSqlTest.java
@@ -153,4 +153,24 @@ public void badSqlSyntaxFails() {
assertTrue(ret.message().length() > 0);
}
+ @Test
+ public void showDataFrame() {
+ InterpreterResult ret;
+ ret = python.interpret("import pandas as pd", context);
+ ret = python.interpret("import numpy as np", context);
+
+ // given a Pandas DataFrame with non-text data
+ ret = python.interpret("d1 = {1 : [np.nan, 1, 2, 3], 'two' : [3., 4., 5., 6.7]}", context);
+ ret = python.interpret("df1 = pd.DataFrame(d1)", context);
+ assertEquals(ret.message(), InterpreterResult.Code.SUCCESS, ret.code());
+
+ // when
+ ret = python.interpret("z.show(df1)", context);
+
+ // then
+ assertEquals(ret.message(), InterpreterResult.Code.SUCCESS, ret.code());
+ assertEquals(ret.message(), Type.TABLE, ret.type());
+ assertTrue(ret.message().indexOf("nan") > 0);
+ assertTrue(ret.message().indexOf("6.7") > 0);
+ }
}
diff --git a/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterTest.java b/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterTest.java
index a4c80ae8966..1228ec46a60 100644
--- a/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterTest.java
+++ b/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterTest.java
@@ -115,7 +115,7 @@ public void testPy4jIsNotInstalled() {
*/
@Test
public void testPy4jInstalled() throws IOException, InterruptedException {
- when(mockPythonProcess.sendAndGetResult(eq("\n\nimport py4j\n"))).thenReturn(">>>");
+ when(mockPythonProcess.sendAndGetResult(eq("\n\nimport py4j\n"))).thenReturn("");
pythonInterpreter.open();
Integer py4jPort = pythonInterpreter.getPy4jPort();
@@ -137,7 +137,7 @@ public void testPy4jInstalled() throws IOException, InterruptedException {
@Test
public void testClose() throws IOException, InterruptedException {
//given: py4j is installed
- when(mockPythonProcess.sendAndGetResult(eq("\n\nimport py4j\n"))).thenReturn(">>>");
+ when(mockPythonProcess.sendAndGetResult(eq("\n\nimport py4j\n"))).thenReturn("");
pythonInterpreter.open();
Integer py4jPort = pythonInterpreter.getPy4jPort();
@@ -210,12 +210,33 @@ private String answerFromPythonMock(InvocationOnMock invocationOnMock) {
String output = "";
for (int i = 0; i < lines.length; i++) {
- output += ">>>" + lines[i];
+ output += lines[i];
}
return output;
} else {
- return ">>>";
+ return "";
}
}
+ @Test
+ public void checkMultiRowErrorFails() {
+ PythonInterpreter pythonInterpreter = new PythonInterpreter(
+ PythonInterpreterTest.getPythonTestProperties()
+ );
+ pythonInterpreter.open();
+ String codeRaiseException = "raise Exception(\"test exception\")";
+ InterpreterResult ret = pythonInterpreter.interpret(codeRaiseException, null);
+
+ assertNotNull("Interpreter result for raise exception is Null", ret);
+
+ assertEquals(InterpreterResult.Code.ERROR, ret.code());
+ assertTrue(ret.message().length() > 0);
+
+ assertNotNull("Interpreter result for text is Null", ret);
+ String codePrintText = "print (\"Exception(\\\"test exception\\\")\")";
+ ret = pythonInterpreter.interpret(codePrintText, null);
+ assertEquals(InterpreterResult.Code.SUCCESS, ret.code());
+ assertTrue(ret.message().length() > 0);
+ }
+
}
diff --git a/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterWithPythonInstalledTest.java b/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterWithPythonInstalledTest.java
index 15787fdce46..38b46e71ca8 100644
--- a/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterWithPythonInstalledTest.java
+++ b/python/src/test/java/org/apache/zeppelin/python/PythonInterpreterWithPythonInstalledTest.java
@@ -42,7 +42,7 @@
public class PythonInterpreterWithPythonInstalledTest {
@Test
- public void badSqlSyntaxFails() {
+ public void badPythonSyntaxFails() {
//given
PythonInterpreter realPython = new PythonInterpreter(
PythonInterpreterTest.getPythonTestProperties());
@@ -58,4 +58,21 @@ public void badSqlSyntaxFails() {
assertTrue(ret.message().length() > 0);
}
+ @Test
+ public void goodPythonSyntaxRuns() {
+ //given
+ PythonInterpreter realPython = new PythonInterpreter(
+ PythonInterpreterTest.getPythonTestProperties());
+ realPython.open();
+
+ //when
+ InterpreterResult ret = realPython.interpret("help()", null);
+
+ //then
+ assertNotNull("Interpreter returned 'null'", ret);
+ //System.out.println("\nInterpreter response: \n" + ret.message());
+ assertEquals(InterpreterResult.Code.SUCCESS, ret.code());
+ assertTrue(ret.message().length() > 0);
+ }
+
}
diff --git a/r/pom.xml b/r/pom.xml
index 0e7d78deb61..2dc2eef5af7 100644
--- a/r/pom.xml
+++ b/r/pom.xml
@@ -380,7 +380,7 @@
scala-2.10
- !scala-2.11 true
src/main/scala-2.10
@@ -390,9 +390,6 @@
scala-2.11
-
- scala-2.11
src/main/scala-2.11
src/test/scala/scala-2.11
diff --git a/r/src/main/scala/org/apache/zeppelin/rinterpreter/RInterpreter.scala b/r/src/main/scala/org/apache/zeppelin/rinterpreter/RInterpreter.scala
index 959f649a157..9504573f40e 100644
--- a/r/src/main/scala/org/apache/zeppelin/rinterpreter/RInterpreter.scala
+++ b/r/src/main/scala/org/apache/zeppelin/rinterpreter/RInterpreter.scala
@@ -17,6 +17,7 @@
package org.apache.zeppelin.rinterpreter
+import java.io.{BufferedInputStream, File, FileInputStream}
import java.nio.file.{Files, Paths}
import java.util._
@@ -110,10 +111,10 @@ object RInterpreter {
// These are the additional properties we need on top of the ones provided by the spark interpreters
lazy val props: Map[String, InterpreterProperty] = new InterpreterPropertyBuilder()
- .add("rhadoop.cmd", SparkInterpreter.getSystemDefault("rhadoop.cmd", "HADOOP_CMD", ""), "Usually /usr/bin/hadoop")
- .add("rhadooop.streamingjar", SparkInterpreter.getSystemDefault("rhadoop.cmd", "HADOOP_STREAMING", ""), "Usually /usr/lib/hadoop/contrib/streaming/hadoop-streaming-.jar")
- .add("rscala.debug", SparkInterpreter.getSystemDefault("rscala.debug","RSCALA_DEBUG", "false"), "Whether to turn on rScala debugging") // TEST: Implemented but not tested
- .add("rscala.timeout", SparkInterpreter.getSystemDefault("rscala.timeout","RSCALA_TIMEOUT", "60"), "Timeout for rScala") // TEST: Implemented but not tested
+ .add("rhadoop.cmd", "HADOOP_CMD", "rhadoop.cmd", "", "Usually /usr/bin/hadoop")
+ .add("rhadooop.streamingjar", "HADOOP_STREAMING", "rhadooop.streamingjar", "", "Usually /usr/lib/hadoop/contrib/streaming/hadoop-streaming-.jar")
+ .add("rscala.debug", "RSCALA_DEBUG", "rscala.debug","false", "Whether to turn on rScala debugging") // TEST: Implemented but not tested
+ .add("rscala.timeout", "RSCALA_TIMEOUT", "rscala.timeout","60", "Timeout for rScala") // TEST: Implemented but not tested
.build
def getProps() = {
@@ -141,8 +142,15 @@ object RInterpreter {
}
def dataURI(file : String, mime : String) : String = {
- val data: String = Source.fromFile(file).getLines().mkString("\n")
- s"""data:${mime};base64,""" + StringUtils.newStringUtf8(Base64.encodeBase64(data.getBytes(), false))
+ val fp = new File(file)
+ val fdata = new Array[Byte](fp.length().toInt)
+ val fin = new BufferedInputStream(new FileInputStream(fp))
+ try {
+ fin.read(fdata)
+ } finally {
+ fin.close()
+ }
+ s"""data:${mime};base64,""" + StringUtils.newStringUtf8(Base64.encodeBase64(fdata, false))
}
// The purpose here is to deal with knitr producing HTML with script and css tags outside the
diff --git a/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile b/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile
new file mode 100644
index 00000000000..450afef94f8
--- /dev/null
+++ b/scripts/docker/spark-cluster-managers/spark_mesos/Dockerfile
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+FROM centos:centos6
+
+ENV SPARK_PROFILE 2.0
+ENV SPARK_VERSION 2.0.0
+ENV HADOOP_PROFILE 2.3
+ENV HADOOP_VERSION 2.3.0
+
+# Update the image with the latest packages
+RUN yum update -y; yum clean all
+
+# Get utils
+RUN yum install -y \
+wget \
+tar \
+curl \
+svn \
+cyrus-sasl-md5 \
+libevent2-devel \
+&& \
+yum clean all
+
+# Remove old jdk
+RUN yum remove java; yum remove jdk
+
+# install jdk7
+RUN yum install -y java-1.7.0-openjdk-devel
+ENV JAVA_HOME /usr/lib/jvm/java
+ENV PATH $PATH:$JAVA_HOME/bin
+
+# install spark
+RUN curl -s http://www.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE.tgz | tar -xz -C /usr/local/
+RUN cd /usr/local && ln -s spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE spark
+
+# update boot script
+COPY entrypoint.sh /etc/entrypoint.sh
+RUN chown root.root /etc/entrypoint.sh
+RUN chmod 700 /etc/entrypoint.sh
+
+# install mesos
+RUN wget http://repos.mesosphere.com/el/6/x86_64/RPMS/mesos-1.0.0-2.0.89.centos65.x86_64.rpm
+RUN rpm -Uvh mesos-1.0.0-2.0.89.centos65.x86_64.rpm
+
+#spark
+EXPOSE 8080 7077 7072 8081 8082
+
+#mesos
+EXPOSE 5050 5051
+
+ENTRYPOINT ["/etc/entrypoint.sh"]
diff --git a/scripts/docker/spark-cluster-managers/spark_mesos/entrypoint.sh b/scripts/docker/spark-cluster-managers/spark_mesos/entrypoint.sh
new file mode 100755
index 00000000000..28d76bf9a1e
--- /dev/null
+++ b/scripts/docker/spark-cluster-managers/spark_mesos/entrypoint.sh
@@ -0,0 +1,48 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+export SPARK_HOME=/usr/local/spark/
+export SPARK_MASTER_PORT=7077
+export SPARK_MASTER_WEBUI_PORT=8080
+export SPARK_WORKER_PORT=8888
+export SPARK_WORKER_WEBUI_PORT=8081
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_HOME/jre/lib/amd64/server/
+
+# spark configuration
+cp $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh
+echo "export MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/libmesos.so" >> $SPARK_HOME/conf/spark-env.sh
+
+cp $SPARK_HOME/conf/spark-defaults.conf.template $SPARK_HOME/conf/spark-defaults.conf
+echo "spark.master mesos://`hostname`:5050" >> $SPARK_HOME/conf/spark-defaults.conf
+echo "spark.mesos.executor.home /usr/local/spark" >> $SPARK_HOME/conf/spark-defaults.conf
+
+# run spark
+cd $SPARK_HOME/sbin
+./start-master.sh
+./start-slave.sh spark://`hostname`:$SPARK_MASTER_PORT
+
+# start mesos
+mesos-master --ip=0.0.0.0 --work_dir=/var/lib/mesos &> /var/log/mesos_master.log &
+mesos-slave --master=0.0.0.0:5050 --work_dir=/var/lib/mesos --launcher=posix &> /var/log/mesos_slave.log &
+
+CMD=${1:-"exit 0"}
+if [[ "$CMD" == "-d" ]];
+then
+ service sshd stop
+ /usr/sbin/sshd -D -d
+else
+ /bin/bash -c "$*"
+fi
diff --git a/scripts/docker/spark-cluster-managers/spark_standalone/Dockerfile b/scripts/docker/spark-cluster-managers/spark_standalone/Dockerfile
new file mode 100644
index 00000000000..fa3078bd75b
--- /dev/null
+++ b/scripts/docker/spark-cluster-managers/spark_standalone/Dockerfile
@@ -0,0 +1,53 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+FROM centos:centos6
+
+ENV SPARK_PROFILE 1.6
+ENV SPARK_VERSION 1.6.2
+ENV HADOOP_PROFILE 2.3
+ENV SPARK_HOME /usr/local/spark
+
+# Update the image with the latest packages
+RUN yum update -y; yum clean all
+
+# Get utils
+RUN yum install -y \
+wget \
+tar \
+curl \
+&& \
+yum clean all
+
+# Remove old jdk
+RUN yum remove java; yum remove jdk
+
+# install jdk7
+RUN yum install -y java-1.7.0-openjdk-devel
+ENV JAVA_HOME /usr/lib/jvm/java
+ENV PATH $PATH:$JAVA_HOME/bin
+
+# install spark
+RUN curl -s http://apache.mirror.cdnetworks.com/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE.tgz | tar -xz -C /usr/local/
+RUN cd /usr/local && ln -s spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE spark
+
+# update boot script
+COPY entrypoint.sh /etc/entrypoint.sh
+RUN chown root.root /etc/entrypoint.sh
+RUN chmod 700 /etc/entrypoint.sh
+
+#spark
+EXPOSE 8080 7077 8888 8081
+
+ENTRYPOINT ["/etc/entrypoint.sh"]
diff --git a/scripts/docker/spark-cluster-managers/spark_standalone/entrypoint.sh b/scripts/docker/spark-cluster-managers/spark_standalone/entrypoint.sh
new file mode 100755
index 00000000000..f4fded0dc9e
--- /dev/null
+++ b/scripts/docker/spark-cluster-managers/spark_standalone/entrypoint.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+export SPARK_MASTER_PORT=7077
+
+# run spark
+cd /usr/local/spark/sbin
+./start-master.sh
+./start-slave.sh spark://`hostname`:$SPARK_MASTER_PORT
+
+CMD=${1:-"exit 0"}
+if [[ "$CMD" == "-d" ]];
+then
+ service sshd stop
+ /usr/sbin/sshd -D -d
+else
+ /bin/bash -c "$*"
+fi
diff --git a/scripts/docker/spark-cluster-managers/spark_yarn_cluster/Dockerfile b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/Dockerfile
new file mode 100644
index 00000000000..712c5b270f9
--- /dev/null
+++ b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/Dockerfile
@@ -0,0 +1,107 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+FROM centos:centos6
+
+ENV SPARK_PROFILE 2.0
+ENV SPARK_VERSION 2.0.0
+ENV HADOOP_PROFILE 2.7
+ENV HADOOP_VERSION 2.7.0
+
+# Update the image with the latest packages
+RUN yum update -y; yum clean all
+
+# Get utils
+RUN yum install -y \
+wget \
+tar \
+curl \
+&& \
+yum clean all
+
+# Remove old jdk
+RUN yum remove java; yum remove jdk
+
+# install jdk7
+RUN yum install -y java-1.7.0-openjdk-devel
+ENV JAVA_HOME /usr/lib/jvm/java
+ENV PATH $PATH:$JAVA_HOME/bin
+
+# install hadoop
+RUN yum install -y curl which tar sudo openssh-server openssh-clients rsync
+
+# hadoop
+RUN curl -s https://archive.apache.org/dist/hadoop/core/hadoop-$HADOOP_VERSION/hadoop-$HADOOP_VERSION.tar.gz | tar -xz -C /usr/local/
+RUN cd /usr/local && ln -s ./hadoop-$HADOOP_VERSION hadoop
+
+ENV HADOOP_PREFIX /usr/local/hadoop
+ENV HADOOP_COMMON_HOME /usr/local/hadoop
+ENV HADOOP_HDFS_HOME /usr/local/hadoop
+ENV HADOOP_MAPRED_HOME /usr/local/hadoop
+ENV HADOOP_YARN_HOME /usr/local/hadoop
+ENV HADOOP_CONF_DIR /usr/local/hadoop/etc/hadoop
+
+RUN sed -i '/^export JAVA_HOME/ s:.*:export JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk.x86_64\nexport HADOOP_PREFIX=/usr/local/hadoop\nexport HADOOP_HOME=/usr/local/hadoop\n:' $HADOOP_PREFIX/etc/hadoop/hadoop-env.sh
+RUN sed -i '/^export HADOOP_CONF_DIR/ s:.*:export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop/:' $HADOOP_PREFIX/etc/hadoop/hadoop-env.sh
+
+RUN mkdir $HADOOP_PREFIX/input
+RUN cp $HADOOP_PREFIX/etc/hadoop/*.xml $HADOOP_PREFIX/input
+
+# hadoop configurations
+ADD hdfs_conf/core-site.xml $HADOOP_PREFIX/etc/hadoop/core-site.xml
+ADD hdfs_conf/hdfs-site.xml $HADOOP_PREFIX/etc/hadoop/hdfs-site.xml
+ADD hdfs_conf/mapred-site.xml $HADOOP_PREFIX/etc/hadoop/mapred-site.xml
+ADD hdfs_conf/yarn-site.xml $HADOOP_PREFIX/etc/hadoop/yarn-site.xml
+
+RUN mkdir /data/
+RUN chmod 777 /data/
+RUN $HADOOP_PREFIX/bin/hdfs namenode -format
+
+RUN rm /usr/local/hadoop/lib/native/*
+RUN curl -Ls http://dl.bintray.com/sequenceiq/sequenceiq-bin/hadoop-native-64-$HADOOP_VERSION.tar|tar -x -C /usr/local/hadoop/lib/native/
+
+# install spark
+RUN curl -s http://archive.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE.tgz | tar -xz -C /usr/local/
+RUN cd /usr/local && ln -s spark-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE spark
+ENV SPARK_HOME /usr/local/spark
+
+ENV YARN_CONF_DIR $HADOOP_PREFIX/etc/hadoop
+ENV PATH $PATH:$SPARK_HOME/bin:$HADOOP_PREFIX/bin
+
+# passwordless ssh
+RUN ssh-keygen -q -N "" -t dsa -f /etc/ssh/ssh_host_dsa_key
+RUN ssh-keygen -q -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key
+RUN ssh-keygen -q -N "" -t rsa -f /root/.ssh/id_rsa
+RUN cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
+
+ADD ssh_config /root/.ssh/config
+RUN chmod 600 /root/.ssh/config
+RUN chown root:root /root/.ssh/config
+RUN chmod +x /usr/local/hadoop/etc/hadoop/*-env.sh
+
+# update boot script
+COPY entrypoint.sh /etc/entrypoint.sh
+RUN chown root.root /etc/entrypoint.sh
+RUN chmod 700 /etc/entrypoint.sh
+
+# Hdfs ports
+EXPOSE 50010 50020 50070 50075 50090
+# Mapred ports
+EXPOSE 9000 9001
+#Yarn ports
+EXPOSE 8030 8031 8032 8033 8040 8042 8088
+#spark
+EXPOSE 8080 7077 8888 8081
+
+ENTRYPOINT ["/etc/entrypoint.sh"]
diff --git a/scripts/docker/spark-cluster-managers/spark_yarn_cluster/entrypoint.sh b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/entrypoint.sh
new file mode 100755
index 00000000000..85b335d6d12
--- /dev/null
+++ b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/entrypoint.sh
@@ -0,0 +1,60 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+echo 'hadoop' |passwd root --stdin
+
+: ${HADOOP_PREFIX:=/usr/local/hadoop}
+
+$HADOOP_PREFIX/etc/hadoop/hadoop-env.sh
+
+rm /tmp/*.pid
+
+# installing libraries if any - (resource urls added comma separated to the ACP system variable)
+cd $HADOOP_PREFIX/share/hadoop/common ; for cp in ${ACP//,/ }; do echo == $cp; curl -LO $cp ; done; cd -
+
+cp $SPARK_HOME/conf/metrics.properties.template $SPARK_HOME/conf/metrics.properties
+
+# start hadoop
+service sshd start
+$HADOOP_PREFIX/sbin/start-dfs.sh
+$HADOOP_PREFIX/sbin/start-yarn.sh
+
+$HADOOP_PREFIX/bin/hdfs dfsadmin -safemode leave && $HADOOP_PREFIX/bin/hdfs dfs -put $SPARK_HOME-$SPARK_VERSION-bin-hadoop$HADOOP_PROFILE/lib /spark
+
+# start spark
+export SPARK_MASTER_OPTS="-Dspark.driver.port=7001 -Dspark.fileserver.port=7002
+ -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004
+ -Dspark.blockManager.port=7005 -Dspark.executor.port=7006
+ -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory"
+export SPARK_WORKER_OPTS="-Dspark.driver.port=7001 -Dspark.fileserver.port=7002
+ -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004
+ -Dspark.blockManager.port=7005 -Dspark.executor.port=7006
+ -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory"
+
+export SPARK_MASTER_PORT=7077
+
+cd /usr/local/spark/sbin
+./start-master.sh
+./start-slave.sh spark://`hostname`:$SPARK_MASTER_PORT
+
+CMD=${1:-"exit 0"}
+if [[ "$CMD" == "-d" ]];
+then
+ service sshd stop
+ /usr/sbin/sshd -D -d
+else
+ /bin/bash -c "$*"
+fi
diff --git a/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf/core-site.xml b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf/core-site.xml
new file mode 100644
index 00000000000..87446337f9a
--- /dev/null
+++ b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf/core-site.xml
@@ -0,0 +1,22 @@
+
+
+
+ fs.defaultFS
+ hdfs://0.0.0.0:9000
+
+
diff --git a/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf/hdfs-site.xml b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf/hdfs-site.xml
new file mode 100644
index 00000000000..b3f88af304c
--- /dev/null
+++ b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf/hdfs-site.xml
@@ -0,0 +1,78 @@
+
+
+
+ dfs.replication
+ 1
+
+
+
+ dfs.data.dir
+ /data/hdfs
+ true
+
+
+
+ dfs.permissions
+ false
+
+
+
+ dfs.client.use.datanode.hostname
+ true
+ Whether clients should use datanode hostnames when
+ connecting to datanodes.
+
+
+
+
+ dfs.datanode.use.datanode.hostname
+ true
+ Whether datanodes should use datanode hostnames when
+ connecting to other datanodes for data transfer.
+
+
+
+
+ dfs.datanode.address
+ 0.0.0.0:50010
+
+ The address where the datanode server will listen to.
+ If the port is 0 then the server will start on a free port.
+
+
+
+
+ dfs.datanode.http.address
+ 0.0.0.0:50075
+
+ The datanode http server address and port.
+ If the port is 0 then the server will start on a free port.
+
+
+
+
+ dfs.datanode.ipc.address
+ 0.0.0.0:50020
+
+ The datanode ipc server address and port.
+ If the port is 0 then the server will start on a free port.
+
+
+
+
+
diff --git a/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf/mapred-site.xml b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf/mapred-site.xml
new file mode 100644
index 00000000000..f8280f74657
--- /dev/null
+++ b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf/mapred-site.xml
@@ -0,0 +1,22 @@
+
+
+
+ mapreduce.framework.name
+ yarn
+
+
diff --git a/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf/yarn-site.xml b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf/yarn-site.xml
new file mode 100644
index 00000000000..89848165c77
--- /dev/null
+++ b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf/yarn-site.xml
@@ -0,0 +1,42 @@
+
+
+
+ yarn.resourcemanager.scheduler.address
+ 0.0.0.0:8030
+
+
+ yarn.resourcemanager.address
+ 0.0.0.0:8032
+
+
+ yarn.resourcemanager.webapp.address
+ 0.0.0.0:8088
+
+
+ yarn.resourcemanager.resource-tracker.address
+ 0.0.0.0:8031
+
+
+ yarn.resourcemanager.admin.address
+ 0.0.0.0:8033
+
+
+ yarn.application.classpath
+ /usr/local/hadoop/etc/hadoop, /usr/local/hadoop/share/hadoop/common/*, /usr/local/hadoop/share/hadoop/common/lib/*, /usr/local/hadoop/share/hadoop/hdfs/*, /usr/local/hadoop/share/hadoop/hdfs/lib/*, /usr/local/hadoop/share/hadoop/mapreduce/*, /usr/local/hadoop/share/hadoop/mapreduce/lib/*, /usr/local/hadoop/share/hadoop/yarn/*, /usr/local/hadoop/share/hadoop/yarn/lib/*, /usr/local/hadoop/share/spark/*
+
+
diff --git a/scripts/docker/spark-cluster-managers/spark_yarn_cluster/ssh_config b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/ssh_config
new file mode 100644
index 00000000000..537a95f48e5
--- /dev/null
+++ b/scripts/docker/spark-cluster-managers/spark_yarn_cluster/ssh_config
@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+Host *
+ UserKnownHostsFile /dev/null
+ StrictHostKeyChecking no
+ LogLevel quiet
\ No newline at end of file
diff --git a/scripts/vagrant/zeppelin-dev/roles/maven/tasks/main.yml b/scripts/vagrant/zeppelin-dev/roles/maven/tasks/main.yml
index e435aac5361..0e21e76723c 100644
--- a/scripts/vagrant/zeppelin-dev/roles/maven/tasks/main.yml
+++ b/scripts/vagrant/zeppelin-dev/roles/maven/tasks/main.yml
@@ -23,24 +23,24 @@
- name: Call apache web service to find preferred maven download mirror
- uri: url=http://www.apache.org/dyn/closer.cgi?path=maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.tar.gz&asjson=1 return_content=yes
+ uri: url=http://www.apache.org/dyn/closer.cgi?path=maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz&asjson=1 return_content=yes
register: webResponse
#- debug: var=webResponse
- name: download maven
get_url: url="{{webResponse.json.preferred}}{{webResponse.json.path_info}}"
- dest=/tmp/apache-maven-3.3.3-bin.tar.gz
+ dest=/tmp/apache-maven-3.3.9-bin.tar.gz
mode=0440
validate_certs=False
- name: extract maven tgz
- unarchive: src=/tmp/apache-maven-3.3.3-bin.tar.gz
+ unarchive: src=/tmp/apache-maven-3.3.9-bin.tar.gz
dest=/usr/local/
copy=no
- creates=/usr/local/apache-maven-3.3.3
+ creates=/usr/local/apache-maven-3.3.9
- name: create symlink to this maven version
- file: src=/usr/local/apache-maven-3.3.3/bin/mvn
+ file: src=/usr/local/apache-maven-3.3.9/bin/mvn
path=/usr/bin/mvn
state=link
diff --git a/spark-dependencies/pom.xml b/spark-dependencies/pom.xml
index 7da976adeea..f320680072b 100644
--- a/spark-dependencies/pom.xml
+++ b/spark-dependencies/pom.xml
@@ -959,26 +959,6 @@
-
- org.apache.rat
- apache-rat-plugin
-
-
- **/.idea/
- **/*.iml
- .gitignore
- **/.settings/*
- **/.classpath
- **/.project
- **/target/**
- **/derby.log
- **/metastore_db/
- **/README.md
- dependency-reduced-pom.xml
-
-
-
-
maven-enforcer-plugin
1.3.1
diff --git a/spark/pom.xml b/spark/pom.xml
index eb208fd1ff9..66d93c42ee6 100644
--- a/spark/pom.xml
+++ b/spark/pom.xml
@@ -51,11 +51,11 @@
slf4j-log4j12
-
+
${project.groupId}
@@ -301,27 +301,6 @@
-
- org.apache.rat
- apache-rat-plugin
-
-
- **/.idea/
- **/*.iml
- .gitignore
- **/.settings/*
- **/.classpath
- **/.project
- **/target/**
- **/derby.log
- **/metastore_db/
- **/README.md
- **/dependency-reduced-pom.xml
- **/interpreter-setting.json
-
-
-
-
maven-enforcer-plugin
1.3.1
@@ -434,11 +413,194 @@
+
+
+ org.apache.maven.plugins
+ maven-compiler-plugin
+
+
+ **/SparkRInterpreter.java
+
+
+ **/SparkRInterpreterTest.java
+ **/ZeppelinRTest.java
+
+
+
+
+ org.scala-tools
+ maven-scala-plugin
+
+
+ **/ZeppelinR.scala
+ **/SparkRBackend.scala
+
+
+
+
+ org.apache.maven.plugins
+ maven-surefire-plugin
+
+
+ **/SparkRInterpreterTest.java
+
+
+
-
+
+ spark-1.1
+
+
+
+
+ 1.1.1
+ 2.2.3-shaded-protobuf
+
+
+
+
+ spark-1.2
+
+
+
+ 1.2.1
+
+
+
+
+ spark-1.3
+
+
+ 1.3.1
+
+
+
+
+
+
+
+
+ spark-1.4
+
+ 1.4.1
+
+
+
+
+
+
+
+ spark-1.5
+
+ 1.5.2
+ com.typesafe.akka
+ 2.3.11
+ 2.5.0
+
+
+
+
+ spark-1.6
+
+ 1.6.1
+ 0.9
+ com.typesafe.akka
+ 2.3.11
+ 2.5.0
+
+
+
+
+ spark-2.0
+
+ true
+
+
+ 2.0.0
+ 2.5.0
+ 0.10.1
+ 2.11.8
+
+
+
+
+ hadoop-0.23
+
+
+
+ org.apache.avro
+ avro
+
+
+
+ 0.23.10
+
+
+
+
+ hadoop-1
+
+ 1.0.4
+ hadoop1
+ 1.8.8
+ org.spark-project.akka
+
+
+
+
+ hadoop-2.2
+
+ 2.2.0
+ 2.5.0
+ hadoop2
+
+
+
+
+ hadoop-2.3
+
+ 2.3.0
+ 2.5.0
+ 0.9.3
+ hadoop2
+
+
+
+
+ hadoop-2.4
+
+ 2.4.0
+ 2.5.0
+ 0.9.3
+ hadoop2
+
+
+
+
+ hadoop-2.6
+
+ 2.6.0
+ 2.5.0
+ 0.9.3
+ hadoop2
+
+
+
+
+ hadoop-2.7
+
+ 2.7.2
+ 2.5.0
+ 0.9.0
+ hadoop2
+
+
+
+
sparkr
@@ -453,36 +615,21 @@
src/main/sparkr-resources
-
-
-
- exclude-sparkr
-
- true
-
-
org.apache.maven.plugins
maven-compiler-plugin
-
- **/SparkRInterpreter.java
-
-
- **/SparkRInterpreterTest.java
- **/ZeppelinRTest.java
-
+
org.scala-tools
maven-scala-plugin
-
- **/ZeppelinR.scala
- **/SparkRBackend.scala
+
@@ -490,8 +637,7 @@
org.apache.maven.plugins
maven-surefire-plugin
-
- **/SparkRInterpreterTest.java
+
@@ -499,5 +645,4 @@
-
diff --git a/spark/src/main/java/org/apache/zeppelin/spark/LogOutputStream.java b/spark/src/main/java/org/apache/zeppelin/spark/LogOutputStream.java
new file mode 100644
index 00000000000..d941cd772c6
--- /dev/null
+++ b/spark/src/main/java/org/apache/zeppelin/spark/LogOutputStream.java
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.spark;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+
+
+/**
+ * Minor modification of LogOutputStream of apache commons exec.
+ * LogOutputStream of apache commons exec has one issue that method flush doesn't throw IOException,
+ * so that SparkOutputStream can not extend it correctly.
+ */
+public abstract class LogOutputStream extends OutputStream {
+ private static final int INTIAL_SIZE = 132;
+ private static final int CR = 13;
+ private static final int LF = 10;
+ private final ByteArrayOutputStream buffer;
+ private boolean skip;
+ private final int level;
+
+ public LogOutputStream() {
+ this(999);
+ }
+
+ public LogOutputStream(int level) {
+ this.buffer = new ByteArrayOutputStream(132);
+ this.skip = false;
+ this.level = level;
+ }
+
+ @Override
+ public void write(int cc) throws IOException {
+ byte c = (byte) cc;
+ if (c != 10 && c != 13) {
+ this.buffer.write(cc);
+ } else if (!this.skip) {
+ this.processBuffer();
+ }
+
+ this.skip = c == 13;
+ }
+
+ @Override
+ public void flush() throws IOException {
+ if (this.buffer.size() > 0) {
+ this.processBuffer();
+ }
+
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (this.buffer.size() > 0) {
+ this.processBuffer();
+ }
+
+ super.close();
+ }
+
+ public int getMessageLevel() {
+ return this.level;
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+ int offset = off;
+ int blockStartOffset = off;
+
+ for (int remaining = len; remaining > 0; blockStartOffset = offset) {
+ while (remaining > 0 && b[offset] != 10 && b[offset] != 13) {
+ ++offset;
+ --remaining;
+ }
+
+ int blockLength = offset - blockStartOffset;
+ if (blockLength > 0) {
+ this.buffer.write(b, blockStartOffset, blockLength);
+ }
+
+ while (remaining > 0 && (b[offset] == 10 || b[offset] == 13)) {
+ this.write(b[offset]);
+ ++offset;
+ --remaining;
+ }
+ }
+
+ }
+
+ protected void processBuffer() {
+ this.processLine(this.buffer.toString());
+ this.buffer.reset();
+ }
+
+ protected void processLine(String line) {
+ this.processLine(line, this.level);
+ }
+
+ protected abstract void processLine(String var1, int var2);
+}
diff --git a/spark/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java b/spark/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
index f63f3d4edf8..ed85558028d 100644
--- a/spark/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
+++ b/spark/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
@@ -75,6 +75,7 @@ public class PySparkInterpreter extends Interpreter implements ExecuteResultHand
private ByteArrayOutputStream input;
private String scriptPath;
boolean pythonscriptRunning = false;
+ private static final int MAX_TIMEOUT_SEC = 10;
public PySparkInterpreter(Properties property) {
super(property);
@@ -179,7 +180,7 @@ private void createGatewayServerAndStartScript() {
cmd.addArgument(Integer.toString(port), false);
cmd.addArgument(Integer.toString(getSparkInterpreter().getSparkVersion().toNumber()), false);
executor = new DefaultExecutor();
- outputStream = new SparkOutputStream();
+ outputStream = new SparkOutputStream(logger);
PipedOutputStream ps = new PipedOutputStream();
in = null;
try {
@@ -316,7 +317,7 @@ public InterpreterResult interpret(String st, InterpreterContext context) {
long startTime = System.currentTimeMillis();
while (pythonScriptInitialized == false
&& pythonscriptRunning
- && System.currentTimeMillis() - startTime < 10 * 1000) {
+ && System.currentTimeMillis() - startTime < MAX_TIMEOUT_SEC * 1000) {
try {
pythonScriptInitializeNotifier.wait(1000);
} catch (InterruptedException e) {
@@ -423,8 +424,15 @@ public List completion(String buf, int cursor) {
}
synchronized (statementFinishedNotifier) {
- while (statementOutput == null) {
+ long startTime = System.currentTimeMillis();
+ while (statementOutput == null
+ && pythonScriptInitialized == false
+ && pythonscriptRunning) {
try {
+ if (System.currentTimeMillis() - startTime < MAX_TIMEOUT_SEC * 1000) {
+ logger.error("pyspark completion didn't have response for {}sec.", MAX_TIMEOUT_SEC);
+ break;
+ }
statementFinishedNotifier.wait(1000);
} catch (InterruptedException e) {
// not working
diff --git a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
index 29c322d78d6..9a54912a35c 100644
--- a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
+++ b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
@@ -33,6 +33,7 @@
import com.google.common.base.Joiner;
+import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
@@ -48,6 +49,7 @@
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.InterpreterContext;
import org.apache.zeppelin.interpreter.InterpreterException;
+import org.apache.zeppelin.interpreter.InterpreterProperty;
import org.apache.zeppelin.interpreter.InterpreterResult;
import org.apache.zeppelin.interpreter.InterpreterResult.Code;
import org.apache.zeppelin.interpreter.InterpreterUtils;
@@ -67,10 +69,8 @@
import scala.collection.Iterator;
import scala.collection.JavaConversions;
import scala.collection.JavaConverters;
-import scala.collection.convert.WrapAsJava;
import scala.collection.Seq;
import scala.collection.convert.WrapAsJava$;
-import scala.collection.convert.WrapAsScala;
import scala.collection.mutable.HashMap;
import scala.collection.mutable.HashSet;
import scala.reflect.io.AbstractFile;
@@ -114,7 +114,7 @@ public class SparkInterpreter extends Interpreter {
/**
* completer - org.apache.spark.repl.SparkJLineCompletion (scala 2.10)
*/
- private Object completer;
+ private Object completer = null;
private Map binder;
private SparkVersion sparkVersion;
@@ -124,7 +124,7 @@ public class SparkInterpreter extends Interpreter {
public SparkInterpreter(Properties property) {
super(property);
- out = new SparkOutputStream();
+ out = new SparkOutputStream(logger);
}
public SparkInterpreter(Properties property, SparkContext sc) {
@@ -238,10 +238,7 @@ public SQLContext getSQLContext() {
*/
private SQLContext getSQLContext_2() {
if (sqlc == null) {
- sqlc = (SQLContext) Utils.invokeMethod(sparkSession, "wrapped");
- if (sqlc == null) {
- sqlc = (SQLContext) Utils.invokeMethod(sparkSession, "sqlContext");
- }
+ sqlc = (SQLContext) Utils.invokeMethod(sparkSession, "sqlContext");
}
return sqlc;
}
@@ -305,7 +302,9 @@ public Object createSparkSession() {
String execUri = System.getenv("SPARK_EXECUTOR_URI");
conf.setAppName(getProperty("spark.app.name"));
- conf.set("spark.repl.class.outputDir", outputDir.getAbsolutePath());
+ if (outputDir != null) {
+ conf.set("spark.repl.class.outputDir", outputDir.getAbsolutePath());
+ }
if (execUri != null) {
conf.set("spark.executor.uri", execUri);
@@ -329,6 +328,7 @@ public Object createSparkSession() {
}
}
+ setupConfForPySpark(conf);
Class SparkSession = Utils.findClass("org.apache.spark.sql.SparkSession");
Object builder = Utils.invokeStaticMethod(SparkSession, "builder");
Utils.invokeMethod(builder, "config", new Class[]{ SparkConf.class }, new Object[]{ conf });
@@ -442,12 +442,17 @@ public SparkContext createSparkContext_1() {
conf.set(key, val);
}
}
+ setupConfForPySpark(conf);
+ SparkContext sparkContext = new SparkContext(conf);
+ return sparkContext;
+ }
- //TODO(jongyoul): Move these codes into PySparkInterpreter.java
- String pysparkBasePath = getSystemDefault("SPARK_HOME", null, null);
+ private void setupConfForPySpark(SparkConf conf) {
+ String pysparkBasePath = new InterpreterProperty("SPARK_HOME", null, null, null).getValue();
File pysparkPath;
if (null == pysparkBasePath) {
- pysparkBasePath = getSystemDefault("ZEPPELIN_HOME", "zeppelin.home", "../");
+ pysparkBasePath =
+ new InterpreterProperty("ZEPPELIN_HOME", "zeppelin.home", "../", null).getValue();
pysparkPath = new File(pysparkBasePath,
"interpreter" + File.separator + "spark" + File.separator + "pyspark");
} else {
@@ -456,7 +461,8 @@ public SparkContext createSparkContext_1() {
}
//Only one of py4j-0.9-src.zip and py4j-0.8.2.1-src.zip should exist
- String[] pythonLibs = new String[]{"pyspark.zip", "py4j-0.9-src.zip", "py4j-0.8.2.1-src.zip"};
+ String[] pythonLibs = new String[]{"pyspark.zip", "py4j-0.9-src.zip", "py4j-0.8.2.1-src.zip",
+ "py4j-0.10.1-src.zip"};
ArrayList pythonLibUris = new ArrayList<>();
for (String lib : pythonLibs) {
File libFile = new File(pysparkPath, lib);
@@ -486,9 +492,6 @@ public SparkContext createSparkContext_1() {
if (getProperty("master").equals("yarn-client")) {
conf.set("spark.yarn.isPython", "true");
}
-
- SparkContext sparkContext = new SparkContext(conf);
- return sparkContext;
}
static final String toString(Object o) {
@@ -499,27 +502,6 @@ private boolean useSparkSubmit() {
return null != System.getenv("SPARK_SUBMIT");
}
- public static String getSystemDefault(
- String envName,
- String propertyName,
- String defaultValue) {
-
- if (envName != null && !envName.isEmpty()) {
- String envValue = System.getenv().get(envName);
- if (envValue != null) {
- return envValue;
- }
- }
-
- if (propertyName != null && !propertyName.isEmpty()) {
- String propValue = System.getProperty(propertyName);
- if (propValue != null) {
- return propValue;
- }
- }
- return defaultValue;
- }
-
public boolean printREPLOutput() {
return java.lang.Boolean.parseBoolean(getProperty("zeppelin.spark.printREPLOutput"));
}
@@ -530,8 +512,8 @@ public void open() {
if (getProperty("master").equals("yarn-client")) {
System.setProperty("SPARK_YARN_MODE", "true");
}
- if (getProperty().contains("spark.yarn.keytab") &&
- getProperty().contains("spark.yarn.principal")) {
+ if (getProperty().containsKey("spark.yarn.keytab") &&
+ getProperty().containsKey("spark.yarn.principal")) {
try {
String keytab = getProperty().getProperty("spark.yarn.keytab");
String principal = getProperty().getProperty("spark.yarn.principal");
@@ -595,7 +577,11 @@ public void open() {
argList.add("-Yrepl-class-based");
argList.add("-Yrepl-outdir");
argList.add(outputDir.getAbsolutePath());
-
+ if (conf.contains("spark.jars")) {
+ String jars = StringUtils.join(conf.get("spark.jars").split(","), File.separator);
+ argList.add("-classpath");
+ argList.add(jars);
+ }
scala.collection.immutable.List list =
JavaConversions.asScalaBuffer(argList).toList();
@@ -720,11 +706,25 @@ public void open() {
logger.error(e.getMessage(), e);
}
}
+ }
+ if (Utils.findClass("org.apache.spark.repl.SparkJLineCompletion", true) != null) {
completer = Utils.instantiateClass(
"org.apache.spark.repl.SparkJLineCompletion",
new Class[]{Utils.findClass("org.apache.spark.repl.SparkIMain")},
new Object[]{intp});
+ } else if (Utils.findClass(
+ "scala.tools.nsc.interpreter.PresentationCompilerCompleter", true) != null) {
+ completer = Utils.instantiateClass(
+ "scala.tools.nsc.interpreter.PresentationCompilerCompleter",
+ new Class[]{ IMain.class },
+ new Object[]{ intp });
+ } else if (Utils.findClass(
+ "scala.tools.nsc.interpreter.JLineCompletion", true) != null) {
+ completer = Utils.instantiateClass(
+ "scala.tools.nsc.interpreter.JLineCompletion",
+ new Class[]{ IMain.class },
+ new Object[]{ intp });
}
if (Utils.isSpark2()) {
@@ -903,6 +903,11 @@ private List classPath(ClassLoader cl) {
@Override
public List completion(String buf, int cursor) {
+ if (completer == null) {
+ logger.warn("Can't find completer");
+ return new LinkedList();
+ }
+
if (buf.length() < cursor) {
cursor = buf.length();
}
@@ -911,22 +916,18 @@ public List completion(String buf, int cursor) {
completionText = "";
cursor = completionText.length();
}
- if (Utils.isScala2_10()) {
- ScalaCompleter c = (ScalaCompleter) Utils.invokeMethod(completer, "completer");
- Candidates ret = c.complete(completionText, cursor);
- List candidates = WrapAsJava$.MODULE$.seqAsJavaList(ret.candidates());
- List completions = new LinkedList();
+ ScalaCompleter c = (ScalaCompleter) Utils.invokeMethod(completer, "completer");
+ Candidates ret = c.complete(completionText, cursor);
- for (String candidate : candidates) {
- completions.add(new InterpreterCompletion(candidate, candidate));
- }
+ List candidates = WrapAsJava$.MODULE$.seqAsJavaList(ret.candidates());
+ List completions = new LinkedList();
- return completions;
- } else {
- return new LinkedList();
+ for (String candidate : candidates) {
+ completions.add(new InterpreterCompletion(candidate, candidate));
}
+ return completions;
}
private String getCompletionTargetString(String text, int cursor) {
diff --git a/spark/src/main/java/org/apache/zeppelin/spark/SparkOutputStream.java b/spark/src/main/java/org/apache/zeppelin/spark/SparkOutputStream.java
index 98a4090b117..e454994aa0d 100644
--- a/spark/src/main/java/org/apache/zeppelin/spark/SparkOutputStream.java
+++ b/spark/src/main/java/org/apache/zeppelin/spark/SparkOutputStream.java
@@ -17,17 +17,20 @@
package org.apache.zeppelin.spark;
import org.apache.zeppelin.interpreter.InterpreterOutput;
+import org.slf4j.Logger;
import java.io.IOException;
-import java.io.OutputStream;
/**
* InterpreterOutput can be attached / detached.
*/
-public class SparkOutputStream extends OutputStream {
+public class SparkOutputStream extends LogOutputStream {
+
+ public static Logger logger;
InterpreterOutput interpreterOutput;
- public SparkOutputStream() {
+ public SparkOutputStream(Logger logger) {
+ this.logger = logger;
}
public InterpreterOutput getInterpreterOutput() {
@@ -40,6 +43,7 @@ public void setInterpreterOutput(InterpreterOutput interpreterOutput) {
@Override
public void write(int b) throws IOException {
+ super.write(b);
if (interpreterOutput != null) {
interpreterOutput.write(b);
}
@@ -47,6 +51,7 @@ public void write(int b) throws IOException {
@Override
public void write(byte [] b) throws IOException {
+ super.write(b);
if (interpreterOutput != null) {
interpreterOutput.write(b);
}
@@ -54,13 +59,20 @@ public void write(byte [] b) throws IOException {
@Override
public void write(byte [] b, int offset, int len) throws IOException {
+ super.write(b, offset, len);
if (interpreterOutput != null) {
interpreterOutput.write(b, offset, len);
}
}
+ @Override
+ protected void processLine(String s, int i) {
+ logger.debug("Interpreter output:" + s);
+ }
+
@Override
public void close() throws IOException {
+ super.close();
if (interpreterOutput != null) {
interpreterOutput.close();
}
@@ -68,6 +80,7 @@ public void close() throws IOException {
@Override
public void flush() throws IOException {
+ super.flush();
if (interpreterOutput != null) {
interpreterOutput.flush();
}
diff --git a/spark/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java b/spark/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
index 8329641d8f3..06139496020 100644
--- a/spark/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
+++ b/spark/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
@@ -21,6 +21,7 @@
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.spark.SparkContext;
import org.apache.spark.SparkRBackend;
import org.apache.zeppelin.interpreter.*;
import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
@@ -70,11 +71,16 @@ public void open() {
int port = SparkRBackend.port();
SparkInterpreter sparkInterpreter = getSparkInterpreter();
- ZeppelinRContext.setSparkContext(sparkInterpreter.getSparkContext());
+ SparkContext sc = sparkInterpreter.getSparkContext();
+ SparkVersion sparkVersion = new SparkVersion(sc.version());
+ ZeppelinRContext.setSparkContext(sc);
+ if (Utils.isSpark2()) {
+ ZeppelinRContext.setSparkSession(sparkInterpreter.getSparkSession());
+ }
ZeppelinRContext.setSqlContext(sparkInterpreter.getSQLContext());
- ZeppelinRContext.setZepplinContext(sparkInterpreter.getZeppelinContext());
+ ZeppelinRContext.setZeppelinContext(sparkInterpreter.getZeppelinContext());
- zeppelinR = new ZeppelinR(rCmdPath, sparkRLibPath, port);
+ zeppelinR = new ZeppelinR(rCmdPath, sparkRLibPath, port, sparkVersion);
try {
zeppelinR.open();
} catch (IOException e) {
diff --git a/spark/src/main/java/org/apache/zeppelin/spark/Utils.java b/spark/src/main/java/org/apache/zeppelin/spark/Utils.java
index 328fa199ca6..78304fd8713 100644
--- a/spark/src/main/java/org/apache/zeppelin/spark/Utils.java
+++ b/spark/src/main/java/org/apache/zeppelin/spark/Utils.java
@@ -56,10 +56,16 @@ static Object invokeStaticMethod(Class c, String name) {
}
static Class findClass(String name) {
+ return findClass(name, false);
+ }
+
+ static Class findClass(String name, boolean silence) {
try {
return Utils.class.forName(name);
} catch (ClassNotFoundException e) {
- logger.error(e.getMessage(), e);
+ if (!silence) {
+ logger.error(e.getMessage(), e);
+ }
return null;
}
}
@@ -83,6 +89,8 @@ static boolean isScala2_10() {
return true;
} catch (ClassNotFoundException e) {
return false;
+ } catch (IncompatibleClassChangeError e) {
+ return false;
}
}
diff --git a/spark/src/main/java/org/apache/zeppelin/spark/ZeppelinR.java b/spark/src/main/java/org/apache/zeppelin/spark/ZeppelinR.java
index 0ff07403911..26488337b11 100644
--- a/spark/src/main/java/org/apache/zeppelin/spark/ZeppelinR.java
+++ b/spark/src/main/java/org/apache/zeppelin/spark/ZeppelinR.java
@@ -36,6 +36,7 @@
public class ZeppelinR implements ExecuteResultHandler {
Logger logger = LoggerFactory.getLogger(ZeppelinR.class);
private final String rCmdPath;
+ private final SparkVersion sparkVersion;
private DefaultExecutor executor;
private SparkOutputStream outputStream;
private PipedOutputStream input;
@@ -107,9 +108,11 @@ public Object getValue() {
* @param rCmdPath R repl commandline path
* @param libPath sparkr library path
*/
- public ZeppelinR(String rCmdPath, String libPath, int sparkRBackendPort) {
+ public ZeppelinR(String rCmdPath, String libPath, int sparkRBackendPort,
+ SparkVersion sparkVersion) {
this.rCmdPath = rCmdPath;
this.libPath = libPath;
+ this.sparkVersion = sparkVersion;
this.port = sparkRBackendPort;
try {
File scriptFile = File.createTempFile("zeppelin_sparkr-", ".R");
@@ -137,9 +140,10 @@ public void open() throws IOException {
cmd.addArgument(Integer.toString(hashCode()));
cmd.addArgument(Integer.toString(port));
cmd.addArgument(libPath);
+ cmd.addArgument(Integer.toString(sparkVersion.toNumber()));
executor = new DefaultExecutor();
- outputStream = new SparkOutputStream();
+ outputStream = new SparkOutputStream(logger);
input = new PipedOutputStream();
PipedInputStream in = new PipedInputStream(input);
diff --git a/spark/src/main/java/org/apache/zeppelin/spark/ZeppelinRContext.java b/spark/src/main/java/org/apache/zeppelin/spark/ZeppelinRContext.java
index 82c320d7f21..935410bdd59 100644
--- a/spark/src/main/java/org/apache/zeppelin/spark/ZeppelinRContext.java
+++ b/spark/src/main/java/org/apache/zeppelin/spark/ZeppelinRContext.java
@@ -27,12 +27,13 @@ public class ZeppelinRContext {
private static SparkContext sparkContext;
private static SQLContext sqlContext;
private static ZeppelinContext zeppelinContext;
+ private static Object sparkSession;
public static void setSparkContext(SparkContext sparkContext) {
ZeppelinRContext.sparkContext = sparkContext;
}
- public static void setZepplinContext(ZeppelinContext zeppelinContext) {
+ public static void setZeppelinContext(ZeppelinContext zeppelinContext) {
ZeppelinRContext.zeppelinContext = zeppelinContext;
}
@@ -40,6 +41,10 @@ public static void setSqlContext(SQLContext sqlContext) {
ZeppelinRContext.sqlContext = sqlContext;
}
+ public static void setSparkSession(Object sparkSession) {
+ ZeppelinRContext.sparkSession = sparkSession;
+ }
+
public static SparkContext getSparkContext() {
return sparkContext;
}
@@ -52,4 +57,7 @@ public static ZeppelinContext getZeppelinContext() {
return zeppelinContext;
}
+ public static Object getSparkSession() {
+ return sparkSession;
+ }
}
diff --git a/spark/src/main/resources/R/zeppelin_sparkr.R b/spark/src/main/resources/R/zeppelin_sparkr.R
index fe2a16b973f..d9517749bbf 100644
--- a/spark/src/main/resources/R/zeppelin_sparkr.R
+++ b/spark/src/main/resources/R/zeppelin_sparkr.R
@@ -21,6 +21,7 @@ args <- commandArgs(trailingOnly = TRUE)
hashCode <- as.integer(args[1])
port <- as.integer(args[2])
libPath <- args[3]
+version <- as.integer(args[4])
rm(args)
print(paste("Port ", toString(port)))
@@ -41,6 +42,10 @@ assign(".scStartTime", as.integer(Sys.time()), envir = SparkR:::.sparkREnv)
# setup spark env
assign(".sc", SparkR:::callJStatic("org.apache.zeppelin.spark.ZeppelinRContext", "getSparkContext"), envir = SparkR:::.sparkREnv)
assign("sc", get(".sc", envir = SparkR:::.sparkREnv), envir=.GlobalEnv)
+if (version >= 200) {
+ assign(".sparkRsession", SparkR:::callJStatic("org.apache.zeppelin.spark.ZeppelinRContext", "getSparkSession"), envir = SparkR:::.sparkREnv)
+ assign("spark", get(".sparkRsession", envir = SparkR:::.sparkREnv), envir = .GlobalEnv)
+}
assign(".sqlc", SparkR:::callJStatic("org.apache.zeppelin.spark.ZeppelinRContext", "getSqlContext"), envir = SparkR:::.sparkREnv)
assign("sqlContext", get(".sqlc", envir = SparkR:::.sparkREnv), envir = .GlobalEnv)
assign(".zeppelinContext", SparkR:::callJStatic("org.apache.zeppelin.spark.ZeppelinRContext", "getZeppelinContext"), envir = .GlobalEnv)
diff --git a/spark/src/main/resources/interpreter-setting.json b/spark/src/main/resources/interpreter-setting.json
index 2343a0f9745..d87a6c7ff75 100644
--- a/spark/src/main/resources/interpreter-setting.json
+++ b/spark/src/main/resources/interpreter-setting.json
@@ -46,7 +46,7 @@
"envName": "ZEPPELIN_SPARK_MAXRESULT",
"propertyName": "zeppelin.spark.maxResult",
"defaultValue": "1000",
- "description": "Max number of SparkSQL result to display."
+ "description": "Max number of Spark SQL result to display."
},
"master": {
"envName": "MASTER",
@@ -77,7 +77,7 @@
"envName": "ZEPPELIN_SPARK_MAXRESULT",
"propertyName": "zeppelin.spark.maxResult",
"defaultValue": "1000",
- "description": "Max number of SparkSQL result to display."
+ "description": "Max number of Spark SQL result to display."
},
"zeppelin.spark.importImplicit": {
"envName": "ZEPPELIN_SPARK_IMPORTIMPLICIT",
diff --git a/spark/src/main/resources/python/zeppelin_pyspark.py b/spark/src/main/resources/python/zeppelin_pyspark.py
index 0380afa7fea..9a405566036 100644
--- a/spark/src/main/resources/python/zeppelin_pyspark.py
+++ b/spark/src/main/resources/python/zeppelin_pyspark.py
@@ -27,19 +27,21 @@
from pyspark.accumulators import Accumulator, AccumulatorParam
from pyspark.broadcast import Broadcast
from pyspark.serializers import MarshalSerializer, PickleSerializer
+import ast
+import traceback
# for back compatibility
from pyspark.sql import SQLContext, HiveContext, Row
class Logger(object):
def __init__(self):
- self.out = ""
+ pass
def write(self, message):
intp.appendOutput(message)
def reset(self):
- self.out = ""
+ pass
def flush(self):
pass
@@ -230,7 +232,7 @@ def getCompletion(self, text_value):
try:
stmts = req.statements().split("\n")
jobGroup = req.jobGroup()
- final_code = None
+ final_code = []
for s in stmts:
if s == None:
@@ -241,15 +243,27 @@ def getCompletion(self, text_value):
if len(s_stripped) == 0 or s_stripped.startswith("#"):
continue
- if final_code:
- final_code += "\n" + s
- else:
- final_code = s
+ final_code.append(s)
if final_code:
- compiledCode = compile(final_code, "", "exec")
+ # use exec mode to compile the statements except the last statement,
+ # so that the last statement's evaluation will be printed to stdout
sc.setJobGroup(jobGroup, "Zeppelin")
- eval(compiledCode)
+ code = compile('\n'.join(final_code), '', 'exec', ast.PyCF_ONLY_AST, 1)
+ to_run_exec, to_run_single = code.body[:-1], code.body[-1:]
+
+ try:
+ for node in to_run_exec:
+ mod = ast.Module([node])
+ code = compile(mod, '', 'exec')
+ exec(code)
+
+ for node in to_run_single:
+ mod = ast.Interactive([node])
+ code = compile(mod, '', 'single')
+ exec(code)
+ except:
+ raise Exception(traceback.format_exc())
intp.setStatementsFinished("", False)
except Py4JJavaError:
diff --git a/spark/src/main/sparkr-resources/interpreter-setting.json b/spark/src/main/sparkr-resources/interpreter-setting.json
index 4902baf9f76..f884fe48702 100644
--- a/spark/src/main/sparkr-resources/interpreter-setting.json
+++ b/spark/src/main/sparkr-resources/interpreter-setting.json
@@ -46,7 +46,7 @@
"envName": "ZEPPELIN_SPARK_MAXRESULT",
"propertyName": "zeppelin.spark.maxResult",
"defaultValue": "1000",
- "description": "Max number of SparkSQL result to display."
+ "description": "Max number of Spark SQL result to display."
},
"master": {
"envName": "MASTER",
@@ -77,7 +77,7 @@
"envName": "ZEPPELIN_SPARK_MAXRESULT",
"propertyName": "zeppelin.spark.maxResult",
"defaultValue": "1000",
- "description": "Max number of SparkSQL result to display."
+ "description": "Max number of Spark SQL result to display."
},
"zeppelin.spark.importImplicit": {
"envName": "ZEPPELIN_SPARK_IMPORTIMPLICIT",
diff --git a/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java b/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java
index badc4e20b0d..1c7979fc428 100644
--- a/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java
+++ b/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java
@@ -19,16 +19,16 @@
import static org.junit.Assert.*;
-import java.io.BufferedReader;
import java.io.File;
import java.util.HashMap;
import java.util.LinkedList;
+import java.util.List;
import java.util.Properties;
import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
-import org.apache.spark.repl.SparkILoop;
import org.apache.zeppelin.display.AngularObjectRegistry;
+import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
import org.apache.zeppelin.resource.LocalResourcePool;
import org.apache.zeppelin.resource.WellKnownResourceName;
import org.apache.zeppelin.user.AuthenticationInfo;
@@ -42,7 +42,6 @@
import org.junit.runners.MethodSorters;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import scala.tools.nsc.interpreter.IMain;
@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class SparkInterpreterTest {
@@ -282,4 +281,10 @@ public void testDisableImplicitImport() {
assertEquals(Code.ERROR, repl2.interpret(ddl, context).code());
repl2.close();
}
+
+ @Test
+ public void testCompletion() {
+ List completions = repl.completion("sc.", "sc.".length());
+ assertTrue(completions.size() > 0);
+ }
}
diff --git a/zeppelin-display/pom.xml b/zeppelin-display/pom.xml
index 47644eefcbf..d33038e88ed 100644
--- a/zeppelin-display/pom.xml
+++ b/zeppelin-display/pom.xml
@@ -81,6 +81,7 @@
org.scala-lang
scala-library
${scala.version}
+ provided
@@ -94,15 +95,12 @@
scala-2.11
-
- scala-2.11
-
org.scala-lang.modules
scala-xml_${scala.binary.version}
1.0.2
+ provided
@@ -110,24 +108,6 @@
-
- org.apache.rat
- apache-rat-plugin
-
-
- **/.idea/
- **/*.iml
- .git/
- .gitignore
- **/.settings/*
- **/.classpath
- **/.project
- **/target/**
- **/README.md
-
-
-
-
maven-failsafe-plugin
2.16
diff --git a/zeppelin-distribution/README.md b/zeppelin-distribution/README.md
index 72fd63077ba..b5dd2e07437 100644
--- a/zeppelin-distribution/README.md
+++ b/zeppelin-distribution/README.md
@@ -15,17 +15,19 @@
limitations under the License.
-->
-# Distribution archive of Zeppelin project #
+# Distribution archive of Apache Zeppelin
-Zeppelin is distributed as a single gzip archive with the following structure:
+Apache Zeppelin is distributed as a single gzip archive with the following structure:
```
-zeppelin
+Zeppelin
├── bin
│ ├── zeppelin.sh
│ └── seppelin-deamon.sh
- ├── lib
├── conf
+ ├── interpreter
+ ├── lib
+ ├── licenses
├── zan-repo
│ ├── txt.wordcount
│ ├── vis.bubble
@@ -33,12 +35,11 @@ zeppelin
│ ├── ml.something
│ └── ...
├── zeppelin-server-.jar
- ├── zeppelin-web-.war
- └── zeppelin-cli-.jar
+ └── zeppelin-web-.war
```
-We use maven-assembly-pugin to build it, see distribution.xml for details
+We use `maven-assembly-plugin` to build it, see `zeppelin-distribution/src/assemble/distribution.xml ` for details.
-**IMPORTANT:** _/lib_ subdirectory contains all transitive dependencies of the zeppelin-distribution module,
-automatically resolved by maven, except for explicitly excluded _server_, _web_ and _cli_ zeppelin sub-modules.
+>**IMPORTANT:** `_/lib_` subdirectory contains all transitive dependencies of the `zeppelin-distribution` module,
+automatically resolved by maven, except for explicitly excluded `_server_` and `_web_` Zeppelin sub-modules.
diff --git a/zeppelin-distribution/build-infrastructure.md b/zeppelin-distribution/build-infrastructure.md
index b8b3e2088d6..676266c36f9 100644
--- a/zeppelin-distribution/build-infrastructure.md
+++ b/zeppelin-distribution/build-infrastructure.md
@@ -15,38 +15,41 @@
limitations under the License.
-->
-Zeppelin dependency graph:
---------------
- hive, hadoop, ...
+# Apache Zeppelin Build Infrastructure
+
+## Dependency graph
+
+```
+ e.g. hive, hadoop, ...
| | |
v v v
- Zeppelin Server <- Zengine -> Zeppelin CLI
+ Zeppelin Server <- Zengine
+ |
zeppeli web v
ZAN
+```
+
+
+## Artifacts
+ - Zeppelin Server : Web UI, server to host it / executable
+ - Zeppelin Web : Web UI, clint-side JS app / HTML+JavaScript; war
+ - Zeppelin Zengine : Main library / java library
+ - ZAN
-Zeppelin artifacts:
-------------------
-Zeppelin CLI - Commandline UI - executable
-Zeppelin Server - Web UI, server to host it - executable
-Zwppwlin Web - Web UI, clint-side JS app - HTML+JavaScript; war
-Zengine - Main library - java library
-ZAN -
+## Build process
+ - compile => *.class, minify *.js
+ - build modules => *.jar, war
+ - test => UnitTest reports
+ - package -P build-distr => final .zip
+ - integration-test => selenium over running zeppelin-server (from package)
-Build process:
--------------
-compile => *.class, minify *.js
-build modules => *.jar, war
-test => UnitTest reports
-package -P build-distr => final .zip
-integration-test => selenium over running zeppelin-server (from package)
+## Verify
-verify:
- pre-inegration-test => start Zeppelin
- integration-test
- post-inegration-test => stop Zeppelin
+ - pre-inegration-test => start Zeppelin
+ - integration-test
+ - post-inegration-test => stop Zeppelin
diff --git a/zeppelin-distribution/pom.xml b/zeppelin-distribution/pom.xml
index f65044e6faf..378c980909a 100644
--- a/zeppelin-distribution/pom.xml
+++ b/zeppelin-distribution/pom.xml
@@ -114,10 +114,6 @@
scala-2.11
-
- scala-2.11
-
diff --git a/zeppelin-distribution/src/bin_license/LICENSE b/zeppelin-distribution/src/bin_license/LICENSE
index ffecbe21ec0..b33ad64b108 100644
--- a/zeppelin-distribution/src/bin_license/LICENSE
+++ b/zeppelin-distribution/src/bin_license/LICENSE
@@ -114,6 +114,7 @@ The following components are provided under Apache License.
(Apache 2.0) Utility classes for Jetty (org.mortbay.jetty:jetty-util:6.1.26 - http://javadox.com/org.mortbay.jetty/jetty/6.1.26/overview-tree.html)
(Apache 2.0) Servlet API (org.mortbay.jetty:servlet-api:2.5-20081211 - https://en.wikipedia.org/wiki/Jetty_(web_server))
(Apache 2.0) Google HTTP Client Library for Java (com.google.http-client:google-http-client-jackson2:1.21.0 - https://github.com/google/google-http-java-client/tree/dev/google-http-client-jackson2)
+ (Apache 2.0) angular-esri-map (https://github.com/Esri/angular-esri-map)
========================================================================
MIT licenses
diff --git a/zeppelin-examples/zeppelin-example-horizontalbar/pom.xml b/zeppelin-examples/zeppelin-example-horizontalbar/pom.xml
index f6a73385931..6305a2e61ed 100644
--- a/zeppelin-examples/zeppelin-example-horizontalbar/pom.xml
+++ b/zeppelin-examples/zeppelin-example-horizontalbar/pom.xml
@@ -105,16 +105,6 @@
-
-
- org.apache.rat
- apache-rat-plugin
-
-
- **/horizontalbar_mockdata.txt
-
-
-
diff --git a/zeppelin-interpreter/pom.xml b/zeppelin-interpreter/pom.xml
index 3309746b527..9a509004224 100644
--- a/zeppelin-interpreter/pom.xml
+++ b/zeppelin-interpreter/pom.xml
@@ -217,24 +217,6 @@
-
- org.apache.rat
- apache-rat-plugin
-
-
- **/.idea/
- **/*.iml
- .gitignore
- **/.settings/*
- **/.classpath
- **/.project
- **/target/**
- *.md
- dependency-reduced-pom.xml
-
-
-
-
org.apache.maven.plugins
maven-shade-plugin
diff --git a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Interpreter.java b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Interpreter.java
index 07f9cbabcad..42caafdfc61 100644
--- a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Interpreter.java
+++ b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/Interpreter.java
@@ -345,10 +345,14 @@ public static void register(String name, String group, String className,
}
public static void register(RegisteredInterpreter registeredInterpreter) {
- // TODO(jongyoul): Error should occur when two same interpreter key with different settings
String interpreterKey = registeredInterpreter.getInterpreterKey();
if (!registeredInterpreters.containsKey(interpreterKey)) {
registeredInterpreters.put(interpreterKey, registeredInterpreter);
+ } else {
+ RegisteredInterpreter existInterpreter = registeredInterpreters.get(interpreterKey);
+ if (!existInterpreter.getProperties().equals(registeredInterpreter.getProperties())) {
+ logger.error("exist registeredInterpreter with the same key but has different settings.");
+ }
}
}
diff --git a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterProperty.java b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterProperty.java
index 488f2a1fa93..c69de5d4a20 100644
--- a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterProperty.java
+++ b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterProperty.java
@@ -70,8 +70,15 @@ public void setDescription(String description) {
this.description = description;
}
+ public int hashCode() {
+ return this.toString().hashCode();
+ }
+
+ public boolean equals(Object o) {
+ return this.toString().equals(o.toString());
+ }
+
public String getValue() {
- //TODO(jongyoul): Remove SparkInterpreter's getSystemDefault method
if (envName != null && !envName.isEmpty()) {
String envValue = System.getenv().get(envName);
if (envValue != null) {
@@ -90,7 +97,7 @@ public String getValue() {
@Override
public String toString() {
- return String.format("{envName=%s, propertyName=%s, defaultValue=%s, description=%20s", envName,
- propertyName, defaultValue, description);
+ return String.format("{envName=%s, propertyName=%s, defaultValue=%s, description=%20s}",
+ envName, propertyName, defaultValue, description);
}
}
diff --git a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterPropertyBuilder.java b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterPropertyBuilder.java
index f077b4e45e0..f33dc7c37e1 100644
--- a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterPropertyBuilder.java
+++ b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterPropertyBuilder.java
@@ -31,6 +31,13 @@ public InterpreterPropertyBuilder add(String name, String defaultValue, String d
return this;
}
+ public InterpreterPropertyBuilder add(String name, String envName, String propertyName,
+ String defaultValue, String description){
+ properties.put(name,
+ new InterpreterProperty(envName, propertyName, defaultValue, description));
+ return this;
+ }
+
public Map build(){
return properties;
}
diff --git a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/AppendOutputBuffer.java b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/AppendOutputBuffer.java
new file mode 100644
index 00000000000..e1484dabaaa
--- /dev/null
+++ b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/AppendOutputBuffer.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.interpreter.remote;
+
+/**
+ * This element stores the buffered
+ * append-data of paragraph's output.
+ */
+public class AppendOutputBuffer {
+
+ private String noteId;
+ private String paragraphId;
+ private String data;
+
+ public AppendOutputBuffer(String noteId, String paragraphId, String data) {
+ this.noteId = noteId;
+ this.paragraphId = paragraphId;
+ this.data = data;
+ }
+
+ public String getNoteId() {
+ return noteId;
+ }
+
+ public String getParagraphId() {
+ return paragraphId;
+ }
+
+ public String getData() {
+ return data;
+ }
+
+}
diff --git a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/AppendOutputRunner.java b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/AppendOutputRunner.java
new file mode 100644
index 00000000000..86ea11a8bda
--- /dev/null
+++ b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/AppendOutputRunner.java
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.interpreter.remote;
+
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This thread sends paragraph's append-data
+ * periodically, rather than continously, with
+ * a period of BUFFER_TIME_MS. It handles append-data
+ * for all paragraphs across all notebooks.
+ */
+public class AppendOutputRunner implements Runnable {
+
+ private static final Logger logger =
+ LoggerFactory.getLogger(AppendOutputRunner.class);
+ public static final Long BUFFER_TIME_MS = new Long(100);
+ private static final Long SAFE_PROCESSING_TIME = new Long(10);
+ private static final Long SAFE_PROCESSING_STRING_SIZE = new Long(100000);
+
+ private final BlockingQueue queue = new LinkedBlockingQueue<>();
+ private final RemoteInterpreterProcessListener listener;
+
+ public AppendOutputRunner(RemoteInterpreterProcessListener listener) {
+ this.listener = listener;
+ }
+
+ @Override
+ public void run() {
+
+ Map > noteMap = new HashMap<>();
+ List list = new LinkedList<>();
+
+ /* "drainTo" method does not wait for any element
+ * to be present in the queue, and thus this loop would
+ * continuosly run (with period of BUFFER_TIME_MS). "take()" method
+ * waits for the queue to become non-empty and then removes
+ * one element from it. Rest elements from queue (if present) are
+ * removed using "drainTo" method. Thus we save on some un-necessary
+ * cpu-cycles.
+ */
+ try {
+ list.add(queue.take());
+ } catch (InterruptedException e) {
+ logger.error("Wait for OutputBuffer queue interrupted: " + e.getMessage());
+ }
+ Long processingStartTime = System.currentTimeMillis();
+ queue.drainTo(list);
+
+ for (AppendOutputBuffer buffer: list) {
+ String noteId = buffer.getNoteId();
+ String paragraphId = buffer.getParagraphId();
+
+ Map paragraphMap = (noteMap.containsKey(noteId)) ?
+ noteMap.get(noteId) : new HashMap();
+ StringBuilder builder = paragraphMap.containsKey(paragraphId) ?
+ paragraphMap.get(paragraphId) : new StringBuilder();
+
+ builder.append(buffer.getData());
+ paragraphMap.put(paragraphId, builder);
+ noteMap.put(noteId, paragraphMap);
+ }
+ Long processingTime = System.currentTimeMillis() - processingStartTime;
+
+ if (processingTime > SAFE_PROCESSING_TIME) {
+ logger.warn("Processing time for buffered append-output is high: " +
+ processingTime + " milliseconds.");
+ } else {
+ logger.debug("Processing time for append-output took "
+ + processingTime + " milliseconds");
+ }
+
+ Long sizeProcessed = new Long(0);
+ for (String noteId: noteMap.keySet()) {
+ for (String paragraphId: noteMap.get(noteId).keySet()) {
+ String data = noteMap.get(noteId).get(paragraphId).toString();
+ sizeProcessed += data.length();
+ listener.onOutputAppend(noteId, paragraphId, data);
+ }
+ }
+
+ if (sizeProcessed > SAFE_PROCESSING_STRING_SIZE) {
+ logger.warn("Processing size for buffered append-output is high: " +
+ sizeProcessed + " characters.");
+ } else {
+ logger.debug("Processing size for append-output is " +
+ sizeProcessed + " characters");
+ }
+ }
+
+ public void appendBuffer(String noteId, String paragraphId, String outputToAppend) {
+ queue.offer(new AppendOutputBuffer(noteId, paragraphId, outputToAppend));
+ }
+
+}
diff --git a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreter.java b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreter.java
index db740f4873a..3e6fa3943ed 100644
--- a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreter.java
+++ b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreter.java
@@ -295,7 +295,9 @@ public void close() {
@Override
public InterpreterResult interpret(String st, InterpreterContext context) {
- logger.debug("st: {}", st);
+ if (logger.isDebugEnabled()) {
+ logger.debug("st:\n{}", st);
+ }
FormType form = getFormType();
RemoteInterpreterProcess interpreterProcess = getInterpreterProcess();
Client client = null;
diff --git a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterEventPoller.java b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterEventPoller.java
index 48c14d50bde..090aeeaaee3 100644
--- a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterEventPoller.java
+++ b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterEventPoller.java
@@ -39,12 +39,18 @@
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
/**
* Processes message from RemoteInterpreter process
*/
public class RemoteInterpreterEventPoller extends Thread {
private static final Logger logger = LoggerFactory.getLogger(RemoteInterpreterEventPoller.class);
+ private static final ScheduledExecutorService appendService =
+ Executors.newSingleThreadScheduledExecutor();
private final RemoteInterpreterProcessListener listener;
private final ApplicationEventListener appListener;
@@ -72,6 +78,9 @@ public void setInterpreterGroup(InterpreterGroup interpreterGroup) {
@Override
public void run() {
Client client = null;
+ AppendOutputRunner runner = new AppendOutputRunner(listener);
+ ScheduledFuture appendFuture = appendService.scheduleWithFixedDelay(
+ runner, 0, AppendOutputRunner.BUFFER_TIME_MS, TimeUnit.MILLISECONDS);
while (!shutdown) {
// wait and retry
@@ -157,7 +166,7 @@ public void run() {
String appId = outputAppend.get("appId");
if (appId == null) {
- listener.onOutputAppend(noteId, paragraphId, outputToAppend);
+ runner.appendBuffer(noteId, paragraphId, outputToAppend);
} else {
appListener.onOutputAppend(noteId, paragraphId, appId, outputToAppend);
}
@@ -192,6 +201,9 @@ public void run() {
logger.error("Can't handle event " + event, e);
}
}
+ if (appendFuture != null) {
+ appendFuture.cancel(true);
+ }
}
private void sendResourcePoolResponseGetAll(ResourceSet resourceSet) {
diff --git a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java
index 173026562f3..7ddb92838f4 100644
--- a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java
+++ b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java
@@ -34,6 +34,7 @@
import org.apache.zeppelin.helium.*;
import org.apache.zeppelin.interpreter.*;
import org.apache.zeppelin.interpreter.InterpreterResult.Code;
+import org.apache.zeppelin.interpreter.dev.ZeppelinDevServer;
import org.apache.zeppelin.interpreter.thrift.*;
import org.apache.zeppelin.resource.*;
import org.apache.zeppelin.scheduler.Job;
@@ -133,7 +134,11 @@ public boolean isRunning() {
public static void main(String[] args)
throws TTransportException, InterruptedException {
- int port = Integer.parseInt(args[0]);
+
+ int port = ZeppelinDevServer.DEFAULT_TEST_INTERPRETER_PORT;
+ if (args.length > 0) {
+ port = Integer.parseInt(args[0]);
+ }
RemoteInterpreterServer remoteInterpreterServer = new RemoteInterpreterServer(port);
remoteInterpreterServer.start();
remoteInterpreterServer.join();
@@ -280,7 +285,9 @@ public void close(String noteId, String className) throws TException {
@Override
public RemoteInterpreterResult interpret(String noteId, String className, String st,
RemoteInterpreterContext interpreterContext) throws TException {
- logger.debug("st: {}", st);
+ if (logger.isDebugEnabled()) {
+ logger.debug("st:\n{}", st);
+ }
Interpreter intp = getInterpreter(noteId, className);
InterpreterContext context = convert(interpreterContext);
diff --git a/zeppelin-interpreter/src/test/java/org/apache/zeppelin/interpreter/remote/AppendOutputRunnerTest.java b/zeppelin-interpreter/src/test/java/org/apache/zeppelin/interpreter/remote/AppendOutputRunnerTest.java
new file mode 100644
index 00000000000..8e9f5b361e6
--- /dev/null
+++ b/zeppelin-interpreter/src/test/java/org/apache/zeppelin/interpreter/remote/AppendOutputRunnerTest.java
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.interpreter.remote;
+
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+import static org.mockito.Matchers.any;
+import static org.mockito.Mockito.atMost;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.log4j.AppenderSkeleton;
+import org.apache.log4j.Level;
+import org.apache.log4j.Logger;
+import org.apache.log4j.spi.LoggingEvent;
+import org.junit.After;
+import org.junit.Test;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+public class AppendOutputRunnerTest {
+
+ private static final int NUM_EVENTS = 10000;
+ private static final int NUM_CLUBBED_EVENTS = 100;
+ private static final ScheduledExecutorService service = Executors.newSingleThreadScheduledExecutor();
+ private static ScheduledFuture future = null;
+ /* It is being accessed by multiple threads.
+ * While loop for 'loopForBufferCompletion' could
+ * run for-ever.
+ */
+ private volatile static int numInvocations = 0;
+
+ @After
+ public void afterEach() {
+ if (future != null) {
+ future.cancel(true);
+ }
+ }
+
+ @Test
+ public void testSingleEvent() throws InterruptedException {
+ RemoteInterpreterProcessListener listener = mock(RemoteInterpreterProcessListener.class);
+ String[][] buffer = {{"note", "para", "data\n"}};
+
+ loopForCompletingEvents(listener, 1, buffer);
+ verify(listener, times(1)).onOutputAppend(any(String.class), any(String.class), any(String.class));
+ verify(listener, times(1)).onOutputAppend("note", "para", "data\n");
+ }
+
+ @Test
+ public void testMultipleEventsOfSameParagraph() throws InterruptedException {
+ RemoteInterpreterProcessListener listener = mock(RemoteInterpreterProcessListener.class);
+ String note1 = "note1";
+ String para1 = "para1";
+ String[][] buffer = {
+ {note1, para1, "data1\n"},
+ {note1, para1, "data2\n"},
+ {note1, para1, "data3\n"}
+ };
+
+ loopForCompletingEvents(listener, 1, buffer);
+ verify(listener, times(1)).onOutputAppend(any(String.class), any(String.class), any(String.class));
+ verify(listener, times(1)).onOutputAppend(note1, para1, "data1\ndata2\ndata3\n");
+ }
+
+ @Test
+ public void testMultipleEventsOfDifferentParagraphs() throws InterruptedException {
+ RemoteInterpreterProcessListener listener = mock(RemoteInterpreterProcessListener.class);
+ String note1 = "note1";
+ String note2 = "note2";
+ String para1 = "para1";
+ String para2 = "para2";
+ String[][] buffer = {
+ {note1, para1, "data1\n"},
+ {note1, para2, "data2\n"},
+ {note2, para1, "data3\n"},
+ {note2, para2, "data4\n"}
+ };
+ loopForCompletingEvents(listener, 4, buffer);
+
+ verify(listener, times(4)).onOutputAppend(any(String.class), any(String.class), any(String.class));
+ verify(listener, times(1)).onOutputAppend(note1, para1, "data1\n");
+ verify(listener, times(1)).onOutputAppend(note1, para2, "data2\n");
+ verify(listener, times(1)).onOutputAppend(note2, para1, "data3\n");
+ verify(listener, times(1)).onOutputAppend(note2, para2, "data4\n");
+ }
+
+ @Test
+ public void testClubbedData() throws InterruptedException {
+ RemoteInterpreterProcessListener listener = mock(RemoteInterpreterProcessListener.class);
+ AppendOutputRunner runner = new AppendOutputRunner(listener);
+ future = service.scheduleWithFixedDelay(runner, 0,
+ AppendOutputRunner.BUFFER_TIME_MS, TimeUnit.MILLISECONDS);
+ Thread thread = new Thread(new BombardEvents(runner));
+ thread.start();
+ thread.join();
+ Thread.sleep(1000);
+
+ /* NUM_CLUBBED_EVENTS is a heuristic number.
+ * It has been observed that for 10,000 continuos event
+ * calls, 30-40 Web-socket calls are made. Keeping
+ * the unit-test to a pessimistic 100 web-socket calls.
+ */
+ verify(listener, atMost(NUM_CLUBBED_EVENTS)).onOutputAppend(any(String.class), any(String.class), any(String.class));
+ }
+
+ @Test
+ public void testWarnLoggerForLargeData() throws InterruptedException {
+ RemoteInterpreterProcessListener listener = mock(RemoteInterpreterProcessListener.class);
+ AppendOutputRunner runner = new AppendOutputRunner(listener);
+ String data = "data\n";
+ int numEvents = 100000;
+
+ for (int i=0; i log;
+
+ int warnLogCounter;
+ LoggingEvent sizeWarnLogEntry = null;
+ do {
+ warnLogCounter = 0;
+ log = appender.getLog();
+ for (LoggingEvent logEntry: log) {
+ if (Level.WARN.equals(logEntry.getLevel())) {
+ sizeWarnLogEntry = logEntry;
+ warnLogCounter += 1;
+ }
+ }
+ } while(warnLogCounter != 2);
+
+ String loggerString = "Processing size for buffered append-output is high: " +
+ (data.length() * numEvents) + " characters.";
+ assertTrue(loggerString.equals(sizeWarnLogEntry.getMessage()));
+ }
+
+ private class BombardEvents implements Runnable {
+
+ private final AppendOutputRunner runner;
+
+ private BombardEvents(AppendOutputRunner runner) {
+ this.runner = runner;
+ }
+
+ @Override
+ public void run() {
+ String noteId = "noteId";
+ String paraId = "paraId";
+ for (int i=0; i log = new ArrayList<>();
+
+ @Override
+ public boolean requiresLayout() {
+ return false;
+ }
+
+ @Override
+ protected void append(final LoggingEvent loggingEvent) {
+ log.add(loggingEvent);
+ }
+
+ @Override
+ public void close() {
+ }
+
+ public List getLog() {
+ return new ArrayList<>(log);
+ }
+ }
+
+ private void prepareInvocationCounts(RemoteInterpreterProcessListener listener) {
+ doAnswer(new Answer() {
+ @Override
+ public Void answer(InvocationOnMock invocation) throws Throwable {
+ numInvocations += 1;
+ return null;
+ }
+ }).when(listener).onOutputAppend(any(String.class), any(String.class), any(String.class));
+ }
+
+ private void loopForCompletingEvents(RemoteInterpreterProcessListener listener,
+ int numTimes, String[][] buffer) {
+ numInvocations = 0;
+ prepareInvocationCounts(listener);
+ AppendOutputRunner runner = new AppendOutputRunner(listener);
+ for (String[] bufferElement: buffer) {
+ runner.appendBuffer(bufferElement[0], bufferElement[1], bufferElement[2]);
+ }
+ future = service.scheduleWithFixedDelay(runner, 0,
+ AppendOutputRunner.BUFFER_TIME_MS, TimeUnit.MILLISECONDS);
+ long startTimeMs = System.currentTimeMillis();
+ while(numInvocations != numTimes) {
+ if (System.currentTimeMillis() - startTimeMs > 2000) {
+ fail("Buffered events were not sent for 2 seconds");
+ }
+ }
+ }
+}
\ No newline at end of file
diff --git a/zeppelin-server/pom.xml b/zeppelin-server/pom.xml
index 76fb72b8b69..862fc30ce35 100644
--- a/zeppelin-server/pom.xml
+++ b/zeppelin-server/pom.xml
@@ -35,6 +35,7 @@
2.7.7
4.3.6
+ 2.6.0
@@ -205,6 +206,61 @@
commons-collections
+
+ org.apache.hadoop
+ hadoop-common
+ ${hadoop-common.version}
+
+
+ com.sun.jersey
+ jersey-core
+
+
+ com.sun.jersey
+ jersey-json
+
+
+ com.sun.jersey
+ jersey-server
+
+
+
+
+ javax.servlet
+ servlet-api
+
+
+ org.apache.avro
+ avro
+
+
+ org.apache.jackrabbit
+ jackrabbit-webdav
+
+
+ io.netty
+ netty
+
+
+ commons-httpclient
+ commons-httpclient
+
+
+ org.apache.zookeeper
+ zookeeper
+
+
+ org.eclipse.jgit
+ org.eclipse.jgit
+
+
+ com.jcraft
+ jsch
+
+
+
+
+
org.quartz-scheduler
quartz
@@ -301,27 +357,6 @@
-
- org.apache.rat
- apache-rat-plugin
-
-
- **/.idea/
- **/*.iml
- .git/
- .gitignore
- **/.settings/*
- **/.classpath
- **/.project
- **/target/**
- **/derby.log
- **/metastore_db/
- **/README.md
- src/test/java/com/webautomation/*
-
-
-
-
maven-failsafe-plugin
2.16
@@ -413,10 +448,6 @@
scala-2.11
-
- scala-2.11
-
diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/realm/ZeppelinHubRealm.java b/zeppelin-server/src/main/java/org/apache/zeppelin/realm/ZeppelinHubRealm.java
new file mode 100644
index 00000000000..cbe490d8de5
--- /dev/null
+++ b/zeppelin-server/src/main/java/org/apache/zeppelin/realm/ZeppelinHubRealm.java
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.realm;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.commons.httpclient.HttpClient;
+import org.apache.commons.httpclient.HttpStatus;
+import org.apache.commons.httpclient.methods.PutMethod;
+import org.apache.commons.httpclient.methods.StringRequestEntity;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.shiro.authc.AccountException;
+import org.apache.shiro.authc.AuthenticationException;
+import org.apache.shiro.authc.AuthenticationInfo;
+import org.apache.shiro.authc.AuthenticationToken;
+import org.apache.shiro.authc.SimpleAuthenticationInfo;
+import org.apache.shiro.authc.UsernamePasswordToken;
+import org.apache.shiro.authz.AuthorizationInfo;
+import org.apache.shiro.realm.AuthorizingRealm;
+import org.apache.shiro.subject.PrincipalCollection;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Joiner;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+
+/**
+ * A {@code Realm} implementation that uses the ZeppelinHub to authenticate users.
+ *
+ */
+public class ZeppelinHubRealm extends AuthorizingRealm {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ZeppelinHubRealm.class);
+ private static final String DEFAULT_ZEPPELINHUB_URL = "https://www.zeppelinhub.com";
+ private static final String USER_LOGIN_API_ENDPOINT = "api/v1/users/login";
+ private static final String JSON_CONTENT_TYPE = "application/json";
+ private static final String UTF_8_ENCODING = "UTF-8";
+ private static final AtomicInteger INSTANCE_COUNT = new AtomicInteger();
+
+ private final HttpClient httpClient;
+ private final Gson gson;
+
+ private String zeppelinhubUrl;
+ private String name;
+
+ public ZeppelinHubRealm() {
+ super();
+ LOG.debug("Init ZeppelinhubRealm");
+ //TODO(anthonyc): think about more setting for this HTTP client.
+ // eg: if user uses proxy etcetc...
+ httpClient = new HttpClient();
+ gson = new Gson();
+ name = getClass().getName() + "_" + INSTANCE_COUNT.getAndIncrement();
+ }
+
+ @Override
+ protected AuthenticationInfo doGetAuthenticationInfo(AuthenticationToken authToken)
+ throws AuthenticationException {
+ UsernamePasswordToken token = (UsernamePasswordToken) authToken;
+ if (StringUtils.isBlank(token.getUsername())) {
+ throw new AccountException("Empty usernames are not allowed by this realm.");
+ }
+ String loginPayload = createLoginPayload(token.getUsername(), token.getPassword());
+ User user = authenticateUser(loginPayload);
+ LOG.debug("{} successfully login via ZeppelinHub", user.login);
+ return new SimpleAuthenticationInfo(user.login, token.getPassword(), name);
+ }
+
+ @Override
+ protected AuthorizationInfo doGetAuthorizationInfo(PrincipalCollection principals) {
+ // TODO(xxx): future work will be done here.
+ return null;
+ }
+
+ protected void onInit() {
+ super.onInit();
+ }
+
+ /**
+ * Setter of ZeppelinHub URL, this will be called by Shiro based on zeppelinhubUrl property
+ * in shiro.ini file.
+ * It will also perform a check of ZeppelinHub url {@link #isZeppelinHubUrlValid},
+ * if the url is not valid, the default zeppelinhub url will be used.
+ *
+ * @param url
+ */
+ public void setZeppelinhubUrl(String url) {
+ if (StringUtils.isBlank(url)) {
+ LOG.warn("Zeppelinhub url is empty, setting up default url {}", DEFAULT_ZEPPELINHUB_URL);
+ zeppelinhubUrl = DEFAULT_ZEPPELINHUB_URL;
+ } else {
+ zeppelinhubUrl = (isZeppelinHubUrlValid(url) ? url : DEFAULT_ZEPPELINHUB_URL);
+ LOG.info("Setting up Zeppelinhub url to {}", zeppelinhubUrl);
+ }
+ }
+
+ /**
+ * Send to ZeppelinHub a login request based on the request body which is a JSON that contains 2
+ * fields "login" and "password".
+ *
+ * @param requestBody JSON string of ZeppelinHub payload.
+ * @return Account object with login, name (if set in ZeppelinHub), and mail.
+ * @throws AuthenticationException if fail to login.
+ */
+ protected User authenticateUser(String requestBody) {
+ PutMethod put = new PutMethod(Joiner.on("/").join(zeppelinhubUrl, USER_LOGIN_API_ENDPOINT));
+ String responseBody = StringUtils.EMPTY;
+ try {
+ put.setRequestEntity(new StringRequestEntity(requestBody, JSON_CONTENT_TYPE, UTF_8_ENCODING));
+ int statusCode = httpClient.executeMethod(put);
+ if (statusCode != HttpStatus.SC_OK) {
+ LOG.error("Cannot login user, HTTP status code is {} instead on 200 (OK)", statusCode);
+ put.releaseConnection();
+ throw new AuthenticationException("Couldnt login to ZeppelinHub. "
+ + "Login or password incorrect");
+ }
+ responseBody = put.getResponseBodyAsString();
+ put.releaseConnection();
+ } catch (IOException e) {
+ LOG.error("Cannot login user", e);
+ throw new AuthenticationException(e.getMessage());
+ }
+
+ User account = null;
+ try {
+ account = gson.fromJson(responseBody, User.class);
+ } catch (JsonParseException e) {
+ LOG.error("Cannot deserialize ZeppelinHub response to User instance", e);
+ throw new AuthenticationException("Cannot login to ZeppelinHub");
+ }
+ return account;
+ }
+
+ /**
+ * Create a JSON String that represent login payload.
+ * Payload will look like:
+ * getUserList(IniRealm r) {
return userList;
}
+
+ /***
+ * Get user roles from shiro.ini
+ * @param r
+ * @return
+ */
+ public List getRolesList(IniRealm r) {
+ List roleList = new ArrayList<>();
+ Map getIniRoles = r.getIni().get("roles");
+ if (getIniRoles != null) {
+ Iterator it = getIniRoles.entrySet().iterator();
+ while (it.hasNext()) {
+ Map.Entry pair = (Map.Entry) it.next();
+ roleList.add(pair.getKey().toString().trim());
+ }
+ }
+ return roleList;
+ }
+
/**
* function to extract users from LDAP
*/
diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/InterpreterRestApi.java b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/InterpreterRestApi.java
index f77aac0c4d1..6025b52c7f1 100644
--- a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/InterpreterRestApi.java
+++ b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/InterpreterRestApi.java
@@ -90,6 +90,9 @@ public Response newSettings(String message) {
try {
NewInterpreterSettingRequest request =
gson.fromJson(message, NewInterpreterSettingRequest.class);
+ if (request == null) {
+ return new JsonResponse<>(Status.BAD_REQUEST).build();
+ }
Properties p = new Properties();
p.putAll(request.getProperties());
InterpreterSetting interpreterSetting = interpreterFactory
diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java
index 700fe1a1c2d..17ec74fa39d 100644
--- a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java
+++ b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java
@@ -20,7 +20,6 @@
import java.io.IOException;
import java.util.HashMap;
import java.util.HashSet;
-import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -39,18 +38,20 @@
import com.google.common.reflect.TypeToken;
import com.google.gson.Gson;
import org.apache.commons.lang3.StringUtils;
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.scheduler.Job;
+import org.apache.zeppelin.utils.InterpreterBindingUtils;
import org.quartz.CronExpression;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.zeppelin.annotation.ZeppelinApi;
-import org.apache.zeppelin.interpreter.InterpreterSetting;
import org.apache.zeppelin.notebook.Note;
import org.apache.zeppelin.notebook.Notebook;
import org.apache.zeppelin.notebook.NotebookAuthorization;
import org.apache.zeppelin.notebook.Paragraph;
import org.apache.zeppelin.rest.message.CronRequest;
-import org.apache.zeppelin.rest.message.InterpreterSettingListForNoteBind;
+import org.apache.zeppelin.types.InterpreterSettingsList;
import org.apache.zeppelin.rest.message.NewNotebookRequest;
import org.apache.zeppelin.rest.message.NewParagraphRequest;
import org.apache.zeppelin.rest.message.RunParagraphWithParametersRequest;
@@ -117,8 +118,8 @@ public Response putNotePermissions(@PathParam("noteId") String noteId, String re
* TODO(jl): Fixed the type of HashSet
* https://issues.apache.org/jira/browse/ZEPPELIN-1162
*/
- HashMap permMap =
- gson.fromJson(req, new TypeToken>() {
+ HashMap> permMap =
+ gson.fromJson(req, new TypeToken>>() {
}.getType());
Note note = notebook.getNote(noteId);
String principal = SecurityUtils.getPrincipal();
@@ -134,9 +135,9 @@ public Response putNotePermissions(@PathParam("noteId") String noteId, String re
ownerPermissionError(userAndRoles, notebookAuthorization.getOwners(noteId))).build();
}
- HashSet readers = permMap.get("readers");
- HashSet owners = permMap.get("owners");
- HashSet writers = permMap.get("writers");
+ HashSet readers = permMap.get("readers");
+ HashSet owners = permMap.get("owners");
+ HashSet writers = permMap.get("writers");
// Set readers, if writers and owners is empty -> set to user requesting the change
if (readers != null && !readers.isEmpty()) {
if (writers.isEmpty()) {
@@ -186,29 +187,9 @@ public Response bind(@PathParam("noteId") String noteId, String req) throws IOEx
@Path("interpreter/bind/{noteId}")
@ZeppelinApi
public Response bind(@PathParam("noteId") String noteId) {
- List settingList = new LinkedList<>();
-
- List selectedSettings = notebook.getBindedInterpreterSettings(noteId);
- for (InterpreterSetting setting : selectedSettings) {
- settingList.add(new InterpreterSettingListForNoteBind(setting.getId(), setting.getName(),
- setting.getInterpreterInfos(), true));
- }
-
- List availableSettings = notebook.getInterpreterFactory().get();
- for (InterpreterSetting setting : availableSettings) {
- boolean selected = false;
- for (InterpreterSetting selectedSetting : selectedSettings) {
- if (selectedSetting.getId().equals(setting.getId())) {
- selected = true;
- break;
- }
- }
-
- if (!selected) {
- settingList.add(new InterpreterSettingListForNoteBind(setting.getId(), setting.getName(),
- setting.getInterpreterInfos(), false));
- }
- }
+ List settingList =
+ InterpreterBindingUtils.getInterpreterBindings(notebook, noteId);
+ notebookServer.broadcastInterpreterBindings(noteId, settingList);
return new JsonResponse<>(Status.OK, "", settingList).build();
}
@@ -338,7 +319,10 @@ public Response cloneNote(@PathParam("notebookId") String notebookId, String mes
throws IOException, CloneNotSupportedException, IllegalArgumentException {
LOG.info("clone notebook by JSON {}", message);
NewNotebookRequest request = gson.fromJson(message, NewNotebookRequest.class);
- String newNoteName = request.getName();
+ String newNoteName = null;
+ if (request != null) {
+ newNoteName = request.getName();
+ }
AuthenticationInfo subject = new AuthenticationInfo(SecurityUtils.getPrincipal());
Note newNote = notebook.cloneNote(notebookId, newNoteName, subject);
notebookServer.broadcastNote(newNote);
@@ -498,7 +482,14 @@ public Response runNoteJobs(@PathParam("notebookId") String notebookId)
return new JsonResponse<>(Status.NOT_FOUND, "note not found.").build();
}
- note.runAll();
+ try {
+ note.runAll();
+ } catch (Exception ex) {
+ LOG.error("Exception from run", ex);
+ return new JsonResponse<>(Status.PRECONDITION_FAILED,
+ ex.getMessage() + "- Not selected or Invalid Interpreter bind").build();
+ }
+
return new JsonResponse<>(Status.OK).build();
}
@@ -550,7 +541,36 @@ public Response getNoteJobStatus(@PathParam("notebookId") String notebookId)
}
/**
- * Run paragraph job REST API
+ * Get notebook paragraph job status REST API
+ *
+ * @param notebookId ID of Notebook
+ * @param paragraphId ID of Paragraph
+ * @return JSON with status.OK
+ * @throws IOException, IllegalArgumentException
+ */
+ @GET
+ @Path("job/{notebookId}/{paragraphId}")
+ @ZeppelinApi
+ public Response getNoteParagraphJobStatus(@PathParam("notebookId") String notebookId,
+ @PathParam("paragraphId") String paragraphId)
+ throws IOException, IllegalArgumentException {
+ LOG.info("get notebook paragraph job status.");
+ Note note = notebook.getNote(notebookId);
+ if (note == null) {
+ return new JsonResponse<>(Status.NOT_FOUND, "note not found.").build();
+ }
+
+ Paragraph paragraph = note.getParagraph(paragraphId);
+ if (paragraph == null) {
+ return new JsonResponse<>(Status.NOT_FOUND, "paragraph not found.").build();
+ }
+
+ return new JsonResponse<>(Status.OK, null, note.generateSingleParagraphInfo(paragraphId)).
+ build();
+ }
+
+ /**
+ * Run asynchronously paragraph job REST API
*
* @param message - JSON with params if user wants to update dynamic form's value
* null, empty string, empty json if user doesn't want to update
@@ -563,7 +583,7 @@ public Response getNoteJobStatus(@PathParam("notebookId") String notebookId)
public Response runParagraph(@PathParam("notebookId") String notebookId,
@PathParam("paragraphId") String paragraphId, String message)
throws IOException, IllegalArgumentException {
- LOG.info("run paragraph job {} {} {}", notebookId, paragraphId, message);
+ LOG.info("run paragraph job asynchronously {} {} {}", notebookId, paragraphId, message);
Note note = notebook.getNote(notebookId);
if (note == null) {
@@ -576,22 +596,60 @@ public Response runParagraph(@PathParam("notebookId") String notebookId,
}
// handle params if presented
- if (!StringUtils.isEmpty(message)) {
- RunParagraphWithParametersRequest request =
- gson.fromJson(message, RunParagraphWithParametersRequest.class);
- Map paramsForUpdating = request.getParams();
- if (paramsForUpdating != null) {
- paragraph.settings.getParams().putAll(paramsForUpdating);
- AuthenticationInfo subject = new AuthenticationInfo(SecurityUtils.getPrincipal());
- note.setLastReplName(paragraph.getId());
- note.persist(subject);
- }
- }
+ handleParagraphParams(message, note, paragraph);
note.run(paragraph.getId());
return new JsonResponse<>(Status.OK).build();
}
+/**
+ * Run synchronously a paragraph REST API
+ *
+ * @param noteId - noteId
+ * @param paragraphId - paragraphId
+ * @param message - JSON with params if user wants to update dynamic form's value
+ * null, empty string, empty json if user doesn't want to update
+ *
+ * @return JSON with status.OK
+ * @throws IOException, IllegalArgumentException
+ */
+ @POST
+ @Path("run/{notebookId}/{paragraphId}")
+ @ZeppelinApi
+ public Response runParagraphSynchronously(@PathParam("notebookId") String noteId,
+ @PathParam("paragraphId") String paragraphId,
+ String message) throws
+ IOException, IllegalArgumentException {
+ LOG.info("run paragraph synchronously {} {} {}", noteId, paragraphId, message);
+
+ Note note = notebook.getNote(noteId);
+ if (note == null) {
+ return new JsonResponse<>(Status.NOT_FOUND, "note not found.").build();
+ }
+
+ Paragraph paragraph = note.getParagraph(paragraphId);
+ if (paragraph == null) {
+ return new JsonResponse<>(Status.NOT_FOUND, "paragraph not found.").build();
+ }
+
+ // handle params if presented
+ handleParagraphParams(message, note, paragraph);
+
+ if (paragraph.getListener() == null) {
+ note.initializeJobListenerForParagraph(paragraph);
+ }
+
+ paragraph.run();
+
+ final InterpreterResult result = paragraph.getResult();
+
+ if (result.code() == InterpreterResult.Code.SUCCESS) {
+ return new JsonResponse<>(Status.OK, result).build();
+ } else {
+ return new JsonResponse<>(Status.INTERNAL_SERVER_ERROR, result).build();
+ }
+ }
+
/**
* Stop(delete) paragraph job REST API
*
@@ -652,7 +710,7 @@ public Response registerCronJob(@PathParam("notebookId") String notebookId, Stri
Map config = note.getConfig();
config.put("cron", request.getCronString());
note.setConfig(config);
- notebook.refreshCron(note.id());
+ notebook.refreshCron(note.getId());
return new JsonResponse<>(Status.OK).build();
}
@@ -680,7 +738,7 @@ public Response removeCronJob(@PathParam("notebookId") String notebookId)
Map config = note.getConfig();
config.put("cron", null);
note.setConfig(config);
- notebook.refreshCron(note.id());
+ notebook.refreshCron(note.getId());
return new JsonResponse<>(Status.OK).build();
}
@@ -783,4 +841,21 @@ public Response search(@QueryParam("q") String queryTerm) {
return new JsonResponse<>(Status.OK, notebooksFound).build();
}
+
+ private void handleParagraphParams(String message, Note note, Paragraph paragraph)
+ throws IOException {
+ // handle params if presented
+ if (!StringUtils.isEmpty(message)) {
+ RunParagraphWithParametersRequest request =
+ gson.fromJson(message, RunParagraphWithParametersRequest.class);
+ Map paramsForUpdating = request.getParams();
+ if (paramsForUpdating != null) {
+ paragraph.settings.getParams().putAll(paramsForUpdating);
+ AuthenticationInfo subject = new AuthenticationInfo(SecurityUtils.getPrincipal());
+ note.setLastReplName(paragraph.getId());
+ note.persist(subject);
+ }
+ }
+ }
+
}
diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/SecurityRestApi.java b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/SecurityRestApi.java
index a079a4460c9..7af52c8c8ba 100644
--- a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/SecurityRestApi.java
+++ b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/SecurityRestApi.java
@@ -18,9 +18,9 @@
package org.apache.zeppelin.rest;
+import org.apache.commons.lang3.StringUtils;
import org.apache.shiro.realm.Realm;
import org.apache.shiro.realm.jdbc.JdbcRealm;
-import org.apache.shiro.realm.ldap.AbstractLdapRealm;
import org.apache.shiro.realm.ldap.JndiLdapRealm;
import org.apache.shiro.realm.text.IniRealm;
import org.apache.zeppelin.annotation.ZeppelinApi;
@@ -98,6 +98,7 @@ public Response ticket() {
public Response getUserList(@PathParam("searchText") final String searchText) {
List usersList = new ArrayList<>();
+ List rolesList = new ArrayList<>();
try {
GetUserList getUserListObj = new GetUserList();
Collection realmsList = SecurityUtils.getRealmsList();
@@ -107,6 +108,7 @@ public Response getUserList(@PathParam("searchText") final String searchText) {
String name = realm.getName();
if (name.equals("iniRealm")) {
usersList.addAll(getUserListObj.getUserList((IniRealm) realm));
+ rolesList.addAll(getUserListObj.getRolesList((IniRealm) realm));
} else if (name.equals("ldapRealm")) {
usersList.addAll(getUserListObj.getUserList((JndiLdapRealm) realm, searchText));
} else if (name.equals("activeDirectoryRealm")) {
@@ -120,8 +122,10 @@ public Response getUserList(@PathParam("searchText") final String searchText) {
} catch (Exception e) {
LOG.error("Exception in retrieving Users from realms ", e);
}
- List autoSuggestList = new ArrayList<>();
+ List autoSuggestUserList = new ArrayList<>();
+ List autoSuggestRoleList = new ArrayList<>();
Collections.sort(usersList);
+ Collections.sort(rolesList);
Collections.sort(usersList, new Comparator() {
@Override
public int compare(String o1, String o2) {
@@ -134,18 +138,28 @@ public int compare(String o1, String o2) {
}
});
int maxLength = 0;
- for (int i = 0; i < usersList.size(); i++) {
- String userLowerCase = usersList.get(i).toLowerCase();
- String searchTextLowerCase = searchText.toLowerCase();
- if (userLowerCase.indexOf(searchTextLowerCase) != -1) {
+ for (String user : usersList) {
+ if (StringUtils.containsIgnoreCase(user, searchText)) {
+ autoSuggestUserList.add(user);
maxLength++;
- autoSuggestList.add(usersList.get(i));
}
if (maxLength == 5) {
break;
}
}
- return new JsonResponse<>(Response.Status.OK, "", autoSuggestList).build();
+
+ for (String role : rolesList) {
+ if (StringUtils.containsIgnoreCase(role, searchText)) {
+ autoSuggestRoleList.add(role);
+ }
+ }
+
+ Map returnListMap = new HashMap<>();
+ returnListMap.put("users", autoSuggestUserList);
+ returnListMap.put("roles", autoSuggestRoleList);
+
+
+ return new JsonResponse<>(Response.Status.OK, "", returnListMap).build();
}
}
diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/server/ActiveDirectoryGroupRealm.java b/zeppelin-server/src/main/java/org/apache/zeppelin/server/ActiveDirectoryGroupRealm.java
index cc868d7a1b4..556f404663c 100644
--- a/zeppelin-server/src/main/java/org/apache/zeppelin/server/ActiveDirectoryGroupRealm.java
+++ b/zeppelin-server/src/main/java/org/apache/zeppelin/server/ActiveDirectoryGroupRealm.java
@@ -16,6 +16,10 @@
*/
package org.apache.zeppelin.server;
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.alias.CredentialProvider;
+import org.apache.hadoop.security.alias.CredentialProviderFactory;
import org.apache.shiro.authc.AuthenticationInfo;
import org.apache.shiro.authc.AuthenticationToken;
import org.apache.shiro.authc.SimpleAuthenticationInfo;
@@ -55,6 +59,13 @@ public class ActiveDirectoryGroupRealm extends AbstractLdapRealm {
private static final String ROLE_NAMES_DELIMETER = ",";
+ String KEYSTORE_PASS = "activeDirectoryRealm.systemPassword";
+ private String hadoopSecurityCredentialPath;
+
+ public void setHadoopSecurityCredentialPath(String hadoopSecurityCredentialPath) {
+ this.hadoopSecurityCredentialPath = hadoopSecurityCredentialPath;
+ }
+
/*--------------------------------------------
| I N S T A N C E V A R I A B L E S |
============================================*/
@@ -91,13 +102,36 @@ public LdapContextFactory getLdapContextFactory() {
defaultFactory.setSearchBase(this.searchBase);
defaultFactory.setUrl(this.url);
defaultFactory.setSystemUsername(this.systemUsername);
- defaultFactory.setSystemPassword(this.systemPassword);
+ defaultFactory.setSystemPassword(getSystemPassword());
this.ldapContextFactory = defaultFactory;
}
return this.ldapContextFactory;
}
+ private String getSystemPassword() {
+ String password = "";
+ if (StringUtils.isEmpty(this.hadoopSecurityCredentialPath)) {
+ password = this.systemPassword;
+ } else {
+ try {
+ Configuration configuration = new Configuration();
+ configuration.set(CredentialProviderFactory.CREDENTIAL_PROVIDER_PATH,
+ this.hadoopSecurityCredentialPath);
+ CredentialProvider provider =
+ CredentialProviderFactory.getProviders(configuration).get(0);
+ CredentialProvider.CredentialEntry credEntry = provider.getCredentialEntry(
+ KEYSTORE_PASS);
+ if (credEntry != null) {
+ password = new String(credEntry.getCredential());
+ }
+ } catch (Exception e) {
+
+ }
+ }
+ return password;
+ }
+
/**
* Builds an {@link AuthenticationInfo} object by querying the active directory LDAP context for
* the specified username. This method binds to the LDAP server using the provided username
@@ -293,3 +327,4 @@ protected Collection getRoleNamesForGroups(Collection groupNames
}
}
+
diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java b/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
index 0f8ce70fc39..19bfba10b54 100644
--- a/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
+++ b/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
@@ -16,19 +16,12 @@
*/
package org.apache.zeppelin.socket;
-import java.io.IOException;
-import java.net.URISyntaxException;
-import java.net.UnknownHostException;
-import java.util.*;
-import java.util.concurrent.ConcurrentLinkedQueue;
-
-import javax.servlet.http.HttpServletRequest;
-
import com.google.common.base.Strings;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.reflect.TypeToken;
+import org.apache.commons.lang.StringUtils;
import org.apache.zeppelin.conf.ZeppelinConfiguration;
import org.apache.zeppelin.conf.ZeppelinConfiguration.ConfVars;
import org.apache.zeppelin.display.AngularObject;
@@ -37,15 +30,13 @@
import org.apache.zeppelin.helium.ApplicationEventListener;
import org.apache.zeppelin.helium.HeliumPackage;
import org.apache.zeppelin.interpreter.InterpreterGroup;
-import org.apache.zeppelin.interpreter.remote.RemoteAngularObjectRegistry;
-import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
-import org.apache.zeppelin.user.AuthenticationInfo;
import org.apache.zeppelin.interpreter.InterpreterOutput;
import org.apache.zeppelin.interpreter.InterpreterResult;
import org.apache.zeppelin.interpreter.InterpreterSetting;
+import org.apache.zeppelin.interpreter.remote.RemoteAngularObjectRegistry;
import org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcessListener;
+import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
import org.apache.zeppelin.notebook.*;
-import org.apache.zeppelin.notebook.repo.NotebookRepo;
import org.apache.zeppelin.notebook.repo.NotebookRepo.Revision;
import org.apache.zeppelin.notebook.socket.Message;
import org.apache.zeppelin.notebook.socket.Message.OP;
@@ -53,6 +44,9 @@
import org.apache.zeppelin.scheduler.Job.Status;
import org.apache.zeppelin.server.ZeppelinServer;
import org.apache.zeppelin.ticket.TicketContainer;
+import org.apache.zeppelin.types.InterpreterSettingsList;
+import org.apache.zeppelin.user.AuthenticationInfo;
+import org.apache.zeppelin.utils.InterpreterBindingUtils;
import org.apache.zeppelin.utils.SecurityUtils;
import org.eclipse.jetty.websocket.servlet.WebSocketServlet;
import org.eclipse.jetty.websocket.servlet.WebSocketServletFactory;
@@ -60,6 +54,13 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import javax.servlet.http.HttpServletRequest;
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.net.UnknownHostException;
+import java.util.*;
+import java.util.concurrent.ConcurrentLinkedQueue;
+
/**
* Zeppelin websocket service.
*/
@@ -131,8 +132,17 @@ public void onMessage(NotebookSocket conn, String msg) {
}
String ticket = TicketContainer.instance.getTicket(messagereceived.principal);
- if (ticket != null && !ticket.equals(messagereceived.ticket))
- throw new Exception("Invalid ticket " + messagereceived.ticket + " != " + ticket);
+ if (ticket != null && !ticket.equals(messagereceived.ticket)){
+ /* not to pollute logs, log instead of exception */
+ if (StringUtils.isEmpty(messagereceived.ticket)) {
+ LOG.debug("{} message: invalid ticket {} != {}", messagereceived.op,
+ messagereceived.ticket, ticket);
+ } else {
+ LOG.warn("{} message: invalid ticket {} != {}", messagereceived.op,
+ messagereceived.ticket, ticket);
+ }
+ return;
+ }
ZeppelinConfiguration conf = ZeppelinConfiguration.create();
boolean allowAnonymous = conf.
@@ -226,7 +236,7 @@ public void onMessage(NotebookSocket conn, String msg) {
listRevisionHistory(conn, notebook, messagereceived);
break;
case NOTE_REVISION:
- getNoteRevision(conn, notebook, messagereceived);
+ getNoteByRevision(conn, notebook, messagereceived);
break;
case LIST_NOTEBOOK_JOBS:
unicastNotebookJobInfo(conn, messagereceived);
@@ -234,6 +244,12 @@ public void onMessage(NotebookSocket conn, String msg) {
case LIST_UPDATE_NOTEBOOK_JOBS:
unicastUpdateNotebookJobInfo(conn, messagereceived);
break;
+ case GET_INTERPRETER_BINDINGS:
+ getInterpreterBindings(conn, messagereceived);
+ break;
+ case SAVE_INTERPRETER_BINDINGS:
+ saveInterpreterBindings(conn, messagereceived);
+ break;
default:
break;
}
@@ -320,7 +336,7 @@ private void broadcastToNoteBindedInterpreter(String interpreterGroupId,
List ids = notebook.getInterpreterFactory().getInterpreters(note.getId());
for (String id : ids) {
if (id.equals(interpreterGroupId)) {
- broadcast(note.id(), m);
+ broadcast(note.getId(), m);
}
}
}
@@ -411,6 +427,29 @@ public void unicastUpdateNotebookJobInfo(NotebookSocket conn, Message fromMessag
.put("notebookRunningJobs", response)));
}
+ public void saveInterpreterBindings(NotebookSocket conn, Message fromMessage) {
+ String noteId = (String) fromMessage.data.get("noteID");
+ try {
+ List settingIdList = gson.fromJson(String.valueOf(
+ fromMessage.data.get("selectedSettingIds")), new TypeToken>() {
+ }.getType());
+ notebook().bindInterpretersToNote(noteId, settingIdList);
+ broadcastInterpreterBindings(noteId,
+ InterpreterBindingUtils.getInterpreterBindings(notebook(), noteId));
+ } catch (Exception e) {
+ LOG.error("Error while saving interpreter bindings", e);
+ }
+ }
+
+ public void getInterpreterBindings(NotebookSocket conn, Message fromMessage)
+ throws IOException {
+ String noteID = (String) fromMessage.data.get("noteID");
+ List settingList =
+ InterpreterBindingUtils.getInterpreterBindings(notebook(), noteID);
+ conn.send(serializeMessage(new Message(OP.INTERPRETER_BINDINGS)
+ .put("interpreterBindings", settingList)));
+ }
+
public List> generateNotebooksInfo(boolean needsReload,
AuthenticationInfo subject) {
@@ -434,11 +473,11 @@ public List> generateNotebooksInfo(boolean needsReload,
for (Note note : notes) {
Map info = new HashMap<>();
- if (hideHomeScreenNotebookFromList && note.id().equals(homescreenNotebookId)) {
+ if (hideHomeScreenNotebookFromList && note.getId().equals(homescreenNotebookId)) {
continue;
}
- info.put("id", note.id());
+ info.put("id", note.getId());
info.put("name", note.getName());
notesInfo.add(info);
}
@@ -447,7 +486,13 @@ public List> generateNotebooksInfo(boolean needsReload,
}
public void broadcastNote(Note note) {
- broadcast(note.id(), new Message(OP.NOTE).put("note", note));
+ broadcast(note.getId(), new Message(OP.NOTE).put("note", note));
+ }
+
+ public void broadcastInterpreterBindings(String noteId,
+ List settingList) {
+ broadcast(noteId, new Message(OP.INTERPRETER_BINDINGS)
+ .put("interpreterBindings", settingList));
}
public void broadcastNoteList(AuthenticationInfo subject) {
@@ -499,7 +544,7 @@ private void sendNote(NotebookSocket conn, HashSet userAndRoles, Noteboo
notebookAuthorization.getReaders(noteId));
return;
}
- addConnectionToNote(note.id(), conn);
+ addConnectionToNote(note.getId(), conn);
conn.send(serializeMessage(new Message(OP.NOTE).put("note", note)));
sendAllAngularObjects(note, conn);
} else {
@@ -523,7 +568,7 @@ private void sendHomeNote(NotebookSocket conn, HashSet userAndRoles,
userAndRoles, notebookAuthorization.getReaders(noteId));
return;
}
- addConnectionToNote(note.id(), conn);
+ addConnectionToNote(note.getId(), conn);
conn.send(serializeMessage(new Message(OP.NOTE).put("note", note)));
sendAllAngularObjects(note, conn);
} else {
@@ -559,7 +604,7 @@ private void updateNote(NotebookSocket conn, HashSet userAndRoles,
note.setName(name);
note.setConfig(config);
if (cronUpdated) {
- notebook.refreshCron(note.id());
+ notebook.refreshCron(note.getId());
}
AuthenticationInfo subject = new AuthenticationInfo(fromMessage.principal);
@@ -598,7 +643,7 @@ private void createNote(NotebookSocket conn, HashSet userAndRoles,
}
note.persist(subject);
- addConnectionToNote(note.id(), (NotebookSocket) conn);
+ addConnectionToNote(note.getId(), (NotebookSocket) conn);
conn.send(serializeMessage(new Message(OP.NEW_NOTE).put("note", note)));
broadcastNoteList(subject);
}
@@ -652,7 +697,7 @@ private void updateParagraph(NotebookSocket conn, HashSet userAndRoles,
p.setTitle((String) fromMessage.get("title"));
p.setText((String) fromMessage.get("paragraph"));
note.persist(subject);
- broadcast(note.id(), new Message(OP.PARAGRAPH).put("paragraph", p));
+ broadcast(note.getId(), new Message(OP.PARAGRAPH).put("paragraph", p));
}
private void cloneNote(NotebookSocket conn, HashSet userAndRoles,
@@ -662,7 +707,7 @@ private void cloneNote(NotebookSocket conn, HashSet userAndRoles,
String name = (String) fromMessage.get("name");
Note newNote = notebook.cloneNote(noteId, name, new AuthenticationInfo(fromMessage.principal));
AuthenticationInfo subject = new AuthenticationInfo(fromMessage.principal);
- addConnectionToNote(newNote.id(), (NotebookSocket) conn);
+ addConnectionToNote(newNote.getId(), (NotebookSocket) conn);
conn.send(serializeMessage(new Message(OP.NEW_NOTE).put("note", newNote)));
broadcastNoteList(subject);
}
@@ -765,12 +810,12 @@ private void angularObjectUpdated(NotebookSocket conn, HashSet userAndRo
List settings = notebook.getInterpreterFactory()
.getInterpreterSettings(note.getId());
for (InterpreterSetting setting : settings) {
- if (setting.getInterpreterGroup(note.id()) == null) {
+ if (setting.getInterpreterGroup(note.getId()) == null) {
continue;
}
- if (interpreterGroupId.equals(setting.getInterpreterGroup(note.id()).getId())) {
+ if (interpreterGroupId.equals(setting.getInterpreterGroup(note.getId()).getId())) {
AngularObjectRegistry angularObjectRegistry = setting
- .getInterpreterGroup(note.id()).getAngularObjectRegistry();
+ .getInterpreterGroup(note.getId()).getAngularObjectRegistry();
// first trying to get local registry
ao = angularObjectRegistry.get(varName, noteId, paragraphId);
@@ -807,17 +852,17 @@ private void angularObjectUpdated(NotebookSocket conn, HashSet userAndRo
List settings = notebook.getInterpreterFactory()
.getInterpreterSettings(note.getId());
for (InterpreterSetting setting : settings) {
- if (setting.getInterpreterGroup(n.id()) == null) {
+ if (setting.getInterpreterGroup(n.getId()) == null) {
continue;
}
- if (interpreterGroupId.equals(setting.getInterpreterGroup(n.id()).getId())) {
+ if (interpreterGroupId.equals(setting.getInterpreterGroup(n.getId()).getId())) {
AngularObjectRegistry angularObjectRegistry = setting
- .getInterpreterGroup(n.id()).getAngularObjectRegistry();
+ .getInterpreterGroup(n.getId()).getAngularObjectRegistry();
this.broadcastExcept(
- n.id(),
+ n.getId(),
new Message(OP.ANGULAR_OBJECT_UPDATE).put("angularObject", ao)
.put("interpreterGroupId", interpreterGroupId)
- .put("noteId", n.id())
+ .put("noteId", n.getId())
.put("paragraphId", ao.getParagraphId()),
conn);
}
@@ -825,10 +870,10 @@ private void angularObjectUpdated(NotebookSocket conn, HashSet userAndRo
}
} else { // broadcast to all web session for the note
this.broadcastExcept(
- note.id(),
+ note.getId(),
new Message(OP.ANGULAR_OBJECT_UPDATE).put("angularObject", ao)
.put("interpreterGroupId", interpreterGroupId)
- .put("noteId", note.id())
+ .put("noteId", note.getId())
.put("paragraphId", ao.getParagraphId()),
conn);
}
@@ -1104,7 +1149,7 @@ private void runParagraph(NotebookSocket conn, HashSet userAndRoles, Not
new InterpreterResult(InterpreterResult.Code.ERROR, ex.getMessage()),
ex);
p.setStatus(Status.ERROR);
- broadcast(note.id(), new Message(OP.PARAGRAPH).put("paragraph", p));
+ broadcast(note.getId(), new Message(OP.PARAGRAPH).put("paragraph", p));
}
}
}
@@ -1136,7 +1181,7 @@ private void checkpointNotebook(NotebookSocket conn, Notebook notebook,
AuthenticationInfo subject = new AuthenticationInfo(fromMessage.principal);
Revision revision = notebook.checkpointNote(noteId, commitMessage, subject);
if (revision != null) {
- List revisions = notebook.listRevisionHistory(noteId, subject);
+ List revisions = notebook.listRevisionHistory(noteId, subject);
conn.send(serializeMessage(new Message(OP.LIST_REVISION_HISTORY)
.put("revisionList", revisions)));
}
@@ -1146,21 +1191,21 @@ private void listRevisionHistory(NotebookSocket conn, Notebook notebook,
Message fromMessage) throws IOException {
String noteId = (String) fromMessage.get("noteId");
AuthenticationInfo subject = new AuthenticationInfo(fromMessage.principal);
- List revisions = notebook.listRevisionHistory(noteId, subject);
+ List revisions = notebook.listRevisionHistory(noteId, subject);
conn.send(serializeMessage(new Message(OP.LIST_REVISION_HISTORY)
.put("revisionList", revisions)));
}
- private void getNoteRevision(NotebookSocket conn, Notebook notebook, Message fromMessage)
+ private void getNoteByRevision(NotebookSocket conn, Notebook notebook, Message fromMessage)
throws IOException {
String noteId = (String) fromMessage.get("noteId");
- Revision revision = (Revision) fromMessage.get("revision");
+ String revisionId = (String) fromMessage.get("revisionId");
AuthenticationInfo subject = new AuthenticationInfo(fromMessage.principal);
- Note revisionNote = notebook.getNoteRevision(noteId, revision, subject);
+ Note revisionNote = notebook.getNoteByRevision(noteId, revisionId, subject);
conn.send(serializeMessage(new Message(OP.NOTE_REVISION)
.put("noteId", noteId)
- .put("revisionId", revision)
+ .put("revisionId", revisionId)
.put("data", revisionNote)));
}
@@ -1264,7 +1309,7 @@ public ParagraphListenerImpl(NotebookServer notebookServer, Note note) {
@Override
public void onProgressUpdate(Job job, int progress) {
notebookServer.broadcast(
- note.id(),
+ note.getId(),
new Message(OP.PROGRESS).put("id", job.getId()).put("progress",
job.progress()));
}
@@ -1339,15 +1384,15 @@ private void sendAllAngularObjects(Note note, NotebookSocket conn) throws IOExce
}
for (InterpreterSetting intpSetting : settings) {
- AngularObjectRegistry registry = intpSetting.getInterpreterGroup(note.id())
+ AngularObjectRegistry registry = intpSetting.getInterpreterGroup(note.getId())
.getAngularObjectRegistry();
- List objects = registry.getAllWithGlobal(note.id());
+ List objects = registry.getAllWithGlobal(note.getId());
for (AngularObject object : objects) {
conn.send(serializeMessage(new Message(OP.ANGULAR_OBJECT_UPDATE)
.put("angularObject", object)
.put("interpreterGroupId",
- intpSetting.getInterpreterGroup(note.id()).getId())
- .put("noteId", note.id())
+ intpSetting.getInterpreterGroup(note.getId()).getId())
+ .put("noteId", note.getId())
.put("paragraphId", object.getParagraphId())
));
}
@@ -1368,7 +1413,7 @@ public void onUpdate(String interpreterGroupId, AngularObject object) {
List notes = notebook.getAllNotes();
for (Note note : notes) {
- if (object.getNoteId() != null && !note.id().equals(object.getNoteId())) {
+ if (object.getNoteId() != null && !note.getId().equals(object.getNoteId())) {
continue;
}
@@ -1379,11 +1424,11 @@ public void onUpdate(String interpreterGroupId, AngularObject object) {
}
broadcast(
- note.id(),
+ note.getId(),
new Message(OP.ANGULAR_OBJECT_UPDATE)
.put("angularObject", object)
.put("interpreterGroupId", interpreterGroupId)
- .put("noteId", note.id())
+ .put("noteId", note.getId())
.put("paragraphId", object.getParagraphId()));
}
}
@@ -1393,7 +1438,7 @@ public void onRemove(String interpreterGroupId, String name, String noteId, Stri
Notebook notebook = notebook();
List notes = notebook.getAllNotes();
for (Note note : notes) {
- if (noteId != null && !note.id().equals(noteId)) {
+ if (noteId != null && !note.getId().equals(noteId)) {
continue;
}
@@ -1401,7 +1446,7 @@ public void onRemove(String interpreterGroupId, String name, String noteId, Stri
for (String id : ids) {
if (id.equals(interpreterGroupId)) {
broadcast(
- note.id(),
+ note.getId(),
new Message(OP.ANGULAR_OBJECT_REMOVE).put("name", name).put(
"noteId", noteId).put("paragraphId", paragraphId));
}
diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/message/InterpreterSettingListForNoteBind.java b/zeppelin-server/src/main/java/org/apache/zeppelin/types/InterpreterSettingsList.java
similarity index 88%
rename from zeppelin-server/src/main/java/org/apache/zeppelin/rest/message/InterpreterSettingListForNoteBind.java
rename to zeppelin-server/src/main/java/org/apache/zeppelin/types/InterpreterSettingsList.java
index 6ec5a54a3ac..e0169167619 100644
--- a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/message/InterpreterSettingListForNoteBind.java
+++ b/zeppelin-server/src/main/java/org/apache/zeppelin/types/InterpreterSettingsList.java
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.zeppelin.rest.message;
+package org.apache.zeppelin.types;
import java.util.List;
@@ -24,13 +24,13 @@
/**
* InterpreterSetting information for binding
*/
-public class InterpreterSettingListForNoteBind {
+public class InterpreterSettingsList {
private String id;
private String name;
private boolean selected;
private List interpreters;
- public InterpreterSettingListForNoteBind(String id, String name,
+ public InterpreterSettingsList(String id, String name,
List interpreters, boolean selected) {
this.id = id;
this.name = name;
diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/utils/InterpreterBindingUtils.java b/zeppelin-server/src/main/java/org/apache/zeppelin/utils/InterpreterBindingUtils.java
new file mode 100644
index 00000000000..9333afd9ef8
--- /dev/null
+++ b/zeppelin-server/src/main/java/org/apache/zeppelin/utils/InterpreterBindingUtils.java
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.utils;
+
+import org.apache.zeppelin.interpreter.InterpreterSetting;
+import org.apache.zeppelin.notebook.Notebook;
+import org.apache.zeppelin.types.InterpreterSettingsList;
+
+import java.util.LinkedList;
+import java.util.List;
+
+/**
+ * Utils for interpreter bindings
+ */
+public class InterpreterBindingUtils {
+ public static List getInterpreterBindings(Notebook notebook,
+ String noteId) {
+ List settingList = new LinkedList<>();
+ List selectedSettings =
+ notebook.getBindedInterpreterSettings(noteId);
+ for (InterpreterSetting setting : selectedSettings) {
+ settingList.add(new InterpreterSettingsList(setting.getId(), setting.getName(),
+ setting.getInterpreterInfos(), true));
+ }
+
+ List availableSettings = notebook.getInterpreterFactory().get();
+ for (InterpreterSetting setting : availableSettings) {
+ boolean selected = false;
+ for (InterpreterSetting selectedSetting : selectedSettings) {
+ if (selectedSetting.getId().equals(setting.getId())) {
+ selected = true;
+ break;
+ }
+ }
+
+ if (!selected) {
+ settingList.add(new InterpreterSettingsList(setting.getId(), setting.getName(),
+ setting.getInterpreterInfos(), false));
+ }
+ }
+
+ return settingList;
+ }
+}
diff --git a/zeppelin-server/src/test/java/org/apache/zeppelin/integration/AuthenticationIT.java b/zeppelin-server/src/test/java/org/apache/zeppelin/integration/AuthenticationIT.java
index 671b213b471..ea3f3637537 100644
--- a/zeppelin-server/src/test/java/org/apache/zeppelin/integration/AuthenticationIT.java
+++ b/zeppelin-server/src/test/java/org/apache/zeppelin/integration/AuthenticationIT.java
@@ -124,7 +124,7 @@ private void logoutUser(String userName) {
userName + "')]")).click();
ZeppelinITUtils.sleep(500, false);
driver.findElement(By.xpath("//div[contains(@class, 'navbar-collapse')]//li[contains(.,'" +
- userName + "')]//a[@ng-click='logout()']")).click();
+ userName + "')]//a[@ng-click='navbar.logout()']")).click();
ZeppelinITUtils.sleep(5000, false);
}
@@ -161,12 +161,12 @@ public void testGroupPermission() throws Exception {
pollingWait(By.xpath("//span[@tooltip='Note permissions']"),
MAX_BROWSER_TIMEOUT_SEC).click();
- pollingWait(By.xpath("//input[@ng-model='permissions.owners']"), MAX_BROWSER_TIMEOUT_SEC)
- .sendKeys("finance");
- pollingWait(By.xpath("//input[@ng-model='permissions.readers']"), MAX_BROWSER_TIMEOUT_SEC)
- .sendKeys("finance");
- pollingWait(By.xpath("//input[@ng-model='permissions.writers']"), MAX_BROWSER_TIMEOUT_SEC)
- .sendKeys("finance");
+ pollingWait(By.xpath(".//*[@id='selectOwners']/following::span//input"),
+ MAX_BROWSER_TIMEOUT_SEC).sendKeys("finance ");
+ pollingWait(By.xpath(".//*[@id='selectReaders']/following::span//input"),
+ MAX_BROWSER_TIMEOUT_SEC).sendKeys("finance ");
+ pollingWait(By.xpath(".//*[@id='selectWriters']/following::span//input"),
+ MAX_BROWSER_TIMEOUT_SEC).sendKeys("finance ");
pollingWait(By.xpath("//button[@ng-click='savePermissions()']"), MAX_BROWSER_TIMEOUT_SEC)
.sendKeys(Keys.ENTER);
diff --git a/zeppelin-server/src/test/java/org/apache/zeppelin/integration/SparkParagraphIT.java b/zeppelin-server/src/test/java/org/apache/zeppelin/integration/SparkParagraphIT.java
index 0ff01353501..500907630de 100644
--- a/zeppelin-server/src/test/java/org/apache/zeppelin/integration/SparkParagraphIT.java
+++ b/zeppelin-server/src/test/java/org/apache/zeppelin/integration/SparkParagraphIT.java
@@ -140,6 +140,25 @@ public void testPySpark() throws Exception {
paragraph1Result.getText().toString(), CoreMatchers.equalTo("test loop 0\ntest loop 1\ntest loop 2")
);
+ // the last statement's evaluation result is printed
+ setTextOfParagraph(2, "%pyspark\\n" +
+ "sc.version\\n" +
+ "1+1");
+ runParagraph(2);
+ try {
+ waitForParagraph(2, "FINISHED");
+ } catch (TimeoutException e) {
+ waitForParagraph(2, "ERROR");
+ collector.checkThat("Paragraph from SparkParagraphIT of testPySpark status: ",
+ "ERROR", CoreMatchers.equalTo("FINISHED")
+ );
+ }
+ WebElement paragraph2Result = driver.findElement(By.xpath(
+ getParagraphXPath(2) + "//div[@class=\"tableDisplay\"]"));
+ collector.checkThat("Paragraph from SparkParagraphIT of testPySpark result: ",
+ paragraph2Result.getText().toString(), CoreMatchers.equalTo("2")
+ );
+
} catch (Exception e) {
handleException("Exception in SparkParagraphIT while testPySpark", e);
}
diff --git a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/InterpreterRestApiTest.java b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/InterpreterRestApiTest.java
index e92432fefaa..c767eb05786 100644
--- a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/InterpreterRestApiTest.java
+++ b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/InterpreterRestApiTest.java
@@ -119,13 +119,22 @@ public void testSettingsCRUD() throws IOException {
delete.releaseConnection();
}
+ @Test
+ public void testSettingsCreateWithEmptyJson() throws IOException {
+ // Call Create Setting REST API
+ PostMethod post = httpPost("/interpreter/setting/", "");
+ LOG.info("testSettingCRUD create response\n" + post.getResponseBodyAsString());
+ assertThat("test create method:", post, isBadRequest());
+ post.releaseConnection();
+ }
+
@Test
public void testInterpreterAutoBinding() throws IOException {
// create note
Note note = ZeppelinServer.notebook.createNote(null);
// check interpreter is binded
- GetMethod get = httpGet("/notebook/interpreter/bind/" + note.id());
+ GetMethod get = httpGet("/notebook/interpreter/bind/" + note.getId());
assertThat(get, isAllowed());
get.addRequestHeader("Origin", "http://localhost");
Map resp = gson.fromJson(get.getResponseBodyAsString(), new TypeToken>() {
diff --git a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/NotebookRestApiTest.java b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/NotebookRestApiTest.java
index 94e7fa8b071..d7f55f54744 100644
--- a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/NotebookRestApiTest.java
+++ b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/NotebookRestApiTest.java
@@ -22,6 +22,7 @@
import com.google.gson.Gson;
import com.google.gson.reflect.TypeToken;
import org.apache.commons.httpclient.methods.GetMethod;
+import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.httpclient.methods.PutMethod;
import org.apache.zeppelin.notebook.Note;
import org.apache.zeppelin.notebook.NotebookAuthorization;
@@ -122,6 +123,54 @@ public void testPermissions() throws IOException {
ZeppelinServer.notebook.removeNote(note2.getId(), null);
}
+
+ @Test
+ public void testGetNoteParagraphJobStatus() throws IOException {
+ Note note1 = ZeppelinServer.notebook.createNote(null);
+ note1.addParagraph();
+
+ String paragraphId = note1.getLastParagraph().getId();
+
+ GetMethod get = httpGet("/notebook/job/" + note1.getId() + "/" + paragraphId);
+ assertThat(get, isAllowed());
+ Map resp = gson.fromJson(get.getResponseBodyAsString(), new TypeToken>() {
+ }.getType());
+ Map> paragraphStatus = (Map>) resp.get("body");
+
+ // Check id and status have proper value
+ assertEquals(paragraphStatus.get("id"), paragraphId);
+ assertEquals(paragraphStatus.get("status"), "READY");
+
+ //cleanup
+ ZeppelinServer.notebook.removeNote(note1.getId(), null);
+
+ }
+
+ @Test
+ public void testCloneNotebook() throws IOException {
+ Note note1 = ZeppelinServer.notebook.createNote(null);
+ PostMethod post = httpPost("/notebook/" + note1.getId(), "");
+ LOG.info("testCloneNotebook response\n" + post.getResponseBodyAsString());
+ assertThat(post, isCreated());
+ Map resp = gson.fromJson(post.getResponseBodyAsString(), new TypeToken>() {
+ }.getType());
+ String clonedNotebookId = (String) resp.get("body");
+ post.releaseConnection();
+
+ GetMethod get = httpGet("/notebook/" + clonedNotebookId);
+ assertThat(get, isAllowed());
+ Map resp2 = gson.fromJson(get.getResponseBodyAsString(), new TypeToken>() {
+ }.getType());
+ Map resp2Body = (Map) resp2.get("body");
+
+ assertEquals((String)resp2Body.get("name"), "Note " + clonedNotebookId);
+ get.releaseConnection();
+
+ //cleanup
+ ZeppelinServer.notebook.removeNote(note1.getId(), null);
+ ZeppelinServer.notebook.removeNote(clonedNotebookId, null);
+
+ }
}
diff --git a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/SecurityRestApiTest.java b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/SecurityRestApiTest.java
index 54c31c1fd6a..b4ecd97e375 100644
--- a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/SecurityRestApiTest.java
+++ b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/SecurityRestApiTest.java
@@ -69,7 +69,7 @@ public void testGetUserList() throws IOException {
get.addRequestHeader("Origin", "http://localhost");
Map resp = gson.fromJson(get.getResponseBodyAsString(),
new TypeToken>(){}.getType());
- List userList = (List) resp.get("body");
+ List userList = (List) ((Map) resp.get("body")).get("users");
collector.checkThat("Search result size", userList.size(),
CoreMatchers.equalTo(1));
collector.checkThat("Search result contains admin", userList.contains("admin"),
@@ -80,7 +80,7 @@ public void testGetUserList() throws IOException {
notUser.addRequestHeader("Origin", "http://localhost");
Map notUserResp = gson.fromJson(notUser.getResponseBodyAsString(),
new TypeToken>(){}.getType());
- List emptyUserList = (List) notUserResp.get("body");
+ List emptyUserList = (List) ((Map) notUserResp.get("body")).get("users");
collector.checkThat("Search result size", emptyUserList.size(),
CoreMatchers.equalTo(0));
diff --git a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
index a65ccbcd762..1250f9ce58b 100644
--- a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
+++ b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
@@ -79,7 +79,7 @@ public void basicRDDTransformationAndActionTest() throws IOException {
waitForFinish(p);
assertEquals(Status.FINISHED, p.getStatus());
assertEquals("55", p.getResult().message());
- ZeppelinServer.notebook.removeNote(note.id(), null);
+ ZeppelinServer.notebook.removeNote(note.getId(), null);
}
@Test
@@ -91,7 +91,7 @@ public void sparkRTest() throws IOException {
if (isSparkR() && sparkVersion >= 14) { // sparkr supported from 1.4.0
// restart spark interpreter
List settings =
- ZeppelinServer.notebook.getBindedInterpreterSettings(note.id());
+ ZeppelinServer.notebook.getBindedInterpreterSettings(note.getId());
for (InterpreterSetting setting : settings) {
if (setting.getName().equals("spark")) {
@@ -100,13 +100,16 @@ public void sparkRTest() throws IOException {
}
}
- // run markdown paragraph, again
+ String sqlContextName = "sqlContext";
+ if (sparkVersion >= 20) {
+ sqlContextName = "spark";
+ }
Paragraph p = note.addParagraph();
Map config = p.getConfig();
config.put("enabled", true);
p.setConfig(config);
p.setText("%r localDF <- data.frame(name=c(\"a\", \"b\", \"c\"), age=c(19, 23, 18))\n" +
- "df <- createDataFrame(sqlContext, localDF)\n" +
+ "df <- createDataFrame(" + sqlContextName + ", localDF)\n" +
"count(df)"
);
note.run(p.getId());
@@ -115,7 +118,7 @@ public void sparkRTest() throws IOException {
assertEquals(Status.FINISHED, p.getStatus());
assertEquals("[1] 3", p.getResult().message());
}
- ZeppelinServer.notebook.removeNote(note.id(), null);
+ ZeppelinServer.notebook.removeNote(note.getId(), null);
}
@Test
@@ -138,7 +141,7 @@ public void pySparkTest() throws IOException {
assertEquals(Status.FINISHED, p.getStatus());
assertEquals("55\n", p.getResult().message());
}
- ZeppelinServer.notebook.removeNote(note.id(), null);
+ ZeppelinServer.notebook.removeNote(note.getId(), null);
}
@Test
@@ -169,7 +172,7 @@ public void pySparkAutoConvertOptionTest() throws IOException {
assertEquals(Status.FINISHED, p.getStatus());
assertEquals("10\n", p.getResult().message());
}
- ZeppelinServer.notebook.removeNote(note.id(), null);
+ ZeppelinServer.notebook.removeNote(note.getId(), null);
}
@Test
@@ -201,7 +204,7 @@ public void zRunTest() throws IOException {
assertEquals(Status.FINISHED, p2.getStatus());
assertEquals("10", p2.getResult().message());
- ZeppelinServer.notebook.removeNote(note.id(), null);
+ ZeppelinServer.notebook.removeNote(note.getId(), null);
}
@Test
@@ -213,7 +216,7 @@ public void pySparkDepLoaderTest() throws IOException {
if (isPyspark() && sparkVersionNumber >= 14) {
// restart spark interpreter
List settings =
- ZeppelinServer.notebook.getBindedInterpreterSettings(note.id());
+ ZeppelinServer.notebook.getBindedInterpreterSettings(note.getId());
for (InterpreterSetting setting : settings) {
if (setting.getName().equals("spark")) {
diff --git a/zeppelin-server/src/test/java/org/apache/zeppelin/server/CorsFilterTest.java b/zeppelin-server/src/test/java/org/apache/zeppelin/server/CorsFilterTest.java
index a76d4c86a6d..df2a6e92e0d 100644
--- a/zeppelin-server/src/test/java/org/apache/zeppelin/server/CorsFilterTest.java
+++ b/zeppelin-server/src/test/java/org/apache/zeppelin/server/CorsFilterTest.java
@@ -1,7 +1,4 @@
-/**
- * Created by joelz on 8/6/15.
- *
- *
+/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
@@ -19,7 +16,6 @@
*/
package org.apache.zeppelin.server;
-import org.apache.zeppelin.socket.TestHttpServletRequest;
import org.junit.Assert;
import org.junit.Test;
import org.mockito.invocation.InvocationOnMock;
@@ -27,14 +23,16 @@
import javax.servlet.FilterChain;
import javax.servlet.ServletException;
+import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
+
import java.io.IOException;
import static org.mockito.Matchers.anyString;
import static org.mockito.Mockito.*;
/**
- * BASIC Cors rest api tests
+ * Basic CORS REST API tests
*/
public class CorsFilterTest {
@@ -42,11 +40,12 @@ public class CorsFilterTest {
public static Integer count = 0;
@Test
+ @SuppressWarnings("rawtypes")
public void ValidCorsFilterTest() throws IOException, ServletException {
CorsFilter filter = new CorsFilter();
HttpServletResponse mockResponse = mock(HttpServletResponse.class);
FilterChain mockedFilterChain = mock(FilterChain.class);
- TestHttpServletRequest mockRequest = mock(TestHttpServletRequest.class);
+ HttpServletRequest mockRequest = mock(HttpServletRequest.class);
when(mockRequest.getHeader("Origin")).thenReturn("http://localhost:8080");
when(mockRequest.getMethod()).thenReturn("Empty");
when(mockRequest.getServerName()).thenReturn("localhost");
@@ -66,11 +65,12 @@ public Object answer(InvocationOnMock invocationOnMock) throws Throwable {
}
@Test
+ @SuppressWarnings("rawtypes")
public void InvalidCorsFilterTest() throws IOException, ServletException {
CorsFilter filter = new CorsFilter();
HttpServletResponse mockResponse = mock(HttpServletResponse.class);
FilterChain mockedFilterChain = mock(FilterChain.class);
- TestHttpServletRequest mockRequest = mock(TestHttpServletRequest.class);
+ HttpServletRequest mockRequest = mock(HttpServletRequest.class);
when(mockRequest.getHeader("Origin")).thenReturn("http://evillocalhost:8080");
when(mockRequest.getMethod()).thenReturn("Empty");
when(mockRequest.getServerName()).thenReturn("evillocalhost");
diff --git a/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java b/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
index bc131131ea9..01a24e2e1de 100644
--- a/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
+++ b/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
@@ -1,7 +1,4 @@
-/**
- * Created by joelz on 8/6/15.
- *
- *
+/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
@@ -35,10 +32,12 @@
import org.apache.zeppelin.rest.AbstractTestRestApi;
import org.apache.zeppelin.server.ZeppelinServer;
import org.junit.AfterClass;
+import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
import javax.servlet.http.HttpServletRequest;
+
import java.io.IOException;
import java.net.InetAddress;
import java.net.UnknownHostException;
@@ -51,12 +50,13 @@
/**
- * BASIC Zeppelin rest api tests
+ * Basic REST API tests for notebookServer
*/
public class NotebookServerTest extends AbstractTestRestApi {
private static Notebook notebook;
private static NotebookServer notebookServer;
private static Gson gson;
+ private HttpServletRequest mockRequest;
@BeforeClass
public static void init() throws Exception {
@@ -71,19 +71,24 @@ public static void destroy() throws Exception {
AbstractTestRestApi.shutDown();
}
+ @Before
+ public void setUp() {
+ mockRequest = mock(HttpServletRequest.class);
+ }
+
@Test
public void checkOrigin() throws UnknownHostException {
NotebookServer server = new NotebookServer();
String origin = "http://" + InetAddress.getLocalHost().getHostName() + ":8080";
assertTrue("Origin " + origin + " is not allowed. Please check your hostname.",
- server.checkOrigin(new TestHttpServletRequest(), origin));
+ server.checkOrigin(mockRequest, origin));
}
@Test
public void checkInvalidOrigin(){
NotebookServer server = new NotebookServer();
- assertFalse(server.checkOrigin(new TestHttpServletRequest(), "http://evillocalhost:8080"));
+ assertFalse(server.checkOrigin(mockRequest, "http://evillocalhost:8080"));
}
@Test
@@ -166,7 +171,7 @@ public void testImportNotebook() throws IOException {
}
@Test
- public void should_bind_angular_object_to_remote_for_paragraphs() throws Exception {
+ public void bindAngularObjectToRemoteForParagraphs() throws Exception {
//Given
final String varName = "name";
final String value = "DuyHai DOAN";
@@ -191,8 +196,7 @@ public void should_bind_angular_object_to_remote_for_paragraphs() throws Excepti
when(paragraph.getCurrentRepl().getInterpreterGroup()).thenReturn(mdGroup);
-
- final AngularObject ao1 = AngularObjectBuilder.build(varName, value, "noteId", "paragraphId");
+ final AngularObject ao1 = AngularObjectBuilder.build(varName, value, "noteId", "paragraphId");
when(mdRegistry.addAndNotifyRemoteProcess(varName, value, "noteId", "paragraphId")).thenReturn(ao1);
@@ -217,7 +221,7 @@ public void should_bind_angular_object_to_remote_for_paragraphs() throws Excepti
}
@Test
- public void should_bind_angular_object_to_local_for_paragraphs() throws Exception {
+ public void bindAngularObjectToLocalForParagraphs() throws Exception {
//Given
final String varName = "name";
final String value = "DuyHai DOAN";
@@ -241,7 +245,7 @@ public void should_bind_angular_object_to_local_for_paragraphs() throws Exceptio
when(paragraph.getCurrentRepl().getInterpreterGroup()).thenReturn(mdGroup);
- final AngularObject ao1 = AngularObjectBuilder.build(varName, value, "noteId", "paragraphId");
+ final AngularObject ao1 = AngularObjectBuilder.build(varName, value, "noteId", "paragraphId");
when(mdRegistry.add(varName, value, "noteId", "paragraphId")).thenReturn(ao1);
@@ -264,7 +268,7 @@ public void should_bind_angular_object_to_local_for_paragraphs() throws Exceptio
}
@Test
- public void should_unbind_angular_object_from_remote_for_paragraphs() throws Exception {
+ public void unbindAngularObjectFromRemoteForParagraphs() throws Exception {
//Given
final String varName = "name";
final String value = "val";
@@ -286,7 +290,7 @@ public void should_unbind_angular_object_from_remote_for_paragraphs() throws Exc
when(paragraph.getCurrentRepl().getInterpreterGroup()).thenReturn(mdGroup);
- final AngularObject ao1 = AngularObjectBuilder.build(varName, value, "noteId", "paragraphId");
+ final AngularObject ao1 = AngularObjectBuilder.build(varName, value, "noteId", "paragraphId");
when(mdRegistry.removeAndNotifyRemoteProcess(varName, "noteId", "paragraphId")).thenReturn(ao1);
NotebookSocket conn = mock(NotebookSocket.class);
NotebookSocket otherConn = mock(NotebookSocket.class);
@@ -309,7 +313,7 @@ public void should_unbind_angular_object_from_remote_for_paragraphs() throws Exc
}
@Test
- public void should_unbind_angular_object_from_local_for_paragraphs() throws Exception {
+ public void unbindAngularObjectFromLocalForParagraphs() throws Exception {
//Given
final String varName = "name";
final String value = "val";
@@ -331,8 +335,7 @@ public void should_unbind_angular_object_from_local_for_paragraphs() throws Exce
when(paragraph.getCurrentRepl().getInterpreterGroup()).thenReturn(mdGroup);
- final AngularObject ao1 = AngularObjectBuilder.build(varName, value, "noteId", "paragraphId");
-
+ final AngularObject ao1 = AngularObjectBuilder.build(varName, value, "noteId", "paragraphId");
when(mdRegistry.remove(varName, "noteId", "paragraphId")).thenReturn(ao1);
@@ -355,12 +358,9 @@ public void should_unbind_angular_object_from_local_for_paragraphs() throws Exce
private NotebookSocket createWebSocket() {
NotebookSocket sock = mock(NotebookSocket.class);
- when(sock.getRequest()).thenReturn(createHttpServletRequest());
+ when(sock.getRequest()).thenReturn(mockRequest);
return sock;
}
- private HttpServletRequest createHttpServletRequest() {
- return mock(HttpServletRequest.class);
- }
}
diff --git a/zeppelin-server/src/test/java/org/apache/zeppelin/socket/TestHttpServletRequest.java b/zeppelin-server/src/test/java/org/apache/zeppelin/socket/TestHttpServletRequest.java
deleted file mode 100644
index 0a176baa4e3..00000000000
--- a/zeppelin-server/src/test/java/org/apache/zeppelin/socket/TestHttpServletRequest.java
+++ /dev/null
@@ -1,372 +0,0 @@
-/**
- * Created by joelz on 8/6/15.
- *
- *
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.zeppelin.socket;
-
-import javax.servlet.*;
-import javax.servlet.http.*;
-import java.io.BufferedReader;
-import java.io.IOException;
-import java.io.UnsupportedEncodingException;
-import java.security.Principal;
-import java.util.Collection;
-import java.util.Enumeration;
-import java.util.Locale;
-import java.util.Map;
-
-/**
- * Created by joelz on 8/6/15.
- * Helps mocking a http servlet request
- */
-public class TestHttpServletRequest implements HttpServletRequest {
- @Override
- public boolean authenticate(HttpServletResponse httpServletResponse) throws IOException, ServletException {
- return false;
- }
-
- @Override
- public String getAuthType() {
- return null;
- }
-
- @Override
- public String getContextPath() {
- return null;
- }
-
- @Override
- public Cookie[] getCookies() {
- return new Cookie[0];
- }
-
- @Override
- public long getDateHeader(String s) {
- return 0;
- }
-
- @Override
- public String getHeader(String s) {
- switch (s) {
- case "Origin":
- return "http://localhost:8080";
- }
-
- return null;
- }
-
- @Override
- public Enumeration getHeaderNames() {
- return null;
- }
-
- @Override
- public Enumeration getHeaders(String s) {
- return null;
- }
-
- @Override
- public int getIntHeader(String s) {
- return 0;
- }
-
- @Override
- public String getMethod() {
- return null;
- }
-
- @Override
- public Part getPart(String s) throws IOException, ServletException {
- return null;
- }
-
- @Override
- public Collection getParts() throws IOException, ServletException {
- return null;
- }
-
- @Override
- public String getPathInfo() {
- return null;
- }
-
- @Override
- public String getPathTranslated() {
- return null;
- }
-
- @Override
- public String getQueryString() {
- return null;
- }
-
- @Override
- public String getRemoteUser() {
- return null;
- }
-
- @Override
- public String getRequestedSessionId() {
- return null;
- }
-
- @Override
- public String getRequestURI() {
- return null;
- }
-
- @Override
- public StringBuffer getRequestURL() {
- return null;
- }
-
- @Override
- public String getServletPath() {
- return null;
- }
-
- @Override
- public HttpSession getSession() {
- return null;
- }
-
- @Override
- public HttpSession getSession(boolean b) {
- return null;
- }
-
- @Override
- public Principal getUserPrincipal() {
- return null;
- }
-
- @Override
- public boolean isRequestedSessionIdFromCookie() {
- return false;
- }
-
- @Override
- public boolean isRequestedSessionIdFromUrl() {
- return false;
- }
-
- @Override
- public boolean isRequestedSessionIdFromURL() {
- return false;
- }
-
- @Override
- public boolean isRequestedSessionIdValid() {
- return false;
- }
-
- @Override
- public boolean isUserInRole(String s) {
- return false;
- }
-
- @Override
- public void login(String s, String s1) throws ServletException {
-
- }
-
- @Override
- public void logout() throws ServletException {
-
- }
-
- @Override
- public AsyncContext getAsyncContext() {
- return null;
- }
-
- @Override
- public Object getAttribute(String s) {
- return null;
- }
-
- @Override
- public Enumeration getAttributeNames() {
- return null;
- }
-
- @Override
- public String getCharacterEncoding() {
- return null;
- }
-
- @Override
- public int getContentLength() {
- return 0;
- }
-
- @Override
- public String getContentType() {
- return null;
- }
-
- @Override
- public DispatcherType getDispatcherType() {
- return null;
- }
-
- @Override
- public ServletInputStream getInputStream() throws IOException {
- return null;
- }
-
- @Override
- public String getLocalAddr() {
- return null;
- }
-
- @Override
- public Locale getLocale() {
- return null;
- }
-
- @Override
- public Enumeration getLocales() {
- return null;
- }
-
- @Override
- public String getLocalName() {
- return null;
- }
-
- @Override
- public int getLocalPort() {
- return 0;
- }
-
- @Override
- public String getParameter(String s) {
- return null;
- }
-
- @Override
- public Map getParameterMap() {
- return null;
- }
-
- @Override
- public Enumeration getParameterNames() {
- return null;
- }
-
- @Override
- public String[] getParameterValues(String s) {
- return new String[0];
- }
-
- @Override
- public String getProtocol() {
- return null;
- }
-
- @Override
- public BufferedReader getReader() throws IOException {
- return null;
- }
-
- @Override
- public String getRealPath(String s) {
- return null;
- }
-
- @Override
- public String getRemoteAddr() {
- return null;
- }
-
- @Override
- public String getRemoteHost() {
- return null;
- }
-
- @Override
- public int getRemotePort() {
- return 0;
- }
-
- @Override
- public RequestDispatcher getRequestDispatcher(String s) {
- return null;
- }
-
- @Override
- public String getScheme() {
- return null;
- }
-
- @Override
- public String getServerName() {
- return "localhost";
- }
-
- @Override
- public int getServerPort() {
- return 0;
- }
-
- @Override
- public ServletContext getServletContext() {
- return null;
- }
-
- @Override
- public boolean isAsyncStarted() {
- return false;
- }
-
- @Override
- public boolean isAsyncSupported() {
- return false;
- }
-
- @Override
- public boolean isSecure() {
- return false;
- }
-
- @Override
- public void removeAttribute(String s) {
-
- }
-
- @Override
- public void setAttribute(String s, Object o) {
-
- }
-
- @Override
- public void setCharacterEncoding(String s) throws UnsupportedEncodingException {
-
- }
-
- @Override
- public AsyncContext startAsync() {
- return null;
- }
-
- @Override
- public AsyncContext startAsync(ServletRequest servletRequest, ServletResponse servletResponse) {
- return null;
- }
-}
diff --git a/zeppelin-web/bower.json b/zeppelin-web/bower.json
index 5d849b3490d..e20493fa7fc 100644
--- a/zeppelin-web/bower.json
+++ b/zeppelin-web/bower.json
@@ -31,7 +31,9 @@
"ng-focus-if": "~1.0.2",
"bootstrap3-dialog": "bootstrap-dialog#~1.34.7",
"handsontable": "~0.24.2",
- "moment-duration-format": "^1.3.0"
+ "moment-duration-format": "^1.3.0",
+ "select2": "^4.0.3",
+ "angular-esri-map": "~2.0.0"
},
"devDependencies": {
"angular-mocks": "1.5.0"
diff --git a/zeppelin-web/src/app/app.controller.js b/zeppelin-web/src/app/app.controller.js
index 8a0466b4cb0..ff30b829c35 100644
--- a/zeppelin-web/src/app/app.controller.js
+++ b/zeppelin-web/src/app/app.controller.js
@@ -13,7 +13,7 @@
*/
'use strict';
-angular.module('zeppelinWebApp').controller('MainCtrl', function($scope, $rootScope, $window) {
+angular.module('zeppelinWebApp').controller('MainCtrl', function($scope, $rootScope, $window, arrayOrderingSrv) {
$scope.looknfeel = 'default';
var init = function() {
@@ -41,6 +41,12 @@ angular.module('zeppelinWebApp').controller('MainCtrl', function($scope, $rootSc
$rootScope.$broadcast('setLookAndFeel', 'default');
});
+ $rootScope.noteName = function(note) {
+ if (!_.isEmpty(note)) {
+ return arrayOrderingSrv.getNoteName(note);
+ }
+ };
+
BootstrapDialog.defaultOptions.onshown = function() {
angular.element('#' + this.id).find('.btn:last').focus();
};
diff --git a/zeppelin-web/src/app/app.js b/zeppelin-web/src/app/app.js
index 98d6b879f09..f0f1d89cee5 100644
--- a/zeppelin-web/src/app/app.js
+++ b/zeppelin-web/src/app/app.js
@@ -33,7 +33,8 @@
'xeditable',
'ngToast',
'focus-if',
- 'ngResource'
+ 'ngResource',
+ 'esri.map'
])
.filter('breakFilter', function() {
return function(text) {
@@ -98,6 +99,13 @@
var baseUrlSrv = angular.injector(['zeppelinWebApp']).get('baseUrlSrv');
// withCredentials when running locally via grunt
$http.defaults.withCredentials = true;
+ jQuery.ajaxSetup({
+ dataType: 'json',
+ xhrFields: {
+ withCredentials: true
+ },
+ crossDomain: true
+ });
return $http.get(baseUrlSrv.getRestApiBase() + '/security/ticket').then(function(response) {
zeppelinWebApp.run(function($rootScope) {
$rootScope.ticket = angular.fromJson(response.data).body;
diff --git a/zeppelin-web/src/app/interpreter/interpreter-create/interpreter-create.html b/zeppelin-web/src/app/interpreter/interpreter-create/interpreter-create.html
index 599248b2e56..1550126b9b7 100644
--- a/zeppelin-web/src/app/interpreter/interpreter-create/interpreter-create.html
+++ b/zeppelin-web/src/app/interpreter/interpreter-create/interpreter-create.html
@@ -72,8 +72,8 @@
@@ -88,6 +88,35 @@
+
+ * {
+ * 'login': 'userLogin',
+ * 'password': 'userpassword'
+ * }
+ *
+ * @param login
+ * @param pwd
+ * @return
+ */
+ protected String createLoginPayload(String login, char[] pwd) {
+ StringBuilder sb = new StringBuilder("{\"login\":\"");
+ return sb.append(login).append("\", \"password\":\"").append(pwd).append("\"}").toString();
+ }
+
+ /**
+ * Perform a Simple URL check by using URI(url).toURL().
+ * If the url is not valid, the try-catch condition will catch the exceptions and return false,
+ * otherwise true will be returned.
+ *
+ * @param url
+ * @return
+ */
+ protected boolean isZeppelinHubUrlValid(String url) {
+ boolean valid;
+ try {
+ new URI(url).toURL();
+ valid = true;
+ } catch (URISyntaxException | MalformedURLException e) {
+ LOG.error("Zeppelinhub url is not valid, default ZeppelinHub url will be used.", e);
+ valid = false;
+ }
+ return valid;
+ }
+
+ /**
+ * Helper class that will be use to deserialize ZeppelinHub response.
+ */
+ protected class User {
+ public String login;
+ public String email;
+ public String name;
+ }
+}
diff --git a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/GetUserList.java b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/GetUserList.java
index 2727fb4a411..f1a895c8bcf 100644
--- a/zeppelin-server/src/main/java/org/apache/zeppelin/rest/GetUserList.java
+++ b/zeppelin-server/src/main/java/org/apache/zeppelin/rest/GetUserList.java
@@ -67,6 +67,25 @@ public ListCreate new interpreter
-
-
+
+
Create new interpreter
pu-elastic-input-minwidth="180px" ng-model="newInterpreterSetting.option.port" />
+
+
+
+
+
+
+
+
+
+
+
+
+
Properties