Notebook: HDFS as a backend storage (Read & Write Mode) #1479

hayssams · 2016-10-02T18:39:59Z

What is this PR for?

Zeppelin currently supports many backends for storing notes through Apache Commons VFS.
Apache Commons VFS supports HDFS in readonly mode.
This PR makes HDFS a first class citizen by allowing users to load notes from / save notes to HDFS.

What type of PR is it?

Improvement

Todos

- Task

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-1515

How should this be tested?

Update zeppelin.notebook.dir property to a value like hdfs://localhost:9000/tmp/notebook and the property zeppelin.notebook.storage to the value org.apache.zeppelin.notebook.repo.HdfsNotebookRepo

check that your notes are loaded from and stored to HDFS by listing notes using the command :
hdfs dfs -ls /tmp/notebook

Screenshots (if appropriate)

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? Yes

Leemoonsoo · 2016-10-05T14:14:52Z

Thanks for the contribution.

How about let user set not only HADOOP_CONF_DIR but also HADOOP_HOME in conf/zeppelin-env.sh to use HDFS notebook storage?

Then hadoop dependencies in zeppelin-zengine can be provided scope, and that'll reduce binary package size. Also that'll allow user set different hadoop version without rebuild.

what do you think?

hayssams · 2016-10-05T15:57:41Z

@Leemoonsoo Just pushed the changes to make hadoop lib provided.

zjffdu · 2016-10-11T08:32:38Z

bin/zeppelin.sh

 addJarInDir "${ZEPPELIN_HOME}/zeppelin-web/target/lib"

+
 CLASSPATH+=":${ZEPPELIN_CLASSPATH}"


unecessary change

zjffdu · 2016-10-11T08:39:31Z

pom.xml

    <gson.version>2.2</gson.version>
    <guava.version>15.0</guava.version>
    <jetty.version>9.2.15.v20160210</jetty.version>
+    <hadoop.hdfs.version>2.6.0</hadoop.hdfs.version>


how about using hadoop.version ? Since in spark-dependencies/pom.xml, we also define hadoop.verison. We can reuse the same property name. Otherwise user need to specify both hadoop.version and hadoop.hdfs.version when building with custom hadoop.

zjffdu · 2016-10-11T08:42:13Z

zeppelin-zengine/pom.xml

+    <dependency>
+      <groupId>org.apache.hadoop</groupId>
+      <artifactId>hadoop-common</artifactId>
+      <version>${hadoop.hdfs.version}</version>


I think you can remove hadoop-common and hadoop-hdfs in parent pom.xml since you define them here.

zjffdu · 2016-10-11T08:42:37Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/HdfsNotebookRepo.java

+ */
+
+public class HdfsNotebookRepo implements NotebookRepo {
+  private Logger logger = LoggerFactory.getLogger(HdfsNotebookRepo.class);


logger can be static

zjffdu · 2016-10-11T08:43:36Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/HdfsNotebookRepo.java

+  public HdfsNotebookRepo(ZeppelinConfiguration conf) throws IOException {
+    this.conf = conf;
+    String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
+    try {


Is the right exception caught when HADOOP_CONF_DIR is not defined ?

Null value is valid for HADOOP_CONF_DIR. Hdfs API will default to the file:// protocol instead of hdfs ://

zjffdu · 2016-10-11T08:46:33Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/util/HdfsUtils.java

+   */
+  public HdfsUtils(String dataPath, String hadoopConfDir) throws URISyntaxException {
+    logger.info("hadoopConfDir:" + hadoopConfDir);
+    if (hadoopConfDir != null && !hadoopConfDir.equals("")) {


Can be replaced with StringUtils.isBlank

zjffdu · 2016-10-11T08:46:38Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/util/HdfsUtils.java

+   */
+  public HdfsUtils(String dataPath, String hadoopConfDir) throws URISyntaxException {
+    logger.info("hadoopConfDir:" + hadoopConfDir);
+    if (hadoopConfDir != null && !hadoopConfDir.equals("")) {


Can be replaced with StringUtils.isBlank

zjffdu · 2016-10-11T08:48:04Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/util/HdfsUtils.java

+ */
+public class HdfsUtils {
+  protected String dataPath;
+  private Logger logger = LoggerFactory.getLogger(HdfsUtils.class);


logger can be static

zjffdu · 2016-10-11T08:51:26Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/HdfsNotebookRepo.java

+  @Override
+  public Revision checkpoint(String noteId, String checkpointMsg, AuthenticationInfo subject)
+      throws IOException {
+    // Auto-generated method stub


these kinds of code can be removed.

Won't compile if note present

zjffdu · 2016-10-11T08:54:00Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/util/HdfsUtils.java

+/**
+ * Utility class to create / move / delete / write to / read from HDFS filessytem
+ */
+public class HdfsUtils {


*Utils class usually provides static method. Either you can change all its method to static or rename it to a more proper name

hayssams · 2016-10-17T08:41:07Z

@Leemoonsoo
Test testClose in org.apache.zeppelin.python.PythonInterpreterTest. Can't understand why it is failing on travis CI but not failing on my laptop.

jongyoul · 2016-10-17T16:59:11Z

@hayssams Why did you removed provided scope? If it doesn't, all kind of hadoop dependencies will be included in zengine.

hayssams · 2016-10-17T20:18:03Z

@jongyoul
When using provided and bringing in all hadoop dependencies to zengine by adding hadoop lib in HADOOP_CONF_DIR to the classpath, Zeppelin raise an exception at runtime.

However by bringing explicitely in scope to the zengine only hadoop-hdfs and hadoop-common, everything works as intended.

jongyoul · 2016-10-18T01:21:03Z

@hayssams AFAIK, those hadoop-* have a lot of dependencies by themselves and it's possible to make a problem like version mismatch even in case that we don't use that feature. What do you think of it?

zjffdu · 2016-10-19T08:43:18Z

Agree with @jongyoul we should make hadoop dependency as provided, @hayssams Could you paste the errors you can see when using provided ? So that someone else can help that.

hayssams · 2016-10-23T15:18:29Z

@jongyoul @zjffdu
This issue is with hadoop 2.7 only. With hadoop 2.6 making hadoop dependency as provided works well.

With Hadoop 2.7 (and only 2.7) I get the following exception
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.tracing.SpanReceiverHost.get(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/String;)Lorg/apache/hadoop/tracing/SpanReceiverHost;
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:634)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at org.apache.zeppelin.util.HdfsSite.mkdirs(HdfsSite.java:98)

This is probably due to the fact that even when compiling with the hadoop.2-7 profile, we still get in the zeppelin lib folder the file hadoop-common-2.6.0.jar

jongyoul · 2016-10-24T03:09:33Z

@hayssams Do you have any clue that you mentioned? I don't know why two different jars exist while building Zeppelin.

aspen01 · 2016-11-01T14:28:33Z

You know, there is WebHDFS Rest API in hadoop.
If we use this, we don’t need to care about library dependency and configuration.
What do you think using WebHDFS Rest API?

HDFS interpreter already use it, so I added HDFSNotebookRepo using this function.
Please check this code : aspen01@4b57a2f

aspen01 · 2016-11-04T15:42:45Z

I made a pull request : #1600

### What is this PR for? This PR implements TableData abstraction that every interpreter can use to share data through resource pool. TableData.java - Definition of TableData abstraction TableDataProxy.java - Proxy object of TableData. This enables data access through ResourcePool InterpreterResultTableData - Convert Table type interpret result into TableData. Each interpreter can implement it's own TableData abstraction. For example JDBCTableData, DataFrameTableData, and so on. ### What type of PR is it? Feature ### Todos * [x] - TableData * [x] - TableDataProxy * [x] - Unittest ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-753 ### How should this be tested? Unittest included ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <[email protected]> Closes #1952 from Leemoonsoo/ZEPPELIN-753 and squashes the following commits: d85964b [Lee moon soo] Proxy for tabledata 2211d30 [Lee moon soo] TableData abstraction 3b49105 [Lee moon soo] make RemoteResource.invokeMethod returns Resource when return resource name is specified 60a1dcb [Lee moon soo] Support method invocation on the resource in local,remote resource pool

### What is this PR for? zeppelin logo image (zepLogo.png) is broken in `web dev` mode like below screenshot. ![screenshot from 2017-01-12 14-50-35](https://cloud.githubusercontent.com/assets/8110458/21883058/948cdd2c-d8f0-11e6-8a48-f738a811259f.png) ### What type of PR is it? [Bug Fix] ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-1943 ### How should this be tested? 1. Run front-end application in [dev mode](https://github.com/apache/zeppelin/tree/master/zeppelin-web) 2. Open console in web developer mode. 3. Connect/refresh localhost:9000 4. Check to be shown Logo image in Zeppelin home. ![image](https://cloud.githubusercontent.com/assets/8110458/21883383/3551188a-d8f2-11e6-980e-cc4045890172.png) 5. Check errors in console. ### Screenshots (if appropriate) ![screenshot from 2017-01-12 16-58-40](https://cloud.githubusercontent.com/assets/8110458/21883263/972a2372-d8f1-11e6-82d5-543d0995bded.png) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: soralee <[email protected]> Author: soralee <[email protected]> Closes #1892 from soralee/ZEPPELIN-1943_logo and squashes the following commits: 7819871 [soralee] extract variable express 41b4be6 [soralee] web dev mode enable context path a09b8fe [soralee] web dev mode enable context path 28ec0e1 [soralee] change zeppelin log path

…in.interpreters property in zeppelin-site.xml ### What is this PR for? Default interpreter list should honour order of zeppelin.interpreters property in zeppelin-site.xml ### What type of PR is it? [Bug Fix] ### What is the Jira issue? * [ZEPPELIN-2083](https://issues.apache.org/jira/browse/ZEPPELIN-2083) ### How should this be tested? Remove default spark interpreter, and and another with a different name say "spark2". This newly created interpreter should show up on top. ### Screenshots (if appropriate) Before: <img width="1439" alt="screen shot 2017-02-08 at 6 58 09 pm" src="https://cloud.githubusercontent.com/assets/674497/22739380/66c50e72-ee31-11e6-83cc-07669d292a12.png"> After: <img width="1434" alt="screen shot 2017-02-08 at 6 56 42 pm" src="https://cloud.githubusercontent.com/assets/674497/22739379/66c377d8-ee31-11e6-965c-f9f61b288d99.png"> ### Questions: * Does the licenses files need update? N/A * Is there breaking changes for older versions? N/A * Does this needs documentation? N/A Author: Prabhjyot Singh <[email protected]> Closes #1992 from prabhjyotsingh/ZEPPELIN-2053 and squashes the following commits: b17d9bd [Prabhjyot Singh] ZEPPELIN-2053: default interpreter list should honour order of zeppelin.interpreters property in zeppelin-site.xml

### What is this PR for? `IMain.interpret` changes current thread's context classloader. It may cause different issues and definitely is the reason of [ZEPPELIN-1738](https://issues.apache.org/jira/browse/ZEPPELIN-1738) test failures. It's a known scala bug. See https://issues.scala-lang.org/browse/SI-9587 As a workaround we need to save and restore context classloader manually ### What type of PR is it? Bug Fix ### What is the Jira issue? [ZEPPELIN-1972](https://issues.apache.org/jira/browse/ZEPPELIN-1972) ### How should this be tested? Run ignite interpreter test and ignite sql interpreter test in the same thread ``` mvn test -pl ignite -am -Pscala-2.11 -Dtest=org.apache.zeppelin.ignite.IgniteInterpreterTest#testInterpret,org.apache.zeppelin.ignite.IgniteSqlInterpreterTest#testSql -DfailIfNoTests=false ``` ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Igor Drozdov <[email protected]> Closes #1911 from DrIgor/ZEPPELIN-1972 and squashes the following commits: 35c5abd [Igor Drozdov] Preserve context classloader

### What is this PR for? This PR reduces travis build time | Test profile | Before (rough avg min) | After (rough avg min) | | ------------- |:-------------:| -----:| | RAT check | 1.5 | 1.5 | | All modules with spark 2.1 and scala 2.11 | 35 | 35 | | All modules with spark 2.0 and scala 2.10 | 35 | 35 | | Spark 1.6 with scala 2.10 | 35 | **15** | | Spark 1.6 with scala 2.11 | 35 | **15** | | Spark 1.5 | 25 | **removed** | | Spark 1.4 | 25 | **removed** | | Selenium | 25 | 25 | | Python2 | 8 | 8 | | Python3 | 8 | 8 | | Livy | 15 | **8** | | Total | 247.5 | 149 | ### What type of PR is it? Improvement ### Todos * [x] - Remove spark 1.5 profile and 1.4 profile * [x] - optimize profiles ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2058 ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <[email protected]> Closes #2003 from Leemoonsoo/ZEPPELIN-2058 and squashes the following commits: 7e50ea5 [Lee moon soo] create zeppelin-web/dist directory for test b14b97e [Lee moon soo] prevent concurrent test execution in DirAccessTest 7860259 [Lee moon soo] reduce build time bdcfd1f [Lee moon soo] Reduce log

…on + sqlcompleter tests ### What is this PR for? This PR changes autocompletion behaviour in jdbc interpeter. There are some changes: * [main change] autocompletion now depends on what are you typing. Now there are four types of competion: schema, table, column and keywords. If you typing new word then autocompetion suggests only keywords and schema names. If you are typing after schema name with point then you get list of tables in that schema. Also if you typing a name after point after a table name you will get a list of column names of this table. * autocomption now supports aliases in sql. If you write alias for a table in sql you will get a list of columns for an aliased table if you write down alias and point. * autocompletion now load keywords only in low case (otherwise there are so many keywords in a list of autocompletion that it is becomes uncomfortable) ### What type of PR is it? Improvement ### Todos * [ ] - sort names in the output of autocompletion * [ ] - list only schema names if we are typing a schema name (for example after keywork FROM) * [ ] - add description in autocompletion list for schema names - schema, for table names - table and so * [ ] - autocompletion must initialize on opening of interpeter (not only after execution of sql) * [ ] - update autocompletion schemas only after execute update sql, not after every sql ??? * [ ] - new option for postgresql interpreter: postgresql.completer.schema.filter. Filter schema names loaded into autocompletion (no more garbage schema names). ### What is the Jira issue? * Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/ * Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533] ### How should this be tested? Outline the steps to test the PR here. ### Screenshots (if appropriate) https://issues.apache.org/jira/secure/attachment/12845228/auto1.JPG https://issues.apache.org/jira/secure/attachment/12845229/auto2.JPG https://issues.apache.org/jira/secure/attachment/12845230/auto3.JPG ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Sotnichenko Sergey <[email protected]> Closes #1886 from sotnich/jdbc-1876 and squashes the following commits: 2db7ed8 [Sotnichenko Sergey] [ZEPPELIN-1876] add support for databases with only catalogs without schemas (like mysql) a048ef2 [Sotnichenko Sergey] [ZEPPELIN-1876] add support for databases with only catalogs without schemas (like mysql) f4b03df [Sotnichenko Sergey] [ZEPPELIN-1876] add support for databases with only catalogs without schemas (like mysql) 675c629 [Sotnichenko Sergey] [ZEPPELIN-1876] Adding licence header 9fac1d0 [Sotnichenko Sergey] [ZEPPELIN-1876] SQLException instead of Throwable 895c35a [Sotnichenko Sergey] [ZEPPELIN-1876] SQLException instead of Throwable 7d40166 [Sotnichenko Sergey] [ZEPPELIN-1876] improved comptetion with schema/table/column separation + sqlcompleter tests

### What is this PR for? https://github.com/apache/zeppelin/blob/master/kylin/src/main/java/org/apache/zeppelin/kylin/KylinInterpreter.java line 103, the getProject's input should be sql, not const string KYLIN_QUERY_PROJECT. Otherwise no project name could be retrieved. And also the SQL should exclude the project part. public HttpResponse prepareRequest(String sql) throws IOException { String kylinProject = getProject(KYLIN_QUERY_PROJECT); ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2078 ### How should this be tested? Has unit test. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Billy Liu <[email protected]> Closes #1986 from yiming187/ZEPPELIN-2078 and squashes the following commits: 1af8493 [Billy Liu] [ZEPPELIN-2078] bug fix in getProject in Kylin interpreter

### What is this PR for? The home screen is a notebook for use as homepage. It is the zeppelin.notebook.homescreen variable in `conf/zeppelin-site.xml` or `conf/zeppelin-env.sh`. But currently zeppelin.notebook.homescreen variable doesn't work on shiro until refreshing window. ### What type of PR is it? [Bug Fix] ### What is the Jira issue? * [ZEPPELIN-2072](https://issues.apache.org/jira/browse/ZEPPELIN-2072) ### How should this be tested? 1. Set zeppelin.notebook.homescreen variable in `conf/zeppelin-site.xml` or `conf/zeppelin-env.sh`. 2. Turn on shiro. 3. Start zeppelin-daemon. 4. Login your account and make sure whether homepage is changed or not. ### Screenshots (if appropriate) [ Before ] ![z_2072_b](https://cloud.githubusercontent.com/assets/8110458/22730568/dd60512e-ee2a-11e6-8e7e-f379b1eb0e58.gif) [ After ] ![z_2072_a](https://cloud.githubusercontent.com/assets/8110458/22730590/04482604-ee2b-11e6-9db6-d2656b252274.gif) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: soralee <[email protected]> Closes #1990 from soralee/ZEPPELIN-2072_homescreen_shiro and squashes the following commits: 35625c4 [soralee] fix homescreen on shiro

### What is this PR for? After appveyor ci is enabled for windows, PR build status shows red cross because of missing .appveyor. This PR add skeleton appveyor.yml. ### What type of PR is it? Feature ### Todos * [x] - Add .appveyor.yml ### What is the Jira issue? https://issues.apache.org/jira/browse/INFRA-14019 ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <[email protected]> Closes #2303 from Leemoonsoo/appveyor and squashes the following commits: 5d13259 [Lee moon soo] add basic .appveyor.yml

### What is this PR for? Minor code fixes for the livy package. The code fixes include : Fixing a typo in a classname - BaseLivyInterprereter to BaseLivyInterpreter Removing an unused variable in BaseLivyInterpreter Removing unused imports in a few classes ### What type of PR is it? Refactoring ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2465 ### How should this be tested? No need to test as there is no change in funcionality ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? NO Author: Benoy Antony <[email protected]> Closes #2297 from benoyantony/refactor and squashes the following commits: bd6d8ff [Benoy Antony] ZEPPELIN-2465 Minor code fixes for the livy package

### What is this PR for? After the merge of #2302 breaks authentication. ### What type of PR is it? [Hot Fix] ### Todos * [x] - Fix selenium logout issue (flaky test) ### Questions: * Does the licenses files need update? * Is there breaking changes for older versions? * Does this needs documentation? Author: Prabhjyot Singh <[email protected]> Closes #2304 from prabhjyotsingh/hotfix/ImproveSession and squashes the following commits: 94ba9e7 [Prabhjyot Singh] check if modal window is present, and then close it f993b59 [Prabhjyot Singh] improve session

### What is this PR for? The example in the doc would fail due to typo, straightforward fix. ### What type of PR is it? [Hot Fix] ### Todos * [ ] - Task ### How should this be tested? Outline the steps to test the PR here. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #2307 from zjffdu/doc_hotfix and squashes the following commits: 133fe72 [Jeff Zhang] [HOTFIX]. Minor doc fix for back-end-angular.md

Change-Id: Iaad10a69983036e84b766a22fbc32113b926b60d ### What is this PR for? With ZEPPELIN-2288 we restored the check of the Origin field for websocket requests. Unfortunately the current implementation will deny the request if the Origin HTTP header is empty, even if the zeppelin.server.allowed.origins is * This patch enables websocket requests without Origin in the HTTP header if the zeppelin.server.allowed.origins=*. This fixes the work behind a restrictive reverse proxy (or behind Apache Knox) ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2468 ### How should this be tested? It could be tested with curl as described in ZEPPELIN-2288, but I added additional unit test, so the change has been covered on unit test level. ### Screenshots (if appropriate) N/A ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? NO Author: Elek, Márton <[email protected]> Closes #2299 from elek/ZEPPELIN-2468 and squashes the following commits: d95bb41 [Elek, Márton] [ZEPPELIN-2468] Enable websocket without Origin if allowed.origins is *

### What is this PR for? Just noticed some grammar inconsistencies that I thought I could clean up while getting myself started on Zeppelin. Thanks for a great project! ### What type of PR is it? Documentation Author: Michael Ressler <[email protected]> Closes #2308 from mressler/master and squashes the following commits: aa39597 [Michael Ressler] Some grammar updates.

### What is this PR for? Move cursor to second line for new paragraphs ### What type of PR is it? Improvement ### Todos * [ ] - Task ### What is the Jira issue? ZEPPELIN-2459 ### How should this be tested? For a new paragraph he initial cursor position should be on the second line. ### Screenshots (if appropriate) **Before** <img width="324" alt="screen shot 2017-04-26 at 10 27 48 pm" src="https://cloud.githubusercontent.com/assets/2031306/25446579/9dcdeb2a-2acf-11e7-883d-e0ccf2ef834d.png"> **After** <img width="394" alt="screen shot 2017-04-26 at 10 27 33 pm" src="https://cloud.githubusercontent.com/assets/2031306/25446593/a66289da-2acf-11e7-8d8e-2029339f0133.png"> ### Questions: * Does the licenses files need update? n/a * Is there breaking changes for older versions? n/a * Does this needs documentation? n/a Author: Renjith Kamath <[email protected]> Closes #2291 from r-kamath/ZEPPELIN-2459 and squashes the following commits: 4c6525d [Renjith Kamath] ZEPPELIN-2459 Zeppelin Usability Improvement for new paragraph

…arameter ### What is this PR for? Security conscious organisations does not want to reveal the Application Server name and version to prevent Script-kiddies from finding the information easily when fingerprinting the Application. The exact version number can tell an Attacker if the current Application Server is patched for or vulnerable to certain publicly known CVE associated to it. ### What type of PR is it? [Improvement | Feature] ### What is the Jira issue? * [ZEPPELIN-2461](https://issues.apache.org/jira/browse/ZEPPELIN-2461) ### How should this be tested? Providing a value in zeppelin-site.xml will replace the actual Jetty server version found in HTTP Header with provided value. E.g. - edit zeppelin-site.xml and add a property `zeppelin.server.jetty.name` and with value say `TOMCAT` - restart the server - open the app in browser then observe the Response Headers for the key "Server" this should now reflect "TOMCAT" ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? N/A * Is there breaking changes for older versions? N/A * Does this needs documentation? N/A Author: krishna-pandey <[email protected]> Closes #2293 from krishna-pandey/ZEPPELIN-2461 and squashes the following commits: b071f7a [krishna-pandey] Set App Server name to config value

…without kerberos ### What is this PR for? Hive JDBC proxy user option should be available generically. ### What type of PR is it? [Improvement] ### What is the Jira issue? * [Zeppelin 2367](https://issues.apache.org/jira/browse/ZEPPELIN-2367) ### How should this be tested? Enable Shiro authentication and set `zeppelin.jdbc.auth.type` as `SIMPLE` in the interpreter setting, and observe the connection string for the Hive. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need an update? N/A * Is there breaking changes for older versions? N/A * Does this needs documentation? Yes Author: Prabhjyot Singh <[email protected]> Closes #2229 from prabhjyotsingh/ZEPPELIN-2367 and squashes the following commits: 84b5e55 [Prabhjyot Singh] add logger.warn for hive and impersonation 45c90a8 [Prabhjyot Singh] improve doc 9fee9d2 [Prabhjyot Singh] replace hive with generic method a348e96 [Prabhjyot Singh] revert "zeppelin.jdbc.auth.kerberos.proxy.enable" behaviour e2bdbb2 [Prabhjyot Singh] include e as inner exception c180f5c [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-2367 1802b45 [Prabhjyot Singh] remove hive string from logger 513987a [Prabhjyot Singh] apply genric logic to appendProxyUserToURL 3fa2b1e [Prabhjyot Singh] change name to appendProxyUserToURL a751674 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-2367 4c382ee [Prabhjyot Singh] log user details as well d51e770 [Prabhjyot Singh] add doc in jdbc.md 01b18b9 [Prabhjyot Singh] add doc (reverted from commit ee8a6b5) 40489c8 [Prabhjyot Singh] Merge remote-tracking branch 'origin/master' into ZEPPELIN-2367 ee8a6b5 [Prabhjyot Singh] add doc 8999d93 [Prabhjyot Singh] ZEPPELIN-2367: Hive JDBC proxy user option should be avail even without kerberos

### What is this PR for? `ng-change` can not capture event, so that means `$event.timeStamp` is undefined. angular/angular.js#6370 This cause AngularElem's onChange is only invoked once. This PR use another approach to update the angularObject. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2486 ### How should this be tested? Tested manually ### Screenshots (if appropriate) Before ![zeppelin_before](https://cloud.githubusercontent.com/assets/164491/25662178/227ec2ec-3046-11e7-9852-9e041d008698.gif) After ![zeppelin_after](https://cloud.githubusercontent.com/assets/164491/25662177/2108e898-3046-11e7-816f-7685480df83e.gif) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #2312 from zjffdu/ZEPPELIN-2486 and squashes the following commits: 4039d6b [Jeff Zhang] ZEPPELIN-2486. AngularElem's onChange is only invoked once

### What is this PR for? Requests interpreter editor mode information in revision mode. However, no information is needed in revision mode. Produces misleading logs with misleading requests to the server. ``` ERROR [2017-04-26 18:07:48,718] ({qtp1273765644-60} NotebookServer.java[onMessage]:383) - Can't handle message java.lang.NullPointerException ERROR [2017-04-26 18:07:49,081] ({qtp1273765644-109} NotebookServer.java[onMessage]:383) - Can't handle message java.lang.NullPointerException ERROR [2017-04-26 18:07:49,083] ({qtp1273765644-14} NotebookServer.java[onMessage]:383) - Can't handle message java.lang.NullPointerException ``` This log is requested as many as the number of paragraphs contained in the note. ### What type of PR is it? Bug fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2452 ### How should this be tested? 1. Create a git-repo based note. (Supported by default) 2. Modify the paragraph and commit. 3. Go to the committed revision. 4. Reload the page. 5. Check the server logs. ``` NotebookServer.java[onMessage]:383) - Can't handle message java.lang.NullPointerException ``` ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: CloverHearts <[email protected]> Closes #2289 from cloverhearts/fix/incorrectcall-geteditorsetting-on-revisionmode and squashes the following commits: 4b19fb0 [CloverHearts] fix indent 8c77f76 [CloverHearts] fix call geteditorsetting

### What is this PR for? In revision mode using git-repository, by default all paragraphs must remain unmodifiable. However, we are currently performing an incorrect update. ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2452 ### How should this be tested? 1. create note and modify to paragraph and version control commit on web 2. modify to paragraph and execute 3. open your 2 browser - one browser is HEAD, and other browser move to before commit 4. insert paragarph or modify and execute. check update ### Screenshots (if appropriate) #### problem ![incorrectrevisionupdate](https://cloud.githubusercontent.com/assets/10525473/25425492/1a0ebcce-2aa7-11e7-9a06-cfc84a1c1fe3.gif) #### fixed (this pr) ![correctrevisionupdate](https://cloud.githubusercontent.com/assets/10525473/25425498/1dce1bfc-2aa7-11e7-816c-c25a64963475.gif) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: CloverHearts <[email protected]> Closes #2288 from cloverhearts/fix/invalidupdateonrevision and squashes the following commits: 8fe72c9 [CloverHearts] add block event on revision mode 19e7479 [CloverHearts] block update paragraph event on revision mode

### What is this PR for? Adding user paragraph while creating paragraph makes default paragraph and user paragraph have same reference. It breaks personalized mode works well. ### What type of PR is it? [Bug Fix | Hot Fix] ### Todos * [x] - Remove `addUser` from `createParagraph` * [x] - Add test case for it ### What is the Jira issue? N/A ### How should this be tested? N/A ### Screenshots (if appropriate) N/A ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jongyoul Lee <[email protected]> Closes #2292 from jongyoul/hotfix/remove-wrong-adduser-and-add-testcase and squashes the following commits: 6ed521a [Jongyoul Lee] Removed adduser while creating note Added test to guarantee paragraph.getUserParagraph is different from paragraph itself

…graph ### What is this PR for? Prevent NullPointerException during check to determine whether a new paragraph needs to added. The fix is to switch order of null check and trim operation so that null check is performed before trim() ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2484 ### How should this be tested? This can be tested with a hive interpreter Create a note and add a paragraph with some query in it. Run all paragraphs. A new paragraph is automatically added. Run all paragraphs again. A NullPointerException is logged in the logs. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Benoy Antony <[email protected]> Closes #2310 from benoyantony/ZEPPELIN-2484 and squashes the following commits: 1cc2f8b [Benoy Antony] ZEPPELIN-2484 do a Null check before calling trim on paragraph's text

…menu ### What is this PR for? This is to fix incorrect appearance of tooltip inside version control menu ### What type of PR is it? Improvement ### Todos * [x] - fix/remove tooltip ### What is the Jira issue? [ZEPPELIN-2479](https://issues.apache.org/jira/browse/ZEPPELIN-2479) ### How should this be tested? go to `version control` menu in notebook action bar of note ### Screenshots (if appropriate) before: <img width="689" alt="screen shot 2017-05-02 at 3 32 23 pm" src="https://cloud.githubusercontent.com/assets/1642088/25606467/62f0901c-2f4c-11e7-95ef-c22f1df410be.png"> after: ![screen shot 2017-05-02 at 6 55 24 pm](https://cloud.githubusercontent.com/assets/1642088/25613839/aac216a4-2f6a-11e7-8cab-a77ddb48382b.png) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Khalid Huseynov <[email protected]> Closes #2306 from khalidhuseynov/fix/ZEPPELIN-2479 and squashes the following commits: ac4e651 [Khalid Huseynov] add css class

### What is this PR for? RemoteInterpreter is only used in the server side then zeppelin-interpreter doesn't have to include this class. Moving this class helps to reduce interpreter binary size and change RemoteInterpreter without adding more dependencies if we want ### What type of PR is it? [Refactoring] ### Todos * [x] - Move RemoteInterpreter and related files out of zeppelin-interpreter module ### What is the Jira issue? N/A ### How should this be tested? N/A ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jongyoul Lee <[email protected]> Closes #2320 from jongyoul/minor/move-remoteinterpreter-into-zengine and squashes the following commits: 8097991 [Jongyoul Lee] Removed author tag e1425df [Jongyoul Lee] Adopted DummyInterpreter 99c0932 [Jongyoul Lee] Made DummyInterpreter 5ac8dfb [Jongyoul Lee] Moved RemoteInterpreterServer to zeppelin-interpreter 0a881c1 [Jongyoul Lee] Removed unused package imported Removed unnecessary classes imported b7e0b94 [Jongyoul Lee] moved some files related remote interpreter and fix some minor things 7e87215 [Jongyoul Lee] move some files of remote packages from zeppelin-interpreter to zeppelin-zengine

### What is this PR for? Use single `Dockerfile` for each release since [apache infra uses tag pushes](https://issues.apache.org/jira/browse/INFRA-12781) to build an image. - https://issues.apache.org/jira/browse/INFRA-12781 After release process finishes, dockerhub will build using the pushed tag. ### What type of PR is it? [Improvement] ### What is the Jira issue? [ZEPPELIN-2492](https://issues.apache.org/jira/browse/ZEPPELIN-2492) ### How should this be tested? 1. `./dev/change_zeppelin_version.sh 0.8.0-SNAPSHOT 0.7.1` 2. Check that version is properly set: `vi scrtips/docker/zeppelin/bin/Dockerfile` 3. Build docker image `cd scripts/docker/zeppelin/bin; docker build -t zeppelin:0.7.1 ./` 4. Run the image: `docker run -p 8080:8080 --rm --name zeppelin zeppelin:0.7.1` ### Screenshots (if appropriate) NONE ### Questions: * Does the licenses files need update? - NO * Is there breaking changes for older versions? - NO * Does this needs documentation? - NO Author: 1ambda <[email protected]> Closes #2318 from 1ambda/ZEPPELIN-2492/use-single-dockerfile-for-each-tag and squashes the following commits: 483bec3 [1ambda] docs: Update README for Dockerfile 5826c8c [1ambda] fix: Use single dockerfile for tag push

### What is this PR for? Fixing ZEPPELIN-2382: A fresh install of Zeppelin 0.7.1 failed to start on a Windows due to the use of [getPosixFilePermissions](https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#getPosixFilePermissions(java.nio.file.Path,%20java.nio.file.LinkOption...)) This method fails on Windows with an UnsupportedOperationException - however, recovering (that is ignoring the exception) appears to have no negative effects. (For a discussion on this related to a different project see, e.g., GoogleCloudPlatform/cloud-code-intellij#937 ) ### What type of PR is it? [Bug Fix] ### Todos * [x] - Have the binary distribution checked by the reporter of [ZEPPELIN-2382] ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2382 ### How should this be tested? Check if clean distribution runs on Windows. Important: Ensure that HADOOP_HOME points to a folder containing bin/winutils.exe since this pull request does not include a fix for https://issues.apache.org/jira/browse/ZEPPELIN-2438 (which is part of a separate pull request). ### Screenshots (if appropriate) N/A ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Christian Fries <[email protected]> Closes #2305 from cfries/windows-compatiblity-interpreter-binding and squashes the following commits: e3b5738 [Christian Fries] Added graceful exception handling if posix permission is not available 096fed6 [Christian Fries] Merge branch 'master' into apache/master 3635ed8 [Christian Fries] [ZEPPELIN-2375]: Avoid modification of CLASSPATH variable.

### What is this PR for? Second travis job didn't work correctly. It suppose to test all core unit tests plus integration test (except for spark related test) but it was doing nothing. It was because of the exclamation mark in **-Dtest** property: ``` -Dtest='!ZeppelinSparkClusterTest,!org.apache.zeppelin.spark.*' ``` which is not supported by maven-surefire-plugin of version 2.17 (which is used in Zeppelin). Exclamation mark is supported started from 2.19 but still does not work properly. I've added plugin **exclude** configuratin instead of **-Dtest** property. After travis job was restored I had found that not all core tests were working properly. I have excluded them from the travis job for now and created jira issues: https://issues.apache.org/jira/browse/ZEPPELIN-2469 https://issues.apache.org/jira/browse/ZEPPELIN-2470 https://issues.apache.org/jira/browse/ZEPPELIN-2471 https://issues.apache.org/jira/browse/ZEPPELIN-2473 ### What type of PR is it? [Bug Fix] ### What is the Jira issue? [ZEPPELIN-2467](https://issues.apache.org/jira/browse/ZEPPELIN-2467) ### Questions: * Does the licenses files need update? **no** * Is there breaking changes for older versions? **no** * Does this needs documentation? **no** Author: Alexander Shoshin <[email protected]> Closes #2300 from AlexanderShoshin/ZEPPELIN-2467 and squashes the following commits: 78771e9 [Alexander Shoshin] made a comment about excluded tests dfa332f [Alexander Shoshin] changed -Dtest flag to -Dtests.to.exclude and excluded unstable tests 0448c4a [Alexander Shoshin] added ability to exclude some unit tests using command line

### What is this PR for? To allow helium modules to import CSS that contains references to png or jpg images. ### What type of PR is it? Improvement ### Todos * [ ] - Task ### What is the Jira issue? * [ZEPPELIN-2268] ### How should this be tested? The volume-leaflet helium module [https://github.com/volumeint/helium-volume-leaflet] imports the Leaflet css file. That css file references a couple of png files. Before this change, the helium module would fail to enable, after the change it succeeds. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No. * Is there breaking changes for older versions? No. * Does this needs documentation? No. Author: Thomas Grant <[email protected]> Closes #2313 from volumeint/master and squashes the following commits: 672b789 [Thomas Grant] Merge branch 'master' of https://github.com/apache/zeppelin c736dbe [Thomas Grant] Merge branch 'master' of https://github.com/apache/zeppelin 19500d5 [Thomas Grant] ZEPPELIN-2268. Adding png and jpg support for helium module imports.

@AhyoungRyu

### What is this PR for? Fix JDBC doc after #2229. ### What type of PR is it? [Bug Fix] ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? * Is there breaking changes for older versions? * Does this needs documentation? Author: Prabhjyot Singh <[email protected]> Closes #2314 from prabhjyotsingh/minor/jdbc-doc-zeppelin-2367 and squashes the following commits: e54a3a2 [Prabhjyot Singh] @AhyoungRyu review comments 0f396ac [Prabhjyot Singh] fix doc for zeppelin-2367

…into branch-0.6

hayssams · 2017-05-10T13:11:52Z

Replaced by PR #2333

zjffdu reviewed Oct 11, 2016

View reviewed changes

aspen01 mentioned this pull request Nov 4, 2016

Using HDFS to backup and restore notebook #1600

Closed

1 task

Leemoonsoo and others added 8 commits February 10, 2017 10:18

Leemoonsoo and others added 28 commits May 1, 2017 07:50

Merge branch 'master' of https://github.com/apache/incubator-zeppelin …

2858a09

…into branch-0.6

no message

dcd3c68

no message

4b1f7c4

Get This WEBHDFS from PR1600

1ae68c6

Implement HDFS Storage using WebHDFS Client

94f56ad

default Scala version

4643379

no message

8058e77

hayssams closed this May 10, 2017

		addJarInDir "${ZEPPELIN_HOME}/zeppelin-web/target/lib"


		CLASSPATH+=":${ZEPPELIN_CLASSPATH}"

Notebook: HDFS as a backend storage (Read & Write Mode) #1479

Notebook: HDFS as a backend storage (Read & Write Mode) #1479

Uh oh!

Conversation

hayssams commented Oct 2, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

Uh oh!

Leemoonsoo commented Oct 5, 2016

Uh oh!

hayssams commented Oct 5, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hayssams commented Oct 17, 2016

Uh oh!

jongyoul commented Oct 17, 2016

Uh oh!

hayssams commented Oct 17, 2016

Uh oh!

jongyoul commented Oct 18, 2016

Uh oh!

zjffdu commented Oct 19, 2016

Uh oh!

hayssams commented Oct 23, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jongyoul commented Oct 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aspen01 commented Nov 1, 2016

Uh oh!

aspen01 commented Nov 4, 2016

Uh oh!

hayssams commented May 10, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

hayssams commented Oct 2, 2016 •

edited

Loading

hayssams commented Oct 23, 2016 •

edited

Loading

jongyoul commented Oct 24, 2016 •

edited

Loading