ZEPPELIN-3051. Support Interpreter Process Recovery #2668

zjffdu · 2017-11-15T06:12:10Z

What is this PR for?

This PR is for the purpose of recover running interpreter process when zeppelin server is restarted. This would be useful when restarting zeppelin without interrupt current running interpreter processes, should be useful when admin do maintenance or upgrading.

Interface RecoveryStorage is used for storing the information of running interpreter process.
Currently it only has one implementation FileSystemRecoveryStorage, other implementation could be done later (such as zookeeper based). InterpreterLauncher is the component where to recover the running interpreter process.

Test:

RecoveryTest.java
FileSystemRecoveryStorageTest.java

Design Doc:

https://docs.google.com/document/d/1Plm3Hd40aGdNaXmjdsoY4ek3f-gTijTMGMkNjAZN39Y/edit?usp=sharing

What type of PR is it?

[Feature]

Todos

- Task

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-3051

How should this be tested?

Unit test & Integration Test is added. Also manually verified.

Screenshots (if appropriate)

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

zjffdu · 2017-11-15T07:10:42Z

@Leemoonsoo @jongyoul @felixcheung @prabhjyotsingh Could you help review it ? Thanks

felixcheung

overall looks right to me.
it seems a bit nasty to not reset interpretersettingmanager though, is it going to cause issue? or staleness of data? is there a way to force it to reset/close/restart?

felixcheung · 2017-11-15T18:13:00Z

...elin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/ShellScriptLauncher.java

+            LOGGER.info("Recover InterpreterProcess: " + client.getHost() + ":" + client.getPort());
+            return (RemoteInterpreterRunningProcess) client;
+          } else {
+            LOGGER.warn("Can not recovery interpreter process: " + client.getHost() + ":"


"Cannot recover ..."

so when it cannot recover, should it delete the recovery data or should it try again the next time?

For FileSystemRecoveryStorage, it would refresh all the recovery data when interpreter process is started or stopped. So when recovery is failed, new interpreter process is created and recovery data will be refreshed, the old recovery data will be deleted as well.

felixcheung · 2017-11-15T18:16:10Z

...engine/src/main/java/org/apache/zeppelin/interpreter/recovery/FileSystemRecoveryStorage.java

+    for (ManagedInterpreterGroup interpreterGroup : interpreterSetting.getAllInterpreterGroups()) {
+      RemoteInterpreterProcess interpreterProcess = interpreterGroup.getInterpreterProcess();
+      if (interpreterProcess != null) {
+        recoveryContent.add(interpreterGroup.getId() + "\t" + interpreterProcess.getHost() + ":" +


what's the \t for?

how much of this kind of custom format is needed? should we look into a well known format instead?
(I guess this is a CSV/TSV file then)

yes, it is tsv file, \t is used as delimited. The format is interpreterGroupId \t host:port

felixcheung · 2017-11-15T18:17:16Z

...engine/src/main/java/org/apache/zeppelin/interpreter/recovery/FileSystemRecoveryStorage.java

+    LOGGER.debug("Updating recovery data for interpreterSetting: " + interpreterSettingName);
+    LOGGER.debug("Recovery Data: " + StringUtils.join(recoveryContent, "\n"));
+    Path recoveryFile = new Path(recoveryDir, interpreterSettingName + ".recovery");
+    fs.writeFile(StringUtils.join(recoveryContent, "\n"), recoveryFile, true);


really nit: System.lineSeparator()

felixcheung · 2017-11-15T18:20:07Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/FileSystemStorage.java

+        }
+        if (fs.isFile(dir)) {
+          throw new IOException("{} is file instead of directory, please remove it or " +
+              "specify another directory");


missing a parameter? {} is there but there's nothing passed

zjffdu · 2017-11-16T00:24:46Z

Closing InterpreterSettingManager is for shutting down interpreter processes, otherwise we would get dangling processes. There's no other side effect or resource leakage if we don't close InterpreterSettingManager except keeping interpreter process alive.

felixcheung · 2017-11-16T05:21:05Z

I see. I think I see close/restarting the process as a last-ditch effort to get around issue with the JVM; is there such a way to force restart?

zjffdu · 2017-11-20T00:43:19Z

is there such a way to force restart?

Sorry, @felixcheung , I didn't get you, can you explain it more ?

felixcheung · 2017-11-20T08:51:08Z

Is there a way to restart the interpreter process?

zjffdu · 2017-11-20T09:17:35Z

Currently, there's no api for restarting interpreter process. Restarting interpreter setting just terminate interpreter process, and zeppelin will restart interpreter process when user run paragraph.

zjffdu · 2017-11-28T02:50:35Z

@felixcheung @Leemoonsoo @jongyoul Any more comments ?

Leemoonsoo · 2017-11-28T18:47:54Z

Doesn't https://github.com/apache/zeppelin/blob/master/bin/zeppelin-daemon.sh#L222 need to aware of recovery? Otherwise, it terminates all the interpreters on zeppelin-deamon.sh stop

zjffdu · 2017-11-28T23:34:02Z

@Leemoonsoo Recovery feature is controlled zeppelin.recovery.enabled by in zeppelin-site.xml, if it is enabled, interpreter process won't be killed.
see https://github.com/apache/zeppelin/pull/2668/files#diff-3477d79f3012453f20c3727ec60338e9R230

Leemoonsoo · 2017-11-29T00:51:58Z

Correct me if i'm wrong. https://github.com/apache/zeppelin/pull/2668/files#diff-3477d79f3012453f20c3727ec60338e9R230 skips interpreter close when zeppelin.recovery.enabled is true. However, zeppelin-deamon.sh script still kill each individual interpreter process using PID file created by interpreter.sh on restart (and stop). Isn't it?

zjffdu · 2017-11-29T01:25:33Z

oops, didn't notice zeppelin-daemon.sh will also kill interpreter process, but according my experiment, it seems zeppelin-daemon.sh won't kill interpreter process, let me double check that.

zjffdu · 2017-11-29T03:25:21Z

@Leemoonsoo Just find out that zeppelin didn't capture the correct pid of interpreter process, it capture the parent process of interpreter process (interpreter.sh) instead of the jvm process. I fix that in the new commit, but even that it still won't work when using isolated mode. All the interpreter process of the same group would share the same interpreter process pid file, which mean we would still have process leakage. Of course we can create one pid file for each process, but I am afraid it would take a long time to shutdown zeppelin, because as my experience, each call of wait_for_zeppelin_to_die in zeppelin-daemon.sh would sleep 3 seconds.

Not sure the history of killing interpreter process in zeppelin-daemon.sh, it might be safe to remove that, because ZeppelinServer will terminate all the interpreter process in the shutdown hook.

felixcheung

what if I want to clean shutdown everything with zeppelin-daemon and isRecoveryEnabled is true? does ZeppelinServer terminate the interpreter processes then?

zjffdu · 2017-11-29T05:54:29Z

There's no way to shutdown everything when zeppelin.recovery.enabled is true. Are you concerning orphan process ?

felixcheung · 2017-11-29T16:24:06Z

Yes!

zjffdu · 2017-12-04T12:38:52Z

@felixcheung I added bin/stop-interpreter.sh so that user can force kill interpreter process even when recovery is enabled.

felixcheung · 2017-12-05T07:47:34Z

bin/stop-interpreter.sh

+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Run Zeppelin


Run -> Stop? Kill?

felixcheung · 2017-12-05T07:50:04Z

docs/usage/interpreter/overview.md

+In such cases, interpreter process recovery is necessary. Starting from 0.8.0, user can enable interpreter process recovering via setting `zeppelin.recovery.storage.class` as 
+`org.apache.zeppelin.interpreter.recovery.FileSystemRecoveryStorage` or other implementations if available in future, by default it is `org.apache.zeppelin.interpreter.recovery.NullRecoveryStorage`
+ which means recovery is not enabled. Enable recover means shutting down zeppelin would not terminating interpreter process,
+and when zeppelin is restarted, it would try to reconnect to the existing running interpreter process. If you want to kill all the interpreter process after terminating zeppelin even when recovery is enabled, you can run `bin/stop-interpreter.sh` 


process -> processes?

also upper case Zeppelin? (multiple)

felixcheung · 2017-12-05T07:54:23Z

zeppelin-server/notebook/python.recovery

@@ -0,0 +1 @@
+2CZA1DVUG:shared_process	192.168.3.2:55410


should this be in the commit? and recovery.crc?

Will delete it

felixcheung · 2017-12-05T08:00:31Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/recovery/StopInterpreter.java

+    if (restoredClients != null) {
+      for (InterpreterClient client : restoredClients.values()) {
+        LOGGER.info("Stop Interpreter Process: " + client.getHost() + ":" + client.getPort());
+        client.stop();


try catch around this?

There's no exception throw from stop method, so no try catch needed here.

felixcheung · 2017-12-05T08:00:57Z

...ne/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterRunningProcess.java

+            }
+          });
+        } catch (Exception e) {
+          logger.warn("ignore the exception when shutting down");


log the exception?

zjffdu · 2017-12-13T10:51:22Z

Any more comments ? I will merge if no more comments.

This PR is for the purpose of recover running interpreter process when zeppelin server is restarted. This would be useful when restarting zeppelin without interrupt current running interpreter processes, should be useful when admin do maintenance or upgrading. Interface `RecoveryStorage` is used for storing the information of running interpreter process. Currently it only has one implementation `FileSystemRecoveryStorage`, other implementation could be done later (such as zookeeper based). `InterpreterLauncher` is the component where to recover the running interpreter process. Test: * RecoveryTest.java * FileSystemRecoveryStorageTest.java Design Doc: https://docs.google.com/document/d/1Plm3Hd40aGdNaXmjdsoY4ek3f-gTijTMGMkNjAZN39Y/edit?usp=sharing [Feature] * [ ] - Task * https://issues.apache.org/jira/browse/ZEPPELIN-3051 Unit test & Integration Test is added. Also manually verified. * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes apache#2668 from zjffdu/ZEPPELIN-3051 and squashes the following commits: a4c9b9c [Jeff Zhang] address comments 575b7b9 [Jeff Zhang] fix the pid of interpreter process id 02b118f [Jeff Zhang] address comments da7cbb9 [Jeff Zhang] ZEPPELIN-3051. Support Interpreter Process Recovery

### What is this PR for? This PR is for the purpose of recover running interpreter process when zeppelin server is restarted. This would be useful when restarting zeppelin without interrupt current running interpreter processes, should be useful when admin do maintenance or upgrading. Interface `RecoveryStorage` is used for storing the information of running interpreter process. Currently it only has one implementation `FileSystemRecoveryStorage`, other implementation could be done later (such as zookeeper based). `InterpreterLauncher` is the component where to recover the running interpreter process. Test: * RecoveryTest.java * FileSystemRecoveryStorageTest.java Design Doc: https://docs.google.com/document/d/1Plm3Hd40aGdNaXmjdsoY4ek3f-gTijTMGMkNjAZN39Y/edit?usp=sharing ### What type of PR is it? [Feature] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3051 ### How should this be tested? Unit test & Integration Test is added. Also manually verified. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes apache#2668 from zjffdu/ZEPPELIN-3051 and squashes the following commits: a4c9b9c [Jeff Zhang] address comments 575b7b9 [Jeff Zhang] fix the pid of interpreter process id 02b118f [Jeff Zhang] address comments da7cbb9 [Jeff Zhang] ZEPPELIN-3051. Support Interpreter Process Recovery

* [ZEPPELIN-3062] Removes ctrl+s default behavior from notebook ### What is this PR for? As a programmer I habitually press CTRL + S out of fear for my sanity. Other web text editing tools (google docs, jupyter notebooks) ignore the base functionality of CTRL + S of saving the HTML page, and do nothing instead. This is a small change that will makes the user experience of Zeppelin notebooks much much better. ### What type of PR is it? Improvement ### Todos ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3062 ### How should this be tested? * I tested manually: 1. Ran `yarn run dev` 2. Went to `localhost:9000` 3. Created a new note, now anytime a user presses 'CTRL+S' in the scope of the note nothing happens, and the user can continue coding uninterrupted. ### Screenshots (if appropriate) ### Questions: Author: Tess <[email protected]> Closes apache#2677 from tessbianchi/ZEPPELIN-3062 and squashes the following commits: db80533 [Tess] Removes ctrl+s default behavior from notebook * ZEPPELIN-3072: Zeppelin UI becomes slow/unresponsive if there are too many notebooks Zeppelin UI becomes slow/unresponsive if there are too many notebooks Have attached a notebook directory in [JIRA](https://issues.apache.org/jira/secure/attachment/12898650/notebook.zip) with 500+ notebooks, now with these notebooks, every time user goes to homepage Zeppelin UI becomes unresponsive for few seconds. [Bug Fix | Improvement] * [x] - Fix search box * [x] - Order of notebook * [ZEPPELIN-3072](https://issues.apache.org/jira/browse/ZEPPELIN-3072) Create 500+ notebook, or import it from [JIRA](https://issues.apache.org/jira/secure/attachment/12898650/notebook.zip), now observe UI it becomes slow/laggy while homepage is rendering. Before: ![before](https://user-images.githubusercontent.com/674497/33070354-c36acdfa-cedd-11e7-81f9-ff0b526622f3.gif) After: ![after](https://user-images.githubusercontent.com/674497/33070353-c3317988-cedd-11e7-9431-fcf596928c3b.gif) * Does the licenses files need update? N/A * Is there breaking changes for older versions? N/A * Does this needs documentation? N/A Author: prabhjyotsingh <[email protected]> Closes apache#2683 from prabhjyotsingh/ZEPPELIN-3072 and squashes the following commits: 06b8ef8 [prabhjyotsingh] add license file 483a3ff [prabhjyotsingh] navbar dropdown bug eb506ba [prabhjyotsingh] fix test 00ec295 [prabhjyotsingh] fix sort and serch b4cbba8 [prabhjyotsingh] ZEPPELIN-3072: Zeppelin UI becomes slow/unresponsive if there are too many notebooks Change-Id: Ibc157312b726b9704cab088192a39e942d8da43d * Add missing build dependencies. Add `apt-get install` lines for R dependencies that are needed to build successfully. ### What is this PR for? This pull request adds installation instructions for building which relate to missing dependencies resulting in failing to build (if using R). ### What type of PR is it? [Documentation] Author: Jens Grassel <[email protected]> Closes apache#2674 from jan0sch/docs-missing-build-deps and squashes the following commits: aa581c1 [Jens Grassel] Add missing build dependencies. * ZEPPELIN-3037 Configure Http Request Header Size Limit for Jetty ### What is this PR for? In some deployment scenarios it is necessary to increase jetty.request.header.size, which default value is 8192. This will reduce the chance of HTTP Error 413 Request entity too large. There should be a mechanism to configure this setting. ### What type of PR is it? [Feature] ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3037 ### How should this be tested? * There is an integration test (automated unit test included) for testing this feature. * To test manually, after increasing setting, make any http request to zeppelin with a request header bigger than 8K , you should not get an HTTP Error 413 Request entity too large. ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? Yes. Author: byamthev <[email protected]> Closes apache#2663 from byamthev/zeppelin3037 and squashes the following commits: 8ff2620 [byamthev] [ZEPPELIN-3037] Configure Http Request Header Size Limit for Jetty * ZEPPELIN-2984: allow enter from username ### What is this PR for? Allow an "Enter" keypress in the username field during login ### What type of PR is it? Improvement ### Todos * N/A ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-2984 ### How should this be tested? * Pull up login screen. Fill in username and password. Click in username field. Press enter. We would expect the login handler to be called. * Pull up login screen. Fill in username and password. Press enter (while still in password field). We would expect the login handler to be called just as before. * Alternatively, use a password manager (e.g. LastPass). Save the password. Pull up the login screen. Press Enter. We would expect the login handler to be called. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Gabe Fernando <[email protected]> Closes apache#2616 from gef756/dev-login-enter and squashes the following commits: 301d398 [Gabe Fernando] MRG: Merge remote-tracking branch 'origin/master' into dev-login-enter 2a1865a [Gabe Fernando] (ZEPPELIN-2984) ENH: allow enter from username * [ZEPPELIN-3042] updating jgit to support post-commit hooks ### What is this PR for? Git functionality in Zeppelin is provided by `jgit`, but it's fixed to an old version. A newer version of said library supports new things, primarily post-commit hooks. ### What type of PR is it? Improvement ### Todos * [ x ] - CI ([two failed, but due to yarn](https://travis-ci.org/kokes/zeppelin)) ### What is the Jira issue? [ZEPPELIN-3042](https://issues.apache.org/jira/browse/ZEPPELIN-3042) ### Questions: * Does the licenses files need update? -- Yes, only because the license file notes the exact version of included libraries. `jgit` hasn't changed, license-wise, so it's only a matter of updating the version information * Is there breaking changes for older versions? -- nope * Does this needs documentation? -- possibly, to let people know they get use post-commit hooks now? Author: Ondrej Kokes <[email protected]> Closes apache#2658 from kokes/jgit-update and squashes the following commits: a2cde16 [Ondrej Kokes] [ZEPPELIN-3042] newer jgit supports post-commit hooks a4fc816 [Ondrej Kokes] [ZEPPELIN-3042] newer jgit supports post-commit hooks * [ZEPPELIN-3076]Chart field is also draggable and sortable in the 'keys', 'groups' and 'values' ### What is this PR for? Current the `keys`, `groups` and `values` is only droppable. It is a little inconvenient if I want to drag it to other place or sort it. This feature let `keys`, `groups` and `values` not only `droppable`, but also `draggable` and `sorttable`. ### What type of PR is it? [Feature] ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3076 [ZEPPELIN-3076] ### How should this be tested? I have Add the Screenshots for the test. Just as the Screenshots showing, all the element in `keys`, `groups` and `values` is draggable and sorttable. ![untitled project4](https://user-images.githubusercontent.com/5969176/33252922-ed1ab0b4-d37b-11e7-8a5c-b3dbb6765d18.gif) ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? NO Author: liguohui <[email protected]> Closes apache#2686 from liguohuicmss/draggable-field-keys-groups-values2 and squashes the following commits: 08b86b8 [liguohui] add a new line ebb8436 [liguohui] Chart field is also draggable and sortable in the 'keys', 'groups' and 'values' * [hotfix] Downgrade JGit from 4.9.0 to 4.5.4 ### What is this PR for? The version of JGit was updated to 4.9.0 at apache#2658. However, this version does not support Java 7 and Travis CI test always fails now: https://travis-ci.org/kjmrknsn/zeppelin/builds/310104872 To fix this issue, downgrade JGit from 4.9.0 to 4.5.4 which is the latest JGit version which supports Java 7: https://projects.eclipse.org/projects/technology.jgit. I confirmed that Travis CI test was passed with JGit 4.5.4: https://travis-ci.org/kjmrknsn/zeppelin/builds/310107611 Now, all PRs to Zeppelin don't pass Travis CI tests, so it's preferable that this PR is merged soon. ### What type of PR is it? [Bug Fix] ### Todos ### What is the Jira issue? ### How should this be tested? I confirmed that Travis CI test passed with JGit 4.5.4: https://travis-ci.org/kjmrknsn/zeppelin/builds/310107611 ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? Yes. zeppelin-distribution/src/bin_license/LICENSE was updated at this PR. * Is there breaking changes for older versions? No. * Does this needs documentation? No. Author: Keiji Yoshida <[email protected]> Closes apache#2693 from kjmrknsn/ci-test-against-jgit-4.5.4 and squashes the following commits: 27b08c9 [Keiji Yoshida] CI Test against JGit 4.5.4 * [Zeppelin-2571] & [Zeppelin-465] Run paragraphs: from first/current to current/last ### What is this PR for? This pr add the ability to run all paragraphs from the first to the current and from the current to the last. This makes it easier to update the data if changes are made in one of the related paragraphs. ### What type of PR is it? Feature ### What is the Jira issue? [ZEPPELIN-2571](https://issues.apache.org/jira/browse/ZEPPELIN-2571) - (Add a "Run to here" option on paragraphs) [ZEPPELIN-465](https://issues.apache.org/jira/browse/ZEPPELIN-465) - (Capability to run all cells below the current cell) ### How should this be tested? 1. Click on the "Run: from first to this" or "Run: from this to last" in the dropdown menu. Paragraphs from the first/current to the current/last will be started in order. 2. Press the key combination: CTRL + SHIFT + ENTER. A window will appear with the choice of the desired action. ![capture2](https://user-images.githubusercontent.com/25951039/33269914-58eff828-d393-11e7-9ebf-6437ec11c8f2.PNG) Choose one of two actions. The selected action will be performed. ### Screenshots (if appropriate) ![capture1_edit](https://user-images.githubusercontent.com/25951039/33269915-5951b19e-d393-11e7-831b-42d4523908ae.png) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: tinkoff-dwh <[email protected]> Closes apache#2688 from tinkoff-dwh/ZEPPELIN-2571&465 and squashes the following commits: 304c196 [tinkoff-dwh] [ZEPPELIN-2517]&[ZEPPELIN-465] fix shortcut description 3d57b30 [tinkoff-dwh] [ZEPPELIN-2517]&[ZEPPELIN-465] save paragraph's focus 70f130c [tinkoff-dwh] [Zeppelin-2517]&[Zeppelin-465] some text change 8ca1df2 [tinkoff-dwh] [Zeppelin-2517]&[Zeppelin-465] line reduction a7a4158 [tinkoff-dwh] [ZEPPELIN-2517]&[ZEPPELIN-465] logic change 9fbcdb6 [tinkoff-dwh] [ZEPPELIN-2517]&[ZEPPELIN-465] add keyboard shortcut bfc3891 [tinkoff-dwh] [ZEPPELIN-2517]&[ZEPPELIN-465] Run paragraphs from 1st to this. From this to last * [ZEPPELIN-3075]Fix unqiue algo for the web side in pivot.js file. ### What is this PR for? unique() algorithm is not correct in pivot.js file. If the input is `[2, 3, 3, 3, 4, 5]` and the output will be `[2, 3, 3, 4, 5]`. The number `3` is still duplicated. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3075 [ZEPPELIN-3075] ### How should this be tested? This is very easy and no need test. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: liguohui <[email protected]> Closes apache#2685 from liguohuicmss/pivot-unqiue-algo and squashes the following commits: 2063175 [liguohui] delete the multi empty line ea582d9 [liguohui] delete some spaces at the end of the line 79c763a [liguohui] add a empty line 99cf93d [liguohui] Revert "Chart field is also draggable and sortable in the 'keys', 'groups' and 'values'" fdde39f [liguohui] add unit test for unique algo in pivot.js f996747 [liguohui] Chart field is also draggable and sortable in the 'keys', 'groups' and 'values' 943e80a [liguohui] Fix unqiue algo for the web side in pivot.js file. * [FIX] fix autocomplete ### What is this PR for? After refactoring of Interpreter autocomplete (from server side) not works without first Run of interpreter. This PR fix it. ### What type of PR is it? [Fix] ### How should this be tested? * Create new Note (JDBC interpreter), try to use autocomplete (schema, tables) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: tinkoff-dwh <[email protected]> Closes apache#2691 from tinkoff-dwh/fix_autocomplete and squashes the following commits: e9bad01 [tinkoff-dwh] remove trim from completion 141dff5 [tinkoff-dwh] [FIX] fix autocomplete * [HOTFIX]: Fix IPythonInterpreter unit test ### What is this PR for? This is for hotfix of `IPythonInterpreter` unit test failure. Just specify the version of ipython in `install_external_dependencies.sh`, otherwise latest ipython version will be installed, and the behavior may change. ### What type of PR is it? [Hot Fix] ### Todos * [ ] - Task ### What is the Jira issue? * ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes apache#2703 from zjffdu/ipython_version and squashes the following commits: b4c7b42 [Jeff Zhang] HotFix: Fix IPythonInterpreter unit test * [ZEPPELIN-3014] NPE bug fix and Error message enhancement with Kylin Interpreter ### What is this PR for? A few sentences describing the overall goals of the pull request's commits. First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html ### What type of PR is it? Bug Fix ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3014 ### How should this be tested? * Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration * Use existing unit tests in kylin module. ### Screenshots (if appropriate) #### before: NPE when result set is empty ![image](https://user-images.githubusercontent.com/18542573/32154048-f1b8ba58-bcfb-11e7-98cc-98cdf484f2d5.png) #### after: no NPE when result set is empty, just an empty table ![image](https://user-images.githubusercontent.com/18542573/32154069-110215d0-bcfc-11e7-87e9-cc049001f1c7.png) #### before: when query fails, only error code is returned, no error message ![image](https://user-images.githubusercontent.com/18542573/32154088-29651938-bcfc-11e7-9e66-cd2cfccba054.png) #### after: when query fails, both error code and error message are displayed to users ![image](https://user-images.githubusercontent.com/18542573/32154096-3d3ab01c-bcfc-11e7-8cf3-d710d96b8c5a.png) ### Questions: * Does the licenses files need update? No. * Is there breaking changes for older versions? No. * Does this needs documentation? No. Author: Liu <[email protected]> Closes apache#2645 from jinxliu/kylin-intp-new and squashes the following commits: d5692bf [Liu] refactor 85b6424 [Liu] add test for empty result set 4596470 [Liu] ZEPPELIN-3014: NPE bug fix and Error message enhancement with Kylin Interpreter * ZEPPELIN-3085 Introduce generic ConfInterpreter for more fine-grained control of interpreter setting ### What is this PR for? Zeppelin's interpreter setting is shared by all the users and notes, if you want to have different setting you have to create new interpreter, e.g. you can create `spark_jar1` for running spark with dependency jar1 and `spark_jar2` for running spark with dependency jar2. This approach works, but not so convenient. `ConfInterpreter` can provide more fine-grained control on interpreter setting and more flexibility. `ConfInterpreter` is a generic interpreter that could be used by any interpreters. The input format should be property file format. In the first paragraph, we use ConfInterpreter to make custom configuration of spark interpreter (set app name, yarn-client mode & add spark-csv dependencies). Then you can run the second paragraph which use spark-csv. ![conf_interpreter](https://user-images.githubusercontent.com/164491/33419465-74a3fae8-d5e5-11e7-8b25-76407804d979.png) It can be used to make custom setting for any interpreter. `ConfInterpreter` would run in the zeppelin server side, it would update the interpreter properties before you launch the interpreter process, so it needs to run before interpreter process launched. And when interpreter process is launched is determined by interpreter mode setting. So users needs to understand the interpreter mode setting of zeppelin and be aware when interpreter process is launched. E.g. If we set spark interpreter setting as isolated per note. Under this setting, each note will launch one interpreter process. In this scenario, user need to put `ConfInterpreter` as the first paragraph as the above example. Otherwise the customized setting can not be applied (The paragraph using ConfInterpreter will report ERROR). ### What type of PR is it? [Feature | Documentation] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3085 ### How should this be tested? * Unit test, System test is added, also manually verified it. ### Screenshots (if appropriate) ![conf_interpreter](https://user-images.githubusercontent.com/164491/33419465-74a3fae8-d5e5-11e7-8b25-76407804d979.png) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes apache#2692 from zjffdu/ZEPPELIN-3085 and squashes the following commits: 87ce20f [Jeff Zhang] ZEPPELIN-3085. Introduce generic ConfInterpreter for more fine-grained control of interpreter setting * ZEPPELIN-3051. Support Interpreter Process Recovery ### What is this PR for? This PR is for the purpose of recover running interpreter process when zeppelin server is restarted. This would be useful when restarting zeppelin without interrupt current running interpreter processes, should be useful when admin do maintenance or upgrading. Interface `RecoveryStorage` is used for storing the information of running interpreter process. Currently it only has one implementation `FileSystemRecoveryStorage`, other implementation could be done later (such as zookeeper based). `InterpreterLauncher` is the component where to recover the running interpreter process. Test: * RecoveryTest.java * FileSystemRecoveryStorageTest.java Design Doc: https://docs.google.com/document/d/1Plm3Hd40aGdNaXmjdsoY4ek3f-gTijTMGMkNjAZN39Y/edit?usp=sharing ### What type of PR is it? [Feature] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3051 ### How should this be tested? Unit test & Integration Test is added. Also manually verified. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes apache#2668 from zjffdu/ZEPPELIN-3051 and squashes the following commits: a4c9b9c [Jeff Zhang] address comments 575b7b9 [Jeff Zhang] fix the pid of interpreter process id 02b118f [Jeff Zhang] address comments da7cbb9 [Jeff Zhang] ZEPPELIN-3051. Support Interpreter Process Recovery * [MINOR] Fix notebook title bar margin ### What is this PR for? Notebook title bar location is little bit shifted to left. This minor fix notebook title bar margin. See screenshots below. ### What type of PR is it? Bug Fix ### Todos * [x] - fix margin ### Screenshots (if appropriate) Before ![image](https://user-images.githubusercontent.com/1540981/33742650-3469c2fe-db5e-11e7-8aa6-936d0a28de3a.png) After ![image](https://user-images.githubusercontent.com/1540981/33742640-250c3dc8-db5e-11e7-8af3-e8a9d7105963.png) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Lee moon soo <[email protected]> Closes apache#2699 from Leemoonsoo/minor_noteaction_margin and squashes the following commits: 703f458 [Lee moon soo] fix noteAction magin * [ZEPPELIN-3091] Correct aggregation functionality in charts ### What is this PR for? The aggregation functions interpret NaN columns as 1 which leads to incorrect output being shown in charts. This PR fixes this by correcting the sum, min, max and average aggregation method. ### What type of PR is it? Bug Fix ### Todos ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3091 ### How should this be tested? * Update the data field of a paragraph results with %table to contain null values, e.g. `"data": "age\tvalue\n19\t4\n20\t3\n21\t7\n22\t9\n23\t20\n24\t24\n25\t44\n26\t77\n27\t94\n28\t103\n29\t97\n20\t5\n20\tnull\n"` The "null" values should be ignored for sum, min, max but included for count (and hence average). ### Screenshots (if appropriate) ![correct_sum](https://user-images.githubusercontent.com/6438072/33609178-14e05988-d9ed-11e7-9f1b-99e0141c5153.png) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Naman Mishra <[email protected]> Closes apache#2696 from namanmishra91/ZEPPELIN-3091 and squashes the following commits: d8a57c2 [Naman Mishra] Add test 38ad39c [Naman Mishra] Merge branch 'master' into ZEPPELIN-3091 568ae3f [Naman Mishra] Correct aggregation functionality in charts * [ZEPPELIN-3101] updated network label, added link to network display in index.md ### What is this PR for? The docs index must show the reference to the network visualization as for the the other types ### What type of PR is it? [Improvement] ### Todos * [x] - Add missing link ### What is the Jira issue? [ZEPPELIN-3101](https://issues.apache.org/jira/projects/ZEPPELIN/issues/ZEPPELIN-3101) ### How should this be tested? 1. cd `docs/` 2. build: `bundle exec jekyll build --safe` 3. check the link is present ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Andrea Santurbano <[email protected]> Closes apache#2702 from conker84/graph-docs and squashes the following commits: 58d58cc [Andrea Santurbano] updated network label, added link to network display in index.md

### What is this PR for? This PR is for the purpose of recover running interpreter process when zeppelin server is restarted. This would be useful when restarting zeppelin without interrupt current running interpreter processes, should be useful when admin do maintenance or upgrading. Interface `RecoveryStorage` is used for storing the information of running interpreter process. Currently it only has one implementation `FileSystemRecoveryStorage`, other implementation could be done later (such as zookeeper based). `InterpreterLauncher` is the component where to recover the running interpreter process. Test: * RecoveryTest.java * FileSystemRecoveryStorageTest.java Design Doc: https://docs.google.com/document/d/1Plm3Hd40aGdNaXmjdsoY4ek3f-gTijTMGMkNjAZN39Y/edit?usp=sharing ### What type of PR is it? [Feature] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3051 ### How should this be tested? Unit test & Integration Test is added. Also manually verified. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes apache#2668 from zjffdu/ZEPPELIN-3051 and squashes the following commits: a4c9b9c [Jeff Zhang] address comments 575b7b9 [Jeff Zhang] fix the pid of interpreter process id 02b118f [Jeff Zhang] address comments da7cbb9 [Jeff Zhang] ZEPPELIN-3051. Support Interpreter Process Recovery

felixcheung reviewed Nov 15, 2017

View reviewed changes

zjffdu force-pushed the ZEPPELIN-3051 branch from b76fcf0 to 696f61e Compare November 16, 2017 01:07

zjffdu force-pushed the ZEPPELIN-3051 branch from 696f61e to de69c67 Compare November 17, 2017 03:35

zjffdu closed this Nov 28, 2017

zjffdu reopened this Nov 28, 2017

felixcheung reviewed Nov 29, 2017

View reviewed changes

zjffdu force-pushed the ZEPPELIN-3051 branch from 6098ad4 to a7431ad Compare December 4, 2017 12:38

zjffdu force-pushed the ZEPPELIN-3051 branch 5 times, most recently from 3532676 to 3523cac Compare December 5, 2017 03:09

felixcheung reviewed Dec 5, 2017

View reviewed changes

zjffdu force-pushed the ZEPPELIN-3051 branch from 9092af2 to 138e7d2 Compare December 14, 2017 01:08

zjffdu force-pushed the ZEPPELIN-3051 branch 2 times, most recently from 77f8b6a to 32839d7 Compare December 14, 2017 04:56

zjffdu added 4 commits December 14, 2017 14:22

ZEPPELIN-3051. Support Interpreter Process Recovery

da7cbb9

address comments

02b118f

fix the pid of interpreter process id

575b7b9

address comments

a4c9b9c

zjffdu force-pushed the ZEPPELIN-3051 branch from 32839d7 to a4c9b9c Compare December 14, 2017 06:27

asfgit closed this in 4c8f20a Dec 14, 2017

		@@ -0,0 +1 @@
		2CZA1DVUG:shared_process 192.168.3.2:55410 No newline at end of file

ZEPPELIN-3051. Support Interpreter Process Recovery #2668

ZEPPELIN-3051. Support Interpreter Process Recovery #2668

Uh oh!

Conversation

zjffdu commented Nov 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

Uh oh!

zjffdu commented Nov 15, 2017

Uh oh!

felixcheung left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjffdu commented Nov 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felixcheung commented Nov 16, 2017

Uh oh!

zjffdu commented Nov 20, 2017

Uh oh!

felixcheung commented Nov 20, 2017 via email

Uh oh!

zjffdu commented Nov 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zjffdu commented Nov 28, 2017

Uh oh!

Leemoonsoo commented Nov 28, 2017

Uh oh!

zjffdu commented Nov 28, 2017

Uh oh!

Leemoonsoo commented Nov 29, 2017

Uh oh!

zjffdu commented Nov 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zjffdu commented Nov 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felixcheung left a comment

Choose a reason for hiding this comment

Uh oh!

zjffdu commented Nov 29, 2017

Uh oh!

felixcheung commented Nov 29, 2017 via email

Uh oh!

zjffdu commented Dec 4, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjffdu commented Nov 15, 2017 •

edited

Loading

zjffdu commented Nov 16, 2017 •

edited

Loading

zjffdu commented Nov 20, 2017 •

edited

Loading

zjffdu commented Nov 29, 2017 •

edited

Loading

zjffdu commented Nov 29, 2017 •

edited

Loading