[ZEPPELIN-1046] bin/install-interpreter.sh for netinst package #1042

Leemoonsoo · 2016-06-18T03:37:29Z

What is this PR for?

Implementation of bin/install-interpreter.sh for netinst package which suggested in the discussion.

Some usages will be

# download all interpreters provided by Apache Zeppelin project
bin/install-interpreter.sh --all

# download an interpreter with name (for example markdown interpreter)
bin/install-interpreter.sh --name md

# download an (3rd party) interpreter with specific maven artifact name
bin/install-interpreter.sh --name md -t org.apache.zeppelin:zeppelin-markdown:0.6.0-SNAPSHOT

If it looks fine, i'll continue the work (refactor code, and add test)

What type of PR is it?

Feature

Todos

- working implementation
- refactor
- add test

What is the Jira issue?

Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
Put link here, and add [ZEPPELIN-Jira number] in PR title, eg. [ZEPPELIN-533]

How should this be tested?

Outline the steps to test the PR here.

Screenshots (if appropriate)

Questions:

Does the licenses files need update?
Is there breaking changes for older versions?
Does this needs documentation?

prabhjyotsingh · 2016-06-18T08:19:07Z

One more feature request for this, with downloading an interpreter, creation/insertion of relevant entry in conf/interpreter.json

felixcheung · 2016-06-20T08:49:40Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/install/InstallInterpreter.java

+    "md",     "zeppelin-markdown",   "Markdown interpreter",
+    "sh",     "zeppelin-shell",      "Allows shell command",
+    "jdbc",   "zeppelin-jdbc",
+    "Generic JDBC interpreter for hive, phoenix, postsgresql, mysql, etc",


do you think we would need to update this list in the future?
is there a way to have a text lookup table with a list of short name instead of hardcoding it in .java file?

bzz · 2016-06-21T04:31:47Z

Both suggestions are nice addition, and there is also a refactoring on TODOs so I guess it's WIP now.

👍 for making it part of the release, as discussed [email protected]

Leemoonsoo · 2016-06-21T21:40:20Z

@prabhjyotsingh I think that's good idea.

If this script runs before start Zeppelin, right after download and unarchive, Zeppelin will auto create conf/interpreter.json with all installed interpreters.

If this script runs after conf/interpreter.json created, it'll not update conf/interpreter.json. And i think it make sense while conf/interpreter.json might customized by user at this time, and user may not want to touch/add something into it

I think we'll have better way to install interpreter once Zeppelin have dynamic interpreter loading capability (https://issues.apache.org/jira/browse/ZEPPELIN-598)

@felixcheung I have removed hardcoded list and updated to read list from text file conf/interpreter-list.

minahlee · 2016-06-21T22:09:56Z

zeppelin-interpreter/src/main/java/org/apache/zeppelin/dep/DependencyResolver.java

    return libs;
  }

+  public List<File> load(String artifact, File destPath) throws IOException, RepositoryException {


Seems like this method with same signature already exists above.

AhyoungRyu · 2016-06-22T05:03:53Z

conf/interpreter-list

+alluxio         org.apache.zeppelin.zeppelin-alluxio:0.6.0              Alluxio interpreter
+angular         org.apache.zeppelin.zeppelin-angular:0.6.0              HTML and AngularJS view rendering
+cassandra       org.apache.zeppelin.zeppelin-cassandra:0.6.0            Cassandra interpreter
+elasticsearch   org.apache.zeppelin.zeppelin-cassandra:0.6.0            Elasticsearch interpreter


@Leemoonsoo Do you mean org.apache.zeppelin.zeppelin-elasticsearch:0.6.0 ?

Leemoonsoo · 2016-06-22T22:20:51Z

It's ready to review and merge.
Tried updated create_release.sh script. zeppelin-0.6.0-bin-all.tgz is about 498M while zeppelin-0.6.0-netinst.tgz is about 208MB.

minahlee · 2016-06-23T05:01:44Z

@Leemoonsoo Could you rebase and resolve conflicts?

…EPPELIN_HOME

minahlee · 2016-06-23T06:21:45Z

docs/manual/interpreterinstallation.md

+  <tr>
+    <td>cassandra</td>
+    <td>org.apache.zeppelin.zeppelin-cassandra:0.6.0</td>
+    <td>HTML and AngularJS view rendering</td>


Seems like description needs to be updated

minahlee · 2016-06-23T07:15:51Z

conf/interpreter-list

+#
+# [name]  [maven artifact]  [description]
+
+alluxio         org.apache.zeppelin.zeppelin-alluxio:0.6.0              Alluxio interpreter


After org.apache.zeppelin, : instead of .. Same for next three lines

Leemoonsoo · 2016-06-23T16:03:36Z

@minahlee addressed your comments

minahlee · 2016-06-23T20:06:04Z

docs/manual/interpreterinstallation.md

+#### Install 3rd party interpreters
+
+```
+./bin/install-interpreter.sh --name interpreter1 --repository groupId1:artifact1:version1


--repository -> --artifact

@minahlee

### What is this PR for? Implementation of bin/install-interpreter.sh for netinst package which suggested in the [discussion](http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Ask-opinion-regarding-0-6-0-release-package-tp3298p3314.html). Some usages will be Unknown option --all Install md(org.apache.zeppelin:zeppelin-markdown:0.6.0-SNAPSHOT ) to /Users/mina/Develop/zeppelin/interpreter/md ... If it looks fine, i'll continue the work (refactor code, and add test) ### What type of PR is it? Feature ### Todos * [x] - working implementation * [x] - refactor * [x] - add test ### What is the Jira issue? * Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/ * Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533] ### How should this be tested? Outline the steps to test the PR here. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? * Is there breaking changes for older versions? * Does this needs documentation? Author: Lee moon soo <[email protected]> Author: AhyoungRyu <[email protected]> Closes apache#1042 from Leemoonsoo/netinst and squashes the following commits: 049bc89 [Lee moon soo] Update docs 7307c67 [Lee moon soo] Merge remote-tracking branch 'AhyoungRyu/netinst-docs' into netinst 7e749ad [Lee moon soo] Address mina's comment 0eedd2a [AhyoungRyu] Address @minahlee feedback 13f2d04 [Lee moon soo] generate netinst package 03c664e [AhyoungRyu] Add a new line 5d0a971 [AhyoungRyu] Revert install.md to latest version 13899fb [AhyoungRyu] Reorganize interpreter installation docs 4c1f029 [Lee moon soo] Proxy support 9079580 [Lee moon soo] fix artifact name 1077296 [Lee moon soo] update test aebca17 [Lee moon soo] Add docs d547551 [Lee moon soo] Remove test entries 6ee06b8 [Lee moon soo] Make DependencyResolver in zeppelin-interpreter module not aware of ZEPPELIN_HOME 7b1b36a [Lee moon soo] update usage 49f0568 [Lee moon soo] Add conf/interpreter-list 1b558fd [Lee moon soo] update some text ec7d152 [Lee moon soo] add tip 2c81a3f [Lee moon soo] update 78a7c52 [Lee moon soo] Refactor and add test 47f5706 [Lee moon soo] Install multiple interpreters at once 38e2556 [Lee moon soo] Initial implementation of install-interpreter.sh

minahlee · 2016-06-23T21:37:24Z

Thanks for the effort @Leemoonsoo, @AhyoungRyu I've tested this branch and it works like a charm! I left few more comments, once they are addressed it's good to merge :)

Leemoonsoo · 2016-06-23T22:03:39Z

Thanks @minahlee for the review. Addressed your comments!

minahlee · 2016-06-24T01:18:12Z

Merging if there is no more discussion

@minahlee

### What is this PR for? Implementation of bin/install-interpreter.sh for netinst package which suggested in the [discussion](http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Ask-opinion-regarding-0-6-0-release-package-tp3298p3314.html). Some usages will be ``` # download all interpreters provided by Apache Zeppelin project bin/install-interpreter.sh --all # download an interpreter with name (for example markdown interpreter) bin/install-interpreter.sh --name md # download an (3rd party) interpreter with specific maven artifact name bin/install-interpreter.sh --name md -t org.apache.zeppelin:zeppelin-markdown:0.6.0-SNAPSHOT ``` If it looks fine, i'll continue the work (refactor code, and add test) ### What type of PR is it? Feature ### Todos * [x] - working implementation * [x] - refactor * [x] - add test ### What is the Jira issue? * Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/ * Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533] ### How should this be tested? Outline the steps to test the PR here. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? * Is there breaking changes for older versions? * Does this needs documentation? Author: Lee moon soo <[email protected]> Author: AhyoungRyu <[email protected]> Closes #1042 from Leemoonsoo/netinst and squashes the following commits: f81d16e [Lee moon soo] address mina's comment 049bc89 [Lee moon soo] Update docs 7307c67 [Lee moon soo] Merge remote-tracking branch 'AhyoungRyu/netinst-docs' into netinst 7e749ad [Lee moon soo] Address mina's comment 0eedd2a [AhyoungRyu] Address @minahlee feedback 13f2d04 [Lee moon soo] generate netinst package 03c664e [AhyoungRyu] Add a new line 5d0a971 [AhyoungRyu] Revert install.md to latest version 13899fb [AhyoungRyu] Reorganize interpreter installation docs 4c1f029 [Lee moon soo] Proxy support 9079580 [Lee moon soo] fix artifact name 1077296 [Lee moon soo] update test aebca17 [Lee moon soo] Add docs d547551 [Lee moon soo] Remove test entries 6ee06b8 [Lee moon soo] Make DependencyResolver in zeppelin-interpreter module not aware of ZEPPELIN_HOME 7b1b36a [Lee moon soo] update usage 49f0568 [Lee moon soo] Add conf/interpreter-list 1b558fd [Lee moon soo] update some text ec7d152 [Lee moon soo] add tip 2c81a3f [Lee moon soo] update 78a7c52 [Lee moon soo] Refactor and add test 47f5706 [Lee moon soo] Install multiple interpreters at once 38e2556 [Lee moon soo] Initial implementation of install-interpreter.sh (cherry picked from commit 4efb39f) Signed-off-by: Mina Lee <[email protected]>

@felixcheung

…docs website ### What is this PR for? I added auto TOC(Table of Contents) generator for Zeppelin documentation website. TOC can help people looking through whole contents at a glance and finding what they want quickly. I just added `<div id="toc"></div>` to the each documentation header. [`toc`](https://github.com/apache/zeppelin/compare/master...AhyoungRyu:ZEPPELIN-1018?expand=1#diff-85af09fb498a5667ea455391533f945dR3) recognize `<h2>` & `<h3>` as a title in the docs and it automatically generate TOC. So I set a rule for this work. (I'll write this rule on `docs/CONTRIBUTING.md` or [docs/howtocontributewebsite](https://zeppelin.apache.org/docs/0.6.0-SNAPSHOT/development/howtocontributewebsite.html)). ``` # Level-1 Heading <- Use only for the main title of the page ## Level-2 Heading <- Start with this one ### Level-3 heading <- Only use this one for child of Level-2 toc only recognize Level-2 & Level-3 ``` Please see the below attached screenshot image. ### What type of PR is it? Improvement & Documentation ### Todos * [x] - Add TOC generator * [x] - Apply TOC(`<div id="toc"></div>`) to every documentation and reorganize each headers(apply the above rule) * [x] - Fix some broken code block in several docs * [x] - Apply TOC to `r.md` (Currently R docs has some duplicated info since [this one](d5e87fb) and [this one](7d6cc7e) ) * [x] - Apply TOC to `install.md` after #1010 merged * [x] - Apply TOC to `interpreterinstallation.md` after #1042 merged ### What is the Jira issue? [ZEPPELIN-1018](https://issues.apache.org/jira/browse/ZEPPELIN-1018) ### How should this be tested? 1. Apply this patch and build `docs/` with [this guide](https://github.com/apache/zeppelin/tree/master/docs#build-documentation) 2. Visit some docs page. Then you can see TOC in the header of page. ### Screenshots (if appropriate) - Automatically generated TOC in Spark interpreter docs page <img width="831" alt="screen shot 2016-06-16 at 9 37 18 pm" src="https://cloud.githubusercontent.com/assets/10060731/16140902/945b9c7a-340a-11e6-91f3-b6174738bed0.png"> ### Questions: * Does the licenses files need update? No. Actually I used [jekyll-table-of-contents#copyright](https://github.com/ghiculescu/jekyll-table-of-contents#copyright). But I don't need to add a license for this :) * Is there breaking changes for older versions? No * Does this needs documentation? Maybe Author: AhyoungRyu <[email protected]> Closes #1031 from AhyoungRyu/ZEPPELIN-1018 and squashes the following commits: e66397b [AhyoungRyu] Apply TOC to interpreterinstallation.md 009579b [AhyoungRyu] Add more info to 'What is the next?' in install.md 04cf501 [AhyoungRyu] Revert 'where to start' section b7cbe5f [AhyoungRyu] Fix typo cf0911c [AhyoungRyu] Rename license file 388f35a [AhyoungRyu] Add jekyll-table-of-contents license info 6394c70 [AhyoungRyu] Fix image path in python.md d00e4b1 [AhyoungRyu] Move interpreter/screenshot/ -> asset/../img/docs-img/ 3ffb383 [AhyoungRyu] Remove duplicated info in r.md & apply toc a03ca99 [AhyoungRyu] Exclude toc.js from pom.xml 3fae7df [AhyoungRyu] Apply auto generated toc to install.md d114a9d [AhyoungRyu] Address @felixcheung feedback 6a788fe [AhyoungRyu] Resize TOC tab indent 6760c00 [AhyoungRyu] Apply auto TOC to all of docs under docs/storage/ fbde57f [AhyoungRyu] Apply auto TOC to all of docs under docs/quickstart/ db76eb6 [AhyoungRyu] Apply auto TOC to all of docs under docs/install/ f35db47 [AhyoungRyu] Apply auto TOC to all of docs under docs/displaysystem/ b05365f [AhyoungRyu] Apply auto TOC to all of docs under docs/rest-api/ 163691c [AhyoungRyu] Apply auto TOC to all of docs under docs/manual/ bef398e [AhyoungRyu] Apply auto TOC to all of docs under docs/development/ 9c5f76b [AhyoungRyu] Apply auto TOC to all of docs under docs/interpreter/ 587d4ba [AhyoungRyu] Apply auto TOC to all of docs under docs/security/ 1f10b97 [AhyoungRyu] Change toc configuration 78dca9e [AhyoungRyu] Add toc.js for auto generating TOC

@felixcheung

…docs website ### What is this PR for? I added auto TOC(Table of Contents) generator for Zeppelin documentation website. TOC can help people looking through whole contents at a glance and finding what they want quickly. I just added `<div id="toc"></div>` to the each documentation header. [`toc`](https://github.com/apache/zeppelin/compare/master...AhyoungRyu:ZEPPELIN-1018?expand=1#diff-85af09fb498a5667ea455391533f945dR3) recognize `<h2>` & `<h3>` as a title in the docs and it automatically generate TOC. So I set a rule for this work. (I'll write this rule on `docs/CONTRIBUTING.md` or [docs/howtocontributewebsite](https://zeppelin.apache.org/docs/0.6.0-SNAPSHOT/development/howtocontributewebsite.html)). ``` # Level-1 Heading <- Use only for the main title of the page ## Level-2 Heading <- Start with this one ### Level-3 heading <- Only use this one for child of Level-2 toc only recognize Level-2 & Level-3 ``` Please see the below attached screenshot image. ### What type of PR is it? Improvement & Documentation ### Todos * [x] - Add TOC generator * [x] - Apply TOC(`<div id="toc"></div>`) to every documentation and reorganize each headers(apply the above rule) * [x] - Fix some broken code block in several docs * [x] - Apply TOC to `r.md` (Currently R docs has some duplicated info since [this one](d5e87fb) and [this one](7d6cc7e) ) * [x] - Apply TOC to `install.md` after #1010 merged * [x] - Apply TOC to `interpreterinstallation.md` after #1042 merged ### What is the Jira issue? [ZEPPELIN-1018](https://issues.apache.org/jira/browse/ZEPPELIN-1018) ### How should this be tested? 1. Apply this patch and build `docs/` with [this guide](https://github.com/apache/zeppelin/tree/master/docs#build-documentation) 2. Visit some docs page. Then you can see TOC in the header of page. ### Screenshots (if appropriate) - Automatically generated TOC in Spark interpreter docs page <img width="831" alt="screen shot 2016-06-16 at 9 37 18 pm" src="https://cloud.githubusercontent.com/assets/10060731/16140902/945b9c7a-340a-11e6-91f3-b6174738bed0.png"> ### Questions: * Does the licenses files need update? No. Actually I used [jekyll-table-of-contents#copyright](https://github.com/ghiculescu/jekyll-table-of-contents#copyright). But I don't need to add a license for this :) * Is there breaking changes for older versions? No * Does this needs documentation? Maybe Author: AhyoungRyu <[email protected]> Closes #1031 from AhyoungRyu/ZEPPELIN-1018 and squashes the following commits: e66397b [AhyoungRyu] Apply TOC to interpreterinstallation.md 009579b [AhyoungRyu] Add more info to 'What is the next?' in install.md 04cf501 [AhyoungRyu] Revert 'where to start' section b7cbe5f [AhyoungRyu] Fix typo cf0911c [AhyoungRyu] Rename license file 388f35a [AhyoungRyu] Add jekyll-table-of-contents license info 6394c70 [AhyoungRyu] Fix image path in python.md d00e4b1 [AhyoungRyu] Move interpreter/screenshot/ -> asset/../img/docs-img/ 3ffb383 [AhyoungRyu] Remove duplicated info in r.md & apply toc a03ca99 [AhyoungRyu] Exclude toc.js from pom.xml 3fae7df [AhyoungRyu] Apply auto generated toc to install.md d114a9d [AhyoungRyu] Address @felixcheung feedback 6a788fe [AhyoungRyu] Resize TOC tab indent 6760c00 [AhyoungRyu] Apply auto TOC to all of docs under docs/storage/ fbde57f [AhyoungRyu] Apply auto TOC to all of docs under docs/quickstart/ db76eb6 [AhyoungRyu] Apply auto TOC to all of docs under docs/install/ f35db47 [AhyoungRyu] Apply auto TOC to all of docs under docs/displaysystem/ b05365f [AhyoungRyu] Apply auto TOC to all of docs under docs/rest-api/ 163691c [AhyoungRyu] Apply auto TOC to all of docs under docs/manual/ bef398e [AhyoungRyu] Apply auto TOC to all of docs under docs/development/ 9c5f76b [AhyoungRyu] Apply auto TOC to all of docs under docs/interpreter/ 587d4ba [AhyoungRyu] Apply auto TOC to all of docs under docs/security/ 1f10b97 [AhyoungRyu] Change toc configuration 78dca9e [AhyoungRyu] Add toc.js for auto generating TOC (cherry picked from commit 5975125) Signed-off-by: Mina Lee <[email protected]>

minahlee mentioned this pull request Jun 20, 2016

[ZEPPELIN-986] Create publish release script #994

Closed

3 tasks

felixcheung reviewed Jun 20, 2016
View reviewed changes

minahlee reviewed Jun 21, 2016
View reviewed changes

Leemoonsoo changed the title ~~[WIP] bin/install-interpreter.sh for netinst package~~ [ZEPPELIN-1046] bin/install-interpreter.sh for netinst package Jun 22, 2016

Leemoonsoo force-pushed the netinst branch from ccf1465 to b68530f Compare June 22, 2016 04:42

AhyoungRyu reviewed Jun 22, 2016
View reviewed changes

Leemoonsoo and others added 17 commits June 22, 2016 22:24

Initial implementation of install-interpreter.sh

38e2556

Install multiple interpreters at once

47f5706

Refactor and add test

78a7c52

update

2c81a3f

add tip

ec7d152

update some text

1b558fd

Add conf/interpreter-list

49f0568

update usage

7b1b36a

Make DependencyResolver in zeppelin-interpreter module not aware of Z…

6ee06b8

…EPPELIN_HOME

Remove test entries

d547551

Add docs

aebca17

update test

1077296

fix artifact name

9079580

Proxy support

4c1f029

Reorganize interpreter installation docs

13899fb

Revert install.md to latest version

5d0a971

Add a new line

03c664e

Leemoonsoo force-pushed the netinst branch from 2ad2d24 to 13f2d04 Compare June 23, 2016 05:54

generate netinst package

13f2d04

minahlee reviewed Jun 23, 2016
View reviewed changes

Address @minahlee feedback

0eedd2a

minahlee reviewed Jun 23, 2016
View reviewed changes

Leemoonsoo added 3 commits June 23, 2016 09:04

Address mina's comment

7e749ad

Merge remote-tracking branch 'AhyoungRyu/netinst-docs' into netinst

7307c67

Update docs

049bc89

minahlee reviewed Jun 23, 2016
View reviewed changes

address mina's comment

f81d16e

asfgit closed this in 4efb39f Jun 24, 2016

AhyoungRyu mentioned this pull request Jun 24, 2016

[ZEPPELIN-1018] Apply auto "Table of Contents" generator to Zeppelin docs website #1031

Closed

6 tasks

[ZEPPELIN-1046] bin/install-interpreter.sh for netinst package #1042

[ZEPPELIN-1046] bin/install-interpreter.sh for netinst package #1042

Uh oh!

Conversation

Leemoonsoo commented Jun 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

Uh oh!

prabhjyotsingh commented Jun 18, 2016

Uh oh!

felixcheung Jun 20, 2016

Choose a reason for hiding this comment

Uh oh!

bzz commented Jun 21, 2016

Uh oh!

Leemoonsoo commented Jun 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

minahlee Jun 21, 2016

Choose a reason for hiding this comment

Uh oh!

AhyoungRyu Jun 22, 2016

Choose a reason for hiding this comment

Uh oh!

Leemoonsoo commented Jun 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

minahlee commented Jun 23, 2016

Uh oh!

minahlee Jun 23, 2016

Choose a reason for hiding this comment

Uh oh!

minahlee Jun 23, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Leemoonsoo commented Jun 23, 2016

Uh oh!

minahlee Jun 23, 2016

Choose a reason for hiding this comment

Uh oh!

minahlee commented Jun 23, 2016

Uh oh!

Leemoonsoo commented Jun 23, 2016

Uh oh!

minahlee commented Jun 24, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Leemoonsoo commented Jun 18, 2016 •

edited

Loading

Leemoonsoo commented Jun 21, 2016 •

edited

Loading

Leemoonsoo commented Jun 22, 2016 •

edited

Loading

minahlee Jun 23, 2016 •

edited

Loading