Zeppelin-531 ] Job management #921

cloverhearts · 2016-05-27T11:48:25Z

What is this PR for?

Zeppelin job management.
Zeppelin job monitoring and control page for Notebooks and Paragraphs.

You can check the cron notebook.
You can easily understand the entire current 'job.
It can easily run the Notebook necessary.

What type of PR is it?

Feature

Todo

What is the Jira issue?

ZEPPELIN-531

How should this be tested?

click to job menu.

filter - write to notebook name and selected interpreter type.
check to error. - Interpreter and problem notification about the execution impossible 'Notebook.
For example, you run the R tutorial 'in the absence of R Interpreter'.
run always top option
The check box on the right exists.
Try clicking while the notebook is running.
execute to notebook and stop
Click on the notebook name and paragraph to see if that move.

and using zeppelin. (make notebook, and run paragraph Misc...)

Screenshots (if appropriate)

(Animated Gif, size 3.7MB)

Questions:

Does the licenses files need update? no
Is there breaking changes for older versions? no
Does this needs documentation? yes

Conflicts: zeppelin-web/test/karma.conf.js

Conflicts: zeppelin-web/.jshintrc

minahlee · 2016-05-28T03:03:34Z

I cannot see any job when I visit job tab

cloverhearts · 2016-05-28T08:46:08Z

@minahlee
Is this some notebooks ?
Can you give me a log ?

minahlee · 2016-05-28T08:55:23Z

This is what I get when I direct to Job menu.

 INFO [2016-05-28 01:51:45,603] ({main} Server.java[doStart]:379) - Started @6372ms
 INFO [2016-05-28 01:51:45,603] ({main} ZeppelinServer.java[main]:117) - Done, zeppelin server started
 INFO [2016-05-28 01:51:55,589] ({qtp1439541970-36} AbstractValidatingSessionManager.java[enableSessionValidation]:230) - Enabling session validation scheduler...
 WARN [2016-05-28 01:51:55,641] ({qtp1439541970-36} SecurityRestApi.java[ticket]:79) - {"status":"OK","message":"","body":{"principal":"anonymous","ticket":"anonymous","roles":"[]"}}
 INFO [2016-05-28 01:51:55,790] ({qtp1439541970-15} NotebookServer.java[onOpen]:95) - New connection from 0:0:0:0:0:0:0:1 : 54959
ERROR [2016-05-28 01:51:59,300] ({qtp1439541970-11} NotebookServer.java[onMessage]:212) - Can't handle message
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at java.util.LinkedList.checkElementIndex(LinkedList.java:553)
    at java.util.LinkedList.get(LinkedList.java:474)
    at org.apache.zeppelin.socket.NotebookServer.generateNotebooksJobInfo(NotebookServer.java:434)
    at org.apache.zeppelin.socket.NotebookServer.unicastNotebookJobInfo(NotebookServer.java:493)
    at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:141)
    at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:56)
    at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
    at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
    at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
    at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
    at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
    at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
    at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:745)
ERROR [2016-05-28 01:52:00,301] ({qtp1439541970-18} NotebookServer.java[onMessage]:212) - Can't handle message

cloverhearts · 2016-05-30T02:55:27Z

Modifications completed.
Add an exception to the no interpreter notebook.
@minahlee I will wait for the reviews.
Thanks to a good point.

skanda83 · 2016-05-30T03:02:19Z

Where is the job history stored?

minahlee · 2016-05-30T03:13:57Z

@cloverhearts Seems like large number of notebooks/paragraphs slows down loading job manage page. Any idea to solve this issue?

cloverhearts · 2016-05-30T03:38:05Z

@minahlee
Okay, I will optimize to job manage page.
How many are you have a Notebook?

cloverhearts · 2016-05-30T03:39:54Z

@skanda83
Sorry. This feature does not save the history.
Provide information only about the current.
It will be offered in the future.

astroshim · 2016-05-30T04:16:55Z

@cloverhearts Really nice job. but I got an follwing error when i click a paragraph in a job

cloverhearts · 2016-05-31T02:25:59Z

@astroshim Thank you for feedback.
I'll check it.

prabhjyotsingh · 2016-05-31T10:59:29Z

@cloverhearts have tested this, works fine with me. Problem that I found for now is the list is case sensitive sort, where as every where else its case insensitive sort.

cloverhearts · 2016-05-31T12:25:33Z

@prabhjyotsingh Thank you for your feedback. While I can solve some problems with the current optimization. Your opinion is right. I will solve the problem. Thanks again.

have a good day.

cloverhearts · 2016-06-04T18:22:25Z

@Leemoonsoo
Sorry, my mistake.
Delete and rebase.
Thank you.

felixcheung · 2016-06-05T19:59:17Z

in the screenshots, what does "interpreter is not set" mean?

cloverhearts · 2016-06-06T01:30:27Z

@felixcheung

If you do not select an interpreter of the notebook.

BASIC Interpreter notebook is displayed in the job manager.
(Eg - notebook name - spark)

bzz · 2016-06-06T04:15:45Z

zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRestApi.java

+    } catch (InterpreterException intpException) {
+      return new JsonResponse<>(Status.INTERNAL_SERVER_ERROR, intpException.getMessage()).build();
+    } catch (Exception e) {
+      return new JsonResponse<>(Status.INTERNAL_SERVER_ERROR, e.getMessage()).build();


What is the difference in handling this 2 exceptional conditions?

bzz · 2016-06-06T04:48:16Z

Great feature @cloverhearts ! I'v left some comments above.

BTW I think we might want to treat websocket protocol as public API and track modifications to it, so this PR adds 3 new messages:

LIST_NOTEBOOK_JOBS,     // [c-s] get notebook job management infomations
LIST_UPDATE_NOTEBOOK_JOBS, // [c-s] get job management informations for until unixtime
UNSUBSCRIBE_JOBMANAGER

\cc @corneadoug for review of the frontend code

cloverhearts · 2016-06-06T09:07:26Z

@bzz Thank you! i will check it.

cloverhearts · 2016-06-07T06:22:15Z

The 'PR' is very huge.
This makes it difficult to review.
I divided the 'PR', will allow for easy review.
this 'pr' is closed.

### What is this PR for? Job management basic backend. was divided into smaller " PR ". - #921 Receive the basic data of Job manager api and the backend have been implemented. ### What type of PR is it? Feature ### Todos - [x] - Basic backend for job manager. - [x] - Basic backend api ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-963 ### How should this be tested? #### step 1. First, calling the rest api as follows receives the data of "job" of the whole. ```shell curl -H "Content-Type: application/json" -X GET http://127.0.0.1:8080/api/notebook/jobmanager/ ``` result ```json { "status":"OK", // result for api request "body":{ "lastResponseUnixTime":1465289938763, // last get server unixtime stamp. "jobs":[ // job list { "notebookId":"2BMQZ9QP6", // notebook id. "unixTimeLastRun":1465289017310, // notebook last running unixtime. "notebookType":"normal", // cron or normal "isRunningJob":false, // is Running? "notebookName":"Untitled Note 1226", // notebook name. "interpreter":"spark", // default interpreter group name or If you have not selected it does not exist. "paragraphs":[ { "name":"20160607-174331_232775609", // paragraph name 'undefined is notebook id' "id":"20160607-174331_232775609", // paragraph id "status":"FINISHED" // paragraph job status } ] } ] } } ``` #### step 2. For example, it showed the result of receiving the information as one of the notebook. focus on "lastResponseUnixTime" value. This value is inserted as an argument when you call the following restapi, us to be able to get the updated data. **Create a Notebook, or run the Paragraph.** And, **call updated notebook api** **"lastResponseUnixTime": 1465289938763** (step 1 get value) ```shell curl -H "Content-Type: application/json" -X GET http://127.0.0.1:8080/api/notebook/jobmanager/1465289938763 ``` #### step 3. If you created a Notebook, a new Notebook information is displayed. If there is a Notebook that if there is a Paragraph of the running, the information will also be displayed. #### step 4. When you restart the step1 of api, if you created a Notebook, the number of Notebook is +1 than when the first call. ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: CloverHearts <[email protected]> Closes #972 from cloverhearts/feat/sm/ZEPPELIN-531-basic-backend and squashes the following commits: 330ed73 [CloverHearts] change indent for jobmanager backend c7bf75a [CloverHearts] Jobmanager basic backend.

### What is this PR for? Job management basic backend. was divided into smaller " PR ". - apache#921 Receive the basic data of Job manager api and the backend have been implemented. ### What type of PR is it? Feature ### Todos - [x] - Basic backend for job manager. - [x] - Basic backend api ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-963 ### How should this be tested? #### step 1. First, calling the rest api as follows receives the data of "job" of the whole. ```shell curl -H "Content-Type: application/json" -X GET http://127.0.0.1:8080/api/notebook/jobmanager/ ``` result ```json { "status":"OK", // result for api request "body":{ "lastResponseUnixTime":1465289938763, // last get server unixtime stamp. "jobs":[ // job list { "notebookId":"2BMQZ9QP6", // notebook id. "unixTimeLastRun":1465289017310, // notebook last running unixtime. "notebookType":"normal", // cron or normal "isRunningJob":false, // is Running? "notebookName":"Untitled Note 1226", // notebook name. "interpreter":"spark", // default interpreter group name or If you have not selected it does not exist. "paragraphs":[ { "name":"20160607-174331_232775609", // paragraph name 'undefined is notebook id' "id":"20160607-174331_232775609", // paragraph id "status":"FINISHED" // paragraph job status } ] } ] } } ``` #### step 2. For example, it showed the result of receiving the information as one of the notebook. focus on "lastResponseUnixTime" value. This value is inserted as an argument when you call the following restapi, us to be able to get the updated data. **Create a Notebook, or run the Paragraph.** And, **call updated notebook api** **"lastResponseUnixTime": 1465289938763** (step 1 get value) ```shell curl -H "Content-Type: application/json" -X GET http://127.0.0.1:8080/api/notebook/jobmanager/1465289938763 ``` #### step 3. If you created a Notebook, a new Notebook information is displayed. If there is a Notebook that if there is a Paragraph of the running, the information will also be displayed. #### step 4. When you restart the step1 of api, if you created a Notebook, the number of Notebook is +1 than when the first call. ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: CloverHearts <[email protected]> Closes apache#972 from cloverhearts/feat/sm/ZEPPELIN-531-basic-backend and squashes the following commits: 330ed73 [CloverHearts] change indent for jobmanager backend c7bf75a [CloverHearts] Jobmanager basic backend.

### What is this PR for? Job manger basic front end. You can check to paragraph and notebook information. It is created with the following additional functions PR. (E. G., Filter and sort) this PR is divided from the #921 ### What type of PR is it? Featrue ### Todos - [x] - notebook information - [x] - viewing status for paragraph - [x] - feat. running progress bar. - [x] - added job menu in navbar. (It can be modified by the following (#1013) ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-964 ### How should this be tested? 1. create and running, modified notebook and paragraph. 2. check to cron or normal notebook status in job manager. ### Screenshots <img width="683" alt="job manger-basic" src="https://cloud.githubusercontent.com/assets/10525473/16113612/0120dec8-33f8-11e6-8dec-c74048fae637.png"> ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: CloverHearts <[email protected]> Author: CloverHearts <[email protected]> This patch had conflicts when merged, resolved by Committer: Lee moon soo <[email protected]> Closes #1025 from cloverhearts/dev/jobmanager/step/02-basic-front and squashes the following commits: aa7f502 [CloverHearts] Merge branch 'master' into dev/jobmanager/step/02-basic-front c759cff [CloverHearts] icon spin animation support for chrome. f183d73 [CloverHearts] job paragraph information tooltip top to top-left db76838 [CloverHearts] restore location for job menu in navbar bb8858c [CloverHearts] Merge branch 'master' into dev/jobmanager/step/02-basic-front 5d75520 [CloverHearts] Merge branch 'master' into dev/jobmanager/step/02-basic-front 0313cfa [CloverHearts] Merge branch 'master' into dev/jobmanager/step/02-basic-front baf3ec6 [CloverHearts] Merge branch 'master' into dev/jobmanager/step/02-basic-front 9ee539b [CloverHearts] Merge branch 'master' into dev/jobmanager/step/02-basic-front 5d64018 [CloverHearts] Merge branch 'master' into dev/jobmanager/step/02-basic-front 75186c7 [CloverHearts] implement basic - frontend for job manager. 18db280 [CloverHearts] modifed get run status for paragraph and bug fixed.

### What is this PR for? Job manger basic front end. You can check to paragraph and notebook information. It is created with the following additional functions PR. (E. G., Filter and sort) this PR is divided from the apache#921 ### What type of PR is it? Featrue ### Todos - [x] - notebook information - [x] - viewing status for paragraph - [x] - feat. running progress bar. - [x] - added job menu in navbar. (It can be modified by the following (apache#1013) ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-964 ### How should this be tested? 1. create and running, modified notebook and paragraph. 2. check to cron or normal notebook status in job manager. ### Screenshots <img width="683" alt="job manger-basic" src="https://cloud.githubusercontent.com/assets/10525473/16113612/0120dec8-33f8-11e6-8dec-c74048fae637.png"> ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? yes Author: CloverHearts <[email protected]> Author: CloverHearts <[email protected]> This patch had conflicts when merged, resolved by Committer: Lee moon soo <[email protected]> Closes apache#1025 from cloverhearts/dev/jobmanager/step/02-basic-front and squashes the following commits: aa7f502 [CloverHearts] Merge branch 'master' into dev/jobmanager/step/02-basic-front c759cff [CloverHearts] icon spin animation support for chrome. f183d73 [CloverHearts] job paragraph information tooltip top to top-left db76838 [CloverHearts] restore location for job menu in navbar bb8858c [CloverHearts] Merge branch 'master' into dev/jobmanager/step/02-basic-front 5d75520 [CloverHearts] Merge branch 'master' into dev/jobmanager/step/02-basic-front 0313cfa [CloverHearts] Merge branch 'master' into dev/jobmanager/step/02-basic-front baf3ec6 [CloverHearts] Merge branch 'master' into dev/jobmanager/step/02-basic-front 9ee539b [CloverHearts] Merge branch 'master' into dev/jobmanager/step/02-basic-front 5d64018 [CloverHearts] Merge branch 'master' into dev/jobmanager/step/02-basic-front 75186c7 [CloverHearts] implement basic - frontend for job manager. 18db280 [CloverHearts] modifed get run status for paragraph and bug fixed.

cloverhearts and others added 13 commits May 9, 2016 19:48

implement job manager for webapp.

8f7b26d

implement logic web for jobmanger.

2fb5f7a

add notebook filter in jab manager page

d23fb7b

Merge branch 'master' into feat/jobmanagement

f51baaa

Conflicts: zeppelin-web/test/karma.conf.js

add sort to job item

7b4db8d

add momentjs in .jshintrc

b24b4e7

implement notebook job management backend.

b90e320

Implement cs transition for jobmanagement.

35887c0

Merge branch 'master' into feat/jobmanagement

5b20b03

improve jobmanagement ui and timestamp

f87217c

bug fixed and improve run status to always top for job manager page.

f1296a8

remove semicolon for job manager

75c03ef

Merge branch 'master' into feat/jobmanagement

f3575f9

Conflicts: zeppelin-web/.jshintrc

cloverhearts added 2 commits May 30, 2016 11:44

Additional feedback in the absence of the interpreter for jobmanager

0bcb9ff

Merge branch 'master' into ZEPPELIN-531

4695c64

optimize job management - slow down issue.

877b203

cloverhearts added 2 commits May 31, 2016 11:46

add comment and change variable name.

5a78136

was divided zeppelin-notebook server - job manager server.

caf4f31

remove bin/derby.log

5590b66

remove head tag - job manager.

bc443da

cloverhearts closed this Jun 5, 2016

cloverhearts reopened this Jun 5, 2016

cloverhearts closed this Jun 6, 2016

cloverhearts reopened this Jun 6, 2016

Merge branch 'master' into ZEPPELIN-531

49422f0

cloverhearts closed this Jun 6, 2016

cloverhearts reopened this Jun 6, 2016

bzz reviewed Jun 6, 2016
View reviewed changes

cloverhearts closed this Jun 7, 2016

cloverhearts mentioned this pull request Jun 7, 2016

ZEPPELIN-963 ] Jobmanagement - (1) basic backend. #972

Closed

2 tasks

cloverhearts mentioned this pull request Jun 16, 2016

ZEPPELIN-964 ] Jobmanagement - (2) basic front-end #1025

Closed

4 tasks

cloverhearts mentioned this pull request Jul 8, 2016

ZEPPELIN-531 ] Job management - Filter and Run/stop controller #1152

Closed

4 tasks

Zeppelin-531 ] Job management #921

Zeppelin-531 ] Job management #921

Uh oh!

Conversation

cloverhearts commented May 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this PR for?

What type of PR is it?

Todo

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

Uh oh!

minahlee commented May 28, 2016

Uh oh!

cloverhearts commented May 28, 2016

Uh oh!

minahlee commented May 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloverhearts commented May 30, 2016

Uh oh!

skanda83 commented May 30, 2016

Uh oh!

minahlee commented May 30, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloverhearts commented May 30, 2016

Uh oh!

cloverhearts commented May 30, 2016

Uh oh!

astroshim commented May 30, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloverhearts commented May 31, 2016

Uh oh!

prabhjyotsingh commented May 31, 2016

Uh oh!

cloverhearts commented May 31, 2016

Uh oh!

cloverhearts commented Jun 4, 2016

Uh oh!

felixcheung commented Jun 5, 2016

Uh oh!

cloverhearts commented Jun 6, 2016

Uh oh!

bzz Jun 6, 2016

Choose a reason for hiding this comment

Uh oh!

bzz commented Jun 6, 2016

Uh oh!

cloverhearts commented Jun 6, 2016

Uh oh!

cloverhearts commented Jun 7, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

cloverhearts commented May 27, 2016 •

edited

Loading

minahlee commented May 28, 2016 •

edited

Loading

minahlee commented May 30, 2016 •

edited

Loading

astroshim commented May 30, 2016 •

edited

Loading