ZEPPELIN-1643:Make spark web UI accesible from interpreters page #1613

karuppayya · 2016-11-08T18:07:16Z

What is this PR for?

Make spark web UI accesible from interpreters page

What type of PR is it?

Improvement

Todos

NA

What is the Jira issue?

ZEPPELIN-1643

How should this be tested?

Start sparkcontext.
Goto interpreters page. Corresponding to the spark interpreter used in notebook, there will be spark ui button.
Clicking the button should open a new tab with the application's spark web UI.

Screenshots (if appropriate)

Questions:

Does the licenses files need update? NA
Is there breaking changes for older versions? No
Does this needs documentation? No

Leemoonsoo · 2016-11-08T20:08:05Z

zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterGroup.java

  ResourcePool resourcePool;
  boolean angularRegistryPushed = false;

+  private RemoteInterpreterEventClient eventClient;


InterpreterGroup is not only used in RemoteInterpreter process but also in ZeppelinServer side, too.
So having reference to eventClient in InterpreterGroup is not optimal i think.

I think InterpreterContext is better place to have such a reference, and you can inject your object when InterpreterContext is being created.

InterpreterContext is not passed to interpreter in open(). So getting routines that getting SparkUI and update meta need to be moved from open() to interpret(). I think it's not a big problem in this use case.

Also i think it'll be better passing wrapper object instead of eventClient directly. Because of we don't want to let interpreter directly access all eventClient's methods.

Leemoonsoo

Thanks @karup1990 for the very useful contribution.
Left some comments. please check.

karuppayya · 2016-11-09T16:30:17Z

@Leemoonsoo Thanks for the suggestions.
I have refactored the code now.

karuppayya · 2016-11-10T15:28:22Z

Ready for review

karuppayya · 2016-11-13T04:20:06Z

Failure seems unrelated.
03:55:19,906 ERROR org.apache.zeppelin.AbstractZeppelinIT:136 - Exception in ParagraphActionsIT while testEditOnDoubleClick

Leemoonsoo · 2016-11-13T15:20:07Z

LGTM. Merge if there're no further discussions.

### What is this PR for? Recently, zeppelin added some amazing features. #1613 You can easily access spark ui from the zeppelin web. However, in the implementation of the function, the Thrift interface seems to have arbitrarily modified the automatically generated code. So if you recreate the Thift interface by calling genthrift.sh. Zeppelin is not built. I modified this to get the same result with auto-generated code. ### What type of PR is it? Bug Fix ### Todos - [x] add event META_INFOS in RemoteInterpreterService.thrift - [x] import generate code ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1679 ### How should this be tested? 1. First you need to install thift 0.9.2. 2. Run zeppelin-interpreter / src / main / thrift / generate_thrift.sh. 3. Build full zeppelin. 4. After build, please check the interpreter spark setting in our web ui. 5. Try using the spark ui button. If everything is running normally, it is a success. Problems occur in the current master 3. ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: CloverHearts <[email protected]> Closes #1650 from cloverhearts/fix-thift and squashes the following commits: ebd2838 [CloverHearts] fixed incorrect thrift implementation.

zjffdu · 2016-11-23T04:58:47Z

If I read it correctly, this PR assume each interpreter setting has one spark ui url ? But I think this is wrong assumption, as we may have multiple spark applications for single interpreter setting. @karup1990 @Leemoonsoo

Leemoonsoo · 2016-11-23T05:15:33Z

@zjffdu Right. Isolated mode can create multiple spark applications.
@karup1990 @zjffdu how can we handle multiple spark applications?

zjffdu · 2016-11-23T05:18:50Z

I think the right place to put spark ui link is in note page instead of interpreter setting page.

Leemoonsoo · 2016-11-23T05:31:47Z

Make sense. @karup1990 what do you think?

karuppayya · 2016-11-23T07:09:15Z

@zjffdu @Leemoonsoo Thanks for letting me know.
I will make the changes to handle isolated scope as well and create a PR

### What is this PR for? Make spark web UI accesible from interpreters page ### What type of PR is it? Improvement ### Todos NA ### What is the Jira issue? ZEPPELIN-1643 ### How should this be tested? Start sparkcontext. Goto interpreters page. Corresponding to the spark interpreter used in notebook, there will be *spark ui* button. Clicking the button should open a new tab with the application's spark web UI. ### Screenshots (if appropriate) ![zscpr58qdi](https://cloud.githubusercontent.com/assets/5082742/20110797/c6852202-a60b-11e6-8264-93437a58f752.gif) ### Questions: * Does the licenses files need update? NA * Is there breaking changes for older versions? No * Does this needs documentation? No Author: karuppayya <[email protected]> Author: Karup <[email protected]> Closes apache#1613 from karup1990/ZEPPELIN-1643 and squashes the following commits: 29e9812 [karuppayya] Fix checkstyle 350d665 [karuppayya] Fix coding err 42dde76 [Karup] Fix typo 5cfed2a [karuppayya] Fix test failure 55c45c9 [karuppayya] Address feedback 4d97196 [karuppayya] Remove comments a1304a2 [karuppayya] Change to make spark web UI accesible from interpreters page

### What is this PR for? Recently, zeppelin added some amazing features. apache#1613 You can easily access spark ui from the zeppelin web. However, in the implementation of the function, the Thrift interface seems to have arbitrarily modified the automatically generated code. So if you recreate the Thift interface by calling genthrift.sh. Zeppelin is not built. I modified this to get the same result with auto-generated code. ### What type of PR is it? Bug Fix ### Todos - [x] add event META_INFOS in RemoteInterpreterService.thrift - [x] import generate code ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1679 ### How should this be tested? 1. First you need to install thift 0.9.2. 2. Run zeppelin-interpreter / src / main / thrift / generate_thrift.sh. 3. Build full zeppelin. 4. After build, please check the interpreter spark setting in our web ui. 5. Try using the spark ui button. If everything is running normally, it is a success. Problems occur in the current master 3. ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: CloverHearts <[email protected]> Closes apache#1650 from cloverhearts/fix-thift and squashes the following commits: ebd2838 [CloverHearts] fixed incorrect thrift implementation.

karuppayya · 2016-11-23T18:44:29Z

I did a code reading to see how to fix this.. Looks like I will need to handle it per user as well.
To complete this PR, I can quickly make the changes to show the link in interpreter page, only for per note scoped and global mode.
Else , I will try to fix all modes(user, note, global) and then create a PR.
@zjffdu @Leemoonsoo let me know your thoughts.

karuppayya · 2016-11-24T16:55:27Z

@zjffdu @Leemoonsoo I have handled the cases in #1678

Make spark web UI accesible from interpreters page Improvement NA ZEPPELIN-1643 Start sparkcontext. Goto interpreters page. Corresponding to the spark interpreter used in notebook, there will be *spark ui* button. Clicking the button should open a new tab with the application's spark web UI. ![zscpr58qdi](https://cloud.githubusercontent.com/assets/5082742/20110797/c6852202-a60b-11e6-8264-93437a58f752.gif) * Does the licenses files need update? NA * Is there breaking changes for older versions? No * Does this needs documentation? No Author: karuppayya <[email protected]> Author: Karup <[email protected]> Closes apache#1613 from karup1990/ZEPPELIN-1643 and squashes the following commits: 29e9812 [karuppayya] Fix checkstyle 350d665 [karuppayya] Fix coding err 42dde76 [Karup] Fix typo 5cfed2a [karuppayya] Fix test failure 55c45c9 [karuppayya] Address feedback 4d97196 [karuppayya] Remove comments a1304a2 [karuppayya] Change to make spark web UI accesible from interpreters page (cherry picked from commit 96ca84d)

Leemoonsoo reviewed Nov 8, 2016

View reviewed changes

Leemoonsoo suggested changes Nov 8, 2016

View reviewed changes

karuppayya closed this Nov 13, 2016

karuppayya reopened this Nov 13, 2016

karuppayya and others added 6 commits November 14, 2016 22:59

Change to make spark web UI accesible from interpreters page

a1304a2

Remove comments

4d97196

Address feedback

55c45c9

Fix test failure

5cfed2a

Fix typo

42dde76

Fix coding err

350d665

karuppayya force-pushed the ZEPPELIN-1643 branch from 538225c to 350d665 Compare November 14, 2016 17:30

Fix checkstyle

29e9812

asfgit closed this in 96ca84d Nov 15, 2016

cloverhearts mentioned this pull request Nov 17, 2016

[ZEPPELIN-1679] Invalid thrift implementation (developer critical) #1650

Closed

2 tasks

ZEPPELIN-1643:Make spark web UI accesible from interpreters page #1613

ZEPPELIN-1643:Make spark web UI accesible from interpreters page #1613

Uh oh!

Conversation

karuppayya commented Nov 8, 2016

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

Uh oh!

Leemoonsoo Nov 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Leemoonsoo left a comment

Choose a reason for hiding this comment

Uh oh!

karuppayya commented Nov 9, 2016

Uh oh!

karuppayya commented Nov 10, 2016

Uh oh!

karuppayya commented Nov 13, 2016

Uh oh!

Leemoonsoo commented Nov 13, 2016

Uh oh!

zjffdu commented Nov 23, 2016

Uh oh!

Leemoonsoo commented Nov 23, 2016

Uh oh!

zjffdu commented Nov 23, 2016

Uh oh!

Leemoonsoo commented Nov 23, 2016

Uh oh!

karuppayya commented Nov 23, 2016

Uh oh!

karuppayya commented Nov 23, 2016

Uh oh!

karuppayya commented Nov 24, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Leemoonsoo Nov 8, 2016 •

edited

Loading