[ZEPPELIN-2645] Adding way to register RemoteInterpreterServer's port into InterpreterProcess

[jongyoul]

What is this PR for?

This is a thrift version related to #2410

What type of PR is it?

[Feature]

Todos

• - Add opposite set of server/client

What is the Jira issue?

• https://issues.apache.org/jira/browse/ZEPPELIN-2645

How should this be tested?

N/A

Screenshots (if appropriate)

N/A

Questions:

• Does the licenses files need update? No
• Is there breaking changes for older versions? No
• Does this needs documentation? No

[zjffdu]

@jongyoul Thanks for the PR.

1. It looks there will be one server per RemoterInterpreterProcess, will it been shutdown after it get the port and host info ? And what will happens if the interpreter process fails to launch ?
2. What is the advantage of netty over thrift ? Since this relates with the how we communicate between zeppelin server and interpreter process, we need to consider which one is better.

[zjffdu]

@jongyoul I think there's 2 follow up things to do.

• Unify the communication frame between zeppelin server & interpreter process.
• Change the pull mode to push mode so that interpreter process can push data to zeppelin-server.

When we consider to choose netty or thrift, we need to consider these 2 things. Any thoughts ?

[jongyoul]

@zjffdu Agreed.

[zjffdu]

Sorry @jongyoul I did a investigation on netty, it looks like we could use Channel directly. In that way I would prefer to use netty. But still I think we need to have a good design for the communication framework change from thrift to netty considering the 2 things I mentioned above.

[jongyoul]

@zjffdu Sounds good. I totally agree with your point of view. We need to add pushing way from interpreter to server, and also abstract that layer carefully to guess most of situation. This is a kind of PoC level. I'll make a design docs for handling it.

[echarles]

I'd like adding more points to the discussion:

• Do we need to support kerberos-like auth in the communication protocol (you can achieve this with Thrift, not sure with Netty)?
• Do we target a single implementation dropping the other or something that can be configured with the burden to maintain multiple implementations?
• What about REST (for more easy and universal clients) or WebSocket (for push) as more alternatives?

[zjffdu]

Do we need to support kerberos-like auth in the communication protocol (you can achieve this with Thrift, not sure with Netty)?

This is internal communication channel between zeppelin server and interpreter process, so I don't think it is necessary to use kerberos here. But definitely, we need to make sure this channel secure, I think both thrift and netty support security channel.

Do we target a single implementation dropping the other or something that can be configured with the burden to maintain multiple implementations?

We would use just one, that's why I mention in the above comment that before we choose thrift or netty, we need to consider the whole picture.

What about REST (for more easy and universal clients) or WebSocket (for push) as more alternatives?

I don't see much advantages of using websocket. Netty and thrift both could do the push. And the communication channel here is one-to-one, not one-to-many.

[echarles]

In case of SNPEGO Web Auth (not yet supported by Zeppelin0,, you could consider transfert the Kerberos ticket from front-end to back-end and use it to , this the usecase I was thinking to.

Well, multiple and pluggable communication between frontend and interpreters would be great and useful, but I am aware of the impact on code level.

If we have to move to something else than thrift, what would be the added value of netty (just asking).

[jongyoul]

@echarles Hi, at first, if we enable front-end to access to interpreter directly, we have to consider that each interpreter has to know whole ACL information and etc. I don't think it's good design. Second, netty has a benefit while making connections between server and interpreter. It can be handled one connection for bidirectional communication.

[echarles]

@jongyoul sorry, I was not clear enough. With frontend, I was meaning the web layer (so jetty). Jetty could communicate via a variety of protocols to the interpreters. Now going further in that direction, having protocols like REST or WebSocket would allow any king of clients to access the interpeters in direct - Those clients could be the current javascript of Zeppelin webapp, or any other kind of client (could be bash or python script...)

This is unique opportunity to extend the Zeppelin user base - Btw, having the AAA (Authentication, Authorization, Auditing) functions at Interpreter level is IMHO a good design, even at the cost of some extra development or refactoring.

I am curious to know what the community thinks about this...

[zjffdu]

@echarles Thanks for the feedback. Currently Interpreter Process can only communicate with Zeppelin Server process, and it is a internal communication channel which is not exposed to users and other components. Regarding your comments Those clients could be the current javascript of Zeppelin webapp, or any other kind of client (could be bash or python script...), Could you give some scenarios that Interpreter Process needs to communicate with other clients ?

[echarles]

@zjffdu I am thinking to automation (using the notes in an scheduled or batch process). Does it make sense to you?

[zjffdu]

For that purpose, you can use zeppelin's rest api or note's cron scheduler
https://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/usage/rest_api/notebook.html