ZEPPELIN-773 : Livy interperter

[prabhjyotsingh]

What is this PR for?

As Zeppelin evolves its notebook, for large scale data analysis, multiple zeppelin users are expected to use and connect to the same set of data repositories within an enterprise. Since Zeppelin notebooks could affect data, state and its lineage, it is important to have separation of users, provide them with appropriate sandboxes, in addition to capturing the right audit details. Further, the IT within the organization would prefer to support fewer Zeppelin instances (preferably one) to support its customers. Therefore, the objectives of creating a multi-tenant zeppelin are:
● Supporting workloads of multiple customers
● Supporting multiple LOBs (lines of business), on a single data systems
● Support fine grained audits
As a natural evolution of Zeppelin Authentication and Authorization design, partly user awareness in downstream data systems such as Spark/Hive and others, is essential to achieve the above stated objectives.

What type of PR is it?

Feature

Todos

• - Test case
• - Review Comments
• - Documentation

What is the Jira issue?

ZEPPELIN-773

How should this be tested?

• Install Livy by following steps on https://github.com/cloudera/livy
• Start the Livy server
• Now by using Zeppelin-Livy interpreter, run any of the spark, pyspark or R commands.

Screenshots (if appropriate)

### Questions: - Does the licenses files need update? n/a - Is there breaking changes for older versions? n/a - Does this needs documentation? yes

[prabhjyotsingh]

Thank you @Leemoonsoo for the review, on debugging I found out that the Livy-Server that https://github.com/cloudera/livy and https://github.com/cloudera/hue/tree/master/apps/spark/java works differently.

https://github.com/cloudera/livy accepts request as curl -X POST --data '{"kind": "pyspark", "conf":{"spark.master": "local[*]"}}' -H "Content-Type: application/json" localhost:8998/sessions

and https://github.com/cloudera/hue/tree/master/apps/spark/java accepts the same as curl -X POST --data '{"kind": "pyspark", "master": "local[*]"}' -H "Content-Type: application/json" localhost:8998/sessions

Have put up an extra check 44c5e82#diff-6590a16671e62be8da95e793c364d739R72 to fix this.

[Leemoonsoo]

@prabhjyotsingh Thanks for explanation. I have tried with https://github.com/cloudera/hue/tree/master/apps/spark/java and LivyInterpreter works greatly.
However, still no luck with https://github.com/cloudera/livy. i'm getting the same error.

[prabhjyotsingh]

@Leemoonsoo can you try one of this

curl -X POST --data '{"kind": "spark", "master": "local[*]", "proxyUser": "null"}' -H "Content-Type: application/json" localhost:8998/sessions if trying on https://github.com/cloudera/hue/tree/master/apps/spark/java

or

curl -X POST --data '{"kind": "spark", "conf":{"spark.master": "local[*]"},"proxyUser": "null"}' -H "Content-Type: application/json" localhost:8998/sessions if on https://github.com/cloudera/livy.

Most likely this is known issue 3deca71#diff-427ba78b970ccdf0a95f5176f693a377R88 and your server is running from https://github.com/cloudera/livy.

or if it is something else can you paste the output, so I can try the same to debug and fix.

[Leemoonsoo]

@prabhjyotsingh Thanks for the help. Commenting out spark.master from https://github.com/cloudera/livy/conf/spark-blacklist.conf made it work.

Looks good to me!

[prabhjyotsingh]

@Leemoonsoo thanks a lot for reviewing it.

CI fails for unrelated issue at Job #4237.1

[felixcheung]

"Known Issue"?

[prabhjyotsingh]

Thank you @AhyoungRyu for the review. Have addressed those.

[prabhjyotsingh]

Shall I merge this, if no more discussion.

[Leemoonsoo]

+1 for merge!

[felixcheung]

+1

[ghost]

@prabhjyotsingh @Leemoonsoo @felixcheung awesome!