[ZEPPELIN-773] Adding support for spark interpreter to work with yarn…

[mfelgamal]

What is this PR for?

Adding support for zeppelin to run in yarn-cluster mode through Livy

• Creating new interpreter (livy-spark) with (spark, pyspark, sparkr)
• Extending the interpreter to allow manipulation in the configurations of Spark from zeppelin web ui.

What type of PR is it?

Feature

Todos

• [Test case ] - Task
• [Documentation ] - Task

What is the Jira issue?

• [ZEPPELIN-773]

How should this be tested?

• Install Livy on https://github.com/cloudera/hue/tree/master/apps/spark/java
• Start the Livy server
• Start the Zeppelin and go to interpreter page, create livy interpreter with supported configurations
• Run livy-interpreter on any of three modes (%livy.spark to spark, %livy.pyspark to python, %livy.sparkr to R)

_If you run on a cluster, the cluster should support yarn, and run livy-server in yarn mode_

Screenshots (if appropriate)

Questions:

• Does the licenses files need update? no
• Is there breaking changes for older versions? no
• Does this needs documentation? yes

[prabhjyotsingh]

Hi @mfelgamal, A livy interpreter has recently been added in zeppelin with #827. It works with both the installation of https://github.com/cloudera/hue/tree/master/apps/spark/java and https://github.com/cloudera/livy, is enabled with user impersonation as well. Can you take a look at that as well, and in case if you see short coming in it, may be file a jira for it.

[felixcheung]

@mfelgamal thank you for your contributions. As you can see we started a PR on Livy interpreter about a month and a half ago, and it has recently been merged. You are very welcome to take a look at that and integrate additional features that we might have missed. More contributions are welcome!

[mfelgamal]

Hi @prabhjyotsingh @felixcheung , the PR in #827 sound like it is the same as our PR. So, I agree with you that we might be duplicating the feature. Anyway, I have suggested a small change in your PR. You can find it at #944 . It mainly adds some important configuration to spark in Yarn mode.