[ZEPPELIN-871] [WIP] spark 2.0 interpreter on scala 2.11

[echarles]

What is this PR for?

Spark interpreter for spark version 2.0.0 and scala 2.11 (implemented in Scala)

What type of PR is it?

[Feature]

Todos

• - Clean code
• - Test SQL
• - Test Python
• - Test R

What is the Jira issue?

• https://issues.apache.org/jira/browse/ZEPPELIN--871

How should this be tested?

Build it with

mvn install   -Pscala-2.11 -Dscala.binary.version=2.11 -Dscala.version=2.11.8   -Pspark-2.0 -Dspark.version=2.0.0-SNAPSHOT   -Phadoop-2.6 -Dhadoop.version=2.7.2 -Pyarn   -Dmaven.findbugs.enable=false   -Drat.skip=true   -Ppyspark   -Psparkr   -Dcheckstyle.skip=true   -Dcobertura.skip=true   -Pbuild-distr   -pl '!alluxio,!cassandra,!elasticsearch,!file,!flink,!hbase,!hive,!ignite,!jdbc,!kylin,!lens,!phoenix,!postgresql,!tajo'   -DskipTests

Run and test the spark paragraph.

Screenshots (if appropriate)

Questions:

• Does the licenses files need update? N
• Is there breaking changes for older versions? N
• Does this needs documentation? Y

[echarles]

This is WIP.

I'd like to get feedback on the approach (scala implementation in the current spark module), taking into consideration:

• Spark 2.0 will ship by defaulit on scala 2.11 (makes sense to me to develop the interpeter on scala 2.11)
• SqlContext and HiveContext are deprected in favor of SparkSession.
• HttpServer is removed and some classes methods have access restricted to spark only packages.
• SpakIMain and SparkJLineCompletion don't exist anymore.

Building on the current java classes with method invocation may be possible, but would make the code more than difficult to read and develop.

This PR proposes separated scala classes for this very specific 2.0 API breaking changes.

WDYT?

Based on feedback, I will further validate the functionalities (for now, a simple spark 2.0 call works well on my local env).

[Leemoonsoo]

@echarles Thanks for the contribution.

How about we divide problem into

• scala 2.11 support
• spark 2.0 support
• reimplement spark interpreter in scala

@lresende is working on scala 2.11 support on #747. I'm also trying to help merge code for scala 2.10 and 2.11 into one via lresende#1
One of the goal in #747 is support scala 2.11 and 2.10 from the single Zeppelin binary package.
Once #747 is done, this PR can take scala 2.11 support from #747.

Regarding spark 2.0 and reimplementation in scala,
How do you think about support spark 2.0 as well as spark 1.x while many users will use Zeppelin with spark 1.x for a some time?

[echarles]

Sure, we can wait on #747 merge.

I see there is a spark/src/main/scala-2.11/org/apache/zeppelin/spark/SparkInterpreter.java: does this mean that there will be 2 separate implementations: one for 2.10 et and for 2.11?

[Leemoonsoo]

I'm trying to combine 2.10 and 2.11 into one implementation in lresende#1

[felixcheung]

there's a new version of py4j too

[echarles]

thx, taking this into account in next push.

[felixcheung]

From what I see, we should still be able to use most of the existing code. Could you elaborate more on how bad it would be to support Spark 1.x and 2.x in the same interpreter code?

[echarles]

Dealing with mixed scala 2.10/2.11 and spark 1.x/2.x with a same implementation is always possible but drives to code plenty of method invocation (see e.g. https://github.com/lresende/incubator-zeppelin/pull/1/files#diff-dbda0c4083ad9c59ff05f0273b5e760fR216).

At the end, you have an implementation which a succession of if (...) then invokeMethod...

On the other hand, having multiple implementations drives to maintenance overhead and a risk of divergence.

In this particular case, I was thinking that having 2 lines was something worth to discuss:

1.- Spark 1.x on scala 2.10
2.- Spark 2.x on scala 2.11

I also don't see why we should still support the DepInterpreter in future developments (normally, deps should be configured via the interperter settings). Also, the 2 lines approach would make easier other evolutions such as for example having a ZeppelinContext available for all interpreters.

Also, we have kind-off already more than one spark impementation: The Livy one has its own implementation and features.

[Leemoonsoo]

Thanks for elaborate on idea about 2 lines approach.

I think there're always pros and cons.

If we go to 2 lines approach, code will be cleaner, simpler and more easier to read.

But this means Zeppelin generally enables create binary per interpreter dependency because we'll have the same policy for all interpreters. In the end, i'm afraid we end up with making bunch of binary packages like

• zeppelin with spark 1.x on scala 2.10
• zeppelin with spark 1.x on scala 2.11
• zeppelin with spark 2.0 on scala 2.10
• zeppelin with spark 2.0 on scala 2.11
• zeppelin with spark 2.1 on scala 2.10
• zeppelin with spark 2.1 on scala 2.10
• zeppelin with python3.x
• zeppelin with python2.x
• zeppelin with hive1
• zeppelin with hive2
• .....

In the perspective of user, it can not be an improvement. A single Zeppelin binary used to work with everything, but now user have to understand difference between of packages and able to select them before uses.

I think code complexity will not endlessly increase if we cut the spark support for last x releases.

[Leemoonsoo]

When @minahlee contributed dependency loading through interpreter setting, we also thought DepInterpreter can be deprecated and removed. That's why current documentation of spark interpreter mark %dep is deprecated.

But since that, there were some strong feedbacks from users that they likes DepInterpreter especially it enables self documenting. i.e. Working code and it's dependency can be in the same notebook. And i think it really make sense.

So i think we need to reconsider deprecating DepInterpreter.

[echarles]

Agree with your agurment. If we push a bit further the discussion, we can not have a single binary which fits all users needs. What if I want spark-2-scala-2.11 with flink-0.9-scala-2.12 ? You single binary will not give me any solution...

I would rather see a zeppelin-kernel (nearly naked without interpeters) with the ability for the user to enable any interpreter via a shell command.

Regarding DepIntepreter, I can understand what users says, but for some packages, it simply does not work (example: spark-csv running on yarn if I remember well) - Not sure if it can be easily fixed.

Bottom line: I am no fan of the invoke coding style, but I will wait on the other PR and try to build on top of this.

[Leemoonsoo]

Why not single binary works for spark-2-scala-2.11 and flink-0.9-scala-2.12?
I'm also not a fan of reflection. But if i have to choose either complexity of code or complexity usage, then i would like choose keeping usage simple, even though code becomes more complex. I believe this is whole point of making a software.

And i 100% agree. Once we have https://issues.apache.org/jira/browse/ZEPPELIN-598 and proper UI / command line for list/download interpreter from maven repository, then i think things will become much more easier and flexible.

[felixcheung]

+1 on all of that. I think we could work together on a proposal to see what is the best way to go forward.
I think we are way overdue on separating "programming languages" (and versions) and "frameworks" (and versions).

[echarles]

@Leemoonsoo

• I was just saying that a single binary will not fit all various users need (multiple combination of interpeters and versions).
• ZEPPELIN-598 ] Dynamic loading for Interpreter #629 is merged. What remains to get ZEPPELIN-598 operational?

@felixcheung Is there already a jira for the proposal you are thinkg to? I feel we have nearly everything open, but maybe we see an umbrella?

[echarles]

@Leemoonsoo Oh, #908 is the one I was looking for

[Leemoonsoo]

@echarles

I was just saying that a single binary will not fit all various users need (multiple combination of interpeters and versions).

If single SparkInterpreter binary is compatible with various spark versions and scala versions, and single FlinkInterpreter binary is compatible with various flink versions and scala versions, user can choose any combination without rebuild. isn't it?

#629 is merged. What remains to get ZEPPELIN-598 operational?

After #908 is merged, we'll need proper commandline or GUI to access this feature.
And publish all interpreters to maven repository and remove interpreter dependencies from binary package.

[echarles]

• If you ship in the same folder scala libs from different versions, it will fail at runtime - You don't have to rebuilt, but you have to arrange classpath is built with scala libs of the same version
• Gui, cli... Makes sense. Is t is what @felixcheung was refereing to?

[rnirmal]

@echarles @Leemoonsoo do you know where the direction lies for Spark 2.0 with scala 2.11 support.. i.e as separate items or continued effort on this PR? Since Spark 2.0 is going to be release in the next week or two, would be great to see this made available on Zeppelin as a quick follow on.

I can help with any testing if needed, not as familiar with Zeppelin code yet to help with reviews.

[Leemoonsoo]

@rnirmal Vote for 0.6.0-rc1 is about to start. We'll need 0.6.1 release immediately after we have scala 2.11 and spark 2.0 support.

Currently, @lresende and me are trying to make Scala 2.11 support green.
@echarles made great work here. Spark 2.0 interpreter implementation in scala.

For 0.6.1 release, I'd like to add scala 2.11 and spark 2.0 support while keeping current way of implementation, to keep previous spark version support the same, more conservatively.

Meanwhile i think we can continue to discuss and working on scala implementation of spark interpreter, easier spark interpreter maintenance, and so on, on the master branch so they can be included in 0.7.0 release.

[rnirmal]

Sounds good, I'll keep a lookout for it to land.

[echarles]

Closing this. Spark 2.0 is implemented with #1195
Optionally, a new PR can be opened to port the Spark interpreter from Java to Scala if interest.