Initial support for CassandraSQLContext for %sql queries

[benbromhead]

Allow users to use %sql syntax to query Cassandra tables with SparkSQL without having to manually create a CassandraSQLContext.

Expanding on "Add build profile for Spark/Cassandra integration #79"

To enable Cassandra SQL context support add -Dzeppelin.spark.useCassandraContext=true to the ZEPPELIN_JAVA_OPTS parameter in conf/zeppelin-env.sh file. Alternatively you can add this parameter in the parameter list of the Spark interpreter on the GUI.

Also supported are "spark.cassandra.connection.host", "spark.cassandra.auth.username" and "spark.cassandra.auth.password" parameters.

[boneill42]

Dude, nice work.

[Leemoonsoo]

I think line 268 sets all property into conf that property starts with 'spark.'. So isn't it duplicated?

[Leemoonsoo]

Thanks for great contribution!

And how about add zeppelin.spark.useCassandraContext into property builder?

  static {
     Interpreter.register(
         "spark",
         "spark",
         SparkInterpreter.class.getName(),
         new InterpreterPropertyBuilder()
             .add("spark.app.name", "Zeppelin", "The name of spark application.")
             ...
             .add("zeppelin.spark.useHiveContext", "true",
                  "Use HiveContext instead of SQLContext if it is true.")
             ...

both zeppelin.spark.useCassandraContext, zeppelin.spark.useHiveContext can be true or false. In this case, in user point of view, it's very unclear to see what's going to happen. Do you have an idea to handle it?

And Is it going to be difficult to add a test for it?

[devsprint]

While I have tried to make Zeppelin to work with DSE 4.6 I have integrated also this change. I'm able to properly query cassandra db. However, if there is an error, the notebook will display a generic error that is not helping at all to identify the root cause. (java.lang.reflect.InvocationTargetException).

if zeppelin.spark.useCassandraContext is set to true, then the value of zeppelin.spark.useHiveContext it doesn't matter. Even if it set to true it will still bind to cassandra context.
I think that there is a need for a small change to make sure that just on of the context could be set to true and not both....

[benbromhead]

Apologies I haven't had a whole heap of bandwidth to address the comments above, after chatting @Leemoonsoo at the spark conference the other week I have a good idea about integrating some tests and cleaning up the above mentioned issues.

Stay tuned.

[devsprint]

@benbromhead let me know if I can help. I have played with it all day long today, using zeppelin forms and registering custom functions to be applied on results. Every thing works fine so far.

[benbromhead]

Ok I finally got around to completing some unit tests and changing the default values for the various interpreters.

In this PR I've changed the default behaviour of the SparkInterpreter to use the standard SQLContext by default rather than the HiveContext. This will potentially break peoples notebooks on upgrade if they are relying on Hive syntax/functionality rather than Spark SQL syntax without realising it.

[doanduyhai]

@benbromhead, I'm writing a Cassandra interpreter (connecting Zeppelin directly to C* without going through Spark). Does it worth the effort to have a CassandraSQL interpreter ?

[benbromhead]

@doanduyhai I think it is... particularly if you want to do things like joins :)

[Leemoonsoo]

Is there any good way to keep this value unchanged?
Changing default value might confuse user. what do you think?

[benbromhead]

@Leemoonsoo yeah I struggled to come up with a good answer for this. On the one hand I think the default behaviour should be to use the standard SQLSpark context, but by changing this it will break the default behaviour for users that rely on Hive features.

You will notice that the sql context tests actually rely on Hive syntax rather than spark sql compliant syntax. My vote would be to go with what is correct early on in this projects life, but I know this will be irritating for existing users.

Happy to revert to the previous behaviour.

[Leemoonsoo]

zeppelin.spark.useHiveContext was originally 'false' but changed to 'true'. The reason was to give the same experience with spark-shell, which uses HiveContext by default.

I think changing zeppelin.spark.useHiveContext need to be handled in separate issue if it is not breaking this contribution.

[benbromhead]

Ok makes sense, changed back to use the HiveContext by default

[Leemoonsoo]

+1

[benbromhead]

Anything left to do on this before it gets merged?

[falconair]

Looks like this request was close to getting merged months ago, any status update on this?

[jongyoul]

@Leemoonsoo @benbromhead Do you guys have any update for this? It looks like pending state.

[Leemoonsoo]

Can someone provide additional review?

[bbromhead]

The codebase has probably shifted a little bit since this last passed tests etc, do you want me to update and test against the latest master?

[corneadoug]

@bbromhead @jongyoul @Leemoonsoo any more feedback on changes needed on this PR?

[benbromhead]

I've updated this PR in the past and also asked for further interest.

I'm happy to update against the latest if there is an appetite to actually merge this into the project, it would be great to get feedback either way!

[doanduyhai]

@benbromhead some remarks before rebasing

1. maybe upgrade C* version and Spark/Cassandra connector version
2. maybe use Achilles-embedded (look Cassandra interpreter unit tests) instead of cassandra-unit. The project is no longer actively maintained
3. the SparkInterpreter class has changed a lot, probably a good idea to cherry-pick your changes and apply them instead of going through the complete rebase
4. you've hard-coded the Spark/Cassandra connector dependency directly in spark/pom.xml. I'm not sure it's the cleanest way because people who don't use Cassandra still need to pull this dependency. Maybe it's better to create a separate Maven build profile for this. Until now it is done in spark-dependencies/pom.xml but since it will be removed soon by @AhyoungRyu (see [ZEPPELIN-1332] Remove spark-dependencies & suggest new way #1339) we'll need to find an alternative

[benbromhead]

I'll look to address the remarks. I've also noticed the project now has a formal tracker via Jira. I'll also look to complete the requirements listed in https://zeppelin.apache.org/contribution/contributions.html