[MINOR] Removed unused profiles from spark/pom.xml

[jongyoul]

What is this PR for?

Making spark/pom.xml simple

What type of PR is it?

[Refactoring]

Todos

• - Simplified pom.xml in spark

What is the Jira issue?

N/A

How should this be tested?

No test. CI should be green

Screenshots (if appropriate)

Questions:

• Does the licenses files need update? No
• Is there breaking changes for older versions? No
• Does this needs documentation? No

[jongyoul]

https://travis-ci.org/apache/zeppelin/builds/150548730

[jongyoul]

https://travis-ci.org/apache/zeppelin/builds/150645554

[bzz]

Simplification looks great to me!

But one thing is not clear - i.e a MapR profiles are not useful any more?

IDK but my guess is that they were added there for the reason (compatibility \w MapR build of Hadoop distro) - we should remove those at least with explanation of the reason, why that is done and how it affects people \w those distro, how do you think?

Looks like candidate for flaky-test labeled JIRA issue in zeppelin-server:

Failed tests: 
  ZeppelinSparkClusterTest.zRunTest:204 expected:<FINISHED> but was:<ERROR>

Tests run: 65, Failures: 1, Errors: 0, Skipped: 0

[jongyoul]

@bzz Basically, Spark module doesn't have to need hadoop version because we only use Spark repl, core and so on not including specific hadoop version. spark-dependencies only needs specific hadoop version. Thus, specifying hadoop version is not necessary for it.

[bzz]

Thank you for explanation! Makes sense.

Looks great to me.

May be it's worth to git rebase master as CI seems to be recently fixed.

[zjffdu]

Definitely should remove profile yarn. I got classpath issue when enabling yarn profile.

And we need to update the docs accordingly.

[bzz]

@jongyoul do you want to take of docs update that @zjffdu mention in this PR as well?

[jongyoul]

@zjffdu @bzz I've already removed yarn profile. Have I missed something?

[jongyoul]

And about docs, it never change some behaviour. I think another issue linked above is a better point to update docs.

[zjffdu]

Is sparkr needed here ? Because I notice there's another sparkr profile in spark-dependencies/pom.xml

[jongyoul]

This is because of selecting interpreter-setting.json if/if not we use sparkR and this is because we have two different SparkR versions.

[felixcheung]

I thought we do need cassandra-spark-* @doanduyhai ?

[jongyoul]

@felixcheung What I want to remove is duplicated from spark-dependencies/pom.xml. This deletion never hurt current features. It just removes duplicated codes.

[jongyoul]

Merging if there's no more discussion.

[bzz]

Quick question: just to double-check - this should not affect users following published build instructions like https://www.mapr.com/blog/building-apache-zeppelin-mapr-using-spark-under-yarn , right?

In a way like in ZEPPELIN-1353

[jongyoul]

@bzz Yes, what you mentioned is related to #1339 which will make a change to build custom Spark.