[SPARK-21578][CORE] Add JavaSparkContextSuite

[dongjoon-hyun]

What changes were proposed in this pull request?

Due to SI-8479, SPARK-1093 introduced redundant SparkContext constructors. However, SI-8479 is already fixed in Scala 2.10.5 and Scala 2.11.1.

The real reason to provide this constructor is that Java code can access SparkContext directly. It's Scala behavior, SI-4278. So, this PR adds an explicit testsuite, JavaSparkContextSuite to prevent future regression, and fixes the outdate comment, too.

How was this patch tested?

Pass the Jenkins with a new test suite.

[SparkQA]

Test build #80061 has finished for PR 18778 at commit ad99361.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

The Python failure is irrelevant.

ERROR: test_toPandas_arrow_toggle (pyspark.sql.tests.ArrowTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/anaconda/envs/py3k/lib/python3.4/site-packages/pandas/__init__.py", line 25, in <module>
    from pandas import hashtable, tslib, lib
ImportError: cannot import name 'hashtable'

[dongjoon-hyun]

Retest this please

[gatorsmile]

Could you check whether JAVA APIs still work? Could you add the related test cases?

[SparkQA]

Test build #80062 has finished for PR 18778 at commit ad99361.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Thank you for review, @gatorsmile .
Do you mean JavaSparkContext? This PR didn't change it here.

[dongjoon-hyun]

Retest this please.

[gatorsmile]

JavaSparkContext is the JAVA friendly SparkContext. Is that possible the JAVA programs could still directly use the SparkContext APIs. I am afraid these changes could break them.

[dongjoon-hyun]

Do you want to create a new test suite for that under https://github.com/apache/spark/tree/master/core/src/test/java/test/org/apache/spark?

The following suite seems to be irrelevant for that purpose because they uses JavaSparkContext only.

• JavaAPISuite.java
• Java8RDDAPISuite.java

[dongjoon-hyun]

@gatorsmile . I see what your did concern here.

If we are locked in here in order to preserve the previous behavior, what about updating those comments? It's not about SI-8479 anymore. Now, it's just for backward compatibility.

// NOTE: The below constructors could be consolidated using default arguments. Due to		
// Scala bug SI-8479, however, this causes the compile step to fail when generating docs.		
// Until we have a good workaround for that bug the constructors remain broken out.

[SparkQA]

Test build #80069 has finished for PR 18778 at commit ad99361.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

@gatorsmile . Three usages (the comment-out lines) were broken here as you concerned.

  @Test
  public void scalaSparkContext() {
    List<String> jars = List$.MODULE$.empty();
    Map<String, String> environment = Map$.MODULE$.empty();

    new SparkContext(new SparkConf().setMaster("local").setAppName("name")).stop();
    new SparkContext("local", "name", new SparkConf()).stop();
    // new SparkContext("local", "name").stop();
    // new SparkContext("local", "name", "sparkHome").stop();
    // new SparkContext("local", "name", "sparkHome", jars).stop();
    new SparkContext("local", "name", "sparkHome", jars, environment).stop();
  }

This Scala behavior is due to SI-4278 instead of SI-8479. Since SI-4278 is Won'tFix, may I update the description more clearly, or simply remove it?

[dongjoon-hyun]

@gatorsmile . I fixes the comment and adds explicit test suite to prevent future regression.
How do you think about this?

[SparkQA]

Test build #80071 has finished for PR 18778 at commit 0111c14.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Nit: just import these classes as usual?

[dongjoon-hyun]

Thank you for review, @srowen !
This is due to the import conflicts on Map. Java code still doesn't allow import aliasing. We can do aliasing only in Scala code.

[dongjoon-hyun]

Hi, @gatorsmile and @srowen .
Could you review this PR again?

[gatorsmile]

LGTM

[gatorsmile]

Thanks! Merging to master.

[dongjoon-hyun]

Thank you, @gatorsmile and @srowen !