[SPARK-4003] [SQL] add 3 types for java SQL context

[adrian-wang]

In JavaSqlContext, we need to let java program use big decimal, timestamp, date types.

[SparkQA]

QA tests have started for PR 2850 at commit 3c58b0d.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2850 at commit 3c58b0d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21904/
Test PASSed.

[marmbrus]

Can you add some tests too?

[SparkQA]

QA tests have started for PR 2850 at commit bb0508f.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2850 at commit bb0508f.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21968/
Test FAILed.

[adrian-wang]

retest this please.

[SparkQA]

QA tests have started for PR 2850 at commit bb0508f.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2850 at commit bb0508f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21975/
Test PASSed.

[liancheng]

Good catch! LGTM, thanks.

[marmbrus]

Thanks for working on this! In order for the BigDecimals to work we are going to need to do some conversions as well. This is because the expected internal representation of BigDecimal is the scala class, not the java one.

Consider the following test case:

    assert(
      javaSqlCtx.sql("SELECT bigDecimalField + bigDecimalField FROM allTypes").collect.head.row ===
        Seq(new java.math.BigDecimal(0)))

Which will throw

[info]   org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0 (TID 5, localhost): java.lang.ClassCastException: java.math.BigDecimal cannot be cast to scala.math.BigDecimal
[info]         scala.math.Numeric$BigDecimalIsFractional$.plus(Numeric.scala:182)
[info]         org.apache.spark.sql.catalyst.expressions.Add$$anonfun$eval$3.apply(arithmetic.scala:72)

I think we'll need to do something similar to convertToCatalyst but for Java.

[marmbrus]

@adrian-wang any chance you will have time to update this before the 1.2 code freeze this Friday?

[adrian-wang]

Sure, I'll deal with that soon.

[SparkQA]

Test build #22449 has started for PR 2850 at commit 4c4292c.

• This patch merges cleanly.

[SparkQA]

Test build #22449 has finished for PR 2850 at commit 4c4292c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22449/
Test PASSed.