[SPARK-12000] Fix API doc generation issues

[JoshRosen]

This pull request fixes multiple issues with API doc generation.

• Modify the Jekyll plugin so that the entire doc build fails if API docs cannot be generated. This will make it easy to detect when the doc build breaks, since this will now trigger Jenkins failures.
• Change how we handle the -target compiler option flag in order to fix javadoc generation.
• Incorporate doc changes from @thunterdb (in [SPARK-12000] Fixes compilation issues with java8 and sbt. #10048).

Closes #10048.

[JoshRosen]

Chatted with @thunterdb and we figured out that the Scala compiler error reported in SPARK-12000 does not occur under Java 8. Tim's Javadoc changes in his PR are required for Javadoc compilation to work under JDK 8, but do not address the issue where Scaladoc compilation fails under Java 7. I'm going to see if I can fix the Java 7 doc build, since we use Java 7 for release and snapshot builds.

In principle, we could use Java 8 to do the snapshot builds while targeting Java 7, but then we'd have to configure out build to use a Java 7 class library in oder to avoid the introduction of subtle incompatibilities caused by Java 8 standard library classes being used.

[JoshRosen]

It looks like Jenkins is able to successfully generate the Scaladocs using Java 7, but my Mac laptop hits that compiler error. I'm going to try upgrading to a newer Java 7 JDK to see if that resolves things.

For posterity, I'm currently encountering the compiler issues while using the following JDK:

[joshrosen ~]$ java -version
java version "1.7.0_65"
Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

[JoshRosen]

Yep, looks like upgrading to a newer JDK fixed things. Here's what worked for me:

java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

Given that using a newer JDK seems to be how we've been able to successfully publish Scaladoc for 1.5.x, I'm going to go ahead and recommend a JDK upgrade as the solution to the Scaladoc issue. Therefore, I think that we're good to merge this and resolve the JIRA issue.

[marmbrus]

LGTM

[JoshRosen]

Going to merge this now into master and branch-1.6.

[SparkQA]

Test build #46907 has finished for PR 10049 at commit 38c1390.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Whoops, forgot the quotes here. Will fix in a followup PR or a hotfix.

[JoshRosen]

Hmm, now I seem to have started having problems even in spite of the JDK update. Will investigate.

[JoshRosen]

Now I'm very confused: it looks like Jenkins is actually using a different Java 7 JDK:

Using /home/jenkins/tools/hudson.model.JDK/JDK_7u60 as default JAVA_HOME.