Add scala docs to javadocs

[jakelandis]

This commit introduces a 3rd party plugin to co-locate
scala docs with java docs. https://github.com/lightbend/genjavadoc

Fixes: #1363

[jakelandis]

Use the following commands to test:

./gradlew elasticsearch-spark-20:javadoc
open spark/sql-20/build/docs/javadoc/index.html

./gradlew elasticsearch-spark-13:javadoc
open spark/sql-13/build/docs/javadoc/index.html

[mark-vieira]

While reading some of the documentation I gotta say this sounds pretty gnarly. What's the main motivation of this over just using scaladoc? Certainly there's awkwardness here since we are presenting a Scala API as though it were a Java API? Not to mention we are adding to the already atrocious compile times of the Scala compiler by adding this Java source generation that is only ever used for JavaDoc generation but now runs on every compile.

This motivation also makes no sense to me:

This project’s goal is the creation of real Javadoc for Scala projects. While Scaladoc—the native API documentation format of Scala—has several benefits over Javadoc, Java programmers are very much used to having access to API documentation in a syntax matching their programming language of choice.

But, this isn't Java code, it's Scala. So why are they doing all this hackiness to make a Scala API more amenable to Java programmers? I mean, aren't the consumers of this API Scala developers, and thereby more inclined to want the more robust Scaladoc vs a watered-down and potentially inaccurate Javadoc representation of the API? Is the assumption here that while we are implementing the API in Scala, it can still technically be consumed by Java code thereby targeting a Java audience? Is this accurate in this case? Do we expect the consumers of this stuff to be primarily Java folks or Scala folks?

[mark-vieira]

To add, my comments are mainly rooted in my ignorance of the Scala ecosystem. If this is a common pattern for documentation of APIs implemented in Scala then by all means, I'll defer to the Scala experts here on whether this is a good idea. I simply found this method of JavaDoc generation to be a bit odd and want to ensure we have strong motivations for introducing this since it carries some cost.

[mark-vieira]

We don't need to run scaladoc to generate the javadocs. We just need to compile. This should depend on compileScala instead.

[jakelandis]

since nothing else actually runs the scala docs, I thought that generating them next to the java docs (even if not used) would serve as a good as test. But it is unnecessary and I will change it.

[jakelandis]

I mean, aren't the consumers of this API Scala developers

Nope. Any consumer that wants to use this library with Spark will need to use this API. That could Java or Scala devs. My guess is most are Java devs. If we think most are Scala devs we should look at publishing the scala docs jar.

If this is a common pattern for documentation of APIs implemented in Scala then by all means

Not sure if this is a common pattern, but the library is by from Lightbend formerly known as Typesafe, which is well known.

[mark-vieira]

Nope. Any consumer that wants to use this library with Spark will need to use this API. That could Java or Scala devs. My guess is most are Java devs. If we think most are Scala devs we should look at publishing the scala docs jar.

Gotcha, understood. Yeah, in that case I would agree then that having the API in Javadoc format would be most accessible. I assume while you were testing this the compile-time cost wasn't appreciable? If it is, there are perhaps things we could do, like make the compiler plugin conditional on the javadoc task being part of the task graph.

[jbaiera]

Nope. Any consumer that wants to use this library with Spark will need to use this API. That could Java or Scala devs. My guess is most are Java devs. If we think most are Scala devs we should look at publishing the scala docs jar.

We should probably avoid making these assumptions without any data. My gut feeling is that it's split 40/40/20 across Scala, Python, and Java (respectively), but I don't have any data to back this up other than past issues I can think of.

While reading some of the documentation I gotta say this sounds pretty gnarly.

From what I'm reading, it "emits structurally equivalent Java code for all Scala sources of a project, keeping the Scaladoc comments (with a few format adaptions)." Is this structural Java code created alongside the regular compiler class file output, or is this structural Java code an intermediary step? I'm assuming its the former, but I want us to be sure. Scala has some whacky language features in it that just do not translate well to structural Java code.

[jbaiera]

Nope. Any consumer that wants to use this library with Spark will need to use this API. That could Java or Scala devs.

Is the Java API documentation missing because it's nested under the Scala source? I want to make sure I'm understanding this clearly since I admittedly haven't spent much time looking into the Javadoc/Scaladoc problem for this part of the project.

[jakelandis]

I assume while you were testing this the compile-time cost wasn't appreciable?

The javadoc task doesn't get called until 'pack' , which should only happen during the release, and doesn't add any appreciable time. I also checked compileScala in case the new params impacted it in a funny way, which it does not.

Is this structural Java code created alongside the regular compiler class file output, or is this structural Java code an intermediary step?

I'm not sure I completely follow the question, but the plugin puts the generated java code in a defined directory.

$ls spark/sql-20/build/classes
scala

$ls spark/sql-20/build/generated
java    sources

Just to be certain, I diffed the generated artifacts and the only difference is the addition of the scala javadoc. (the left is with the changes here)

Is the Java API documentation missing because it's nested under the Scala source?

Where is the Java API documentation missing ?

My gut feeling is that it's split 40/40/20 across Scala, Python, and Java (respectively)

We should probably consider adding the proper scala docs jar to the things that published ? However, I don't think this PR and that are mutually exclusive things.

[mark-vieira]

The javadoc task doesn't get called until 'pack' , which should only happen during the release, and doesn't add any appreciable time. I also checked compileScala in case the new params impacted it in a funny way, which it does not.

Right, since this is implemented as a Scala compiler plugin those Java source files are generated whenever the Scala source is compiled, regardless of whether we're actually generated JavaDocs or not, so that's the bit I would expect to have a cost. I suspect one of the most costly parts of the Scala compile process is building the AST, which needs to be done regardless of this generation step so in way it makes sense that it doesn't add much time to the build. 👍

[jbaiera]

I'm not sure I completely follow the question, but the plugin puts the generated java code in a defined directory.

Ok cool, this was my major paranoia. I was mostly worried that the compiler plugin was taking the scala source, turning it into java source, and then converting the resulting java source into byte code/class files. But it sounds like the scala source still compiles to byte code and class files directly, with the added generation of some Java files for use in the javadoc process.

[jbaiera]

One small nit, but it looks like an easy change. Thanks for diving into this!

[jbaiera]

I see that in the sql-13 build file we add these with tasks.withType(...) instead of appending here. I think I'm fine with either, but we should try to do the same in both files.

[jakelandis]

done 2e42e73

[jakelandis]

@jbaiera - should be ready for another look. thanks!

[jbaiera]

LGTM