Spark javadoc is incomplete

[jakelandis]

The elasticsearch-spark-* javadoc is missing some packages. For example https://www.javadoc.io/doc/org.elasticsearch/elasticsearch-spark-20_2.11/7.4.0 is missing sql and streaming.

Cracking open the javadoc jar, it appears it is not being included in the jar. https://repo1.maven.org/maven2/org/elasticsearch/elasticsearch-spark-20_2.11/7.4.0/elasticsearch-spark-20_2.11-7.4.0-javadoc.jar

$ls org/elasticsearch/spark/
cfg rdd

[jbaiera]

There is an additional problem here: Last time I checked, the Scaladoc and Javadoc tools do not play well together in terms of making a fully realized documentation artifact from a project that contains both Java and Scala sources. IIRC, we currently rely on only Javadoc outputs which does not include any documentation for Scala classes or packages.

I wonder if its worth it to explore creating both a scaladoc and javadoc output for the project.

[jakelandis]

I started to play around with https://github.com/lightbend/genjavadoc ... so far it looks promising. Assuming I can figure out some gradle weirdness, I will submit a PR soon.

An example screen shot: