Inheriting Javadoc from the JDK is broken (as was _linking to_ the JDK)

[cpovirk]

e.g., https://guava.dev/BiMap has blank descriptions for put, putAll, and values, and it has no links for types like java.lang.Object and java.util.Map

[cpovirk]

I can reproduce this locally with ./mvnw clean javadoc:javadoc with JAVA_HOME pointed to Java 11. I can also reproduce it with mvn, so it's not likely to be related to my recent Maven Wrapper change.

[cpovirk]

The info disappeared at bdbcb37, so it's been almost a year :\

A change from around that time that looks plausibly related is 8a676ad. I could believe that the cause is java.specification.version.

[cpovirk]

Yes, I can fix it by removing the java.specification.version change. But that point, Javadoc prints a bunch of warnings under JDK 11, and it fails entirely (or at least pretty badly) under JDK 17.

I could believe that the problem is somehow related to modules. Roughly, the JDK sources might be "hidden" by the JDK modules?

I briefly tried messing with the path we pass to Javadoc, passing ${project.build.directory}/jdk-sources/java.base instead of just ${project.build.directory}/jdk-sources, but that didn't help.

I tried removing our exclusion of the JDK's module-info files, but then I got:

[ERROR] Creating an aggregated report for both named and unnamed modules is not possible.
[ERROR] Ensure that every module has a module descriptor or is a jar with a MANIFEST.MF containing an Automatic-Module-Name.
[ERROR] Fix the following projects:
[ERROR]  - com.google.guava:guava

So it's possible that things would go better if Guava were a real module (#2970). (Presumably Javadoc would be willing to inherit Javadoc across module boundaries.)

I could probably "fix" this by setting the Javadoc source version back to 8 for JDK 11 (where we know that it works). But we know that it doesn't work for JDK 17, so this would be only a temporary fix.

[cpovirk]

(Presumably Javadoc would be willing to inherit Javadoc across module boundaries.)

I guess I'm somewhat pessimistic about that, given that the sources probably are being hidden entirely.

But I could believe that we could do something with --patch-module or something. And in fact, after poking around a little, I found https://bugs.openjdk.org/browse/JDK-8300694. (That bug even notes that this worked for Guava at the latest release but not at head at the time :))

I'm not excited about trying to get this to work, but if anyone wants to try to wire it up, either the temporary solution of using <source>8</source> under JDK11 or the permanent(?) solution of figuring out --module-path, please be my guest.

[cpovirk]

Oh, and in all this, I've been forgetting about the problem linking to the JDK. I would guess (but haven't checked) that that problem is a result of the same commit as the other problem. It may be related to how we still link (well, try to link) to JDK 9, which of course supports the module system but does not yet include the module name in the path of the Javadoc it produces.

[cpovirk]

It looks like #7087 fixes linking in snapshot Javadoc by switching to Java 21 for snapshot-Javadoc generation.

If we want to make things work for our releases, then we can do that by using Java 21 for our releases, too. (Now, I've always been in the habit of using snapshot Javadoc, myself. But I don't know what most users do.)

If we switch to Java 21, we'll have to be careful to continue to generate bytecode that works under older versions (which -target doesn't guarantee). Fortunately, Animal Sniffer usually accomplishes that, such as with a recent problem with SequencedCollection in Java 21+. (See also #6903 and #6832.) That's more important for releases than for snapshots, but we would of course prefer not to break snapshots under older JDKs, either :)

[cpovirk]

(Snapshot docs are built by a different workflow task/whatever than snapshots themselves, so we could also use Java 21 for one but not the other. Releases currently use the same Java version for each.)

[cpovirk]

Wait, no, #7087 (aka #7102 aka f2b8c4f) causes Javadoc generation to fail?

Error: Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/javadoc/Doclet
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1027)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
	at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:524)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:427)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:421)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:420)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
	at jdk.javadoc/jdk.javadoc.internal.tool.Start.loadDocletClass(Start.java:800)
	at jdk.javadoc/jdk.javadoc.internal.tool.Start.preprocess(Start.java:771)
	at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:376)
	at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:347)
	at jdk.javadoc/jdk.javadoc.internal.tool.Main.execute(Main.java:57)
	at jdk.javadoc/jdk.javadoc.internal.tool.Main.main(Main.java:46)
Caused by: java.lang.ClassNotFoundException: com.sun.javadoc.Doclet
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
	... 16 more

I had tested something locally, but I guess it wasn't that. For now, I'll roll back to JDK 11.

[cpovirk]

Linking to the JDK works with a Java 21 (and perhaps earlier versions), so that's a reason to generate docs with a newer version once we can. (We'd want to think about both what we do for snapshots and what we do for releases.) Inheriting docs from the JDK doesn't work automatically, so I'd probably continue not to worry about it.

The only javadoc-21 blocker I remember offhand is JDiff, which I'm assuming is responsible for the error reported just above.

[cpovirk]

I tried using the Java 21 version of Javadoc again after #7516, and it looks like that still works OK (but presumably also still fails if we use it for JDiff). But I do see an error from Javadoc itself if I go all the way to Java 23, mirroring our approach of running the newest javac:

[ERROR] error: Unexpected redirection for URL https://docs.oracle.com/javase/21/docs/api/element-list to https://docs.oracle.com/en/java/javase/21/docs/api/index.html

Changing our configuration to link to Java 23 doesn't help:

[ERROR] error: Unexpected redirection for URL https://docs.oracle.com/javase/23/docs/api/element-list to https://docs.oracle.com/en/java/javase/23/docs/api/index.html

So we probably wouldn't go past Java 21 for Javadoc yet.

(There are also some warnings, which might already arise with Java 11's Javadoc but which (if so) are hidden by repeated warnings: "[WARNING] javadoc: warning - The code being documented uses packages in the unnamed module, but the packages defined in https://docs.oracle.com/en/java/javase/21/docs/api/ are in named modules.")

[edit: As already discussed somewhere or other, using jdk-based activation is the wrong approach for setting (e.g.) Javadoc flags now that we use toolchains. We'd likely have to modify that if we were to change the version we use and/or the JDK that our doc-generating CI job uses to start Maven.]

[cpovirk]

The inability to inherit also leads to these warnings (if we dial up Javadoc warnings enough):

guava/src/com/google/common/util/concurrent/AbstractFuture.java:433: warning: overridden methods do not document exception type java.util.concurrent.CancellationException (package java.util.concurrent class CancellationException)
   * @throws CancellationException {@inheritDoc}
     ^
guava/src/com/google/common/util/concurrent/AbstractFuture.java:547: warning: overridden methods do not document exception type java.util.concurrent.CancellationException (package java.util.concurrent class CancellationException)
   * @throws CancellationException {@inheritDoc}
     ^

I wondered if we could avoid that by omitting the @throws clauses entirely, but it turns out that we can't: I confirmed with a self-contained project that unchecked exceptions appear in the subclass Javadoc only if we use @throws there.

[edit: similarly in ListeningExecutorService with RejectedExecutionException]