Feature request: Separate compilation of module-info.java

[tlinkowski]

Overview

I'd like to request a feature for library developers who need to target JDK 6-8 but who also want to provide a module-info.class (as per Option 3 by @jodastephen).

The proposed feature has two parts:

1. Low-level API: separate compilation of module-info.java.
2. High-level API: easy setting of javac --release option, separately for main Java code and for module-info.java.

Justification

General

Library maintainers need to target JDK 6-8 because these JDKs (especially JDK 8) are still very popular (taking ~95% share in 2018, and predicted to take ~90% in 2019).

Still, those maintainers could make the most of JPMS by:

• providing module-info.class for consumers who put the library on module path,
• compiling module-info.java against the remaining classes of this module and against other modules (which provides better encapsulation and prevents split packages).

So providing a module-info.class is like saying:

Hey, I'm JPMS-compatible! You can safely put me on the module path — I guarantee to have no split packages!

To sum up, I see this feature request as great means of promoting JPMS! Would you agree, @paulbakker, @sandermak?

Examples

Here are some popular libraries that provide module-info.class while targeting JDK 6-8, e.g.:

• Maven:
  • ThreeTen-extra (JDK 9 module-info.class in root dir)
  • Google Gson - not released yet (JDK 9 module-info.class in root dir)
  • SLF4J - not released yet (JDK 9 module-info.class in META-INF/versions/9 - Multi-Release JAR)
• Ant:
  • Lombok (JDK 9 module-info.class in root dir)

There are also some libraries that:

• consider doing that: Add module-info.java google/guava#2970 (in Maven)
• wanted (want?) to do that: Multi-Release JAR that contains Java 8 binaries + Java 9 module-info.class vavr-io/vavr#2230 (in Gradle)
• or even have a separate modular artifact (although I'm not sure why that's necessary): log4j-api-java9 (in Maven)

Vavr /ˈveɪ.vɚ/

Let's have a closer look at vavr's case:

1. It's a quite popular library (3000+ Github stars) targeting JDK 8.
2. Its creator, @danieldietrich, is (or was?) a believer in JPMS.
3. However, as Multi-Release JAR that contains Java 8 binaries + Java 9 module-info.class vavr-io/vavr#2230 (comment) shows, he backed out of it (partly because the required Gradle configuration was too complex).
4. I personally think it's a shame if such a popular library cannot be easily shipped with a module-info.class while still targeting JDK 8.

Proposed DSL

Note: Everything in this section (especially naming) is TBD.

Low-level API

Extra property on moduleOptions extension:

build.gradle

compileJava.moduleOptions.compileModuleInfoSeparately = true

build.gradle.kts

tasks {
  compileJava.moduleOptions.compileModuleInfoSeparately = true
}

High-level API

Special modularity extension with two functions:

• for releasing to JDK 6-8:
  • mixedJavaRelease(int mainJavaRelease, int moduleInfoJavaRelease = 9)
• for releasing to JDK 9+
  • standardJavaRelease(int mainJavaRelease)

For example, the most common configuration (JDK 8 + JDK 9 module-info.class) would be a one-liner:

build.gradle

modularity.mixedJavaRelease 8

build.gradle.kts

modularity.mixedJavaRelease(8)

Proposed behavior

module-info.java location

I propose to leave module-info.java in src/main/java (I don't see any need to put it in a separate source set directory).

module-info.class location

There are two places where module-info.class can end up:

1. in the root output directory (natural; corresponds to module-info.java location)
2. in META-INF/versions/9 (Multi-Release JAR, AKA MRJAR)

Having read a post on MRJARs by @melix, I'm rather suspicious of MRJARs, and so I prefer option 1.

On the other hand, @gunnarmorling claims that it's better to use option 2 because module-info.class "will be out of the way in most cases when being used on an older Java version". But I don't really know why, because as far as I understand, any tool that scans the classpath (like Guava's ClassPath) will return module-info.class no matter where it is (based on its .class extension). @gunnarmorling, would you care to give me a pointer here?

Alternatives

ModiTect

It's fair to mention ModiTect (by @gunnarmorling) and its Gradle plugin (by @siordache) here.

However, ModiTect does much more than I request here: ModiTect actually generates module-info.class from a special notation (there's even no module-info.java there edit: it can actually parse module-info.java). Personally, I find it too complex for my needs.

Badass-Jar plugin

@siordache also created a Gradle plugin that lets one "seamlessly create modular jars that target a Java release before 9".

It looks quite nice, however:

• I'm not sure how it plays along with org.javamodularity.moduleplugin (and I'd like to use it for patching modules, testing on module-path, etc.)
• in order to build the proper JAR and make sure your module-info.java is correct, you actually have to run the build twice (./gradlew jar and ./gradlew -PjavaCompatibility=9 jar) - too complex
• it doesn't use javac's --release option, which (as opposed to using -source and -target) guarantees that only the right APIs are referenced (e.g. it throws when compiling optional.isEmpty() with --release 8)
• it produces only Multi-Release JARs, and I'm somewhat skeptical about them (as I mentioned before)

Related StackOverflow questions

• Compile a JDK 8 project + a JDK 9 “module-info.java” in Gradle (mine)
• Gradle: Building a modularized library that is compatible with Java 8
• Compile Java codebase into 10 and 8 with Gradle
• Java 9 --release flag doesn't appear to be working in Gradle
• Gradle Multi Project with Java 9 and Java 8
• Is it possible to mix Java 8 and Java 9 source code in the same project without using compiler flags?

[sandermak]

Thanks for the great overview. Definitely a good usecase to support!

[danieldietrich]

Its creator, @danieldietrich, is (or was?) a believer in JPMS.

It seems no one is adopting JPMS... Some believe it has no future.

[siordache]

@tlinkowski The badass-jar plugin is probably what you're looking for.

[tlinkowski]

It seems no one is adopting JPMS... Some believe it has no future.

@danieldietrich I guess it's because the adoption of JDK 9+ seems very small yet. But if it was easy to bundle a module-info.class and the popular libraries started doing so, then perhaps in 5-10 years JPMS could be in common use 🙂

[tlinkowski]

@siordache Thanks! I forgot to mention your plugin (I think I stumbled upon it before). I'll edit the overview.

[gunnarmorling]

Hey, thanks for the mention! On this one:

ModiTect actually generates module-info.class from a special notation (there's even no module-info.java there). Personally, I find it too complex for my needs.

This is only one mode of operation in ModiTect. You also can give the contents of a module-info.java file verbatim in your pom.xml or build.gradle file; alternatively, its contents can also be obtained from a given file in the project. This will be compiled as is to the class file, using ASM. This is also the mode that works on JDK < 9. So it seems it already pretty much does what you are after (IIUC)?

[gunnarmorling]

But I don't really know why, because as far as I understand, any tool that scans the classpath (like Guava's ClassPath) will return module-info.class no matter where it is (based on its .class extension).

It depends on the specific implementation of scanning. But we had definitely issues with that, e.g. with older versions of Jetty stumbling upon the file. Note that scanning often isn't done using reflection on the classpath but by "manually" reading class files using ASM or similar. Older tools doing that will usually not be aware of classfiles under META-INF/versions.

[siordache]

I'm not sure how it plays along with org.javamodularity.moduleplugin (and I'd like to use it for patching modules, testing on module-path, etc.)

It actually uses the org.javamodularity.moduleplugin, as mentioned in the README:

"If the plugin detects that at least one of sourceCompatibility and targetCompatibility has an effective value >= 9, it automatically applies the gradle-modules-plugin, which adds support for the JPMS."

it produces only Multi-Release JARs, and I'm somewhat skeptical about them (as I mentioned before)

You can prevent the creation of a multi-release jar by setting multiRelease to false in the jar section of your build script:

jar {
    ...
    multiRelease = false
    ...
}

However, you are right about the plugin not using the --release option and the need to run the build twice to make sure that the module-info.java is correct.

[tlinkowski]

its (=module-info.java file) contents can also be obtained from a given file in the project. This will be compiled as is to the class file, using ASM.

@gunnarmorling Thanks, I didn't know ModiTect can do it!

Still, I'm afraid that using it that way would be a bit like using a sledgehammer to crack a nut 😉 Plus, I'd feel safer to have my module-info.class generated by javac.

---

It depends on the specific implementation of scanning. But we had definitely issues with that, e.g. with older versions of Jetty stumbling upon the file. Note that scanning often isn't done using reflection on the classpath but by "manually" reading class files using ASM or similar. Older tools doing that will usually not be aware of classfiles under META-INF/versions.

As far as I understand, we're talking about tools that scan the classpath for .class files, but deliberately omit META-INF directory. Makes sense! On the other hand, an "older" tool like Guava's ClassPath (which doesn't use reflection - it uses URLClassLoader.getURLs) doesn't make this assumption, and breaks in both cases 😉

To sum up, if I understand correctly:

• Jetty is already fixed to omit module-info.class
• releasing as Multi-Release JAR becomes increasingly less useful (but might still be useful, true)

[tlinkowski]

With #73, I provided a complete implementation of this proposal (with tests & documentation).

Of course, this issue is still open for discussion, and I'm ready to adapt the provided PR as needed.

[gunnarmorling]

Still, I'm afraid that using it that way would be a bit like using a sledgehammer to crack a nut 😉 Plus, I'd feel safer to have my module-info.class generated by javac.

I think there's nothing to be worried about, ASM works just fine. The reasoning for using it btw. was to enable the build itself running on older JDKs, which naturally can't be done when using javac to compile the descriptor. Another thing I'm curious about: how does the suggest approach work exactly, will the classes be compiled twice, once with module descriptor and once without?

Personally, I find [ModiTect] too complex for my needs.

That's interesting, because it's actually intended to reduce complexity. E.g. my thinking is that specifying exported packages via wildcards can help to reduce efforts, similar to "requires" clauses derived from dependencies. So I'm curious which complexity you actually perceive?

Not I'm welcoming this effort and obviously I've no stake in deciding whether this should be included here or not; I'm just curious to see what could be improved in ModiTect to further improve it.

[tlinkowski]

Thanks for your response, Gunnar!

I think there's nothing to be worried about, ASM works just fine.

Agreed that ASM works fine 🙂 What I'm worried about is that Java compiler (javac) is a really complex tool. Here, I simply propose a thin wrapper over this tool. ModiTect, on the other hand, is another complex tool that (in this use case) duplicates what javac does.

Still, ModiTect is a great tool, and I'd use it if the benefits for me:

• "defining dependence clauses based on your project's dependencies" 👍
• "describing exported and opened packages with patterns" 👍
• "auto-detecting service usages" 👍
• "generating module-info.java descriptors" 👍
• complex, highly configurable tool:
  • I can probably do with it what I need 👍

outweighed the costs for me:

• no IntelliJ support for indirect module declarations (I'm spoiled about such things):
  • no instant static type safety 👎
  • no click-through to a given package/class 👎
• complex, highly configurable tool:
  • another thing to learn & maintain its usage 👎
  • higher probability of having issues, e.g. Moditect removes import clauses from input file moditect/moditect#90 👎

    ModiTect is at an early stage of development and it still has some rough edges. Use it at your own risk.

• (strictly project-specific) I have small projects where I export, require, use, and provide little 👎

---

The reasoning for using it btw. was to enable the build itself running on older JDKs, which naturally can't be done when using javac to compile the descriptor.

This is certainly an advantage of ModiTect! Only, I don't need to run the build on an older JDK. I only need to target users who might run on an older JDK.

Another thing I'm curious about: how does the suggest approach work exactly, will the classes be compiled twice, once with module descriptor and once without?

A good question! Actually, every class is compiled only once:

1. First, all classes except module-info.java are compiled with --release 6-8.
2. Then, only module-info.java is compiled with --release 9+ while all the previously compiled classes are accessible to it on the classpath (because both compilation tasks share the same output directory).

That's interesting, because it's actually intended to reduce complexity. E.g. my thinking is that specifying exported packages via wildcards can help to reduce efforts, similar to "requires" clauses derived from dependencies. So I'm curious which complexity you actually perceive?

Yes, these are cool features that reduce complexity of usage (if we don't count lack of IDE support). The complexity I perceive is primarily the complexity of the tool itself (another thing to learn, higher risk of issues).

Not I'm welcoming this effort and obviously I've no stake in deciding whether this should be included here or not; I'm just curious to see what could be improved in ModiTect to further improve it.

ModiTect is a great tool but:

• until Oracle picks your ideas up,
• or at least until there's an IntelliJ plugin for it,
• or unless I had a larger, more complex project where the benefits could be higher,

I'd probably decide against using it.

[paulbakker]

@tlinkowski This is great! Thanks for the detailed writeup and PR. I'm fully onboard with supporting this, seems like a great feature to further increase JPMS adoption.
I'll need a few days before I can give it the attention it deserves (trip coming up this weekend), but I'll get to it as soon as possible.

[paulbakker]

Merged the work done by @tlinkowski! Will be in the next release (which I plan to publish today)