External Java command line tool "launcher" feature

[sormuras]

As a follow-up from #10 I'd like to discuss what's the best (read: pro) way to implement an external Java command line tool launcher. A requirement for such a feature is that it resolves tool's dependencies and executes the tool in its own VM with custom main parameters.

What I'd like to write in a "build.pro" file is something like this pseudo-code:

ExternalTool junit = ExternalTool.builder().withName("junit").withDependencies(list(
  "org.junit.platform:junit-platform-console-standalone:1.0.0-SNAPSHOT"
)).build()
int code = junit.start("--selectTag", "fast")
if (code == 0) code = junit.start("--selectTag", "slow")
// ... handle failures/erros/etc

Or with google-java-format as another example:

ExternalTool format= ExternalTool.builder().withName("format").withDependencies(list(
  "com.google.googlejavaformat:google-java-format:1.3"
)).build()
format.start(Pro.allJavaSourceFilesAsFlatStringArray())

On the disk, I imagine following runtime directories/files:

deps/
  external-tool/
    junit/ (name either generated or given by user)
      junit.platform.console.standalone.jar
   format/
      google.java.format.jar
      guava.jar
      errorprone.jar

Can I (re-)use the existing "runner" plugin to achive this goal?

Note: there's no dependency of project's modules to the external tools.

[forax]

Your external launcher is more or less pro itself.

For google.java.format, you can create a plugin just for it, formatter ?
For junit, it should be another plugin, tester?, that find all modules in the tests have a dependency on junit (on junit.platform.console.standalone ?) and run them.

You can not reuse directly the runner plugin, because i want each plugin to be independent, obviously you can cut&paste the code you want, if there is too much duplication, we will move the common part in some helpers.

[sormuras]

Started to work on formatter plugin: master...sormuras:formatter
If I drop the "all-deps.jar" from here into pro's deps/ folder, I get:

[pro] registered plugins compiler, convention, formatter, linker, modulefixer, packager, resolver, runner, uberpackager
[modulefixer] multiple modules [java.xml.ws.annotation@9-ea, google.java.format@1.3-all-deps] contains package javax/annotation

[sormuras]

That's why I'm proposing a generic "java tool downloader, resolver and launcher" plugin that an user may configure at "script execution time" ... separated from pro itself by using its own namespace (read module path) below "deps/external-tool/".

[sormuras]

Now. Having Pro.local(action): how do I implement a "local tool runner"? ;)

[forax]

Before doing that, i want to experiment how to load a plugin without having its dependencies to be melt with the dependencies of another plugin.

One way to do that without local() is to do a call that will exec pro in another VM.

[forax]

Ok, i've commited 5dd83b0 which allows to load different plugins at runtime, each one being sandboxed
(see the runner plugin for an exemple)

[sormuras]

Great! Locally, pro builds.

Why does Travis fail: https://travis-ci.org/forax/pro/builds/225126486#L235 ?
Just saw your latest commit: cd49be1

[sormuras]

I started with a fresh "formatter" skeleton at master...sormuras:plugins_formatter -- which already prints:

[pro] registered plugins compiler, convention, formatter, linker, modulefixer, packager, resolver, runner, uberpackager
Formatting... soon.
[pro] DONE !          elapsed time 0 ms

But as soon as I drop an external artifact (non modular) JAR into {PRO_HOME}/plugins/formatter/deps, the pro binary gets corrupted. Pro fails in static initializer in line Plugins.getAllPlugins() without a stacktrace.

I guess, the artifacts created by https://github.com/google/google-java-format are not "auto-module-able". Will try to use the old classpath here...

[forax]

at least
set("compiler.moduleDependencyPath", path("deps", "plugins/formatter/../../target/main/artifact/", "plugins/formatter/../../deps"));
should be
set("compiler.moduleDependencyPath", path("plugins/formatter/deps", "plugins/formatter/../../target/main/artifact/", "plugins/formatter/../../deps"));

also, you can put a giant try/catch around the static initializer to get the right exception

[sormuras]

Ja, that fixed it. Now, the external module is found.

Why is the "runner" line at https://github.com/forax/pro/blob/master/src/main/java/com.github.forax.pro.bootstrap/com/github/forax/pro/bootstrap/Bootstrap.java#L94 only ... path("deps", ...

[forax]

because the runner has currently no external dependency

[sormuras]

Understood.

As I have currently more knowledge (and control) over JUnit, I started the "tester" plugin in parallel. It almost builds. It does build, when you drop the standalone artifact from here into the plugins/tester/deps directory manually.
Is the "resolver" not yet working with "local" or are core module/plugin dependencies not resolved by the bootstrap process?

[forax]

it should work :(
your junit.platform.console.standalone is a modular jar or not ?

[sormuras]

It's a plain old jar. You may drop junit-platform-console-standalone-1.0.0-20170422.164334-149.jar into plugins/tester/deps ... and the modulefixer injects the module-info.class as expected. And the tester plugin prints out the help text.

[sormuras]

After the help text, a simple Pro.run("tester") outputs locally:

org.junit.platform.commons.util.PreconditionViolationException: Cannot create Launcher without at least one TestEngine; consider adding an engine implementation JAR to the classpath
        at junit.platform.console.standalone@1.0.0-20170422.164334-149/org.junit.platform.commons.util.Preconditions.condition(Preconditions.java:285)
        at junit.platform.console.standalone@1.0.0-20170422.164334-149/org.junit.platform.launcher.core.DefaultLauncher.<init>(DefaultLauncher.java:55)
        at junit.platform.console.standalone@1.0.0-20170422.164334-149/org.junit.platform.launcher.core.LauncherFactory.create(LauncherFactory.java:53)
        at junit.platform.console.standalone@1.0.0-20170422.164334-149/org.junit.platform.console.tasks.ConsoleTestExecutor.executeTests(ConsoleTestExecutor.java:61)
        at junit.platform.console.standalone@1.0.0-20170422.164334-149/org.junit.platform.console.tasks.ConsoleTestExecutor.lambda$execute$0(ConsoleTestExecutor.java:57)
        at junit.platform.console.standalone@1.0.0-20170422.164334-149/org.junit.platform.console.tasks.CustomContextClassLoaderExecutor.invoke(CustomContextClassLoaderExecutor.java:33)
        at junit.platform.console.standalone@1.0.0-20170422.164334-149/org.junit.platform.console.tasks.ConsoleTestExecutor.execute(ConsoleTestExecutor.java:57)
        at junit.platform.console.standalone@1.0.0-20170422.164334-149/org.junit.platform.console.ConsoleLauncher.executeTests(ConsoleLauncher.java:85)
        at junit.platform.console.standalone@1.0.0-20170422.164334-149/org.junit.platform.console.ConsoleLauncher.execute(ConsoleLauncher.java:75)
        at junit.platform.console.standalone@1.0.0-20170422.164334-149/org.junit.platform.console.ConsoleLauncher.execute(ConsoleLauncher.java:48)
        at com.github.forax.pro.plugin.tester@0.9/com.github.forax.pro.plugin.tester.TesterPlugin.execute(TesterPlugin.java:35)
        at com.github.forax.pro@0.9/com.github.forax.pro.Pro.execute(Pro.java:325)
        at com.github.forax.pro@0.9/com.github.forax.pro.Pro.runAll(Pro.java:305)
        at com.github.forax.pro@0.9/com.github.forax.pro.Pro.run(Pro.java:295)
        at REPL.$JShell$14.do_it$($JShell$14.java:4)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:563)
        at jdk.jshell/jdk.jshell.execution.DirectExecutionControl.invoke(DirectExecutionControl.java:198)
        at jdk.jshell/jdk.jshell.execution.RemoteExecutionControl.invoke(RemoteExecutionControl.java:107)
        at jdk.jshell/jdk.jshell.execution.DirectExecutionControl.invoke(DirectExecutionControl.java:108)
        at jdk.jshell/jdk.jshell.execution.ExecutionControlForwarder.processCommand(ExecutionControlForwarder.java:134)
        at jdk.jshell/jdk.jshell.execution.ExecutionControlForwarder.commandLoop(ExecutionControlForwarder.java:225)
        at jdk.jshell/jdk.jshell.execution.Util.forwardExecutionControl(Util.java:76)
        at jdk.jshell/jdk.jshell.execution.Util.forwardExecutionControlAndIO(Util.java:137)
        at jdk.jshell/jdk.jshell.execution.RemoteExecutionControl.main(RemoteExecutionControl.java:70)

The generated "module-info.class" looks good, though:

open module junit.platform.console.standalone {
    requires java.xml;
    requires java.base;
    requires java.logging;
    requires java.sql;

    exports org.junit.experimental.categories;
    exports org.junit.platform.console.options;
    exports org.junit.platform.engine.support.hierarchical;
    exports org.junit.platform.console;
[...]
    exports org.junit.runners.parameterized;
    exports org.junit.internal;

    uses org.junit.platform.engine.TestEngine;
    uses org.junit.jupiter.api.extension.Extension;
    uses org.junit.platform.launcher.TestExecutionListener;

    provides org.junit.platform.engine.TestEngine with org.junit.jupiter.engine.JupiterTestEngine, org.junit.vintage.engine.VintageTestEngine;
}