Documentation on testing toolchain changes is out of date

[cushon]

@comius

This page is out of date: https://github.com/bazelbuild/java_tools/blob/master/docs/behind-the-release.md#manually-trying-out-java_tools-before-the-release-process

• The //src:java_tools_javaX.zip target doesn't exist anymore (I think it's been replaced by e.g. //src:java_tools_zip).

• It refers to --java_toolchain, which doesn't work with bazel at head due to the switch to toolchain resolution.

[cushon]

I think the current process is something like the following. @comius does that look right, or is there a better way to do this?

• From the main Bazel repo, build //src:java_tools_zip and stash it somewhere:

  $ bazel build //src:java_tools_zip
  $ cp bazel-bin/src/java_tools.zip /tmp/
  

• Patch distdir_deps.bzl to refer to the local java_tools_zip:

  --- a/distdir_deps.bzl
  +++ b/distdir_deps.bzl
  @@ -267,11 +267,10 @@ DIST_DEPS = {
               "remote_java_tools_test",
               "remote_java_tools_for_testing",
           ],
  -        "archive": "java_tools-v11.3.zip",
  -        "sha256": "52b66d8df456f2ce057d8e435904789463df8414af8110aa13af68ce32d8c4cc",
  +        "archive": "java_tools3.zip",
  +        "sha256": "6e59059c571571528d7e5e264d6ec6bba4ba570241f71fc6d31f17cbbcb30fc8",
           "urls": [
  -            "https://mirror.bazel.build/bazel_java_tools/releases/java/v11.3/java_tools-v11.3.zip",
  -            "https://github.com/bazelbuild/java_tools/releases/download/java_v11.3/java_tools-v11.3.zip",
  +            "file:///tmp/java_tools.zip",
           ],
           "used_in": [
               "additional_distfiles",

• Build and run the patched Bazel

[comius]

I didn't see this issue. I will update docs. Yes, a lot has changed, for example toolchain definitions are not anymore in java_tools.

or is there a better way to do this?

This is the best way to run all the tests.

You could also unzip java_tools and add --override_repository=java_tools=/tmp/java_tools to just use them in a build for arbitrary project.

[cushon]

Does --override_repository=java_tools=/tmp/java_tools still work? I tried that and it doesn't seem to do anything.

The approach in #43 (comment) still works for me.

[cushon]

You could also unzip java_tools and add --override_repository=java_tools=/tmp/java_tools to just use them in a build for arbitrary project.

Does --override_repository=java_tools=/tmp/java_tools still work? I tried that and it doesn't seem to do anything.

@comius ping? It would be nice to have a way to test java_tools changes without editing distdir_deps.bzl and rebuilding bazel. Is --override_repository= still a supported way to do that? Maybe I'm just holding it wrong, but it doesn't work for me.

The following does not pick up any local modifications to java_tools:

bazel build //src:java_tools_zip
unzip -q -o -d /tmp bazel-bin/src/java_tools.zip
bazel build --override_repository=java_tools=/tmp/java_tools //src:bazel

[comius]

The mechanism should still work, except that you need to override both java_tools and java_tools_linux repositories.
You build zips with bazel build //src:java_tools_zip and bazel build //src:java_tools_prebuilt_zip.

[cushon]

Is this the right idea? I can't get it to work:

bazel build //src:java_tools_zip //src:java_tools_prebuilt_zip
unzip -q -o -d /tmp bazel-bin/src/java_tools.zip
unzip -q -o -d /tmp/java_tools_prebuilt bazel-bin/src/java_tools_prebuilt.zip
bazel build --override_repository=java_tools=/tmp/java_tools --override_repository=java_tools_prebuilt=/tmp/java_tools_prebuilt //src:bazel

I'm testing a repro for bazelbuild/bazel#14093, which causes JavaBuilder to unconditionally OOM, and the only way I can see the behaviour I'm introducing is by patching distdir_deps.bzl

[oliviernotteghem]

Facing the same issue. Unfortunately, your workaround @cushon doesn't seem to work anymore (we're on Bazel 5.2). Getting the following error. Does the workaround still work for you?

ERROR: /Users/oliviern/Uber/android/build/.bazelw_cache/91a99194e71852b0e089ca5cc1c48073/external/rules_java/toolchains/BUILD:116:14: no such target '@remote_java_tools_darwin//:ijar_prebuilt_binary': target 'ijar_prebuilt_binary' not declared in package '' defined by /Users/oliviern/Uber/android/build/.bazelw_cache/91a99194e71852b0e089ca5cc1c48073/external/remote_java_tools_darwin/BUILD and referenced by '@rules_java//toolchains:ijar_prebuilt_binary_darwin'

[oliviernotteghem]

OK, there was 2 issues :

1. WORKSPACE file is missing from zip file generate from //src:java_tools_zip (you can copy over manually the default one)
2. --override_repository=java_tools=/tmp/java_tools should be --override_repository=remote_java_tools=/tmp/java_tools

After that, I can verify the files located at /tmp/java_tools are copied over to /external/remote_java_tools/java_tools and are used.

Seeing this error now (which can be temporary worked around by removing ErrorPronePlugin manually) :

Exception in thread "main" java.lang.NoSuchMethodError: 'com.google.common.collect.ImmutableMap com.google.common.collect.ImmutableMap$Builder.buildOrThrow()'
	at com.google.errorprone.scanner.ScannerSupplier.defaultSeverities(ScannerSupplier.java:64)
	at com.google.errorprone.scanner.ScannerSupplier.fromBugCheckerInfos(ScannerSupplier.java:85)
	at com.google.errorprone.scanner.BuiltInCheckerSuppliers.allChecks(BuiltInCheckerSuppliers.java:566)
	at com.google.errorprone.scanner.BuiltInCheckerSuppliers.errorChecks(BuiltInCheckerSuppliers.java:583)
	at com.google.devtools.build.buildjar.javac.plugins.errorprone.ErrorPronePlugin.<init>(ErrorPronePlugin.java:56)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.parse(BazelJavaBuilder.java:126)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.parseAndBuild(BazelJavaBuilder.java:76)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:65)

[cushon]

For posterity, I think the complete steps for java_tools are something like:

bazel build //src:java_tools_zip
mkdir /tmp/java_tools/
unzip -q -o -d /tmp/java_tools/ bazel-bin/src/java_tools.zip
touch /tmp/java_tools/WORKSPACE

bazel build --override_repository=remote_java_tools=/tmp/java_tools ...

And for java_tools_linux:

bazel build //src:java_tools_prebuilt_zip
mkdir /tmp/java_tools_linux/
unzip -q -o -d /tmp/java_tools_linux/ bazel-bin/src/java_tools_prebuilt.zip
touch /tmp/java_tools_linux/WORKSPACE

--override_repository=rules_java~7.3.2~toolchains~remote_java_tools_linux=/tmp/java_tools_linux

I think bzlmod changes mean it's necessary to override version (e.g. the ~7.3.2~ in the name), maybe there's a cleaner alternative for that.

[cushon]

Trying to do this again for bazelbuild/bazel#23246, the incantation for java_tools (I didn't need to modify architecture specific tools this time) was

bazel build --override_repository=rules_java++toolchains+remote_java_tools=/tmp/java_tools ...

To test a local build of turbine I also installed turbine to the local maven repo

mvn install

and updated Bazel to refer to it:

$ git diff MODULE.bazel
diff --git a/MODULE.bazel b/MODULE.bazel
index afabcbe5ff..73b938896c 100644
--- a/MODULE.bazel
+++ b/MODULE.bazel
@@ -113,7 +113,7 @@ maven.install(
         "com.google.http-client:google-http-client:1.42.0",
         "com.google.http-client:google-http-client-gson:1.42.0",
         "com.google.j2objc:j2objc-annotations:1.3",
-        "com.google.turbine:turbine:0.6.0",
+        "com.google.turbine:turbine:HEAD-SNAPSHOT",
         "com.guardsquare:proguard-base:jar:7.5.0",
         "com.ryanharter.auto.value:auto-value-gson-extension:1.3.1",
         "com.ryanharter.auto.value:auto-value-gson-factory:1.3.1",
@@ -199,6 +199,7 @@ maven.install(
     fail_if_repin_required = True,
     lock_file = "//:maven_install.json",
     repositories = [
+        "m2local",
         "https://repo1.maven.org/maven2",
     ],
     strict_visibility = True,

Running REPIN=1 bazel run @maven//:pin failed because mvn install doesn't write checksum files anymore, so I manually added the missing checksums with bash.

Suggestions for a better way I could have accomplished that are welcome.