Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Bzlmod and add rules_scala to bazel-central-registry #1482

Open
sfc-gh-pbennes opened this issue Mar 16, 2023 · 35 comments · Fixed by #1619
Open

Support Bzlmod and add rules_scala to bazel-central-registry #1482

sfc-gh-pbennes opened this issue Mar 16, 2023 · 35 comments · Fixed by #1619

Comments

@sfc-gh-pbennes
Copy link

sfc-gh-pbennes commented Mar 16, 2023

Greetings!

It looks like rules_scala isn't part of the Bzlmod effort or added to bazel-central-registry yet. I've opened an issue both here and in https://github.com/bazelbuild/bazel-central-registry to request it: bazelbuild/bazel-central-registry#522

@chrislovecnm
Copy link

Just a note to anyone that starts the work native.register_toolchains is not supported when using bzlmod.

@sluongng
Copy link

@sluongng
Copy link

Yeah you would only want to prepare the toolchain (as a new repository) inside the extension I think. Registering it must be called from the MODULE.bazel file.

So native.register_toolchains call from the extension's starlark does not work, but having the extension create @pythons_hub//:all and then MODULE.bazel calling register_toolchains("@pythons_hub//:all") worked. 👍

@mateuszkuta256
Copy link
Contributor

I prepared a minimal bzlmod support here and gonna split it into a few PRs
TODOs after merging it:

  • write documentation on how to use rules_scala with bzlmod (some properties like SCALA_VERSION will probably be configurable via repo-env)
  • prepare release notes
  • contribute repo to the Bazel Central Registry
  • add tests that the project builds using both WORKSPACE and MODULE.bazel
  • migrate test/external workspaces to blzmod too

@pcj
Copy link
Member

pcj commented Jun 30, 2024

@mateuszkuta256 thanks for getting this started! Any progress updates or blockers?

@mateuszkuta256
Copy link
Contributor

@mateuszkuta256 thanks for getting this started! Any progress updates or blockers?

hi, unfortunately I'm working on other things now and won't continue with this PR in the foreseeable future
I remember this migration was on hold because interfered with 'cross compilation support'
It looks like great progress was done it this area, so maybe someone can already take over the PR

@mbland
Copy link
Contributor

mbland commented Oct 2, 2024

I'd like to take on the task of Bzlmodifying this repo through a series of pull requests.

I've already created a (mostly) working branch in my own fork. Though I saw a couple draft pull requests here, I ended up taking a different approach and got it mostly working. In fact, I did exactly what @sluongng suggested in #1482 (comment). (I didn't notice this comment before just now—I might've read it, but not understood it at the time—but I did study rules_python and rules_go, and ended up doing exactly that.)

There are still issues I need help to address (recorded in some of the commit messages), I didn't strictly maintain WORKSPACE compatibility (fixable), and I only tested against Scala 2.13. But I can start teasing chunks out of it in a methodical fashion, to ensure that I maintain WORKSPACE and cross-version compatibility.

For an example of what it looks like from a client perspective, here's what the MODULE.bazel stanza from my local EngFlow/example repo looks like (as opposed to the current stanza, explanatory comment notwithstanding):

bazel_dep(name = "rules_scala", repo_name = "io_bazel_rules_scala")
local_path_override(
    module_name = "rules_scala",
    path = "../../bazelbuild/rules_scala"
)   

scala_dev_deps = use_extension(
    "@io_bazel_rules_scala//scala/extensions:deps.bzl",
    "scala_deps",
    dev_dependency = True,
)   
scala_dev_deps.toolchains(
    scalatest = True,
)

So if folks are game for me to do this, I'll start carving off pieces as separate pull requests, and we can resolve any outstanding problems in the process.

cc: @BillyAutrey @jayconrod @benjaminp @TheGrizzlyDev

mbland added a commit to mbland/rules_scala that referenced this issue Oct 3, 2024
This begins the Bzlmod compatibility migration by updating Bazel to
version 7.3.2 and adding initial `MODULE.bazel` and `WORKSPACE.bzlmod`
files.

Part of: bazelbuild#1482

Though Bzlmod remains disabled, updating to Bazel 7.3.2 requred updating
or adding the following packages to maintain `WORKSPACE` compatibility.

In `rules_scala_setup()`:

- bazel_skylib: 1.4.1 => 1.7.1
- rules_cc: 0.0.6 => 0.0.10
- rules_java: 5.4.1 => 7.9.0
- rules_proto: 5.3.0-21.7 => 6.0.2

Dev dependencies in `WORKSPACE`:

- com_google_protobuf: 28.2
- rules_pkg: 1.0.1
- rules_jvm_external: 6.4
- com_google_absl: abseil-cpp-20240722.0
- zlib: 1.3.1

Of all of the new, explicit dev dependencies, only `com_google_protobuf`
will be necessary to include in `MODULE.bazel`. The Bzlmod mechanism
will discover these other transitive dev dependencies automatically.

Also removed the `rules_java_extra` repo from `WORKSPACE`, which
appeared unused.

---

Though the current `rules_java` version is 7.12.1, and largely works
with this repo, it requires a few temporary workarounds. Rather than
commit the workarounds, upgrading only to 7.9.0 now seems less crufty.

What follows is a very detailed explanation of what happens with 7.12.1
with Bazel 7.3.2, just to have it on the record.

---

The workaround is to change a few toolchain and macro file targets from
`@bazel_tools//tools/jdk:` to `@rules_java//toolchains:`. This isn't a
terribly bad or invasive workaround, but `@bazel_tools//tools/jdk:` is
clearly the canonical path. Best to keep it that way, lest we build up
technical debt.

Without the workaround, these targets would fail:

- //test/src/main/resources/java_sources:CompiledWithJava11
- //test/src/main/resources/java_sources:CompiledWithJava8
- //test/toolchains:java21_toolchain
- //test:JunitRuntimePlatform
- //test:JunitRuntimePlatform_test_runner
- //test:scala_binary_jdk_11

with this error:

```txt
ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14:

While resolving toolchains for target
@@rules_java_builtin//toolchains:platformclasspath (096dcc8):

No matching toolchains found for types
@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type.
```

This appears to be a consequence of both upgrading the Bazel version
from 6.3.0 to 7.3.2 and updating `rules_java` to 7.12.1. The
`rules_java_builtin` repo is part of the `WORKSPACE` prefix that adds
implicit dependencies:

- https://bazel.build/external/migration#builtin-default-deps

This repo was added to 7.0.0-pre.20231011.2 in the following change,
mapped to `@rules_java` within the scope of the `@bazel_tools` repo:

- bazelbuild/bazel: Add rules_java_builtin to the users of Java modules
  bazelbuild/bazel@ff1abb2

This change tried to ensure `rules_java` remained compatible with
earlier Bazel versions. However, it changed all instances of
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` to
`//toolchains:bootstrap_runtime_toolchain_type`:

- bazelbuild/rules_java: Make rules_java backwards compatible with Bazel
  6.3.0
  bazelbuild/rules_java@30ecf3f

Bazel has bumped `rules_java` in its `workspace_deps.bzl` from 7.9.0 to
7.11.0, but it's only available as of 8.0.0-pre.20240911.1.

- bazelbuild/bazel: Update rules_java 7.11.1 / java_tools 13.8
  bazelbuild/bazel#23571
  bazelbuild/bazel@f92124a

---

What I believe is happening is, under Bazel 7.3.2 and `rules_java`
7.12.1:

- Bazel creates `rules_java` 7.9.0 as `@rules_java_builtin` in the
  `WORKSPACE` prefix.

- `@bazel_tools` has `@rules_java` mapped to `@rules_java_builtin` when
  initialized during the `WORKSPACE` prefix, during which
  `@bazel_tools//tools/jdk` registers `alias()` targets to
  `@rules_java` toolchain targets. These aliased toolchains specify
  `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` in their
  `toolchains` attribute.

- `WORKSPACE` loads `@rules_java` 7.12.1 and registers all its
  toolchains with type
  `@rules_java//toolchains:bootstrap_runtime_toolchain_type`.

- Some `@rules_java` rules explicitly specifying toolchains from
  `@bazel_tools//tools/jdk` can't find them, because the
  `@bazel_tools//tools/jdk` toolchain aliases expect toolchains of type
  `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`.

This has broken other projects in the same way:

- bazelbuild/bazel: [Bazel CI] Downstream project broken by rules_java
  upgrade #23619
  bazelbuild/bazel#23619

These problems don't appear under Bzlmod, and `@rules_java_builtin` was
never required. This is because `WORKSPACE` executes its statements
sequentially, while Bzlmod builds the module dependency graph _before_
instantiating repositories (within module extensions).

It seems a fix is on the way that removes `@rules_java_builtin` from the
`WORKSPACE` prefix, and adds `@rules_java` to the suffix. At this
moment, though, it's not even in a prerelase:

- bazelbuild/bazel: Remove rules_java_builtin in WORKSPACE prefix
  bazelbuild/bazel@7506690

---

Note that the error message matches that from the following resolved
issue, but that issue was for non-Bzlmod child repos when `WORKSPACE`
was disabled.

- bazelbuild/bazel: Undefined @@rules_java_builtin repository with
  --noenable_workspace option
  bazelbuild/bazel#22754
mbland added a commit to mbland/rules_scala that referenced this issue Oct 5, 2024
Related to bazelbuild#1482, bazelbuild#1618, and bazelbuild#1619. Results from the investigation
documented at:

- bazelbuild#1619 (comment)

Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle
differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources`
values between Bazel 6 and Bazel 7.

Without this change, `_import_paths()` emits incorrect values under
Bazel 7, causing targets containing generated `.proto` inputs to fail,
e.g.  `//test/proto3:test_generated_proto`.

See also:

- Fix paths for sibling repository setup and generated .proto files
  bazelbuild/bazel@6c6c196

- The docstring for `ProtoInfo.proto_source_root` in the Bazel sources:
  https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172

- Remove incompatible_generated_protos_in_virtual_imports
  bazelbuild/bazel@3efaa32

- Comment from: Generated Protos are no longer considered as
  virtual_imports in Bazel 7
  bazelbuild/bazel#21075 (comment)

---

I cherrypicked this commit into bazelbuild#1618. While it fixed the
`//test/proto3` build failure, it does _not_ fix the hanging
scalapb_workers from the ProtoScalaPBRule aspect.

I'll have to investiate further whether than hang is related to Bazel,
rules_proto, com_google_protobuf, or some mixture thereof. Still,
progress!
mbland added a commit to mbland/rules_scala that referenced this issue Oct 5, 2024
Related to bazelbuild#1482, bazelbuild#1618, and bazelbuild#1619. Results from the investigation
documented at:

- bazelbuild#1619 (comment)

Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle
differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources`
values between Bazel 6 and Bazel 7.

Without this change, `_import_paths()` emits incorrect values under
Bazel 7, causing targets containing generated `.proto` inputs to fail,
e.g.  `//test/proto3:test_generated_proto`.

See also:

- Fix paths for sibling repository setup and generated .proto files
  bazelbuild/bazel@6c6c196

- The docstring for `ProtoInfo.proto_source_root` in the Bazel sources:
  https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172

- Remove incompatible_generated_protos_in_virtual_imports
  bazelbuild/bazel@3efaa32

- Comment from: Generated Protos are no longer considered as
  virtual_imports in Bazel 7
  bazelbuild/bazel#21075 (comment)

---

I cherrypicked this commit into bazelbuild#1618. While it fixed the
`//test/proto3` build failure, it does _not_ fix the hanging
scalapb_workers from the ProtoScalaPBRule aspect.

I'll have to investiate further whether than hang is related to Bazel,
rules_proto, com_google_protobuf, or some mixture thereof. Still,
progress!
mbland added a commit to mbland/rules_scala that referenced this issue Oct 6, 2024
Related to bazelbuild#1482, bazelbuild#1618, and bazelbuild#1619. Results from the investigation
documented at:

- bazelbuild#1619 (comment)

Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle
differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources`
values between Bazel 6 and Bazel 7.

Without this change, `_import_paths()` emits incorrect values under
Bazel 7, causing targets containing generated `.proto` inputs to fail,
e.g.  `//test/proto3:test_generated_proto`.

See also:

- Fix paths for sibling repository setup and generated .proto files
  bazelbuild/bazel@6c6c196

- The docstring for `ProtoInfo.proto_source_root` in the Bazel sources:
  https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172

- Remove incompatible_generated_protos_in_virtual_imports
  bazelbuild/bazel@3efaa32

- Comment from: Generated Protos are no longer considered as
  virtual_imports in Bazel 7
  bazelbuild/bazel#21075 (comment)

---

I cherrypicked this commit into bazelbuild#1618. While it fixed the
`//test/proto3` build failure, it does _not_ fix the hanging
scalapb_workers from the ProtoScalaPBRule aspect.

I'll have to investiate further whether than hang is related to Bazel,
rules_proto, com_google_protobuf, or some mixture thereof. Still,
progress!
mbland added a commit to mbland/rules_scala that referenced this issue Oct 7, 2024
Related to bazelbuild#1482, bazelbuild#1618, and bazelbuild#1619. Results from the investigation
documented at:

- bazelbuild#1619 (comment)

Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle
differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources`
values between Bazel 6 and Bazel 7.

Without this change, `_import_paths()` emits incorrect values under
Bazel 7, causing targets containing generated `.proto` inputs to fail,
e.g.  `//test/proto3:test_generated_proto`.

See also:

- Fix paths for sibling repository setup and generated .proto files
  bazelbuild/bazel@6c6c196

- The docstring for `ProtoInfo.proto_source_root` in the Bazel sources:
  https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172

- Remove incompatible_generated_protos_in_virtual_imports
  bazelbuild/bazel@3efaa32

- Comment from: Generated Protos are no longer considered as
  virtual_imports in Bazel 7
  bazelbuild/bazel#21075 (comment)

---

I cherrypicked this commit into bazelbuild#1618. While it fixed the
`//test/proto3` build failure, it does _not_ fix the hanging
scalapb_workers from the ProtoScalaPBRule aspect.

I'll have to investiate further whether than hang is related to Bazel,
rules_proto, com_google_protobuf, or some mixture thereof. Still,
progress!
mbland added a commit to mbland/rules_scala that referenced this issue Oct 7, 2024
Part of bazelbuild#1482.

Splits the last component off of canonical repo names to produce the
expected repo name.

Without Bzlmod, it returns the original name. With Bzlmod enabled, it
avoids generating output like:

    scala_import(
        name = "_main~scala_deps~io_bazel_rules_scala_scala_compiler",
        jars = ["scala-compiler-2.12.18.jar"],
    )

resulting in errors like:

```
ERROR: .../_main~_repo_rules~io_bazel_rules_scala/scala/BUILD:
no such target '@@_main~scala_deps~io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler':
target 'io_bazel_rules_scala_scala_compiler' not declared in package ''
defined by .../_main~scala_deps~io_bazel_rules_scala_scala_compiler/BUILD
and referenced by '@@_main~_repo_rules~io_bazel_rules_scala//scala:default_toolchain_scala_compile_classpath_provider'
```

Also fixes the following error when attaching resources from custom repos to
targets under Bzlmod:

```txt
$ bazel test //test/src/main/scala/scalarules/test/resources:all

1) Scala library depending on resources from external resource-only
  jar::allow to load resources(scalarules.test.resources.ScalaLibResourcesFromExternalDepTest)
  java.lang.NullPointerException
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.get(ScalaLibResourcesFromExternalDepTest.scala:17)
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$3(ScalaLibResourcesFromExternalDepTest.scala:11)
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$2(ScalaLibResourcesFromExternalDepTest.scala:11)
```

Can be replaced with a future bazel-skylib implementation, if accepted
into that repo.

---

We can't rely on the specific canonical repository name format:

> Repos generated by extensions have canonical names in the form of
> `module_repo_canonical_name+extension_name+repo_name`. For extensions
> hosted in the root module, the `module_repo_canonical_name` part is
> replaced with the string `_main`. Note that the canonical name format is
> not an API you should depend on — it's subject to change at any time.
>
> - https://bazel.build/external/extension#repository_names_and_visibility

The change to no longer encode module versions in canonical repo names in
Bazel 7.1.0 is a recent example of Bazel maintainers altering the format:

- bazelbuild/bazel#21316

And the maintainers recently replaced the `~` delimiter with `+` in the
upcoming Bazel 8 release due to build performance issues on Windows:

- bazelbuild/bazel#22865

This function assumes the only valid `repo_name` characters are letters,
numbers, '_', '-', and '.'. It finds the last character not in this set, and
returns the contents of `name` following this character. This is valid so
long as this condition holds:

- https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/java/com/google/devtools/build/lib/cmdline/RepositoryName.java#L159-L162
mbland added a commit to mbland/rules_scala that referenced this issue Oct 7, 2024
Part of bazelbuild#1482.

Splits the last component off of canonical repo names to produce the
expected repo name.

Without Bzlmod, it returns the original name. With Bzlmod enabled, it
avoids generating output like:

    scala_import(
        name = "_main~scala_deps~io_bazel_rules_scala_scala_compiler",
        jars = ["scala-compiler-2.12.18.jar"],
    )

resulting in errors like:

```
ERROR: .../_main~_repo_rules~io_bazel_rules_scala/scala/BUILD:
no such target '@@_main~scala_deps~io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler':
target 'io_bazel_rules_scala_scala_compiler' not declared in package ''
defined by .../_main~scala_deps~io_bazel_rules_scala_scala_compiler/BUILD
and referenced by '@@_main~_repo_rules~io_bazel_rules_scala//scala:default_toolchain_scala_compile_classpath_provider'
```

Also fixes the following error when attaching resources from custom repos to
targets under Bzlmod:

```txt
$ bazel test //test/src/main/scala/scalarules/test/resources:all

1) Scala library depending on resources from external resource-only
  jar::allow to load resources(scalarules.test.resources.ScalaLibResourcesFromExternalDepTest)
  java.lang.NullPointerException
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.get(ScalaLibResourcesFromExternalDepTest.scala:17)
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$3(ScalaLibResourcesFromExternalDepTest.scala:11)
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$2(ScalaLibResourcesFromExternalDepTest.scala:11)
```

Can be replaced with a future bazel-skylib implementation, if accepted
into that repo.

---

We can't rely on the specific canonical repository name format:

> Repos generated by extensions have canonical names in the form of
> `module_repo_canonical_name+extension_name+repo_name`. For extensions
> hosted in the root module, the `module_repo_canonical_name` part is
> replaced with the string `_main`. Note that the canonical name format is
> not an API you should depend on — it's subject to change at any time.
>
> - https://bazel.build/external/extension#repository_names_and_visibility

The change to no longer encode module versions in canonical repo names in
Bazel 7.1.0 is a recent example of Bazel maintainers altering the format:

- bazelbuild/bazel#21316

And the maintainers recently replaced the `~` delimiter with `+` in the
upcoming Bazel 8 release due to build performance issues on Windows:

- bazelbuild/bazel#22865

This function assumes the only valid `repo_name` characters are letters,
numbers, '_', '-', and '.'. It finds the last character not in this set, and
returns the contents of `name` following this character. This is valid so
long as this condition holds:

- https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/java/com/google/devtools/build/lib/cmdline/RepositoryName.java#L159-L162
liucijus pushed a commit that referenced this issue Oct 8, 2024
Related to #1482, #1618, and #1619. Results from the investigation
documented at:

- #1619 (comment)

Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle
differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources`
values between Bazel 6 and Bazel 7.

Without this change, `_import_paths()` emits incorrect values under
Bazel 7, causing targets containing generated `.proto` inputs to fail,
e.g.  `//test/proto3:test_generated_proto`.

See also:

- Fix paths for sibling repository setup and generated .proto files
  bazelbuild/bazel@6c6c196

- The docstring for `ProtoInfo.proto_source_root` in the Bazel sources:
  https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172

- Remove incompatible_generated_protos_in_virtual_imports
  bazelbuild/bazel@3efaa32

- Comment from: Generated Protos are no longer considered as
  virtual_imports in Bazel 7
  bazelbuild/bazel#21075 (comment)

---

I cherrypicked this commit into #1618. While it fixed the
`//test/proto3` build failure, it does _not_ fix the hanging
scalapb_workers from the ProtoScalaPBRule aspect.

I'll have to investiate further whether than hang is related to Bazel,
rules_proto, com_google_protobuf, or some mixture thereof. Still,
progress!
@simuons simuons reopened this Oct 8, 2024
mbland added a commit to mbland/rules_scala that referenced this issue Oct 8, 2024
Part of bazelbuild#1482.

Splits the last component off of canonical repo names to produce the
expected repo name.

Without Bzlmod, it returns the original name. With Bzlmod enabled, it
avoids generating output like:

    scala_import(
        name = "_main~scala_deps~io_bazel_rules_scala_scala_compiler",
        jars = ["scala-compiler-2.12.18.jar"],
    )

resulting in errors like:

```
ERROR: .../_main~_repo_rules~io_bazel_rules_scala/scala/BUILD:
no such target '@@_main~scala_deps~io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler':
target 'io_bazel_rules_scala_scala_compiler' not declared in package ''
defined by .../_main~scala_deps~io_bazel_rules_scala_scala_compiler/BUILD
and referenced by '@@_main~_repo_rules~io_bazel_rules_scala//scala:default_toolchain_scala_compile_classpath_provider'
```

Also fixes the following error when attaching resources from custom repos to
targets under Bzlmod:

```txt
$ bazel test //test/src/main/scala/scalarules/test/resources:all

1) Scala library depending on resources from external resource-only
  jar::allow to load resources(scalarules.test.resources.ScalaLibResourcesFromExternalDepTest)
  java.lang.NullPointerException
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.get(ScalaLibResourcesFromExternalDepTest.scala:17)
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$3(ScalaLibResourcesFromExternalDepTest.scala:11)
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$2(ScalaLibResourcesFromExternalDepTest.scala:11)
```

Can be replaced with a future bazel-skylib implementation, if accepted
into that repo.

---

We can't rely on the specific canonical repository name format:

> Repos generated by extensions have canonical names in the form of
> `module_repo_canonical_name+extension_name+repo_name`. For extensions
> hosted in the root module, the `module_repo_canonical_name` part is
> replaced with the string `_main`. Note that the canonical name format is
> not an API you should depend on — it's subject to change at any time.
>
> - https://bazel.build/external/extension#repository_names_and_visibility

The change to no longer encode module versions in canonical repo names in
Bazel 7.1.0 is a recent example of Bazel maintainers altering the format:

- bazelbuild/bazel#21316

And the maintainers recently replaced the `~` delimiter with `+` in the
upcoming Bazel 8 release due to build performance issues on Windows:

- bazelbuild/bazel#22865

This function assumes the only valid `repo_name` characters are letters,
numbers, '_', '-', and '.'. It finds the last character not in this set, and
returns the contents of `name` following this character. This is valid so
long as this condition holds:

- https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/java/com/google/devtools/build/lib/cmdline/RepositoryName.java#L159-L162
@mbland
Copy link
Contributor

mbland commented Oct 11, 2024

A quick update for visibility: I'm very close to having the Bzlmodified rules_scala passing 100% of the tests, down to the last couple of failures:

  • I need to replace the bind() calls from twitter_scrooge with alias() or some such to get test_version.sh to pass.
  • I need to fix the Scala 3.1.2 and Scala 3.3.3 failures for the Scalafmt targets from test/shell/test_cross_build.sh (corresponding to the test3 and library3 test cases in test_scalafmt()).

I also need to get test_lint.sh working again, and update the dt_patches repos for Bzlmod. But the following all pass:

  • test_rules_scala.sh
  • test_reproducibility.sh
  • test_examples.sh
  • test_coverage.sh

Hopefully I can get these fixed up today, and I'll start peeling off the next pull request or two. And thanks to @simuons and @liucijus for reviewing and merging #1619 and #1620 already.

@mbland
Copy link
Contributor

mbland commented Nov 26, 2024

Weekly update!

#1633 and #1650 are approved and hopefully landing soon. Then I'll fire up the pull request factory again.

But in related news, it's all Protobuf + ScalaPB, all the time, with Bazel 8 and rules_java 8 starting to join in on the fun. #1647 has all the links to all the details, but here's the high level recap.

Upstream ScalaPB and protoc-bridge changes to avoid rules_scala ProtoScalaPBRule build hangs

I filed scalapb/ScalaPB#1771 to volunteer to contribute scalapb.ScalaPbCodeGenerator changes upstream to avoid build hangs due to Protobuf + ScalaPB incompatibilities. Right after that, I noticed the latest protoc-bridge implementation of PluginFrontend.runWithInputStream(). It apparently landed as part of scalapb/protoc-bridge#367, without being specifically called out in the pull request description or discussion. It's not in the latest protoc-bridge 0.9.7 release, however.

Ideally a new protoc-bridge release will come out soon, which might obviate the need to contribute to ScalaPB or maintain the scalapb.ScalaPbCodeGenerator changes from #1648 indefinitely. But I have no idea if or when that might happen.

Bazel 8 and rules_java 8 compatibility requires breaking WORKSPACE builds

Just this morning, I noticed #1652, and experimented with local changes to build using Bazel 8.0.0rc4 on top of my bzlmod working branch. I found (copying from my summarization on the related bazelbuild/bazel#24235) that most of the Bazel 8 and rules_java 8 incompatibilities can be easily addressed once Bzlmodification lands. What I think is the only remaining one may be resolved by a protobuf-java:4.29.0 release (without -RC2). (Granted, I didn't run the full test suite, but bazel build //src/... //test/... //third_party/... //scala_proto/... is a decent smoke test.)

The tradeoffs are that users will be forced to upgrade to Protobuf v28.3 under Bazel 6 and 7 (thanks to ScalaPB 1.0.0-alpha.1), and Protobuf v29 under Bazel 8. , and the load("@rules_java") statements required by Bazel 8 to define JavaInfo and java_common.JavaToolchainInfo will break WORKSPACE users for good. Then again, that may be all the more reason to suggest folks migrate to Bzlmod sooner than later.

Versioning for Bzlmod and Bazel 8 compatibility

I'm thinking, if the maintainers are OK with it, it might be worth including Bzlmodification in the next major release (7.x), and then building up to Bazel 8 compatibility in another major release (8.x). The Bazel 8 compatibility release could happen very soon after, since it won't require that many changes. rules_scala 7.x would give existing WORKSPACE users a chance to migrate to Bzlmod first, using either Bazel 6 or 7. Then rules_scala 8.x would enable them to build using Bazel 8. , but only under Bzlmod

Or maybe one could argue for Bzlmodification being released as rules_scala 6.7.0, since I think the only confirmed breaking change is possibly the "bugfix" from removing external/$REPO_NAME path prefixes from embedded resources in 1621. Or, if we bump Protobuf up to v25.5 (or v24.4, since that's the highest compatible version of the protobuf module in the Bazel Central Registry), we'd have to tell Bazel 6 users to add some compiler flags. But then Bazel 8 compatibility would be a truly breaking change, requiring a major 7.0.0 release.

I'm particularly interested in what @simuons and @liucijus think of sacrificing WORKSPACE users for implementing Bazel 8 compatibility, and when. Curious what others think of the possibility as well.

Update from bazelbuld/rules_scala#1652: I spoke too soon: I can get a WORKSPACE build to use rules_java 8.5.1 and the other updated repo versions, and get to the point where it reaches the Detected mismatched Protobuf Gencode/Runtime version suffixes error.

So it seems we could bump rules_scala to Bazel 8 and rules_java 8 without abandoning WORKSPACE, but we will abandon Bazel 6 and 7 at that point.

One more update: I've created the bazel-8-compatibility working branch in my rules_scala fork with the commit message for mbland@ebe714d describing the current build breakages, which are the same under WORKSPACE and Bzlmod.

Note that this branch/commit is based on my bzlmod working branch, not the current upstream master branch.

@simuons
Copy link
Collaborator

simuons commented Nov 27, 2024

Hi, @mbland, If I understood you correctly I'm leaning towards your first proposal:

Bzlmodification in the next major release (7.x), and then building up to Bazel 8 compatibility in another major release (8.x). The Bazel 8 compatibility release could happen very soon after, since it won't require that many changes. rules_scala 7.x would give existing WORKSPACE users a chance to migrate to Bzlmod first, using either Bazel 6 or 7. Then rules_scala 8.x would enable them to build using Bazel 8.

liucijus pushed a commit that referenced this issue Nov 27, 2024
* Toolchainize //scala:toolchain_type

Moves the `toolchain` targets for `//scala:toolchain_type` to a new
`@io_bazel_rules_scala_toolchains` repository as a step towards
Bzlmodification. Part of #1482.

Instantiating toolchains in their own repository enables module
extensions to define the the repositories required by those toolchains
within the extension's own namespace. Bzlmod users can then register the
toolchains from this repository without having to import all the
toolchains' dependencies into their own namespace via `use_repo()`.

---

The `scala_toolchains_repo()` macro wraps the underlying repository rule
and assigns it the standard name `io_bazel_rules_scala_toolchains`.
Right now it's only instantiating the main Scala toolchain via the
default `scala = True` parameter. Future changes will expand this
macro and repository rule with more boolean parameters to instantiate
other toolchains, specifically:

- `scalatest`
- `junit`
- `specs2`
- `twitter_scrooge`
- `jmh`
- `scala_proto` and `scala_proto_enable_all_options`
- `testing` (includes all of `scalatest`, `junit`, and `specs2`)
- `scalafmt`

---

`WORKSPACE` users will now have to import and call the
`scala_toolchains_repo()` macro to instantiate
`@io_bazel_rules_scala_toolchains`.

```py
load("@io_bazel_rules_scala//:scala_config.bzl", "scala_config")

scala_config()

load(
    "//scala:scala.bzl",
    "rules_scala_setup",
    "rules_scala_toolchain_deps_repositories",
    "scala_toolchains_repo",
)

rules_scala_setup()

rules_scala_toolchain_deps_repositories()

scala_toolchains_repo()

register_toolchains("@io_bazel_rules_scala_toolchains//...:all")
```

This is what the corresponding `MODULE.bazel` setup would look like:

```py
module(name = "rules_scala", version = "7.0.0")

scala_config = use_extension(
    "//scala/extensions:config.bzl", "scala_config"
)
scala_config.settings(scala_version = "2.13.14")

scala_deps = use_extension("//scala/extensions:deps.bzl", "scala_deps")
scala_deps.toolchains()
```

The `register_toolchains()` call in `WORKSPACE` isn't strictly required
at this point, but is recommended. However, all the `WORKSPACE` files in
this repo already register their required toolchains using existing
macros, which have been updated in this change.

In fact, calling `register_toolchains()` right after
`scala_toolchains_repo()` as shown above breaks two tests that depend on
the existing `WORKSPACE` toolchain registration:

- `test_compilation_fails_with_plus_one_deps_undefined` from
  `test/shell/test_compilation.sh` depends on
  `scala_register_unused_deps_toolchains()` setting up its toolchain to
  resolve first. `//scala:unused_dependency_checker_error_toolchain`
  sets the `scala_toolchain()` parameters `dependency_tracking_method =
  "ast-plus"` and `unused_dependency_checker_mode = "error"`, and the
  `@io_bazel_rules_scala_toolchains//scala` toolchains don't.

- `test_scala_binary_allows_opt_in_to_use_of_argument_file_in_runner_for_improved_performance`
  from `test/shell/test_scala_binary.sh` depends on the
  `use_argument_file_in_runner` parameter of `scala_toolchain` being
  `False`. This is the default, but the
  `@io_bazel_rules_scala_toolchains//scala` toolchains explicitly set
  this to `True` instead.

In the Bzlmod case, the `register_toolchains()` call isn't necessary at
all. This is because `@io_bazel_rules_scala_toolchains` includes one
package per set of toolchains, and the rules_scala `MODULE.bazel` calls
`register_toolchains("@io_bazel_rules_scala_toolchains//...:all")`. This
will automatically register all configured rules_scala toolchains, while
allowing users to override any of them using `register_toolchains()` in
their own `MODULE.bazel` files.

Technically, the `scala_deps.toolchains()` call isn't required when only
using the default `scala = True` parameter; the rules_scala
`MODULE.bazel` will instantiate this automatically, too.

* Add scala_toolchains macro for WORKSPACE, Bzlmod

Extracted a single `scala_toolchains` macro to share between `WORKSPACE`
and the `deps.bzl` module extension. This will make it easier to ensure
`WORKSPACE` compatibility, and to add a Bazel module extension as a thin
layer on top. Part of #1482.

This change includes updates to `rules_scala_setup` and
`scala_repositories` to support this, while preserving compatibility
with existing `WORKSPACE` calls. The next commit will replace many
existing `WORKSPACE` calls with `scala_toolchains`.

* Replace WORKSPACE calls with scala_toolchains()

Also added `@io_bazel_rules_scala_toolchains//...:all` to
`register_toolchains()` calls everywhere, even when not specifically
necessary. This proves the mechanism is safe and works with `WORKSPACE`
now, and will make future updates to consolidate other toolchains less
noisy.

* Remove obsolete WORKSPACE toolchain calls

Should've been included in the previous commit. All tests still pass.

* Replace scala_register_unused_deps_toolchains call

Missed this one in third_party/test/proto earlier. Also removed
unnecessary `USE_BAZEL_VERSION` expression in
test_scala_proto_library.sh. If `USE_BAZEL_VERSION` is set, it takes
precedence to begin with, and third_party/test/proto/.bazelversion
exists now after #1629.

* Give scala_toolchains_repo a default name argument

This turns out to be helpful when adapting
`test_version/WORKSPACE.template` to the toolchainized version of
`twitter_scrooge` for testing. In this case, we want `twitter_scooge` to
be in its own customized repo, separate from the one generated by
`scala_toolchains`.

This seems like it might be generally useful when writing module
extensions for alternative toolchains.

* Update Scala toolchainizaion per @simuons in #1633

- Removes an extraneous `compiler_sources_repo` call.

- `scala_toolchains` will now _always_ create the main Scala toolchain
  repository (there's no longer an option to turn it off).

- Registers `@io_bazel_rules_scala_toolchains//...:all` in
  `scala_register_toolchains.

* Extract load_rules_dependencies macro

Requested by @simuons in #1633 to make `rules_scala_setup` and
`scala_repositories` more readable while maintaining the existing APIs.
liucijus pushed a commit that referenced this issue Nov 27, 2024
This presents the same API, remains compatible with both `WORKSPACE` and
Bzlmod, and avoids relying on the canonical repository name format at
all. Part of #1482, and supersedes the addition of `apparent_repo_name`
from #1621.

Also adds the `_SCALA_IMPORT_RULE_LOAD` constant in
`scala/scala_maven_import_external.bzl`, which also removes
`@io_bazel_rules_scala` from the `load`ed file path. It now generates
the correct path with a `Label` instead.

---

The goal is to maintain the ability to generate default target names
from a repository name under Bzlmod. While not documented on
https://bazel.build, it is documented in the Bazel source itself, even
under the current 9.0.0-pre.20241105.2 prerelease:

- https://github.com/bazelbuild/bazel/blob/9.0.0-pre.20241105.2/src/main/java/com/google/devtools/build/lib/cmdline/LabelParser.java#L99

Under `WORKSPACE`, the `repository_ctx.name` value served this purpose
directly, as it would reflect the `name` value as provided by the
caller. Under Bzlmod, this value now contains the canonical repo name,
which is different than that provided by the `name` parameter. The
`apparent_repo_name` utility was my first attempt to resolve this, by
parsing the original `name` parameter from the canonicalized
`repository_ctx.name`.

I also created bazelbuild/bazel-skylib#548 to contribute
`apparent_repo_name` to `bazel_skylib`, along with other proposed
utilities. I quickly found better solutions to obviate the other
utilities, but hung onto `apparent_repo_name`.

After that pull request stagnated, I eventually realized a macro wrapper
could generate a default target name for a repository_rule by
duplicating the `name` attr. This isn't as easy as adding
`apparent_repo_name(repository_ctx.name)` to a repo rule's
implementation, but it's also not that much more work. See also this
thread from the #bzlmod channel in the Bazel Slack workspace about
generating default repo names:

- https://bazelbuild.slack.com/archives/C014RARENH0/p1730909824656159

---

One small downside of this technique is that the macro can't be imported
into a `MODULE.bazel` file directly via `use_repo_rule`. If you did want
to use it in `MODULE.bazel`, you'd have to write a module extension to
call it, or use `modules.as_extension` from `bazel_skylib`. I suppose
that's a small price to pay for squashing a canonical repo name format
dependency forever.

The other small downside may be that the documentation from the original
`repository_rule` doesn't automatically convey to the repo wrapper. In
this case, `_alias_repository` is already internal, and the original
`jvm_import_external` didn't have docstrings to begin with.
mbland added a commit to mbland/rules_scala that referenced this issue Nov 27, 2024
Moves the Scalatest, JUnit, and Specs2 toolchains into
`@io_bazel_rules_scala_toolchains//testing`. Part of bazelbuild#1482.

Updates all `WORKSPACE` files to set the appropriate `scala_toolchains`
parameters and to remove the unnecessary repository import and toolchain
registration macros.

Adds a `fetch_sources_by_id` parameter to `repositories` from
`third_party/repositories/repositories.bzl`. This enables
`scala_toolchains` to build the `artifact_ids_to_fetch_sources` mapping
from artifact ID lists returned by new macros extracted from `WORKSPACE`
macros. The values assigned to each id match the original
`fetch_sources` settings in the corresponding original `WORKSPACE`
macros.

Updates `scala/scala_maven_import_external.bzl` to generate a `load`
line for `//scala:scala_import.bzl` based on the repo's canonical name,
not `@io_bazel_rules_scala`.

As usual, includes several other opportunistic removals of the
`@io_bazel_rules_scala` repo name prefix to avoid an internal dependency
on that name. This means Bzlmod users won't necessarily have to set the
`repo_name` parameter of `bazel_dep` when using `rules_scala`.

---

Introduces more macros to return a framework's Maven artifact
dependencies, rather than inlining them in a `repositories` call. These
inlined lists are replaced by macro invocations, and now the
`scala_toolchains` macro can invoke these macros to collect artifact IDs
to pass to `repositories`. This also allows for future changes to
introduce a `scala_version` parameter if necessary, similar to how
`scalafmt_artifact_ids` already works.

This is important to avoid collisions when creating repositories for
artifacts upon which more than one framework depends under Bzlmod.
`WORKSPACE` doesn't seem affected by these collisions, but Bzlmod will
produce errors like the following, where both `scala_proto` and
`twitter_scrooge` depend upon `io_bazel_rules_scala_guava`:

```txt
$ bazel build //src/...

ERROR: .../scala/scala_maven_import_external.bzl:299:24:
  Traceback (most recent call last):
    File ".../scala/extensions/deps.bzl",
      line 140, column 21, in _scala_deps_impl
        scala_toolchains(
    File ".../scala/private/macros/toolchains.bzl",
      line 140, column 17, in scala_toolchains
        _scrooge(
    File ".../twitter_scrooge/twitter_scrooge.bzl",
      line 96, column 17, in twitter_scrooge
        repositories(
    File ".../third_party/repositories/repositories.bzl",
      line 113, column 37, in repositories
        _scala_maven_import_external(
    File ".../scala/scala_maven_import_external.bzl",
      line 263, column 30, in scala_maven_import_external
        jvm_maven_import_external(
    File ".../scala/scala_maven_import_external.bzl",
      line 299, column 24, in jvm_maven_import_external
        jvm_import_external(jar_urls = jar_urls, srcjar_urls = srcjar_urls, coordinates = artifact, **kwargs)

Error in repository_rule: A repo named
  io_bazel_rules_scala_guava_2_13_15
  is already generated by this module extension at
  .../scala/scala_maven_import_external.bzl:299:24

ERROR: Analysis of target
  '//src/java/io/bazel/rulesscala/worker:worker_test'
  failed; build aborted:
  error evaluating module extension scala_deps in
  //scala/extensions:deps.bzl
```

Recent updates to `scripts/create_repository.py` (bazelbuild#1639, bazelbuild#1642) make it
easy to emit full direct dependency lists for artifacts included in
`third_party/repositories/scala_*.bzl`. This increases the likelihood of
collisions, since this expanded metadata forces the macros that
instantiate artifact repos to instantiate even more repos.

By fetching list of artifact IDs from these macros, `scala_toolchains`
can now consolidate them into dictionary keys. Then it passes these
unique keys to `repositories` directly, avoiding the problem of
instantiating the same repo multiple times in the same module extension.

This, in turn, also avoids the need to add parameters to the original
`WORKSPACE` macros that instantiate dependencies to avoid collisions
under Bzlmod. The `scala_toolchains` macro never needs to call these
original macros, under either `WORKSPACE` or Bzlmod.

Finally, it also reduces duplication between these artifact ID lists and
the `_*_DEPS` symbols originally from `testing/BUILD` (and now in
`testing/deps.bzl`). The dependency labels are now generated
programatically.

(Aside: As I mentioned, we may eventually need to pass a Scala version
argument to these macros. It will be possible to cross that bridge
without too much trouble if and when that day comes. Or I can try to
future proof it in a follow up pull request.)
mbland added a commit to mbland/rules_scala that referenced this issue Nov 27, 2024
Moves the Scalatest, JUnit, and Specs2 toolchains into
`@io_bazel_rules_scala_toolchains//testing`. Part of bazelbuild#1482.

Updates all `WORKSPACE` files to set the appropriate `scala_toolchains`
parameters and to remove the unnecessary repository import and toolchain
registration macros.

Adds a `fetch_sources_by_id` parameter to `repositories` from
`third_party/repositories/repositories.bzl`. This enables
`scala_toolchains` to build the `artifact_ids_to_fetch_sources` mapping
from artifact ID lists returned by new macros extracted from `WORKSPACE`
macros. The values assigned to each id match the original
`fetch_sources` settings in the corresponding original `WORKSPACE`
macros.

Updates `scala/scala_maven_import_external.bzl` to generate a `load`
line for `//scala:scala_import.bzl` based on the repo's canonical name,
not `@io_bazel_rules_scala`.

As usual, includes several other opportunistic removals of the
`@io_bazel_rules_scala` repo name prefix to avoid an internal dependency
on that name. This means Bzlmod users won't necessarily have to set the
`repo_name` parameter of `bazel_dep` when using `rules_scala`.

---

Introduces more macros to return a framework's Maven artifact
dependencies, rather than inlining them in a `repositories` call. These
inlined lists are replaced by macro invocations, and now the
`scala_toolchains` macro can invoke these macros to collect artifact IDs
to pass to `repositories`. This also allows for future changes to
introduce a `scala_version` parameter if necessary, similar to how
`scalafmt_artifact_ids` already works.

This is important to avoid collisions when creating repositories for
artifacts upon which more than one framework depends under Bzlmod.
`WORKSPACE` doesn't seem affected by these collisions, but Bzlmod will
produce errors like the following, where both `scala_proto` and
`twitter_scrooge` depend upon `io_bazel_rules_scala_guava`:

```txt
$ bazel build //src/...

ERROR: .../scala/scala_maven_import_external.bzl:299:24:
  Traceback (most recent call last):
    File ".../scala/extensions/deps.bzl",
      line 140, column 21, in _scala_deps_impl
        scala_toolchains(
    File ".../scala/private/macros/toolchains.bzl",
      line 140, column 17, in scala_toolchains
        _scrooge(
    File ".../twitter_scrooge/twitter_scrooge.bzl",
      line 96, column 17, in twitter_scrooge
        repositories(
    File ".../third_party/repositories/repositories.bzl",
      line 113, column 37, in repositories
        _scala_maven_import_external(
    File ".../scala/scala_maven_import_external.bzl",
      line 263, column 30, in scala_maven_import_external
        jvm_maven_import_external(
    File ".../scala/scala_maven_import_external.bzl",
      line 299, column 24, in jvm_maven_import_external
        jvm_import_external(jar_urls = jar_urls, srcjar_urls = srcjar_urls, coordinates = artifact, **kwargs)

Error in repository_rule: A repo named
  io_bazel_rules_scala_guava_2_13_15
  is already generated by this module extension at
  .../scala/scala_maven_import_external.bzl:299:24

ERROR: Analysis of target
  '//src/java/io/bazel/rulesscala/worker:worker_test'
  failed; build aborted:
  error evaluating module extension scala_deps in
  //scala/extensions:deps.bzl
```

Recent updates to `scripts/create_repository.py` (bazelbuild#1639, bazelbuild#1642) make it
easy to emit full direct dependency lists for artifacts included in
`third_party/repositories/scala_*.bzl`. This increases the likelihood of
collisions, since this expanded metadata forces the macros that
instantiate artifact repos to instantiate even more repos.

By fetching list of artifact IDs from these macros, `scala_toolchains`
can now consolidate them into dictionary keys. Then it passes these
unique keys to `repositories` directly, avoiding the problem of
instantiating the same repo multiple times in the same module extension.

This, in turn, also avoids the need to add parameters to the original
`WORKSPACE` macros that instantiate dependencies to avoid collisions
under Bzlmod. The `scala_toolchains` macro never needs to call these
original macros, under either `WORKSPACE` or Bzlmod.

Finally, it also reduces duplication between these artifact ID lists and
the `_*_DEPS` symbols originally from `testing/BUILD` (and now in
`testing/deps.bzl`). The dependency labels are now generated
programatically.

(Aside: As I mentioned, we may eventually need to pass a Scala version
argument to these macros. It will be possible to cross that bridge
without too much trouble if and when that day comes. Or I can try to
future proof it in a follow up pull request.)
mbland added a commit to mbland/rules_scala that referenced this issue Nov 27, 2024
This is part of the Bzlmod prep work from bazelbuild#1482, but also reduces
duplication between the `dt_patches/test_dt_patches*/WORKSPACE` files.

The structure of the `dt_patches/compiler_sources/extensions.bzl`
declarations accommodates the fact that, unlike `WORKSPACE,
`MODULE.bazel` files cannot import constants or contain conditional
statements. That is to say, there's no way to port the `if
SCALA_VERSION.startswith("2.")` expressions from the previous
`WORKSPACE` files to Bzlmod.

The new `import_compiler_source_repos` macro, however, works for both
`WORKSPACE` and Bzlmod builds, as it's trivial to wrap
`import_compiler_source_repos` in a module extension.
mbland added a commit to mbland/rules_scala that referenced this issue Nov 28, 2024
This is a pretty straightforward and easy update on top of the previous
`setup_toolchains` and `scala_toolchains_repo` changes. Part of bazelbuild#1482.
mbland added a commit to mbland/rules_scala that referenced this issue Nov 28, 2024
Adds scala_proto toolchains to `scala_toolchains()`. Part of bazelbuild#1482.

The most significant part of the change is moving all the toolchain
rules from `scala_proto/BUILD` to `setup_scala_toolchains()` in
`scala_proto/toolchains.bzl`.

Adds the `scala_proto_deps_providers()` macro to replace
`//scala_proto:scalapb_{compile,grpc,worker}_deps_provider` targets in
the `dep_providers` parameter of `scala_proto_deps_toolchain()`.
Examples of this are in `test/proto/custom_generator/BUILD`.

A lot of the other changes are more opportunistic removals of
`@io_bazel_rules_scala` label prefixes and application of `Label()`
where appropriate. Doing this will allow Bzlmod users to use
`rules_scala` without setting `repo_name = "@io_bazel_rules_scala"` in
`bazel_dep()`.
mbland added a commit to mbland/rules_scala that referenced this issue Nov 28, 2024
Adds the jmh toolchain to `scala_toolchains()`. Part of bazelbuild#1482.
mbland added a commit to mbland/rules_scala that referenced this issue Nov 28, 2024
Adds scala_proto toolchains to `scala_toolchains()`. Part of bazelbuild#1482.

The most significant part of the change is moving all the toolchain
rules from `scala_proto/BUILD` to `setup_scala_toolchains()` in
`scala_proto/toolchains.bzl`.

Adds the `scala_proto_deps_providers()` macro to replace
`//scala_proto:scalapb_{compile,grpc,worker}_deps_provider` targets in
the `dep_providers` parameter of `scala_proto_deps_toolchain()`.
Examples of this are in `test/proto/custom_generator/BUILD`.

A lot of the other changes are more opportunistic removals of
`@io_bazel_rules_scala` label prefixes and application of `Label()`
where appropriate. Doing this will allow Bzlmod users to use
`rules_scala` without setting `repo_name = "@io_bazel_rules_scala"` in
`bazel_dep()`.
mbland added a commit to mbland/rules_scala that referenced this issue Nov 28, 2024
Adds the jmh toolchain to `scala_toolchains()`. Part of bazelbuild#1482.
mbland added a commit to mbland/rules_scala that referenced this issue Nov 28, 2024
This is the last of the toolchain to receive the "toolchainization"
treatment prior to Bzlmodification. Part of bazelbuild#1482.
mbland added a commit to mbland/rules_scala that referenced this issue Nov 28, 2024
`dev_deps_repositories()` encapsulates the instantiation of repositories
used only for `rules_scala` development. Also removes the unused
`//private` package and its `WORKSPACE` statements. Part of bazelbuild#1482.

Both `WORKSPACE` and Bzlmod builds can use this macro, though how Bzlmod
will use it will depend on whether we continue building with Bazel 6.

`@bazel_tools//tools/build_defs/repo:local.bzl` isn't available under
Bazel 6. To continue building with Bazel 6 under Bzlmod, we will need to
call `dev_deps_repositories()` from `WORKSPACE.bzlmod` to continue using
`native.{,new_}local_repository()`.

If we switch to Bazel 7, we can load `local.bzl` and strip the `native.`
prefix from the `{,new_}local_repository()` calls. Then we can call
`dev_deps_repositories()` from a module extension instead of from
`WORKSPACE.bzlmod`.

Another alternative would be updating the local repositories to become
proper nested repositories. Then we can call `local_repository()` from
`WORKSPACE` and call `bazel_dep()` and `local_path_override()` from
`MODULE.bazel`. In that case, we'd remove the `{,new_}local_repository`
calls from `dev_deps_dependencies()`, and remove
`proto_cross_repo_boundary_repository()` entirely.
mbland added a commit to mbland/rules_scala that referenced this issue Nov 28, 2024
Removes `native.{,new_}local_repository()` calls in macros in favor of
`local_repository` calls from `WORKSPACE`. Part of bazelbuild#1482.

`native.{,new_}local_repository()` isn't available under Bzlmod,
`@bazel_tools//tools/build_defs/repo:local.bzl` with the Starlarkified
definitions isn't available under Bazel 6, and Bazel 8 compatibility
work is imminent. Redefining the repositories in this way will be
compatible with Bazel 6, 7, and 8, both under `WORKSPACE` and Bzlmod.
(`MODULE.bazel` will use a combination of `bazel_dep()` and
`local_path_override()`.)
mbland added a commit to mbland/rules_scala that referenced this issue Nov 29, 2024
Removes `native.{,new_}local_repository()` calls in macros in favor of
`local_repository` calls from `WORKSPACE`. Part of bazelbuild#1482.

`native.{,new_}local_repository()` isn't available under Bzlmod,
`@bazel_tools//tools/build_defs/repo:local.bzl` with the Starlarkified
definitions isn't available under Bazel 6, and Bazel 8 compatibility
work is imminent. Redefining the repositories in this way will be
compatible with Bazel 6, 7, and 8, both under `WORKSPACE` and Bzlmod.
(`MODULE.bazel` will use a combination of `bazel_dep()` and
`local_path_override()`.)
mbland added a commit to mbland/rules_scala that referenced this issue Nov 29, 2024
`dev_deps_repositories()` encapsulates the instantiation of repositories
used only for `rules_scala` development. Also removes the unused
`//private` package and its `WORKSPACE` statements. Part of bazelbuild#1482.

Both `WORKSPACE` and Bzlmod builds can use this macro, though how Bzlmod
will use it will depend on whether we continue building with Bazel 6.

`@bazel_tools//tools/build_defs/repo:local.bzl` isn't available under
Bazel 6. To continue building with Bazel 6 under Bzlmod, we will need to
call `dev_deps_repositories()` from `WORKSPACE.bzlmod` to continue using
`native.{,new_}local_repository()`.

If we switch to Bazel 7, we can load `local.bzl` and strip the `native.`
prefix from the `{,new_}local_repository()` calls. Then we can call
`dev_deps_repositories()` from a module extension instead of from
`WORKSPACE.bzlmod`.

Another alternative would be updating the local repositories to become
proper nested repositories. Then we can call `local_repository()` from
`WORKSPACE` and call `bazel_dep()` and `local_path_override()` from
`MODULE.bazel`. In that case, we'd remove the `{,new_}local_repository`
calls from `dev_deps_dependencies()`, and remove
`proto_cross_repo_boundary_repository()` entirely.
mbland added a commit to mbland/rules_scala that referenced this issue Nov 29, 2024
Removes `native.{,new_}local_repository()` calls in macros in favor of
`local_repository` calls from `WORKSPACE`. Part of bazelbuild#1482.

`native.{,new_}local_repository()` isn't available under Bzlmod,
`@bazel_tools//tools/build_defs/repo:local.bzl` with the Starlarkified
definitions isn't available under Bazel 6, and Bazel 8 compatibility
work is imminent. Redefining the repositories in this way will be
compatible with Bazel 6, 7, and 8, both under `WORKSPACE` and Bzlmod.
(`MODULE.bazel` will use a combination of `bazel_dep()` and
`local_path_override()`.)
mbland added a commit to mbland/rules_scala that referenced this issue Nov 29, 2024
Every Scala version now uses com.google.protobuf:protobuf-java:4.29.0.
Part of bazelbuild#1482 and bazelbuild#1652.

This is necessary, but not sufficient, to make `rules_scala` Bazel 8
compatible by enabling an update to `protobuf` v29.0. Building with
Bazel 6.5.0 and `protobuf` v21.7 continues to pass all tests after this
change.
mbland added a commit to mbland/rules_scala that referenced this issue Nov 29, 2024
`dev_deps_repositories()` encapsulates the instantiation of repositories
used only for `rules_scala` development. Also removes the unused
`//private` package and its `WORKSPACE` statements. Part of bazelbuild#1482.

Both `WORKSPACE` and Bzlmod builds can use this macro, though how Bzlmod
will use it will depend on whether we continue building with Bazel 6.

`@bazel_tools//tools/build_defs/repo:local.bzl` isn't available under
Bazel 6. To continue building with Bazel 6 under Bzlmod, we will need to
call `dev_deps_repositories()` from `WORKSPACE.bzlmod` to continue using
`native.{,new_}local_repository()`.

If we switch to Bazel 7, we can load `local.bzl` and strip the `native.`
prefix from the `{,new_}local_repository()` calls. Then we can call
`dev_deps_repositories()` from a module extension instead of from
`WORKSPACE.bzlmod`.

Another alternative would be updating the local repositories to become
proper nested repositories. Then we can call `local_repository()` from
`WORKSPACE` and call `bazel_dep()` and `local_path_override()` from
`MODULE.bazel`. In that case, we'd remove the `{,new_}local_repository`
calls from `dev_deps_dependencies()`, and remove
`proto_cross_repo_boundary_repository()` entirely.
mbland added a commit to mbland/rules_scala that referenced this issue Nov 29, 2024
Removes `native.{,new_}local_repository()` calls in macros in favor of
`local_repository` calls from `WORKSPACE`. Part of bazelbuild#1482.

`native.{,new_}local_repository()` isn't available under Bzlmod,
`@bazel_tools//tools/build_defs/repo:local.bzl` with the Starlarkified
definitions isn't available under Bazel 6, and Bazel 8 compatibility
work is imminent. Redefining the repositories in this way will be
compatible with Bazel 6, 7, and 8, both under `WORKSPACE` and Bzlmod.
(`MODULE.bazel` will use a combination of `bazel_dep()` and
`local_path_override()`.)
mbland added a commit to mbland/rules_scala that referenced this issue Nov 29, 2024
Removes `native.{,new_}local_repository()` calls in macros in favor of
`local_repository` calls from `WORKSPACE`. Part of bazelbuild#1482.

`native.{,new_}local_repository()` isn't available under Bzlmod,
`@bazel_tools//tools/build_defs/repo:local.bzl` with the Starlarkified
definitions isn't available under Bazel 6, and Bazel 8 compatibility
work is imminent. Redefining the repositories in this way will be
compatible with Bazel 6, 7, and 8, both under `WORKSPACE` and Bzlmod.
(`MODULE.bazel` will use a combination of `bazel_dep()` and
`local_path_override()`.)
mbland added a commit to mbland/rules_scala that referenced this issue Nov 30, 2024
Updates almost all dependencies on `@bazel_tools//tools/jdk:` toolchain
targets and `.bzl` files to corresponding targets and files in
`@rules_java//toolchains:`. Part of bazelbuild#1482 and bazelbuild#1652.

All dependencies on `@bazel_tools//tools/jdk:toolchain_type` remain,
however, as there is not yet a corresponding target in
`@rules_java//toolchains:`.

Adds the `WORKSPACE` stanza recommended in the `rules_java` release
notes, and removes our own calls to instantiate `rules_java` repos and
to register `rules_java` toolchains.

These changes are required to avoid build breakages when updating to
`rules_scala` 7.10.0 and higher. All targets build and all tests pass
under Bazel 6.5.0 and 7.4.1.

---

I was on the right track in my analysis from bazelbuild#1619 ("Bump to
rules_java 7.9.0 for Bazel 7 compatibility" in the message for
commit cd22d88). However, I thought we
_shouldn't_ update any targets from `@bazel_tools//tools/jdk:` to
`@rules_java//toolchains:`. That's why I thought we were stuck on
`rules_java` 7.9.0.

However, this comment from @fmeum made me think switching to
`@rules_java//toolchains:` actually is the preferred approach:

- bazelbuild/rules_java#214 (comment)

So this is a potentially breaking change, but in the good kind of way,
in that it requires an easy, future proof update.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet