Download GraalVM distributions lazily

[fmeum]

Bazel only fetches repositories when they are actually needed during the build. By moving the toolchain definitions into a separate repo from the actual files comprising the toolchains, this can be used to prevent fetches of large toolchains (such as GraalVM distributions) when not used for a particular build (because the toolchain is never selected).

It would be great if rules_graalvm could support this, potentially by splitting

rules_graalvm/internal/graalvm_bindist.bzl

Lines 462 to 552 in 43adbdc

	# bazel 6+ has support for the `version` attribute on the `java_runtime` rule. earlier
	# versions do not, so we omit it.
	toolchain_template = _JDK_BUILD_TEMPLATE
	if not versions.is_at_least("6", versions.get()):
	toolchain_template = _JDK_BUILD_TEMPLATE_BAZEL5
	toolchain_aliases_template = """
	# Entry Aliases
	alias(
	name = "entry",
	actual = ":{bin_java_path}",
	)
	alias(
	name = "graalvm",
	actual = ":entry",
	)
	alias(
	name = "{name}",
	actual = ":entry",
	)
	# Toolchains
	alias(
	name = "toolchain",
	actual = "@{repo}//:toolchain",
	visibility = ["//visibility:public"],
	)
	{bootstrap_toolchain_alias}
	load(
	"@rules_graalvm//internal:toolchain.bzl",
	"graalvm_sdk",
	"graalvm_engine",
	)
	graalvm_sdk(
	name = "gvm",
	native_image_bin = ":native-image",
	gvm_files = ":files",
	)
	alias(
	name = "sdk",
	actual = "toolchain_gvm",
	)
	toolchain(
	name = "toolchain_gvm",
	exec_compatible_with = [
	{gvm_toolchain_tags_exec}
	],
	target_compatible_with = [
	{gvm_toolchain_tags_target}
	],
	toolchain = ":gvm",
	toolchain_type = "@rules_graalvm//graalvm/toolchain",
	)
	# Tool Aliases
	{rendered_bin_aliases}
	""".format(
	name = ctx_alias,
	repo = ctx.attr.toolchain_config,
	bootstrap_toolchain_alias = bootstrap_toolchain_alias,
	rendered_bin_aliases = rendered_bin_aliases,
	bin_java_path = rendered_bin_paths.java,
	gvm_toolchain_tags_exec = "",
	gvm_toolchain_tags_target = "",
	)
	ctx.file(
	"BUILD.bazel",
	"""
	exports_files(glob(["**/*"]))
	filegroup(
	name = "files",
	srcs = glob(["**/*"]),
	)
	# Tool Targets
	{rendered_bin_targets}
	# Toolchain
	{toolchain}
	# Aliases
	{aliases}
	""".format(
	toolchain = toolchain_template.format(RUNTIME_VERSION = java_version),
	aliases = ctx.attr.enable_toolchain and toolchain_aliases_template or "",
	rendered_bin_targets = rendered_bin_targets,
	),
	)

into two separate repository rules that are instantiated by a macro.

Context: I am trying to add rules_graalvm to Bazel's remote java tools, but currently this would require anyone who builds Bazel (or really any target in the repo) to also download GraalVM.

[sgammon]

@fmeum With regard to Java toolchains, I believe two repos are already configured -- the <name>_toolchain_config_repo repo, and the main <name> repo configured by the user.

repo structure looks like:

flowchart TD
    A(graalvm) -->|Alias| B(graalvm_toolchain_config_repo)
    B-->C[Java toolchain]
    A-->|toolchain target|D[GVM toolchain]

Loading

graalvm_toolchain_config_repo looks like:

@graalvm_toolchain_config_repo//:BUILD.bazel (null)
@graalvm_toolchain_config_repo//:bootstrap_runtime_toolchain (9dc7412)
@graalvm_toolchain_config_repo//:gvm (9dc7412)
@graalvm_toolchain_config_repo//:prefix_version_setting (9dc7412)
@graalvm_toolchain_config_repo//:toolchain (9dc7412)
@graalvm_toolchain_config_repo//:version_or_prefix_version_setting (9af774c)
@graalvm_toolchain_config_repo//:version_or_prefix_version_setting (9dc7412)
@graalvm_toolchain_config_repo//:version_setting (9dc7412)

graalvm looks like:

# ...
@graalvm//:sdk (9af774c) → @graalvm//:toolchain_gvm
@graalvm//:toolchain (9af774c) → @graalvm_toolchain_config_repo//:toolchain
@graalvm//:toolchain_gvm (9dc7412) → (see below)
# ...

toolchain(
  name = "toolchain_gvm",
  toolchain_type = "//graalvm/toolchain:toolchain",
  exec_compatible_with = [],
  target_compatible_with = [],
  toolchain = "@_main~graalvm~graalvm//:gvm",
)

graalvm_sdk(
  name = "gvm",
  native_image_bin = "@_main~graalvm~graalvm//:native-image",
  gvm_files = "@_main~graalvm~graalvm//:files",
)

So, is the above correct with regard to the Java toolchain, or are you referring to the GVM toolchain used to resolve native-image? And if the latter, would moving it to match the Java toolchain structure shown fix the issue? Thanks for your advice here, it is very helpful.

[fmeum]

Yes, I was specifically referring to the toolchain for native_image, which I believe is in a single repo and would benefit from this optimization.

[sgammon]

@fmeum understood, sorry, I was only clarifying because I was wondering if this pattern I saw with Java Toolchains was one and the same with what you're talking about. I should be able to have this change in quickly.