bazel mod tidy fails when nothing changes

[tyler-french]

Description of the bug:

MODULE.bazel is correct, I then run bazel mod tidy and get an error:

% bazel mod --enable_bzlmod tidy       
ERROR: Unexpected error while running buildozer: Process exited with status 3:
. Type 'bazel help mod' for syntax and help.

Next, I mess up the formatting (but not the syntax), and run bazel mod tidy again, and it succeeds:

% bazel mod --enable_bzlmod tidy 

Next, I run it again, and Bazel crashes

% bazel mod --enable_bzlmod tidy
 checking cached actions
    Fetching repository @@bazel_tools; starting
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'com.google.devtools.build.lib.bazel.bzlmod.BazelDepGraphValue$$Lambda/0x0000000800590400@11c76f5f' (requested by nodes 'com.google.devtools.build.lib.bazel.bzlmod.BazelModTidyValue$$Lambda/0x0000000800593800@5733c27d')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:550)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
        at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.NullPointerException: null value in entry: googleapis=null
        at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33)
        at com.google.common.collect.ImmutableMapEntry.<init>(ImmutableMapEntry.java:54)
        at com.google.common.collect.ImmutableMap.entryOf(ImmutableMap.java:341)
        at com.google.common.collect.ImmutableMap$Builder.put(ImmutableMap.java:450)
        at com.google.devtools.build.lib.bazel.bzlmod.Module.getRepoMappingWithBazelDepsOnly(Module.java:67)
        at com.google.devtools.build.lib.bazel.bzlmod.BazelDepGraphFunction.getExtensionUsagesById(BazelDepGraphFunction.java:233)
        at com.google.devtools.build.lib.bazel.bzlmod.BazelDepGraphFunction.compute(BazelDepGraphFunction.java:126)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
        ... 7 more

This will continue with all --enable_bzlmod builds until MODULE.bazel.lock is deleted

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

See description

Which operating system are you running Bazel on?

Linux AMD 64

What is the output of bazel info release?

% bazel info release Starting local Bazel server and connecting to it... INFO: Invocation ID: f7b2a282-a126-4ed2-9b9b-a7eab1906e22 release 7.1.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

Internal Uber codebase

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

https://github.com/bazelbuild/bazel/pull/21265/files

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[fmeum]

@Wyverald Looks like two separate bugs to me. I can fix the exit code behavior, but not sure what the stack trace is about.

[fmeum]

@tyler-french Could you share where Uber Bazel Extensions commands: is coming from? Is that a Bazel wrapper?

[fmeum]

@bazel-io fork 7.1.1

[tyler-french]

@tyler-french Could you share where Uber Bazel Extensions commands: is coming from? Is that a Bazel wrapper?

@fmeum This is just a bug in our wrapper, it is succeeding, just not recognizing mod as a command.

Let me remove this from the Bug

    known_cmds = {
        "analyze-profile", "aquery", "build", "canonicalize-flags", "clean",
        "coverage", "cquery", "dump", "fetch", "help", "info", "license",
        "mobile-install", "modquery", "print_action", "query", "run",
        "shutdown", "sync", "test", "version", "debug"
    }

[tyler-french]

Here's our MODULE.bazel file with many things redacted:

module(name = "go-code")

# Version Declarations

bazel_gazelle_version = "0.35.0"

...

# Bazel Modules
bazel_dep(name = "bazel_skylib", version = bazel_skylib_version)
bazel_dep(name = "com_google_googleapis", version = googleapis_revision, repo_name = "googleapis")
bazel_dep(name = "gazelle", version = bazel_gazelle_version, repo_name = "bazel_gazelle")
bazel_dep(name = "hermetic_cc_toolchain", version = hermetic_cc_toolchain_version)
bazel_dep(name = "platforms", version = platforms_version)
bazel_dep(name = "rules_go", version = rules_go_version, repo_name = "io_bazel_rules_go")
bazel_dep(name = "rules_proto", version = rules_proto_version)
bazel_dep(name = "rules_python", version = rules_python_version)
bazel_dep(name = "rules_python_gazelle_plugin", version = rules_python_version, repo_name = "com_github_bazelbuild_rules_python_gazelle")

# Overrides
# An archive_override is required because we can't patch the MODULE.bazel file using a single_version_override.
archive_override(
    module_name = "rules_go",
    ...
)

single_version_override(
    module_name = "gazelle",
    ...
)

archive_override(
    module_name = "com_google_googleapis",
    patches = [
        "//patches:googleapis_add_bzlmod_support.patch",  # https://github.com/googleapis/googleapis/pull/855
        "//patches:googleapis.patch",
    ],
    ...
)

###########################################
####### SDK/Toolchain Registrations #######
###########################################

go_sdk = use_extension("@io_bazel_rules_go//go:extensions.bzl", "go_sdk")
go_sdk.download(
    name = "go_sdk",
    patches = [
        ...
    ],
    sdks = {
        ...
    }
)
go_sdk.nogo(
    nogo = "//:default_nogo",
)
use_repo(go_sdk, "go_sdk")

toolchains = use_extension("@hermetic_cc_toolchain//toolchain:ext.bzl", "toolchains")
use_repo(toolchains, "zig_sdk")

# These C++ toolchains will only be considered when @zig_sdk//libc:flavor
# is set. That is usually done by providing a `--platforms=...` or by a
# transition.
register_toolchains(
    ...
)

# Repos and Extentions
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    configure_coverage_tool = True,
    is_default = True,
    python_version = python_version,
)

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "pip",
    python_version = python_version,
    requirements_lock = "//third_party/python:requirements.lock",
)

artifactory = use_extension("//third_party:artifactory.bzl", "artifactory")
use_repo(artifactory, "artifactory_urls")

###########################################
############ HTTP DEPENDENCIES ############
###########################################

...

switched_rules = use_extension("@googleapis//:extensions.bzl", "switched_rules")
switched_rules.use_languages()
use_repo(switched_rules, "com_google_googleapis_imports")

###########################################
############# GO DEPENDENCIES #############
###########################################

# Load all go repos from the go.mod file.
go_deps = use_extension("@bazel_gazelle//:extensions.bzl", "go_deps")
go_deps.from_file(go_mod = "//src/code.uber.internal:go.mod")

# Similar to `gazelle update-repos`, apply default attributes to all go_deps.
go_deps.gazelle_default_attributes(
    build_extra_args = ["-go_naming_convention_external=go_default_library"],
    build_file_generation = "on",
    directives = [
        "gazelle:proto disable",
    ],
)

########################################################################
# BELOW are overrides for Go dependencies
# TODO(tfrench): this should automatically populated by https://github.com/bazelbuild/bazel-gazelle/pull/1677

...

# This is a minimal set of dependencies needed to build a few targets in Go Monorepo.
# Once WORKSPACE is removed, this will need to list all directly depended on Go Dependencies.
use_repo(
    go_deps,
    "com_github_gogo_protobuf",
    ...
)

[fmeum]

One half of this is fixed by #21657.

@Wyverald Are you looking into the other failure or should I?

[Wyverald]

sorry, I haven't had time yet (busy with some other stuff today). Will take a look in a bit if you can't get to it!

[fmeum]

I can reproduce this locally. What's quite interesting is that googleapis having a version set in bazel_dep despite being subject to a non-registry override is required for the crash.

It's likely that this regressed when I removed the name attribute from the lockfile. I can dig deeper and submit a fix.

[iancha1992]

A fix for this issue has been included in Bazel 7.1.1 RC1. Please test out the release candidate and report any issues as soon as possible.
If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=7.1.1rc1. Thanks!

[stagnation]

I hit this with 7.1.1
When trying to convert bb-deployments to use only bzlmod.
buildbarn/bb-deployments#136

$ bazel version
Bazelisk version: development
Starting local Bazel server and connecting to it...
Build label: 7.1.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Thu Mar 21 18:08:37 2024 (1711044517)
Build timestamp: 1711044517
Build timestamp as int: 1711044517

bb-deployments $ bazel build //... 
Starting local Bazel server and connecting to it...
Computing main repo mapping:
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'com.google.devtools.build.lib.bazel.bzlmod.BazelDepGraphValue$$Lambda/0x00000008004beb48@6c9547d' (requested by nodes 'Key{repoName=@@, rootModuleShouldSeeWorkspaceRepos=true}')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:550)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
        at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.NullPointerException: null value in entry: googletest=null
        at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33)
        at com.google.common.collect.ImmutableMapEntry.<init>(ImmutableMapEntry.java:54)
        at com.google.common.collect.ImmutableMap.entryOf(ImmutableMap.java:341)
        at com.google.common.collect.ImmutableMap$Builder.put(ImmutableMap.java:450)
        at com.google.devtools.build.lib.bazel.bzlmod.Module.getRepoMappingWithBazelDepsOnly(Module.java:67)
        at com.google.devtools.build.lib.bazel.bzlmod.BazelDepGraphFunction.getExtensionUsagesById(BazelDepGraphFunction.java:233)
        at com.google.devtools.build.lib.bazel.bzlmod.BazelDepGraphFunction.compute(BazelDepGraphFunction.java:126)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
        ... 7 more

Twiddling a little bit with the module file it sometimes prints this warning:

WARNING: For repository 'googletest', the root module requires module version [email protected], but got [email protected] in the resolved dependency graph.

$ rg [email protected]
MODULE.bazel.lock
507:    "[email protected]": {
510:      "key": "[email protected]",
682:        "com_google_googletest": "[email protected]",
1661:        "com_google_googletest": "[email protected]",
2783:        "com_google_googletest": "[email protected]",
3447:        "com_google_googletest": "[email protected]",

[fmeum]

@stagnation Could you try deleting your lockfile once? If the lockfile was generated by pre-7.1.1 bazel mod tidy, you will still see the issue with 7.1.1.

[stagnation]

Cheers, I guess that works or something similar. I just tried a build with last_green which kicked it just right to start working again.