[7.2.0] Unrecoverable error while evaluating node 'REPOSITORY_DIRECTORY:@@WORKSPACE.bazel'

[criemen]

Description of the bug:

My colleague hit this internal exception while developing on our bazel-wrapper script.
Our test run first builds some targets (works), and then runs a test invocation, which immediately crashes.
WORKSPACE.bazel exists, but is empty. Maybe the in-development wrapper messes some paths up, but bazel shouldn't crash like this.
We've not attempted to debug this further, hoping that the exception stack trace is enough for you to identify a problem on your side.

  tools/bazel test --test_tag_filters=glibc-symbols-check //... @codeql//ruby/... @codeql//python/...
  shell: /usr/bin/bash -e {0}
  env:
    JAVA_HOME: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/17.0.11-9/x64
    JAVA_HOME_17_X64: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/17.0.11-9/x64
    pythonLocation: /opt/hostedtoolcache/Python/3.12.3/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.12.3/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.3/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.3/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.3/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.12.3/x64/lib
    DEBIAN_FRONTEND: noninteractive
2024/06/12 16:22:01 Using unreleased version at commit 76286491d0bacaf790ea8e6fa76ae5bdf6dd3db2
INFO: Invocation ID: ebf3771a-239e-44db-a951-5735bdfd10f5
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'REPOSITORY_DIRECTORY:@@WORKSPACE.bazel' (requested by nodes '[/home/runner/.cache/bazel/_bazel_runner/9f4ed60bba64c867c2242caf607221b5]/[external/WORKSPACE.bazel]')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:550)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.ClassCastException: class com.google.devtools.build.lib.packages.InputFile cannot be cast to class com.google.devtools.build.lib.packages.Rule (com.google.devtools.build.lib.packages.InputFile and com.google.devtools.build.lib.packages.Rule are in unnamed module of loader 'app')
	at com.google.devtools.build.lib.packages.Package.getRule(Package.java:615)
	at com.google.devtools.build.lib.repository.ExternalPackageHelper$ExternalPackageRuleExtractor.processAndShouldContinue(ExternalPackageHelper.java:144)
	at com.google.devtools.build.lib.repository.ExternalPackageHelper.iterateWorkspaceFragments(ExternalPackageHelper.java:118)
	at com.google.devtools.build.lib.repository.ExternalPackageHelper.getRuleByName(ExternalPackageHelper.java:52)
	at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.getRepoRuleFromWorkspace(RepositoryDelegatorFunction.java:459)
	at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.getRepositoryRule(RepositoryDelegatorFunction.java:330)
	at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.compute(RepositoryDelegatorFunction.java:155)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
	... 7 more

CC @redsun82

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

If the stack trace isn't enough to debug this, we can try working out a reproducer.

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

development version

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

bazelisk, commit 7628649 (the tip of the 7.2.1 release branch a few days ago), basically 7.2.0 with one commit applied.

What's the output of git remote get-url origin; git rev-parse HEAD ?

N/A

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

We've not tried to identify if this is a regression yet

Have you found anything relevant by searching the web?

Not looked yet

Any other information, logs, or outputs that you want to share?

No response

[meisterT]

It would be great if you could test with Bazel 7.0 and Bazel 7.1 as well

[criemen]

7.1.1 (7.1.0 was deadlocking due to the restartless fetch changes):

2024/06/13 19:40:58 Downloading https://releases.bazel.build/7.1.1/release/bazel-7.1.1-linux-x86_64...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: e4966b3d-3435-4ad5-b0b6-5e931dd62617
Loading: 15 packages loaded
    currently loading:  ... (157 packages)
    Fetching repository @@rules_python~; starting
    Fetching repository @@rules_cc~; starting
    Fetching repository @@contrib_rules_jvm~; starting
    Fetching ...despace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98/external/rules_cc~; Extracting rules_cc-0.0.9.tar.gz
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'REPOSITORY_DIRECTORY:@@WORKSPACE.bazel' (requested by nodes '[/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98]/[external/WORKSPACE.bazel]')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:550)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
        at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.ClassCastException: class com.google.devtools.build.lib.packages.InputFile cannot be cast to class com.google.devtools.build.lib.packages.Rule (com.google.devtools.build.lib.packages.InputFile and com.google.devtools.build.lib.packages.Rule are in unnamed module of loader 'app')
        at com.google.devtools.build.lib.packages.Package.getRule(Package.java:615)
        at com.google.devtools.build.lib.repository.ExternalPackageHelper$ExternalPackageRuleExtractor.processAndShouldContinue(ExternalPackageHelper.java:144)
        at com.google.devtools.build.lib.repository.ExternalPackageHelper.iterateWorkspaceFragments(ExternalPackageHelper.java:118)
        at com.google.devtools.build.lib.repository.ExternalPackageHelper.getRuleByName(ExternalPackageHelper.java:52)
        at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.getRepoRuleFromWorkspace(RepositoryDelegatorFunction.java:451)
        at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.getRepositoryRule(RepositoryDelegatorFunction.java:326)
        at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.compute(RepositoryDelegatorFunction.java:151)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
        ... 7 more

7.0.0

2024/06/13 19:42:11 Downloading https://releases.bazel.build/7.0.0/release/bazel-7.0.0-linux-x86_64...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 9db6154e-a6d1-41ed-89a0-d966df454a39
Loading: 2 packages loaded
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'REPOSITORY_DIRECTORY:@@WORKSPACE.bazel' (requested by nodes '[/home/codespace/.cache/bazel/_bazel_codespace/3c41b00dfcd8c4fb714360e042371c98]/[external/WORKSPACE.bazel]')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:550)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
        at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.ClassCastException: class com.google.devtools.build.lib.packages.InputFile cannot be cast to class com.google.devtools.build.lib.packages.Rule (com.google.devtools.build.lib.packages.InputFile and com.google.devtools.build.lib.packages.Rule are in unnamed module of loader 'app')
        at com.google.devtools.build.lib.packages.Package.getRule(Package.java:625)
        at com.google.devtools.build.lib.repository.ExternalPackageHelper$ExternalPackageRuleExtractor.processAndShouldContinue(ExternalPackageHelper.java:144)
        at com.google.devtools.build.lib.repository.ExternalPackageHelper.iterateWorkspaceFragments(ExternalPackageHelper.java:118)
        at com.google.devtools.build.lib.repository.ExternalPackageHelper.getRuleByName(ExternalPackageHelper.java:52)
        at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.getRepoRuleFromWorkspace(RepositoryDelegatorFunction.java:327)
        at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.compute(RepositoryDelegatorFunction.java:155)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
        ... 7 more

[Wyverald]

Looks like something is depending on the FileValue of $ROOT/external/WORKSPACE.bazel. I've no idea how that could happen, so a minimal repro would be appreciated.

(It's a known problem that trying to refer to a repo named "WORKSPACE" or "WORKSPACE.bazel" would cause a crash, but we never got around to fixing that. Incidentally, --noenable_workspace removes the crash.)

[criemen]

Aaaaah, I think the thing that confused bazel terribly here is that our bazel wrapper gained the following code (python):

output_base_symlink = workspace / "bazel-base"
if not output_base_symlink.is_symlink():
    bazel_output_root = subprocess.check_output([bazelisk, 'info', 'output_base'], stderr=subprocess.DEVNULL).strip()
    output_base_symlink.symlink_to(bazel_output_root)

creating a symlink bazel-base in the root workspace/module folder to the bazel output base (to make it easier to inspect for debugging/accessing the command.log file). Then the //... target tries to recurse into that. With common --noenable_workspace we're getting weird errors, too, with that setup, putting the symlink into .bazelignore solves the crash.

I don't think there's a bazel bug here per se, but crashing with an internal error and a ClassCastException is not exactly helpful either. As a user, my expectation would be that regardless of how I try to break bazel, it's not crashing.