[7.1.0] Sandbox error when building with Docker: Invalid cross-device link

[grische]

Description of the bug:

When building two consecutive builds with Bazel inside a Docker container, the second build fails with:

#14 0.129 Starting local Bazel server and connecting to it...
#14 1.523 Computing main repo mapping: 
#14 2.333 Loading: 
#14 2.336 Loading: 0 packages loaded
#14 2.434 ERROR: Failed to initialize sandbox: /root/.cache/bazel/_bazel_root/20f0ced2adfccd482c7c7a0094d96fba/sandbox/_moved_trash_dir -> /root/.cache/bazel/_bazel_root/20f0ced2adfccd482c7c7a0094d96fba/sandbox/stale-trash-0 (Invalid cross-device link)
#14 2.453 INFO: Elapsed time: 2.303s
#14 2.454 INFO: 0 processes.
#14 2.454 ERROR: Build did NOT complete successfully

See https://github.com/freifunkMUC/wgkex/actions/runs/8317435033/job/22758237573

Which category does this issue belong to?

CLI

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

git clone https://github.com/freifunkMUC/wgkex.git
cd wgkex
git checkout b19e1473df04aa78a3355ad2c36f546e821cf79c
docker build -t wgkex -f Dockerfile .

Which operating system are you running Bazel on?

Ubuntu - Github Actions

What is the output of bazel info release?

release 7.1.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

https://github.com/freifunkMUC/wgkex.git
b19e1473df04aa78a3355ad2c36f546e821cf79c

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

Yes, this regression first occurred in 7.1.0 and is not present in 7.0.2.

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[fmeum]

@bazel-io fork 7.1.1

[fmeum]

Cc @oquenchil

[meteorcloudy]

@grische Can you please use https://github.com/bazelbuild/bazelisk?tab=readme-ov-file#--bisect to identify which commit introduced the breakage?

[T0biii]

I hope this helps:

(RUN ["./bazelisk-linux-amd64", "--bisect=release-7.0.2..7.1.0", "build", "//wgkex/broker:app"])
https://github.com/T0biii/wgkex/actions/runs/8341566011/job/22827898807#step:7:740
#17 333.6 first bad commit is https://github.com/bazelbuild/bazel/commit/fa3491359e1050f25e89ff8f63c3c5f915680b5f

(RUN ["./bazelisk-linux-amd64", "--bisect=release-7.0.2..7.1.0", "build", "//wgkex/worker:app"])
https://github.com/T0biii/wgkex/actions/runs/8341566011/job/22827898807#step:7:953
#18 309.5 first bad commit is https://github.com/bazelbuild/bazel/commit/fa3491359e1050f25e89ff8f63c3c5f915680b5f

[meteorcloudy]

I didn't see the ERROR: Failed to initialize sandbox in the log, but FATAL: Attempted to kill stale server process (pid=484) using SIGKILL, but it did not die in a timely fashion.. So something was wrong. Maybe you can set BAZELISK_CLEAN to true and retry.

Meanwhile, @oquenchil is already looking into this

[iancha1992]

A fix for this issue has been included in Bazel 7.1.1 RC2. Please test out the release candidate and report any issues as soon as possible.
If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=7.1.1rc2. Thanks!

[T0biii]

seems to work fine now:
The Actions is Green now :)
https://github.com/T0biii/wgkex/actions/runs/8351122134/job/22858898347#step:7:460

#17 [builder 8/9] RUN ["./bazelisk-linux-amd64", "build", "//wgkex/broker:app"]
#17 0.054 2024/03/19 22:48:48 Downloading https://releases.bazel.build/7.1.1/rc2/bazel-7.1.1rc2-linux-x86_64...
#17 0.938 Extracting Bazel installation...
#17 2.417 Starting local Bazel server and connecting to it...
#17 3.882 Computing main repo mapping: 
#17 3.892 WARNING: --enable_bzlmod is set, but no MODULE.bazel file was found at the workspace root. Bazel will create an empty MODULE.bazel file. Please consider migrating your external dependencies from WORKSPACE to MODULE.bazel. For more details, please refer to [https://github.com/bazelbuild/bazel/issues/18958.](https://github.com/bazelbuild/bazel/issues/18958.?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc)
#17 4.885 Computing main repo mapping: 
#17 5.663 Loading: 
#17 5.669 Loading: 0 packages loaded
#17 6.886 Loading: 0 packages loaded
#17 6.886     currently loading: wgkex/broker
#17 11.92 Analyzing: target //wgkex/broker:app (1 packages loaded)
#17 12.07 Analyzing: target //wgkex/broker:app (1 packages loaded, 0 targets configured)
#17 12.09 Analyzing: target //wgkex/broker:app (1 packages loaded, 0 targets configured)
#17 12.09 [0 / 1] checking cached actions
#17 13.09 Analyzing: target //wgkex/broker:app (61 packages loaded, 278 targets configured)
#17 13.09 [1 / 1] checking cached actions
#17 13.99 INFO: Analyzed target //wgkex/broker:app (85 packages loaded, 895 targets configured).
#17 14.09 INFO: Found 1 target...
#17 14.10 Target //wgkex/broker:app up-to-date:
#17 14.10   bazel-bin/wgkex/broker/app
#17 14.16 INFO: Elapsed time: 13.152s, Critical Path: 0.06s
#17 14.16 INFO: 6 processes: 6 internal.
#17 14.16 INFO: Build completed successfully, 6 total actions

https://github.com/T0biii/wgkex/actions/runs/8351122134/job/22858898347#step:7:513

#18 [builder 9/9] RUN ["./bazelisk-linux-amd64", "build", "//wgkex/worker:app"]
#18 0.131 Starting local Bazel server and connecting to it...
#18 1.615 Computing main repo mapping: 
#18 2.418 Loading: 
#18 2.420 Loading: 0 packages loaded
#18 2.648 Analyzing: target //wgkex/worker:app (1 packages loaded, 0 targets configured)
#18 2.671 Analyzing: target //wgkex/worker:app (1 packages loaded, 0 targets configured)
#18 2.671 [0 / 1] checking cached actions
#18 3.884 INFO: Analyzed target //wgkex/worker:app (78 packages loaded, 906 targets configured).
#18 3.906 [1 / 5] [Prepa] Expanding template wgkex/worker/app
#18 4.021 INFO: Found 1 target...
#18 4.034 Target //wgkex/worker:app up-to-date:
#18 4.035   bazel-bin/wgkex/worker/app
#18 4.059 INFO: Elapsed time: 3.901s, Critical Path: 0.06s
#18 4.060 INFO: 5 processes: 5 internal.
#18 4.061 INFO: Build completed successfully, 5 total actions