hermetic linux sandbox throws error when mounting sandbox directory #14226

psigen · 2021-11-04T18:32:18Z

Description of the problem / feature request:

When using --experimental_use_hermetic_linux_sandbox, even a basic workspace seems to fail with the following error in almost any build action:

src/main/tools/linux-sandbox-pid1.cc:612: "mount(/home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox, /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox, nullptr, MS_BIND | MS_REC, nullptr)": No such file or directory

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I have created a minimal reproduction in the repository: https://github.com/psigen/bazel-issue-14226

Steps to reproduce:

Create a workspace and .bazelrc file.

In the .bazelrc file add the lines

build --experimental_use_hermetic_linux_sandbox
build --sandbox_add_mount_pair "/:/"

bazel build any target in the workspace

What operating system are you running Bazel on?

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"

What's the output of `bazel info release`?

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"

Have you found anything relevant by searching the web?

I have not found any relevant issues in

StackOverflow: http://stackoverflow.com/questions/tagged/bazel
GitHub issues: https://github.com/bazelbuild/bazel/issues
email threads on https://groups.google.com/forum/#!forum/bazel-discuss

The closest I can find is another sandbox mount issue which seems to have a separate cause.

Any other information, logs, or outputs that you want to share?

Here is the full debug output of trying to run a build action in my test workspace:

$ bazel build //:example --sandbox_debug
INFO: Analyzed target //:example (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/pras/bazel-hermetic-sandbox/BUILD.bazel:1:11: Compiling example.cpp failed: (Exit 1): linux-sandbox failed: error executing command 
  (cd /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox && \
  exec env - \
    PATH=/home/pras/.cache/bazelisk/downloads/bazelbuild/bazel-6.0.0-pre.20211025.1-linux-x86_64/bin:/home/pras/.krew/bin:/home/pras/.nvm/versions/node/v12.18.3/bin:/home/pras/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PWD=/proc/self/cwd \
    TMPDIR=/tmp \
  /home/pras/.cache/bazel/_bazel_pras/install/850276ae221e5ad4603a47e9b49bd43a/linux-sandbox -t 15 -w /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox -w /tmp -w /dev/shm -M / -h /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5 -D -- /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/k8-fastbuild/bin/_objs/example/example.pic.d '-frandom-seed=bazel-out/k8-fastbuild/bin/_objs/example/example.pic.o' -fPIC -iquote . -iquote bazel-out/k8-fastbuild/bin -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c example.cpp -o bazel-out/k8-fastbuild/bin/_objs/example/example.pic.o)
1636050087.194696190: src/main/tools/linux-sandbox.cc:152: calling pipe(2)...
1636050087.194748260: src/main/tools/linux-sandbox.cc:171: calling clone(2)...
1636050087.195282399: src/main/tools/linux-sandbox.cc:180: linux-sandbox-pid1 has PID 13827
1636050087.195306567: src/main/tools/linux-sandbox-pid1.cc:641: Pid1Main started
1636050087.195425296: src/main/tools/linux-sandbox.cc:197: done manipulating pipes
1636050087.196284581: src/main/tools/linux-sandbox-pid1.cc:580: mount: /

1636050087.196513283: src/main/tools/linux-sandbox-pid1.cc:609: writable: /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox
src/main/tools/linux-sandbox-pid1.cc:612: "mount(/home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox, /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox, nullptr, MS_BIND | MS_REC, nullptr)": No such file or directory
1636050087.214234533: src/main/tools/linux-sandbox.cc:233: child exited normally with code 1
Target //:example failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.193s, Critical Path: 0.03s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully

The text was updated successfully, but these errors were encountered:

ulrfa · 2021-11-05T10:37:37Z

The error is probably caused by: --sandbox_add_mount_pair "/:/"

The purpose of the hermetic sandbox is to prevent mounting too much in the sandbox.

Bazel will not consider the mount pairs as dependencies, and will not rebuild if those files are changed. So you should typically not mount files that are changing. And especially not map anything containing bazel’s output root.

Mapping the whole root would not make it more hermetic than using the normal linux-sandbox.

psigen · 2021-11-05T11:11:57Z

I used that line specifically in my example code to make the changes as minimal as possible from the non-hermetic environment. I don't see why this should cause a failure to bind mount a completely separate folder though. The error also does not seem to go away if that line is removed.

…

On Fri, Nov 5, 2021, 06:37 Ulrik Falklöf ***@***.***> wrote: The error is probably caused by: --sandbox_add_mount_pair "/:/" The purpose of the hermetic sandbox is to prevent mounting too much in the sandbox. Bazel will not consider the mount pairs as dependencies, and will not rebuild if those files are changed. So you should typically not mount files that are changing. And especially not map anything containing bazel’s output root. Mapping the whole root would not make it more hermetic than using the normal linux-sandbox. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14226 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVM4K7VWENJ6RGYAVZZFJ3UKOXX3ANCNFSM5HMDMHDQ> .

frazze-jobb · 2021-11-05T11:35:04Z

You can't mount anything on /, since the hermetic linux sandbox creates its own root dir '/', and mounts every mount_pair inside that. You can mount --sandbox_add_mount_pair="/:oldRoot" this will mount / in /oldRoot in the hermetic linux sandbox.

psigen · 2021-11-05T11:58:52Z

Got it! If that's the case, is there a preferred strategy for loading a custom rootfs? Like a separate flag for the root mount? (I feel like that is one of the more common use cases for the hermetic sandbox. Do I just need to mount every subdirectory?)

…

On Fri, Nov 5, 2021, 07:35 Fredrik Frantzen ***@***.***> wrote: You can't mount anything on /, since the hermetic linux sandbox creates its own root dir '/', and mounts every mount_pair inside that. You can mount --sandbox_add_mount_pair="/:oldRoot" this will mount / in /oldRoot in the hermetic linux sandbox. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14226 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVM4K2EUUKXSSXK35T5C7LUKO6PHANCNFSM5HMDMHDQ> .

frazze-jobb · 2021-11-05T13:58:26Z

We did not implement support for a custom rootfs. But I guess you could just run bazel with normal linux sandbox inside there?

(Yes, you can mount every individual directory in '/' that you need in your sandbox under /
i.e. --sandbox_add_mount_pair=/usr:/usr)

psigen · 2021-11-05T17:59:30Z

Well, the normal linux sandbox does not let me use a custom rootfs as far as I know (in my actual use case, I have a rootfs exported from the docker image used in my RBE environment, and I do not want to use anything from the host). I only used my own root partition in the example. This is for developers, not CI/CD, so I do not want them to have to do extra steps to work inside the sandbox.

Maybe I can make a PR into the documentation about how mounting / will cause problems? I could see it being the first thing people try if they want to hermetically mount a rootfs generated by Yocto/Docker/etc.

Does anyone have a link to the appropriate place to update the docs for the CLI flags?

psigen · 2021-11-05T18:53:52Z

Ok, I am clearly missing something in how to use --sandbox_add_mount_pair:

I changed the entry in .bazelrc to be the following:

build --experimental_use_hermetic_linux_sandbox
build --sandbox_add_mount_pair "/usr:/usr"

And now I get the following error:

$ bazel build //:example --sandbox_debug
INFO: Analyzed target //:example (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/pras/bazel-hermetic-sandbox/BUILD.bazel:1:11: Compiling example.cpp failed: (Exit 1): linux-sandbox failed: error executing command 
  (cd /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox && \
  exec env - \
    PATH=/home/pras/.cache/bazelisk/downloads/bazelbuild/bazel-6.0.0-pre.20211025.1-linux-x86_64/bin:/home/pras/.krew/bin:/home/pras/.nvm/versions/node/v12.18.3/bin:/home/pras/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PWD=/proc/self/cwd \
    TMPDIR=/tmp \
  /home/pras/.cache/bazel/_bazel_pras/install/850276ae221e5ad4603a47e9b49bd43a/linux-sandbox -t 15 -w /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox -w /tmp -w /dev/shm -M /usr/bin -h /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5 -D -- /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/k8-fastbuild/bin/_objs/example/example.pic.d '-frandom-seed=bazel-out/k8-fastbuild/bin/_objs/example/example.pic.o' -fPIC -iquote . -iquote bazel-out/k8-fastbuild/bin -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c example.cpp -o bazel-out/k8-fastbuild/bin/_objs/example/example.pic.o)
1636138183.250740003: src/main/tools/linux-sandbox.cc:152: calling pipe(2)...
1636138183.250764346: src/main/tools/linux-sandbox.cc:171: calling clone(2)...
1636138183.251008707: src/main/tools/linux-sandbox.cc:180: linux-sandbox-pid1 has PID 5066
1636138183.251023670: src/main/tools/linux-sandbox-pid1.cc:641: Pid1Main started
1636138183.251090767: src/main/tools/linux-sandbox.cc:197: done manipulating pipes
1636138183.251643293: src/main/tools/linux-sandbox-pid1.cc:580: mount: /usr

1636138183.251753936: src/main/tools/linux-sandbox-pid1.cc:609: writable: /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox
1636138183.251801793: src/main/tools/linux-sandbox-pid1.cc:609: writable: /tmp
1636138183.251822521: src/main/tools/linux-sandbox-pid1.cc:609: writable: /dev/shm
1636138183.252463368: src/main/tools/linux-sandbox-pid1.cc:451: calling fork...
1636138183.252580847: src/main/tools/linux-sandbox-pid1.cc:481: child started with PID 2
src/main/tools/linux-sandbox-pid1.cc:478: "execvp(/usr/bin/gcc, 0x1883de0)": No such file or directory
1636138183.253010098: src/main/tools/linux-sandbox-pid1.cc:498: wait returned pid=2, status=0x100
1636138183.253016714: src/main/tools/linux-sandbox-pid1.cc:516: child exited normally with code 1
1636138183.292290393: src/main/tools/linux-sandbox.cc:233: child exited normally with code 1
Target //:example failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.151s, Critical Path: 0.05s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully

"execvp(/usr/bin/gcc, 0x1883de0)": No such file or directory doesn't seem to make sense. I definitely have gcc installed, and I see mount: /usr immediately above in the logs.

frazze-jobb · 2021-11-05T19:19:37Z

Yes, the message is confusing, but whats actually "No such file or directory" is the things that /usr/bin/gcc depends on.
Run
ldd /usr/bin/gcc and you'll see the dynamically linked things that gcc depends on.

$ ldd /usr/bin/gcc
linux-vdso.so.1 => (0x00007ffe1738f000)
libm.so.6 => /lib64/libm.so.6 (0x00007f8c78e01000)
libc.so.6 => /lib64/libc.so.6 (0x00007f8c78a33000)
/lib64/ld-linux-x86-64.so.2 (0x00007f8c79103000)

In our environment we do at the minimum this:

build --sandbox_add_mount_pair=/etc
build --sandbox_add_mount_pair=/usr
 ## symlinks in to /usr
build --sandbox_add_mount_pair=/usr/bin:/bin
build --sandbox_add_mount_pair=/usr/lib:/lib
build --sandbox_add_mount_pair=/usr/lib64:/lib64

psigen · 2021-11-06T16:01:51Z

Getting closer!

I tried this with my directories, and I ran into the following error:

build --sandbox_add_mount_pair=/usr/lib32:/lib32

Mount target '/lib32' does not exist. Bazel only supports bind mounting on top of existing files/directories. Please create an empty file or directory at the mount target path according to the type of mount source.

The exact same flag with /usr/lib64:lib64 works fine. Is there some step to specify what empty directories will be created in the sandbox to use as mount points?

frazze-jobb · 2021-11-09T15:28:20Z

Sorry for taking my time, I had not run into that issue before. But I can replicate it now.

build --sandbox_add_mount_pair=/usr/lib32:/lib32
This fails for me because /usr/lib32 does not exist
build --sandbox_add_mount_pair=/usr/lib:/lib32
This fails for me because /lib32 does not exist in the original filesystem.
build --sandbox_add_mount_pair=/usr/lib:/lib
This works because both source and target exists in the original filesystem

I guess that in the hermetic sandbox case, we should not validate the target paths, but instead create them.

github-actions · 2023-06-09T01:34:21Z

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

psigen · 2023-06-15T14:26:25Z

So... this is definitely still a bug and we were able to replicate and had a specific path forward for a fix.
Should it be marked as awaiting-bazeler?

github-actions · 2024-08-20T01:31:28Z

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.

github-actions · 2024-11-18T01:33:21Z

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please post @bazelbuild/triage in a comment here and we'll take a look. Thanks!

psigen · 2024-11-18T07:36:37Z

@bazelbuild/triage
This is still an active bug, specifically the behavior described here makes it impossible to mount sandbox directories that don't already match existing paths.

gregestren added team-Local-Exec Issues and PRs for the Execution (Local) team untriaged type: bug labels Nov 18, 2021

meisterT added P3 We're not considering working on this, but happy to review a PR. (No assignee) and removed untriaged labels Dec 15, 2021

github-actions bot added the stale Issues or PRs that are stale (no activity for 30 days) label Jun 9, 2023

github-actions bot removed the stale Issues or PRs that are stale (no activity for 30 days) label Jun 16, 2023

github-actions bot added the stale Issues or PRs that are stale (no activity for 30 days) label Aug 20, 2024

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

hermetic linux sandbox throws error when mounting sandbox directory #14226

hermetic linux sandbox throws error when mounting sandbox directory #14226

psigen commented Nov 4, 2021 •

edited

Loading

ulrfa commented Nov 5, 2021

psigen commented Nov 5, 2021 via email

frazze-jobb commented Nov 5, 2021

psigen commented Nov 5, 2021 via email

frazze-jobb commented Nov 5, 2021 •

edited

Loading

psigen commented Nov 5, 2021 •

edited

Loading

psigen commented Nov 5, 2021 •

edited

Loading

frazze-jobb commented Nov 5, 2021 •

edited

Loading

psigen commented Nov 6, 2021 •

edited

Loading

frazze-jobb commented Nov 9, 2021

github-actions bot commented Jun 9, 2023

psigen commented Jun 15, 2023

github-actions bot commented Aug 20, 2024

github-actions bot commented Nov 18, 2024

psigen commented Nov 18, 2024

hermetic linux sandbox throws error when mounting sandbox directory #14226

hermetic linux sandbox throws error when mounting sandbox directory #14226

Comments

psigen commented Nov 4, 2021 • edited Loading

Description of the problem / feature request:

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

What operating system are you running Bazel on?

What's the output of bazel info release?

Have you found anything relevant by searching the web?

Any other information, logs, or outputs that you want to share?

ulrfa commented Nov 5, 2021

psigen commented Nov 5, 2021 via email

frazze-jobb commented Nov 5, 2021

psigen commented Nov 5, 2021 via email

frazze-jobb commented Nov 5, 2021 • edited Loading

psigen commented Nov 5, 2021 • edited Loading

psigen commented Nov 5, 2021 • edited Loading

frazze-jobb commented Nov 5, 2021 • edited Loading

psigen commented Nov 6, 2021 • edited Loading

frazze-jobb commented Nov 9, 2021

github-actions bot commented Jun 9, 2023

psigen commented Jun 15, 2023

github-actions bot commented Aug 20, 2024

github-actions bot commented Nov 18, 2024

psigen commented Nov 18, 2024

psigen commented Nov 4, 2021 •

edited

Loading

What's the output of `bazel info release`?

frazze-jobb commented Nov 5, 2021 •

edited

Loading

psigen commented Nov 5, 2021 •

edited

Loading

psigen commented Nov 5, 2021 •

edited

Loading

frazze-jobb commented Nov 5, 2021 •

edited

Loading

psigen commented Nov 6, 2021 •

edited

Loading