cant build with bazel and ccache

[kreuzerkrieg]

Description of the problem / feature request:

Bazel fails to build tcmalloc with message

ccache: error: Failed to create temporary file for /home/user/.ccache/tmp/string_uti.stdout: Read-only file system

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I guess no special steps needed. ccache works fine, I'm able to build without using sudo with CMake, for example

What operating system are you running Bazel on?

Ubuntu 20.04

What's the output of bazel info release?

release 3.5.0

Have you found anything relevant by searching the web?

semi-related
#1540
but I installed the Bazel using apt

Any other information, logs, or outputs that you want to share?

worked as expected once I ran Bazel under sudo, but I guess it is not the expected behavior
let me know if you need any additional information

[jiridanek]

My guess is that bazel sandboxing is blocking access. If that is the case, try building with --sandbox_writable_path=$HOME/.ccache

Your workaround using sudo suggests sandboxing may not be to blame, but the option may be worth trying anyways. If file permissions on your file system are to blame, then try sudo chown -R $USER:$USER $HOME/.ccache before running bazel. There may be files/directories owned by root user in your .ccache, which then cannot be written into by regular user.

[jiridanek]

It is usually useful to run bazel with -s --sandbox_debug --verbose_failures and include the output from this.

[kreuzerkrieg]

It is usually useful to run bazel with -s --sandbox_debug --verbose_failures and include the output from this.

sandbox_debug.txt

here you go

[kreuzerkrieg]

My guess is that bazel sandboxing is blocking access. If that is the case, try building with --sandbox_writable_path=$HOME/.ccache

Your workaround using sudo suggests sandboxing may not be to blame, but the option may be worth trying anyways. If file permissions on your file system are to blame, then try sudo chown -R $USER:$USER $HOME/.ccache before running bazel. There may be files/directories owned by root user in your .ccache, which then cannot be written into by regular user.

as of sudo chown -R $USER:$USER $HOME/.ccache cmake with gnumake or ninja works fine with this folder without any sudo, so I guess chown should make no difference, am I missing anything?

[jiridanek]

so I guess chown should make no difference, am I missing anything?

Your reasoning makes sense. I just personally prefer one bold try with chown over doing much up-front thinking about causes; I prefer to do that afterwards. That's all. edit: and if you mix running commands with sudo and without, its easy to create temp files owned by root in various places.

Sandboxing can be switched off, or you may try less powerfull sandbox, by doing either --spawn_strategy=standalone or --spawn_strategy=processwrapper-sandbox. This is what I'd certainly try, besides --sandbox_writable_path=$HOME/.ccache I previously suggested.

Previous instances of similar issues I was able to find were due to sandboxing

• https://stackoverflow.com/questions/49178997/bazel-failed-to-create-temporary-file
• https://stackoverflow.com/questions/52370202/how-to-get-bazel-ccache-and-sandboxing-to-work-together-ccache-read-only-file

edit: in any case, you've attached the verbose log. That should be enough information for somebody more knowledgeable about bazel to figure things out. You could maybe add the command you use to run bazel, and how you tell it to use ccache, as further information.

[jiridanek]

BTW, bazel has a background daemon. It is sometimes useful to kill it and restart it, because it might be remembering some now out-of-date state, with bazel shutdown. Probably not the case here, but its helpful to think of this when you want to try some steps again with a completely clean state.

[kreuzerkrieg]

sudo chown -R $USER:$USER ~/.ccache same result
bazel shutdown same result
bazel build --spawn_strategy=standalone //tcmalloc/... worked as expected

Dont get it, am I the only one who have a problem with sandboxing? it is quite fresh machine, 3 or 4 weeks, I remember having the same problem with previous machine with Ubuntu 18.04. Mystery...

[jiridanek]

In that case, what does using the sandbox writable path argument, that is --sandbox_writable_path=$HOME/.ccache do?

[kreuzerkrieg]

worked too

[jiridanek]

Awesome. That's expected behavior. Sandboxing limits the writable paths the build can write into. If you want ccache to work from inside a Bazel build with sandboxing turned on, you have to add $HOME/.ccache to the write allowlist. This is how the other Bazel+ccache issues I found turned out, too.

One thing I don't understand is why sudo also ended up, in effect, disabling the sandbox.

[kreuzerkrieg]

then it is odd the problem is not easily searchable on the net, since it sounds as something quite usual.
BTW, where is I'm whitelisting .ccache folder? I dont want to (remember and) add it to command line each time I run bazel

[jiridanek]

You can create config file $HOME/.bazelrc with the option, https://docs.bazel.build/versions/master/guide.html#bazelrc-syntax-and-semantics.

[susinmotion]

It looks like this is a documentation request at this point, to clarify that users should pass --sandbox_writeable_path to enable ccache with sandboxing. Does that sound right to you?

[kreuzerkrieg]

Whatever discoverable by google search will do, I guess

[blackliner]

I do have problems with that solution. If I add --sandbox_writable_path=/home/my_user/.ccache to the %workspace%/.bazelrc file, it works.
But since not every developer has ccache installed, I cannot blindly add that option to the repo. So I added it to the $HOME/.bazelrc, but it is not picked up by bazel... Any thoughts?

EDIT: sorry, not even adding it to the %workspace%/.bazelrc helps, it must be passed when running bazel test .... This changed within the last few bazel versions :-(

[jiridanek]

build --sandbox_writable_path=/home/user/.ccache

in bazel.rc should work. What does not work is using env variables, such as $HOME in .bazel.rc. #10904. And env variables there never worked, as far as I remember.

[blackliner]

Thanks, I think I forgot the build...

[irfansharif]

I get that whitelisting this .ccache path works, but this is still pretty surprising behaviour to have by default. We're trying to roll-out bazel usage for engineers at cockroachdb/cockroach, and it's unfortunate to have to ask each person include this (custom, because of $USERNAME) path in their local .bazelrc.user or something.

Fundamentally though, I'm not sure why bazel is relying on the ccache-wrapped compiler doing it's own caching when bazel is perfectly capable of caching the generated artifacts. Is there any benefit I'm no seeing? Very naively, if bazel always exported CCACHE_DISABLE=1, to disable any sort of compiler-initiated caching (with artifacts placed outside the sandbox!), that'd be more in-line with what I'd expect. I think it's also more "bazel"-like, since this external write is clearly unexpected. Perhaps even surfacing this as a "--disable-ccache" option could work, though I'd still push for this to be the default. All that said, I'm glad to have found this issue -- would've spent many more hours on cockroachdb/cockroach#66988 if I hadn't.

[irfansharif]

The workarounds I've seen elsewhere (RobotLocomotion/drake#4464, RobotLocomotion/drake#5130) feel a tad-bit unfortunate. It's similar to what we're thinking of in cockroachdb/dev#11, and feels more appropriate to fix up-stream. Curious to hear your thoughts.

[GMNGeoffrey]

Please don't make disabling the thing the user tried to do the default. The Bazel cache is not something shared between build systems, whereas ccache is. It would be great if Bazel worked better with ccache though, and especially if the known issues were documented (currently the top result is a stackoverflow answer, which isn't the worst)

[diagprov]

Discovered this issue building https://github.com/ankitects/anki on my Fedora 36 system. The command I invoked was:

bazel build --sandbox_writable_path="$HOME/.ccache" --config opt wheels

and ccache complains it cannot locate gcc on the path, which is somewhat amusing. /usr/lib64/ccache/gcc does in fact exist.

The workaround for me was to explicitly define CC=/usr/bin/gcc and CXX=/usr/bin/g++ to force Bazel to not use ccache at all as per https://stackoverflow.com/a/53517061. I have tried all the steps indicated in this thread (server shutdown, and even bazel cleans between runs).

$ lsb_release --all
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	Fedora
Description:	Fedora release 36 (Thirty Six)
Release:	36
Codename:	ThirtySix
$ uname -sm
Linux x86_64
$ bazel version
Bazelisk version: v1.12.0
Build label: 5.2.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jun 7 16:02:26 2022 (1654617746)
Build timestamp: 1654617746
Build timestamp as int: 1654617746

This feels like surprisingly brittle behaviour.

[createyourpersonalaccount]

On Fedora 36 I get

ccache: error: Failed to create temporary file for /run/user/1000/ccache-tmp/tmp.cpp_stdout.6cAvv5: Read-only file system

whether I use the sandbox option or not, installed from npm or binary.

[StLeoX]

Me too.
I remounted the ccache directory to /tmp/ccache, which is rw.
It doesn't work either.

[jiridanek]

@createyourpersonalaccount I have the same problem now, also on Fedora 36. Trying to build for example this target in bazel itself

$ cd bazel
$ bazel build src/tools/remote:worker_deploy.jar
Starting local Bazel server and connecting to it...
INFO: Analyzed target //src/tools/remote:worker_deploy.jar (174 packages loaded, 4597 targets configured).
INFO: Found 1 target...
ERROR: /home/jdanek/.cache/bazel/_bazel_jdanek/45e00946e5a24da511342660c0186815/external/com_google_protobuf/BUILD:155:11: Compiling src/google/protobuf/parse_context.cc failed: (Exit 1): gcc failed: error executing command /usr/lib64/ccache/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 29 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
ccache: error: Failed to create temporary file for /run/user/1000/ccache-tmp/tmp.cpp_stdout.hry1JF: Read-only file system
Target //src/tools/remote:worker_deploy.jar failed to build
Use --verbose_failures to see the command lines of failed build steps.

I can fix this by making the cache directory accessible from sandbox, by adding the appropriate parameter (as discussed in the previous comments on this issue), covering all the paths that the build desires write access to

$ mkdir ~/.ccache
$ bazel build src/tools/remote:worker_deploy.jar --sandbox_writable_path="$HOME/.ccache" --sandbox_writable_path="/run/user/$UID/ccache-tmp/"

(If I did not create .ccache directory beforehand, I'd get IO error about the directory not existing)