-
Notifications
You must be signed in to change notification settings - Fork 546
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sccache-dist
: cache can end up somehow corrupted
#2122
Comments
sccache-dist
: cache can end up somehow corrupted
It can generate errors like:
|
I hit this same exact issue. Tiny object file, no symbols. |
This was running |
I ran 0.7.1 for a few weeks without a problem, I've switched to 0.7.2 last monday and so far so good. I'll keep slowly bisecting ... |
Facing the same issue on a C++ build. Getting random linker errors. 0.7.1 working fine so far. |
I think I've narrowed it down a bit: In this example the first invocation with sccache results in
(This is with 0.7.2, can someone test 0.8.0?) |
I've tested 0.7.1, 0.7.2 and 0.8.0 now. 0.7.1 is not affected. 0.7.2 and 0.8.0 are affected (only when distributed compilation is configured). Steps to reproduce: https://gist.github.com/johnLate/9ea390dfa59a4641cfc8ff20263ce14b |
@johnLate I know it can be painful but having an exact changeset would help a lot :) |
@sylvestre You weren't kidding. This was my first time using the rust compiler and each "cargo build" took more than 6 minutes.
|
FTR I am running on 0.8.0 for a few days and I have that in my config:
So far I think I have not yet hit the issue. |
That was the point: verifying this was the problem. Between reduced performance and broken builds, I've made my choice :) |
@johnLate thanks for all the effort, this is quite helpful. I have limited time to look into this now, but I wanted to at least get an idea. I see that when implemented the PCM I even left a FYI the slowness in compilation is mainly due to linker + linker settings, removing
from the |
|
@Alphare does this help? Locally, I cannot find content matching |
This is caused by this: https://github.com/mozilla/sccache/blob/main/src/compiler/c.rs#L444 |
So, the way the preprocessor cache works, it can't work with sccache-dist. The preprocessor output is not cached (because local compilations don't need it). The immediate fix would be to disable preprocessor cache when distributed compilation is enabled. For a longer term fix, the preprocessor output would need to be added, and it would also need to lead to new preprocessor cache keys, because otherwise the preprocessor cache from a local compilation (or from an older compilation) would break things. |
#2173 will address the immediate problem. |
PoC fix: https://paste.mozilla.org/0g2pVJZw (expires in a week) |
Note to future self: the preprocessor cache is not compressed, and keeping raw preprocessor output in there is not going to go well. |
indeed, 15GB of |
Release 0.8.1 "fixed" it |
I have no STR unfortunately, but I was using
sccache-dist
circa ~0.7.1pre and it was running fine. Recently updated to 0.7.4 and intermittently I end up with mozilla-central failing to perform some link step, at various places.After long and extensible debugging, one should note the following:
.o
from objdir (pppolicy.o
for the example)dist-client
feature set) log shows cache it on the same.o
Looking at it closely:
On a successfull builds, it's bigger:
The bogus one content's through
objdump -tTC
shows no symbolChecking the local
sccache
content,277 bytes is obviously weird. Getting rid of this cache key and the matching objdir file, issuing a rebuild:
And no more linkage failure (on this file at least). I have no idea how it got corrupted at first.
The text was updated successfully, but these errors were encountered: