Firefox build failure for win32 with Rust 1.22 (exit code: 3221225477)

[rillian]

Rust 1.22.0-beta.3 (and I think earlier betas) Fail to link the gkrust-gtest crate as part of Firefox builds targeting i686-pc-windows-msvc.

13:39:52     INFO -     Compiling gkrust-gtest v0.1.0 (file:///Z:/task_1510749903/build/src/toolkit/library/gtest/rust)
13:39:52     INFO -       Running `z:/task_1510749903/build/src/rustc/bin/rustc.exe --crate-name gkrust_gtest 'Z:\task_1510749903\build\src\toolkit\library\gtest\rust\lib.rs' --crate-type staticlib --emit=dep-info,link -C opt-level=2 -C panic=abort -C lto --cfg 'feature="bindgen"' --cfg 'feature="gkrust-shared"' --cfg 'feature="no-static-ideograph-encoder-tables"' --cfg 'feature="quantum_render"' --cfg 'feature="servo"' --cfg 'feature="simd-accel"' -C metadata=dcf04380df960a70 -C extra-filename=-dcf04380df960a70 --out-dir 'z:/task_1510749903/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps' --target i686-pc-windows-msvc -L 'dependency=z:/task_1510749903/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps' -L 'dependency=z:/task_1510749903/build/src/obj-firefox/toolkit/library\release\deps' --extern 'nsstring_gtest=z:/task_1510749903/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libnsstring_gtest-10d93009b4b96094.rlib' --extern 'gkrust_shared=z:/task_1510749903/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libgkrust_shared-a27e948f260cfd40.rlib' --extern 'mp4parse_gtest=z:/task_1510749903/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libmp4parse_gtest-0ee4c20075271ff2.rlib' -C opt-level=2 -C debuginfo=2`
13:39:52     INFO -  error: Could not compile `gkrust-gtest`.
13:39:52     INFO -  Caused by:
13:39:52     INFO -    process didn't exit successfully: `z:/task_1510749903/build/src/rustc/bin/rustc.exe --crate-name gkrust_gtest Z:\task_1510749903\build\src\toolkit\library\gtest\rust\lib.rs --crate-type staticlib --emit=dep-info,link -C opt-level=2 -C panic=abort -C lto --cfg feature="bindgen" --cfg feature="gkrust-shared" --cfg feature="no-static-ideograph-encoder-tables" --cfg feature="quantum_render" --cfg feature="servo" --cfg feature="simd-accel" -C metadata=dcf04380df960a70 -C extra-filename=-dcf04380df960a70 --out-dir z:/task_1510749903/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps --target i686-pc-windows-msvc -L dependency=z:/task_1510749903/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps -L dependency=z:/task_1510749903/build/src/obj-firefox/toolkit/library\release\deps --extern nsstring_gtest=z:/task_1510749903/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libnsstring_gtest-10d93009b4b96094.rlib --extern gkrust_shared=z:/task_1510749903/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libgkrust_shared-a27e948f260cfd40.rlib --extern mp4parse_gtest=z:/task_1510749903/build/src/obj-firefox/toolkit/library\i686-pc-windows-msvc\release\deps\libmp4parse_gtest-0ee4c20075271ff2.rlib -C opt-level=2 -C debuginfo=2` (exit code: 3221225477)
13:39:52     INFO -  z:/task_1510749903/build/src/config/rules.mk:952: recipe for target 'force-cargo-library-build' failed
13:39:52     INFO -  mozmake.EXE[4]: *** [force-cargo-library-build] Error 101
13:39:52     INFO -  mozmake.EXE[4]: Leaving directory 'z:/task_1510749903/build/src/obj-firefox/toolkit/library/gtest/rust'

[rillian]

Sorry for the late report. I tried to debug a bit locally but haven't been successful in generating a better report. Maybe something with -C lto? Debug builds are fine, as are builds targeting x86_64.

[rillian]

This corresponds to Gecko bug 1417268.

[kennytm]

(3221225477 = 0xc0000005 = STATUS_ACCESS_VIOLATION)

[rillian]

Thanks @kennytm. Note this a regression from 1.21.0 stable.

[rillian]

The problem doesn't occur with 1.23.0-nightly from October 31 from from November 15. It's only an issue with the current beta series.

[nikomatsakis]

The problem doesn't occur with 1.23.0-nightly from October 31 from from November 15. It's only an issue with the current beta series.

Well that certainly narrows it down some! Is it possible to narrow down when within the beta series the problem started?

[nikomatsakis]

cc @rust-lang/compiler @alexcrichton -- late-breaking beta regression

[eddyb]

Is it running out of memory? Does it occur after a LLVM update?

[nikomatsakis]

@eddyb I believe we did recently upgrade LLVM... not sure though.

[michaelwoerister]

@rillian can you test with -Ztime-passes somehow? That would print out memory consumption at many points during compilation and give a hint how far compiler is getting.

[Zoxc]

@rillian Do you have a JIT debugger installed so we can get a backtrace?

[rillian]

@michaelwoerister Good idea. Running now. The failure should be near the end.

@Zoxc I don't know. How would I find out? NB I'm not able to reproduce this locally; I only see it in our build automation. I'll try to reproduce in an interactive session on one of those machines tomorrow, so having things to try would be good.

[rillian]

Final line before the failure is

11:34:42     INFO -      time: 101.169; rss: 1771MB	LTO passes

This is running on an AWS c4.4xlarge instance, which apparently had 16 GB of ram (although it's listed as having 30 GB of memory, so running out seems unlikely. OTOH, I'm using the i686-pc-windows-msvc hosted rustc because I had trouble with the 64-bit version calling the wrong linker. Maybe 1.7GB is close enough to exhaust the address space?

This might explain why I couldn't reproduce locally too; I was using the 64-bit rustc.

[michaelwoerister]

Maybe 1.7GB is close enough to exhaust the address space?

Sounds likely to me.

[michaelwoerister]

@rillian When you try to reproduce locally, do you also use a 32-bit version of the compiler?

[rillian]

@michaelwoerister When I tried before I was using the 64-bit version of the compiler. I'm trying with the 32-bit version now, but it will be a while. My local machine is not a c4.4xlarge. :)

[Zoxc]

If you're running Windows x64 the compiler should have 4GB to play with since we link with /LARGEADDRESSAWARE.

If you have Visual Studio installed, you can enable it as a JIT debugger by going to Debug -> Options -> Just-In-Time and check Native. You'll then get an option to debug crashing applications.

[Zoxc]

You can also create a minidump of crashes by enabling that in the registry. You can then open the minidump in a debugger to get a stack trace.

[cuviper]

Note that for exhausting address space, the total virtual size should be considered, not just what's resident. On #45854 with 32-bit ARM Linux, I saw the 3GB address space exhausted with only ~2GB rss.

[rillian]

Just an update: I've still not been able to reproduce with a local 32-bit rustc.

We may be able to work around the issue for Firefox builds by just using 64-bit rust, if we can figure out how to pass the correct msvc locations into build scripts.

[nikomatsakis]

@rillian any update here?

[nikomatsakis]

Just saw this comment from the Gecko bug, seems relevant:

Rust 1.23.0-nightly does work however, so this may be a symptom of extra memory use by llvm in the 1.22 release.

[kennytm]

Just saw this comment from the Gecko bug, seems relevant:

Rust 1.23.0-nightly does work however, so this may be a symptom of extra memory use by llvm in the 1.22 release.

Any chance this is another instance of the bug known as we send too much stuff to LLVM which was improved by #45575 during 1.22 ..= 1.23?

[rillian]

I've still not been able to reproduce, and I haven't been able to get access to one of our build hosts to try there. Since we don't see the problem with Rust 1.23 or in local builds, the plan is to move official 32-bit windows Firefox builds to 1.23.0-beta as a work around.

Given that, I'm going to stop working on this unless we get some new information. Thanks everyone for your help!