Segfault while building the Rust toolchain with a toolchain that uses mimalloc

[gnzlbg]

So I'm trying to build a Rust toolchain with mimalloc that works here: rust-lang/rust#62340 , such that we can use the Rust toolchain compiler benchmark suite to test whether it would be worth it switching to mimalloc from jemalloc.

I think I've managed so far to link mimalloc properly, and built a toolchain that's statically linked with it and uses it. When that toolchain is used to build another toolchain, after building a chunk of it, it segfaults with: signal: 11, SIGSEGV: invalid memory reference. Target is x86_64-unknown-linux-gnu , and I'm using the master branch of mimalloc, via the mimalloc-sys bindings, which are here: https://github.com/gnzlbg/mimallocator/tree/master/mimalloc-sys

I'm building mimalloc following the steps from this script: https://github.com/gnzlbg/mimallocator/blob/master/mimalloc-sys/build.rs

[daanx]

Very nice you are working on integrating this with Rust -- let me know of issues or performance.
With regard to the crash, when integrating with a language, it is best to:

• if the runtime uses plain malloc/free, then first use LD_PRELOAD to see if those programs still work. This lets you know if the bug is in mimalloc or in the integration.
• for a language runtime, it is best to link statically, and have the runtime consistently use the mi_ prefixed API and never mix up pointers gotten through some other API. However, that means special care must be taken that pointers obtained through some C library do not get freed with the new Rust free (which uses mi_free). In particular, watch for uses of strdup, strndup, and realpath ! and use the mi_ prefixed versions instead. Same for posix_align etc. (this will get easier once I get to an update for issue Expose standard C and POSIX APIs even without override #75 )

Do you have a backtrace of the segfault? Was it using the debug version of mimalloc with -DMI_CHECK_FULL=ON enabled?

I will try help out and try to look at the Rust thread tomorrow.

edit: I looked a little and saw rust already works with jemalloc so the above issues should already be handled well. Don't use MI_OVERRIDE and statically link seems the best approach (which you are already doing).

[daanx]

Just a note, ensure the aligment and size arguments are in the right order for the aligned allocation functions in:

        static _F2: unsafe extern fn(*mut *mut c_void, usize, usize) -> c_int =
            mimalloc_sys::posix_memalign;
        static _F3: unsafe extern fn(usize, usize) -> *mut c_void =
            mimalloc_sys::aligned_alloc;

since mimalloc lets alignment always come after the size but posix_memalign and memalign do it the other way around!

[gnzlbg]

let me know of issues or performance.

First step is to get it to work correctly, but that's the next step :)

if the runtime uses plain malloc/free, then first use LD_PRELOAD to see if those programs still work. This lets you know if the bug is in mimalloc or in the integration.

So in the mimalloc-sys crate, I do two things:

• check that mimalloc is actually used: with the MI_OVERRIDE setting, mimalloc replaces the standard symbols, but still provides the mi_ symbols. This was great to be able to test that the library was properly linked, e.g., by calling mi_malloc+free and malloc+mi_free, and making sure that works. (When I wrote the tests the first time, they failed, because I didn't had MI_INTERPOSE enabled on macos, so hurd the system allocator was trying to free memory from mimalloc and vice-versa, which crashed).

• c-ffi-test: for all functions, types, etc. exposed by the Rust bindings to a C API, tests are generated that check that the size, alignment, field offsets, etc. of types are the same, that the addresses of the rust function wrapper and the C functions is the same, that the functions take the same arguments, return types, etc. Essentially, that the ABI of the wrapper matches that of the library being wrapped. This is done by generating C code and Rust code, to compute these things at each side of the FFI, and then collecting the results and compare them.

if the runtime uses plain malloc/free, then first use LD_PRELOAD to see if those programs still work. This lets you know if the bug is in mimalloc or in the integration.

The Rust compiler uses "plain" malloc/free (the C and POSIX apis), but right now jemalloc is statically linked into the binary, overriding these symbols. I know LD_PRELOAD works if the malloc library is linked dynamically, but I'm not sure if it works in this situation.

By default, Rust binaries use the system allocator. I've tried DYLD_LIBRARY_INSERT=.. with mimalloc on some small binary and that just worked (I saw the statistics output), but the Rust compiler is a beast in code size, and number of tests and benchmarks, and other linked libraries, so it

for a language runtime, it is best to link statically, and have the runtime consistently use the mi_ prefixed API and never mix up pointers gotten through some other API. However, that means special care must be taken that pointers obtained through some C library do not get freed with the new Rust free (which uses mi_free). In particular, watch for uses of strdup, strndup, and realpath ! and use the mi_ prefixed versions instead. Same for posix_align etc. (this will get easier once I get to an update for issue #75 )

So I'm statically linking mimalloc into the binary, and hopefully correctly overriding all symbols. That's what we do for jemalloc and it works. So I would hope that this would override strdup and friends as well.

One thing we are doing is compiling LLVM and somehow linking it into rustc. We don't do anything special for jemalloc here, so I don't think we need to do anything special for mimalloc either.

Just a note, ensure the aligment and size arguments are in the right order for the aligned allocation functions in:

Those are wrapped here: https://github.com/gnzlbg/mimallocator/blob/master/mimalloc-sys/src/lib.rs#L49

We just provide the C and POSIX API. I suppose the ones mimalloc defines have the arguments in the standard orders for these. From looking at the code here, they appear to match: https://github.com/microsoft/mimalloc/blob/master/src/alloc-override.c#L138

That is, we are only using the overriden functions from mimalloc, we don't use any of the mi_... APIs directly in the compiler (yet).

Do you have a backtrace of the segfault? Was it using the debug version of mimalloc with -DMI_CHECK_FULL=ON enabled?

AFAICT it was using a release build, but I will enable full checks and try again.

[daanx]

Those are wrapped here: https://github.com/gnzlbg/mimallocator/blob/master/mimalloc-sys/src/lib.rs#L49

We just provide the C and POSIX API. I suppose the ones mimalloc defines have the arguments in the standard orders for these. From looking at the code here, they appear to match: https://github.com/microsoft/mimalloc/blob/master/src/alloc-override.c#L138

Mmm, I mean the mi_malloc_aligned and friends don't match with posix_memalign: the arguments are swapped:

int posix_memalign(void** p, size_t alignment, size_t size);
void* memalign(size_t alignment, size_t size);

versus

void* mi_malloc_aligned(size_t size, size_t alignment);

Be very careful when binding to Rust that it matches what Rust defined.

[gnzlbg]

Mmm, I mean the mi_malloc_aligned and friends don't match with posix_memalign:

Ah ok. Yes, we don't use mi_... and friends, only malloc/posix_memalign, etc. and the APIs of those that we use appear correct.

[daanx]

I understand you want to link statically with mimalloc and override the standard malloc/free. If so, that will still be tricky general. Static + override:

• works on Linux/BSD with weak symbols -- just link carefully to ensure mimalloc comes first
• should work on macOS -- but currently there are issues (OSX debugging #50) but I am confident it will be fixed in the future as other allocators make this work too. More testing is needed here and I need to study how jemalloc does this, tcmalloc uses MacOS zones which I am currently testing with.
• does not work on Windows with msvc! (needs dynamic loading)

Static but not overriding is the safest option but you need to control the runtime system enough to ensure no pointers are mixed -- but that may well be the case for rust.

For now, I would at least try your current approach on Linux first to ensure it all works before moving on to other platforms.

[gnzlbg]

I forgot to mention that, to be able to run Rust compiler benchmarks, we only need to be able to link mimalloc on Linux.

Static but not overriding is the safest option but you need to control the runtime system enough to ensure no pointers are mixed -- but that may well be the case for rust.

I don't think this is an option. We link against LLVM, which uses the malloc APIs to allocate memory. I can't really assume that there aren't any places in the code where memory from one is passed to the other, which afterwards deallocates it. I hope this isn't the case, but making sure that it isn't would be a lot of work at this point (EDIT: the code interfacing with LLVM is compartmentalized in a single place, so while doable, it is still a lot of work to do this).

For now, I would at least try your current approach on Linux first to ensure it all works before moving on to other platforms.

That's the plan. Maybe I wasn't clear. Our current goal isn't to switch from jemalloc to mimalloc in the Rust toolchain, but rather to gather data about the performance of the Rust toolchain when mimalloc is its allocator. Performance is just one of the many data-points that would need to be considered in a discussion about doing a switch, but that discussion hasn't started yet. This is purely exploratory work.

If you want to try my PR branch on Linux, the only thing you have to do is clone it, and execute ./x.py build --stage 1. It should work out-of-the box.

[daanx]

Thanks for explaining :-) I thought that you were working on macOS somehow. Anyway, in that case there should not be segfaults so let's see if we can track down the cause. If can provide me with a test case, or a debug backtrace from the faulty build that will help. If I try your branch, will it cause the error for me investigate?

[gnzlbg]

If I try your branch, will it cause the error for me investigate?

Yes. Briefly, the Rust toolchain bootstraps itself. So the first "stage" is stage0, and here, an older compiler is used to compile the toolchain, producing a stage0 compiler. This stage0 compiler that gets produced here is linked statically with mimalloc in my PR branch. Then, in stage1, this compiler is used to compile the toolchain again, producing a stage1 compiler. This is where the compiler using mimalloc gets exercised, and where the error happens.

The ./x.py script handles everything for you. To build a stage1 compiler, one can write ./x.py build --stage 1. Unless you want to modify the stage0 compiler, e.g., to link mimalloc differently, you probably don't want to re-build stage0 all the time. You can prevent that by doing ./x.py build --stage 1 --keep-stage 0. When compilation fails, you get an error message, providing the CLI invocation that failed to execute. So you can just copy-paste it into your terminal, and re-try, use gdb, etc. Please let me know if you have any questions or need any help. If you start doing this, and happen to have Discord or Zulip installed, you can join the Rust channels there (search for Rust discord and Rust zulip for the links). There are compiler channels in both, where you can ask any questions.

[daanx]

This is still on my todo list -- let me know of further issues but I will get to it soonish :-)

[gnzlbg]

@daanx let me know if you need help setting anything up or feel free to drop by Rust's discord or Zulip chat rooms.

[ArniDagur]

Building rustc seems to work fine for me, am I doing something wrong?

git clone [email protected]:gnzlbg/rust.git rust-mimalloc
cd rust-mimalloc
git checkout mimalloc
./x.py build

Results in:

Build completed successfully in 0:37:10

If I do ./x.py build --stage 1 --keep-stage 0, I get the same.

[daanx]

The issue is fairly old and in the meantime various bugs were addressed. Perhaps it is just working now? :-) If so, please confirm as it would be nice if Rust is able to easily use mimalloc as an allocator. I read somewhere that Rust now uses no default allocator but instead let's the user opt-in to another allocator -- it would be great if mimalloc can be one of those options.

[mati865]

Steps from #77 (comment) give you 2 months old Rust and Mimalloc. There are no fixes included so it shouldn't work meaning something went wrong.

I read somewhere that Rust now uses no default allocator but instead let's the user opt-in to another

Rust compiler uses jemalloc but users can choose which allocator they will use in their binaries (defaults to system allocator).

[gnzlbg]

@mati865 actually, the latest commit in that PR was never "try"ed (some other PR landed before that that made it unmergeable), so... maybe it worked ? I'll rebase and we should schedule a try-build.

[gnzlbg]

So I rebased on top of master, and re-run all tests, and we are still failing with a SIGSEV: https://dev.azure.com/rust-lang/e71b0ddf-dd27-435a-873c-e30f86eea377/_build/results?buildId=7960

[gnzlbg]

@ArniDagur which platform are you testing on ? Those build jobs run on x86_64-linux. It might also be that there are some differences between how those jobs are build there and your machine.

[daanx]

@gnzlbg: thanks for trying again!. I am still very interested to fix this and understand what is happening. Can you try with the latest dev branch too? And if that still fails, perhaps build with -DMI_CHECK_FULL=ON so we can see if any assertions fail. (Perhaps too much work, no worries if you don't get to it)

[ArniDagur]

Here's information about my system:

• OS: Gentoo Linux
• Output of uname -a: Linux arni-pc 5.2.3-ck-x86_64 #1 SMP Wed Sep 11 19:29:52 GMT 2019 x86_64 Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz GenuineIntel GNU/Linux
• Compiler: gcc (Gentoo 9.2.0 p1) 9.2.0
• Linker: GNU ld (Gentoo 2.32 p2) 2.32.0
• Compiler environment variables: RUSTFLAGS=-C target-cpu=native

[mati865]

Builds ran through Docker containers have various config options tweaked: https://github.com/rust-lang/rust/blob/c43d03a19f326f4a323569328cc501e86eb6d22e/src/ci/run.sh
DEPLOY=1 is defined BTW.

[gnzlbg]

Can you try with the latest dev branch too? And if that still fails, perhaps build with -DMI_CHECK_FULL=ON so we can see if any assertions fail. (Perhaps too much work, no worries if you don't get to it)

The results are in here: https://dev.azure.com/rust-lang/rust/_build/results?buildId=7973&view=logs&jobId=d0d954b5-f111-5dc4-4d76-03b6c9d0cf7e&taskId=1094049d-8071-5d7a-acdc-b0909289cb44&lineStart=14107&lineEnd=14107&colStart=1&colEnd=75

An assertion triggers:

mimalloc: assertion failed: at "mimalloc/src/page.c":324, _mi_page_abandon
  assertion: "_mi_ptr_page(block) != page"

[daanx]

Ah very interesting -- that assertion should not trigger! Can you try if this happens with the latest dev branch too? I made various improvements to atomic handling which may fix it. I will try to repro on my local linux box if I can find some time. Thanks again for your work -- hope I can figure out what is causing this.

[gnzlbg]

@daanx that test was done with the latest dev branch that was released at the time of the test (~3 hours ago), but the last commit to the dev branch is from 17 hours ago, so those tests should have used the very latest dev branch. If you update it feel free to let me know and I'll re-run the tests.

[daanx]

Ah I see -- but it says line 324 in the assertion which corresponds to the master branch, in dev it is line 343. Maybe a wrong checkout?

[gnzlbg]

Weird, I noticed that to. I'll investigate.

[gnzlbg]

You were right. I was being bitten by semver, the "0.1.6-dev" version was being resolved as "0.1.6". I had to specify that I really really wanted that one with "=0.1.6-dev". I've restarted the build.

[goffrie]

From the latest run...

2019-09-12T23:14:31.9142450Z mimalloc: warning: possibly trying to mi_free a pointer that does not point to a valid heap region: 0x0x413f1c02000
2019-09-12T23:14:31.9142691Z (this may still be a valid very large allocation (over 64MiB))
2019-09-12T23:14:31.9142806Z mimalloc: warning: (yes, the previous pointer 0x0x413f1c02000 was valid after all)
2019-09-12T23:14:31.9292779Z mimalloc: warning: possibly trying to mi_free a pointer that does not point to a valid heap region: 0x0x41426002000
2019-09-12T23:14:31.9292911Z (this may still be a valid very large allocation (over 64MiB))
2019-09-12T23:14:31.9294975Z mimalloc: warning: (yes, the previous pointer 0x0x41426002000 was valid after all)
2019-09-12T23:18:21.3635930Z [RUSTC-TIMING] rustc test:false 821.997
2019-09-12T23:18:21.3636999Z rustc exited with signal: 11

[gnzlbg]

@goffrie showed the error, here is the link.

[gnzlbg]

Latest commit disables secure mode making it work despite the warnings, you have to revert it to get the crash.

@mati865 while the last commit does do that, in rust-lang/rust CI, the build still fails at some point after the warnings with a segfault, which @daanx is not able to reproduce. I don't understand why this happens. Maybe the resources of the VMs used there play a role here ? (e.g. maybe a higher memory pressure is required to trigger this?)

[daanx]

Thanks everyone for helping to isolate the problem. I hope I can repro at some point and get a trace.
I am focusing on the non-secure build as that still crashed in the CI; Last time I built with make -j1 and will try now with make -j4 -- perhaps it is a concurrency bug in mimalloc that only happens with clang (and not with gcc/msvc as I usually test with those)?

[mati865]

./x.py uses all available threads, so it was 16 in my case.
Clang could be worth trying, although it will use system allocator.

@daanx have you applied all the options from #77 (comment) or only --set rust.jemalloc?

[daanx]

@mati865: I used the full script in comment #77. Maybe using sccache is not good?
I also set export MIMALLOC_VERBOSE=1 so I can observe mimalloc being used. Btw. where is the mimalloc source in the tree -- I want to turn off the threadinit/done messages as there are so many.

[mati865]

Maybe it has something to do with the ancient CentoOS 5 that is used by Rust dist builders?

Btw. where is the mimalloc source in the tree

It's complicated because it's external crate: https://github.com/gnzlbg/mimallocator/tree/master/mimalloc-sys
Moving it into Rust repo for testing seems like a good idea but I'm out of time.

[daanx]

Update: using make -j4 succeeds as well. I will now roll back one commit and use the mimalloc secure build and try to repro with that.
I am wondering what is different on the CI machines? I remember the mprotect warnings too which are strange; I

daan@daan-HP-Z4-G4:~$ clang --version
clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
daan@daan-HP-Z4-G4:~$ gcc --version
gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

daan@daan-HP-Z4-G4:~$ linuxinfo
Linux daan-HP-Z4-G4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019
Eight Intel Unknown 3829MHz processors, 57600.00 total bogomips, 15764M RAM
System library 2.27.0

[daanx]

Moving it into Rust repo for testing seems like a good idea but I'm out of time.
No worries -- thanks for helping out!

[gnzlbg]

Mimalloc is pulled in via an external library, that includes the source code, a script for how to compile it, and Rust C FFI bindings so that it can be called from Rust.

That code is imported from the src/rustc/Cargo.toml file, which points to mimalloc-sys 0.1.6-dev, which lives in the dev branch of this repo: https://github.com/gnzlbg/mimallocator in the mimalloc-sys sub-directory.

[mati865]

I am wondering what is different on the CI machines?

If you have a lot of time to spend there is possibility to run exactly the same environment as Rust's CI (but it takes around twice as long).
You need Linux with working Docker and then just use this in the repo: DEPLOY=1 ./src/ci/docker/run.sh dist-x86_64-linux.

[daanx]

When using the secure build, I can repro. The command fails with:

> rm -rf build
> ./configure --set rustc.jemalloc=1
> make -j4
...
   Compiling rustc_save_analysis v0.0.0 (/home/daan/dev/rust-mimalloc/src/librustc_save_analysis)
   Compiling rustc_plugin v0.0.0 (/home/daan/dev/rust-mimalloc/src/librustc_plugin/deprecated)
mimalloc: warning: possibly trying to mi_free a pointer that does not point to a valid heap region: 0x0x4389b002000
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x4389b002000 was valid after all)
mimalloc: warning: mprotect error: start: 0x0x416d6801000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d6bff000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d7002000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d700f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d701f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d702f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d703f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d704f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d705f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d706f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d707f000, csize: 0x1000, err: -1
error: Could not compile `rustc`.

Caused by:
  process didn't exit successfully: `/home/daan/dev/rust-mimalloc/build/bootstrap/debug/rustc --edition=2018 --crate-name rustc src/librustc/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=2 -C metadata=5e0d3eeff150bb82 -C extra-filename=-5e0d3eeff150bb82 --out-dir /home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps -L dependency=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps --extern arena=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libarena-af9d93ef6dc08aee.rmeta --extern backtrace=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbacktrace-12b311ad047821a8.rmeta --extern bitflags=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbitflags-0185f8c31122856c.rmeta --extern byteorder=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbyteorder-84431642a2d21986.rmeta --extern chalk_engine=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libchalk_engine-62466d355dbdc99a.rmeta --extern fmt_macros=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libfmt_macros-482d5e66bb772b92.rmeta --extern graphviz=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libgraphviz-f6a7dc011264b85b.rmeta --extern jobserver=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libjobserver-c8b46a25bb9da522.rmeta --extern log=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/liblog-bd0a443870ca05a1.rmeta --extern measureme=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libmeasureme-091ecbacf269a237.rmeta --extern num_cpus=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libnum_cpus-88fd2b8b876d6e21.rmeta --extern parking_lot=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libparking_lot-f9292c52c4e814ca.rmeta --extern polonius_engine=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libpolonius_engine-e88075d624331b85.rmeta --extern rustc_rayon=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_rayon-ea44b91f88f116f1.rmeta --extern rustc_rayon_core=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_rayon_core-7809f22d4704262f.rmeta --extern rustc_apfloat=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_apfloat-b58383702b0e9127.rmeta --extern rustc_data_structures=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_data_structures-8fd90ba391486a46.rmeta --extern errors=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_errors-8413b71d09c551f0.rmeta --extern rustc_fs_util=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_fs_util-2b07fd141255082a.rmeta --extern rustc_macros=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps/librustc_macros-35a0cb747106834a.so --extern rustc_target=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_target-a02958114c225952.rmeta --extern scoped_tls=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libscoped_tls-484e840675fbe6fc.rmeta --extern rustc_serialize=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libserialize-f09f146ed063c0de.rmeta --extern smallvec=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libsmallvec-739f6fdb85f68aa2.rmeta --extern syntax=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libsyntax-0fd6e4d7aa6beaaa.rmeta --extern syntax_pos=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libsyntax_pos-255f37cb2a800a63.rmeta -Zbinary-dep-depinfo -L native=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/backtrace-sys-1d74c3d84afd8f29/out` (signal: 11, SIGSEGV: invalid memory reference)
command did not execute successfully: "/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-j" "8" "--release" "--features" "jemalloc" "--manifest-path" "/home/daan/dev/rust-mimalloc/src/rustc/Cargo.toml" "--message-format" "json-render-diagnostics"
expected success, got: exit code: 101
failed to run: /home/daan/dev/rust-mimalloc/build/bootstrap/debug/bootstrap build
Build completed unsuccessfully in 0:27:14
Makefile:18: recipe for target 'all' failed
make: *** [all] Error 1

[daanx]

Now, I am trying to run the last failed command in the debugger (/home/daan/dev/rust-mimalloc/build/bootstrap/debug/rustc ...) but I cannot as it expects all kinds of environment definitions, like RUSTC_STAGE, RUSTC_REAL, RUSTC_LIBDIR etc. Does anyone know how to set these correctly? The goal is to run the last command under a debugger and set breakpoints at the warnings in mimalloc and see a stack trace at the fault.

[gnzlbg]

@daanx I don't know how to do that, but we should document how to do that: there is a tracking issue for documenting this here (rust-lang/rustc-dev-guide#443), and a solution for a different case which sadly I don't think will work here.

[gnzlbg]

@daanx what you can do is add the --on-fail=bash flag to the x.py invocation. Once something fails, it will create an interactive shell with the environment properly set up. In that shell you can then spawn a debugger to run the command that failed.

[gnzlbg]

@daanx if --on-fail=bash does not work, you can use --on-fail=env to print the environment that's used by x.py. And then set it up yourself before trying to reproduce by executing the command that failed.

[gnzlbg]

Has there been any progress on this?

[daanx]

I would like to try this again with the latest dev (or dev-abandon) mimalloc branches. However, it would be great if I can control the mimalloc sources; as I understand it is now pulled from https://github.com/gnzlbg/mimallocator. What is the best way to change this so I can change the source or (cmake) compilation flags? Can it point to a local folder?

[ArniDagur]

Try this patch:

diff --git a/src/rustc/Cargo.toml b/src/rustc/Cargo.toml
index 825e3655ac..7a4f81ce59 100644
--- a/src/rustc/Cargo.toml
+++ b/src/rustc/Cargo.toml
@@ -22,7 +22,7 @@ rustc_codegen_ssa = { path = "../librustc_codegen_ssa" }
 #features = ['unprefixed_malloc_on_supported_platforms']

 [dependencies.mimalloc-sys]
-version = '=0.1.6-dev'
+path="../../../mimallocator/mimalloc-sys"
 optional = true

 [features]

After something like:

git clone [email protected]:gnzlbg/rust.git
cd rust
git checkout mimalloc
cd ..
git clone [email protected]:gnzlbg/mimallocator.git
cd mimallocator/mimalloc-sys
rmdir mimalloc
git clone [email protected]:microsoft/mimalloc.git
cd mimalloc
git checkout dev

Remember to enable jemalloc in config.toml or using the configure script.

[ArniDagur]

For what it's worth, I got Rust to compile with only these messages from mimalloc:

   Compiling rustc_resolve v0.0.0 (/sdb/src/rust/src/librustc_resolve)
   Compiling rustc_privacy v0.0.0 (/sdb/src/rust/src/librustc_privacy)
   Compiling rustc_plugin v0.0.0 (/sdb/src/rust/src/librustc_plugin/deprecated)
   Compiling rustc_codegen_ssa v0.0.0 (/sdb/src/rust/src/librustc_codegen_ssa)
   Compiling rustc_save_analysis v0.0.0 (/sdb/src/rust/src/librustc_save_analysis)
mimalloc: warning: possibly trying to free a pointer that does not point to a valid heap region: 0x0x40030000100
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x40030000100 was valid after all)
mimalloc: warning: possibly trying to free a pointer that does not point to a valid heap region: 0x0x40040000100
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x40040000100 was valid after all)
mimalloc: warning: possibly trying to free a pointer that does not point to a valid heap region: 0x0x40074400100
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x40074400100 was valid after all)
   Compiling rustc-main v0.0.0 (/sdb/src/rust/src/rustc)
    Finished release [optimized] target(s) in 9m 48s
Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage1 codegen artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu, llvm)
   Compiling cc v1.0.35

[daanx]

Thank you @ArniDagur, @gnzlbg: with the above instructions I successfully built using the latest dev-abandon branch (on its way to become version 2.0 with dev to 1.3). However, when enabling the secure build it still fails! The error was:

mimalloc: warning: mprotect error: start: 0x0x4ab9e75f000, csize: 0x1000, err: 12

(actually, the real error occurred later with a panic message a thread could not start due to resource exhaustion)

Some careful reading shows errno 12 is ENOMEM, and the man page says that that might be caused by creating too many mappings. Well, of course, full secure mode creates guard pages between every mimalloc "page" thus creating many mappings, and I guessed the biggest compiled sources may exceed the default (64k mappings) so I increased it to 256k:

sudo sysctl -w vm.max_map_count=262144

and that was it! After that it built in secure mode with full debug checking. :-)
Great. I will add a better warning message and add documentation; and perhaps a secure mode that only uses guard pages around full segments instead of every mimalloc page.

Pfff, glad this got sorted out -- would love to see some benchmarks with rust.

[jq-rs]

I just opened a new test try with another crate: rust-lang/rust#81782. Let's see do we get benchmarks at some point.

Edit: It looks good at the moment!

[jq-rs]

And here are the results: https://perf.rust-lang.org/compare.html?start=f9435f4c92651d67d5dbaba13c5606c4c4fc1327&end=86f3a470dd7fa102be01dc6d84ecc6920f8af32a

Pretty good and only a few regressions. Props for mimalloc!

I think this issue can be closed now.

[jq-rs]

I tried also the MacOS rustc-build and there the override did not work as expected, statistics show zeroes.

heap stats:     peak      total      freed    current       unit      count  
    normal:       0 b        0 b        0 b        0 b        1 b              ok
      huge:       0 b        0 b        0 b        0 b        1 b              ok
     giant:       0 b        0 b        0 b        0 b        1 b              ok
     total:       0 b        0 b        0 b        0 b        1 b              ok
malloc req:       0 b        0 b        0 b        0 b        1 b              ok

  reserved:       0 b        0 b        0 b        0 b        1 b              ok
 committed:       0 b        0 b        0 b        0 b        1 b              ok
     reset:       0 b        0 b        0 b        0 b        1 b              ok
   touched:       0 b        0 b        0 b        0 b        1 b              ok
  segments:       0          0          0          0                           ok
-abandoned:       0          0          0          0                           ok
   -cached:       0          0          0          0                           ok
     pages:       0          0          0          0                           ok
-abandoned:       0          0          0          0                           ok
 -extended:       0   
 -noretire:       0   
     mmaps:       0   
   commits:       0   
   threads:       0          0          0          0                           ok
  searches:     0.0 avg
numa nodes:       1
   elapsed:      10.628 s
   process: user: 29.280 s, system: 0.574 s, faults: 679, rss: 154.8 mb

I did enable debugs and ran it with env MIMALLOC_SHOW_STATS=1 ./x.py build

@daanx Anything more to try here?

[jq-rs]

Seems that MacOS support for rustc is difficult to achieve in practice. And jemalloc does not support it either. So this is a known issue.

[jq-rs]

Jemalloc for macOS was just broken and can be fixed via simple changes provided in rust-lang/rust#82423.

[fee1-dead]

@daanx said:

Static + override:
does not work on Windows with msvc! (needs dynamic loading)

Is this still true today? To be able to static link and override the libc symbols is still preferable.