Cargo eating all RAM during aarch64 build under Docker on x86-64

[nazar-pc]

Problem

I'm trying to build an application, this time to build aarch64 container image on x86-64 machine.

My machine is a beefy 5900X with 128G of RAM, but my system runs out of memory (Cargo eats it all) when cross-compiling aarch64 container.

Steps

1. Clone this branch: https://github.com/nazar-pc/subspace/tree/cargo-memory-bug-aarch64
2. Have Docker installed on Linux x86-64 machine
3. Run docker buildx build --platform linux/arm64 -t test -f Dockerfile-farmer .
4. Observe system freezing/crashing because of running out of memory

Possible Solution(s)

No response

Notes

x86-64 build on the same machine works fine with ~20G of system memory.

Not entirely sure if this is Cargo's fault or QEMU or something else, but I used similar QEMU-based setups before and never seen anything remotely like this.

Version

cargo 1.60.0-nightly (c082648 2022-02-08)
release: 1.60.0-nightly
commit-hash: c082648646cbb2be266df9ecbcdc253058158d68
commit-date: 2022-02-08
host: aarch64-unknown-linux-gnu
libgit2: 1.3.0 (sys:0.13.23 vendored)
libcurl: 7.80.0-DEV (sys:0.4.51+curl-7.80.0 vendored ssl:OpenSSL/1.1.1m)
os: Linux [64-bit]

UPD: Tried the most recent version, still the same issue:

cargo 1.62.0-nightly (dba5baf 2022-04-13)
release: 1.62.0-nightly
commit-hash: dba5baf4345858c591517b24801902a062c399f8
commit-date: 2022-04-13
host: aarch64-unknown-linux-gnu
libgit2: 1.4.2 (sys:0.14.2 vendored)
libcurl: 7.80.0-DEV (sys:0.4.51+curl-7.80.0 vendored ssl:OpenSSL/1.1.1m)
os: Linux [64-bit]

[epage]

#8405 is one potential solution for this.

[nazar-pc]

Hm, I feel like this is something different because I only observe this issue in above scenario with QEMU-based emulation. Let me know if I can add some instrumentation to help debugging this (though reproduction steps should be reliable too as everything is containerized).

[nazar-pc]

I'm fairly certain that memory consumption happens during repo cloning (dependencies from git), before doing any compilation

[weihanglo]

If it is related to git clone, perhaps set the config net.git-fetch-with-cli = true to see whether libgit2 is the suspect.

[nazar-pc]

Thank you @weihanglo, it helped!

[Flawm]

I'm seeing the exact same behavior as OP, however setting the net git fetch flag didn't seem to reduce the memory consumption

In my case, I was able to get around it by binding my cargo caches into the container:

    -v $HOME/.cargo/git:/usr/local/cargo/git \
    -v $HOME/.cargo/registry:/usr/local/cargo/registry \

I suppose the docker-compose.yml equivalent would work as well, but in some CI scenarios this may not be possible

[msrd0]

I just ran into this issue as well trying to execute cargo fetch on an alpine aarch64 guest system running on an x86_64 archlinux host. It never made it past cloning the crates.io index.

[ThomasWaldmann]

Maybe related: pyca/cryptography#8640