Docker in Docker doesn't seem to find / forward the volume mounts

[nuket]

Checklist

• I've looked through the issues and pull requests for similar reports

Describe your issue

Using a clean Ubuntu Server 22.04.2 LTS system, running cross inside of one Docker container gives me the same error Build fails with sh: 1: cargo: not found listed in #260.

If /var/lib/docker is not bind mounted to the outer Docker instance, when cross goes to forward some of those mounts, there is nothing to forward to the inner Docker instance:

rustuser@rust-build:~/workspace$ cross build -vv --target aarch64-unknown-linux-gnu
+ cargo metadata --format-version 1 --filter-platform aarch64-unknown-linux-gnu
+ rustc --print sysroot
+ rustup toolchain list
+ rustup target list --toolchain 1.70.0-x86_64-unknown-linux-gnu
+ rustup component list --toolchain 1.70.0-x86_64-unknown-linux-gnu
+ /usr/bin/docker
+ /usr/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=rustuser' -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library --rm --user 1000:1000 -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/.xargo:/xargo:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/opt/cargo:/cargo:z -v /cargo/bin -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu:/rust:z,ro -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/target:/target:z -w /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace -i -t ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.5 sh -c 'PATH=$PATH:/rust/bin cargo build -vv --target aarch64-unknown-linux-gnu'
sh: 1: cargo: not found
+ rustup component list --toolchain 1.70.0-x86_64-unknown-linux-gnu

If you run the container above and drop into a shell, you can examine the paths available in /var/lib and see what is missing.

rustuser@rust-build:~/workspace$  /usr/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=rustuser' -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library --rm --user 1000:1000 -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/.xargo:/xargo:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/opt/cargo:/cargo:z -v /cargo/bin -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu:/rust:z,ro -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/target:/target:z -w /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace -i -t ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.5 sh
$ ls
target
$ id
uid=1000 gid=1000 groups=1000
$ ls /
bin  boot  cargo  common.sh  dev  etc  home  lib  lib.sh  lib64  linux-image.sh  linux-runner  media  mnt  opt  proc  qemu  root  run  rust  sbin  srv  sys  target  tmp  usr  var  xargo
rustuser@rust-build:~/workspace$ ls -l /var/lib
total 48
drwxr-xr-x 1 root root 4096 Jun 28 14:25 apt
drwxr-xr-x 1 root root 4096 Jun 28 14:26 dpkg
drwxr-xr-x 2 root root 4096 Feb 22 10:51 git
drwxr-xr-x 2 root root 4096 Apr  2 13:55 misc
drwxr-xr-x 2 root root 4096 Jun 12 02:00 pam
drwxr-xr-x 2 root root 4096 Jun 13 05:29 python
drwxr-xr-x 3 root root 4096 Jun 28 14:25 sudo
drwxr-xr-x 1 root root 4096 Jun 13 05:29 systemd
drwxr-xr-x 3 root root 4096 Jun 13 05:29 ucf
rustuser@rust-build:~/workspace$ ls -l /var/lib/docker
ls: cannot access '/var/lib/docker': No such file or directory

I don't have a solution to this yet. It should be possible to bind mount /var/lib/docker, but the root:root permissions will be an issue.

System Info

Host system:

$ uname -a
Linux boombox 5.15.0-72-generic #79-Ubuntu SMP Wed Apr 19 08:22:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy

$ docker --version
Docker version 24.0.2, build cb74dfc

Outer Docker is rust:latest (debian:11, 11.7, bullseye, bullseye-20230612):

$ docker --version
Docker version 20.10.5+dfsg1, build 55c4c88

$ cross --version
cross 0.2.5
[cross] note: Falling back to `cargo` on the host.
cargo 1.70.0 (ec8a8a0ca 2023-04-25)

Inner Docker is the aarch64-unknown-linux-gnu image.

What target(s) are you cross-compiling for?

aarch64-unknown-linux-gnu

Which operating system is the host (e.g computer cross is on) running?

• macOS
• Windows
• Linux / BSD
• other OS (specify in description)

What architecture is the host?

• x86_64 / AMD64
• arm32
• arm64 (including Mac M1)

What container engine is cross using?

• docker
• podman
• other container engine (specify in description)

cross version

cross 0.2.5

Example

#260 (comment)

Examining the Docker command line that is run by cross, it finds the /var/lib/docker mounts, I believe by querying the docker.sock, but because the outer Docker container does not mount /var/lib/docker, the inner Docker can't then access any of those mounts.

It would be an issue anyways as the outer Docker container marks all mounts root:root in /var/lib/docker.

Additional information / notes

I'm trying to think of workarounds.

One of which would be to run rustup in my non-admin account in the host system, and then run the aarch64-unknown-linux-gnu container.

Another workaround might be Cross.toml offering a way to specify what folders to mount for which tools.

Another workaround might be to use a Docker managed volume for the tools and cross.

Final option would be to reproduce the mounts in a docker-compose.yml and reuse the cross-rs containers from the host system.

[Emilgardis]

Can you please input the details of the error into the description instead of having it in another issue? Thanks!

what virtual filesystem does docker end up using? Does it work better when using cross from the main branch? cargo install cross --git https://github.com/cross-rs/cross

[nuket]

Sure. Docker is using overlay2. I tried this all again using the cross main branch and it gives the same error.

   Installed package `cross v0.2.5 (https://github.com/cross-rs/cross#1511a287)` (executables `cross`, `cross-util`)                                                                                                                                         rustuser@rust-build:~/rustuser/hello-world$ cross build --target aarch64-unknown-linux-gnu                                                                                                                                                                   info: downloading component 'rust-std' for 'aarch64-unknown-linux-gnu'
info: installing component 'rust-std' for 'aarch64-unknown-linux-gnu'
 32.8 MiB /  32.8 MiB (100 %)  19.5 MiB/s in  1s ETA:  0s
Unable to find image 'ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main' locally
main: Pulling from cross-rs/aarch64-unknown-linux-gnu
56e0351b9876: Pull complete
4af1bd48d3cc: Pull complete
4aca212696d7: Pull complete
9b86d293c3aa: Pull complete
135d8bb64bfe: Pull complete
f8d31578d148: Pull complete
c31c5054dcb5: Pull complete
8b0ee46a57da: Pull complete
9ac65f18b85a: Pull complete
9c167934d71a: Pull complete
f071157538a0: Pull complete
bf17075c21f0: Pull complete
72024d4b9b48: Pull complete
670aea1b2e5b: Pull complete
3fa7105e6443: Pull complete
5d921008a306: Pull complete
fd46145107df: Pull complete
5df2ce8167df: Pull complete
Digest: sha256:69aff27a8d7e3fca1cd62d4537085f52ab6dd10a212f7fbc302193475fbe9e4f
Status: Downloaded newer image for ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main
sh: 1: cargo: not found
rustuser@rust-build:~/rustuser/hello-world$ cross build -vv --target aarch64-unknown-linux-gnu
+ cargo metadata --format-version 1 --filter-platform aarch64-unknown-linux-gnu
+ rustc --print sysroot
+ /usr/bin/docker
+ /usr/bin/docker version -f '{{ .Server.Os }},,,{{ .Server.Arch }}'
+ rustup toolchain list
+ rustup target list --toolchain 1.70.0-x86_64-unknown-linux-gnu
+ rustup component list --toolchain 1.70.0-x86_64-unknown-linux-gnu
+ rustup --verbose component add rust-src --toolchain 1.70.0-x86_64-unknown-linux-gnu
verbose: read metadata version: '12'
info: downloading component 'rust-src'
verbose: downloading file from: 'https://static.rust-lang.org/dist/2023-06-01/rust-src-1.70.0.tar.xz'
verbose: downloading with reqwest
verbose: checksum passed
info: installing component 'rust-src'
verbose: creating temp directory: /usr/local/rustup/tmp/mn72qzlcdzrry82n_dir
verbose: creating temp file: /usr/local/rustup/tmp/8shhcidths2cr1wx_file
verbose: creating temp file: /usr/local/rustup/tmp/6rfg9gh32kmpz1vv_file
verbose: deleted temp directory: /usr/local/rustup/tmp/mn72qzlcdzrry82n_dir
verbose: creating temp file: /usr/local/rustup/tmp/h43iqhs92bv3mr6i_file
verbose: creating temp file: /usr/local/rustup/tmp/53revexnd2nzs38s_file
verbose: deleted temp file: /usr/local/rustup/tmp/8shhcidths2cr1wx_file
verbose: deleted temp file: /usr/local/rustup/tmp/6rfg9gh32kmpz1vv_file
verbose: deleted temp file: /usr/local/rustup/tmp/h43iqhs92bv3mr6i_file
verbose: deleted temp file: /usr/local/rustup/tmp/53revexnd2nzs38s_file
+ /usr/bin/docker inspect rust-build
+ /usr/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/home/rustuser/.xargo' -e 'CARGO_HOME=/usr/local/cargo' -e 'CROSS_RUST_SYSROOT=/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-
x86_64-unknown-linux-gnu' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e CROSS_CONTAINER_IN_CONTAINER -e 'USER=rustuser' -e 'CROSS_RUSTC_MAJOR_VERSION=1' -e 'CROSS_RUSTC_MINOR_VERSION=70' -e 'CROSS_RUSTC_PATCH_VERSION=0' --name cross-1.70.0
-x86_64-unknown-linux-gnu-4f472-90c541806-aarch64-unknown-linux-gnu-8913b-1688026898002 --rm --user 1000:1000 -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/.xargo:/home/rustuser/.xargo:z -v /var/lib/docker/overlay2/kusnn1ksi
tcoorg62e3s43xo4/merged/usr/local/cargo:/usr/local/cargo:z -v /usr/local/cargo/bin -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world:/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/ru
stuser/hello-world:z -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu:/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-lin
ux-gnu:z,ro -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world/target:/target:z -w /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world -t ghcr.io/cross-rs/aarch64-unkn
own-linux-gnu:main sh -c 'PATH="$PATH":"/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu/bin" cargo build -vv --target aarch64-unknown-linux-gnu'
sh: 1: cargo: not found
+ rustup component list --toolchain 1.70.0-x86_64-unknown-linux-gnu
rustuser@rust-build:~/rustuser/hello-world$ env
HOSTNAME=rust-build
PWD=/home/rustuser/rustuser/hello-world
TZ=Europe/Berlin
HOME=/home/rustuser
CARGO_HOME=/usr/local/cargo
LESSCLOSE=/usr/bin/lesspipe %s %s
TERM=xterm
RUSTUP_HOME=/usr/local/rustup
LESSOPEN=| /usr/bin/lesspipe %s
CROSS_CONTAINER_IN_CONTAINER=true
SHLVL=1
RUST_VERSION=1.70.0
PATH=/home/rustuser/.local/bin:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env

If I look at the GraphDriver output from docker inspect:

"GraphDriver": {
    "Data": {
        "LowerDir": "/var/lib/docker/overlay2/lcict9yuf79atqnw3wmyaf88n/diff:/var/lib/docker/overlay2/s5s3g442kwq94uk2i8dl2zhf3/diff:/var/lib/docker/overlay2/233a539d76681edec13902221be6618919d025f20b6d815f3eb7e8a8142a98fe/diff:/var/lib/docker/overlay2/912d79a6732c346fc70b67c0a28a698e236ae11fe0f390b3f2b9515969b531f9/diff:/var/lib/docker/overlay2/eb2a67b15cd69939d3dc9730271610df85f519476b78a0567d391681518db33d/diff:/var/lib/docker/overlay2/862a3c349e660a3834e24ca97b91e1af7ed52eccd7d5477d35fff3fd8a8c44aa/diff:/var/lib/docker/overlay2/e15b634681e07fca087589cbe2be164100de48183587c9311fcfa1653f2db143/diff",
        "MergedDir": "/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged",
        "UpperDir": "/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/diff",
        "WorkDir": "/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/work"
    },
    "Name": "overlay2"
},

And again if I look at /var/lib from the outer Docker container, there's no /var/lib/docker, so I think when it tries to pass those into the inner Docker container running cross, it can't. But perhaps I don't understand how Docker resolves those overlays.

rustuser@rust-build:~/rustuser/hello-world$ ll /var/lib
total 64
drwxr-xr-x 1 root root 4096 Jun 29 09:44 ./
drwxr-xr-x 1 root root 4096 Jun 12 02:00 ../
drwxr-xr-x 1 root root 4096 Jun 29 09:44 apt/
drwxr-xr-x 1 root root 4096 Jun 29 09:44 dpkg/
drwxr-xr-x 2 root root 4096 Feb 22 10:51 git/
drwxr-xr-x 2 root root 4096 Apr  2 13:55 misc/
drwxr-xr-x 2 root root 4096 Jun 12 02:00 pam/
drwxr-xr-x 2 root root 4096 Jun 13 05:29 python/
drwxr-xr-x 3 root root 4096 Jun 29 09:44 sudo/
drwxr-xr-x 1 root root 4096 Jun 13 05:29 systemd/
drwxr-xr-x 3 root root 4096 Jun 13 05:29 ucf/
rustuser@rust-build:~/rustuser/hello-world$

If I run the cross image with -it and check what is mounted, there's essentially nothing there.

rustuser@rust-build:~/rustuser/hello-world$ /usr/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/home/rustuser/.xargo' -e 'CARGO_HOME=/usr/local/cargo' -e 'CROSS_RUST_SYSROOT=/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e CROSS_CONTAINER_IN_CONTAINER -e 'USER=rustuser' -e 'CROSS_RUSTC_MAJOR_VERSION=1' -e 'CROSS_RUSTC_MINOR_VERSION=70' -e 'CROSS_RUSTC_PATCH_VERSION=0' --name cross-1.70.0-x86_64-unknown-linux-gnu-4f472-90c541806-aarch64-unknown-linux-gnu-8913b-1688026898002 --rm --user 1000:1000 -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/.xargo:/home/rustuser/.xargo:z -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/cargo:/usr/local/cargo:z -v /usr/local/cargo/bin -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world:/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world:z -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu:/var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu:z,ro -v /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world/target:/target:z -w /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world -i -t ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main sh
$ id
uid=1000 gid=1000 groups=1000
$ mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/ELEJ3ASXQUUB73VF6WJEB4IKBR:/var/lib/docker/overlay2/l/72G7LQQ6RMELO4SA4LZYAMFVXF:/var/lib/docker/overlay2/l/MEWX6BY6RA6H3474OXWS2XX7QE:/var/lib/docker/overlay2/l/MW7BH3LP75LSGEMMOT3HOASW4S:/var/lib/docker/overlay2/l/24FK3GPTOYU4LEHLPSEX34JKWC:/var/lib/docker/overlay2/l/QMAOC23WTEP3XE7FA3F4EWVSHX:/var/lib/docker/overlay2/l/LII7A4KYM6SQPO65RN4W3V5VMY:/var/lib/docker/overlay2/l/KZ2GJ557I3IAETSNDOFIUHFOFB:/var/lib/docker/overlay2/l/7SI72CJ5KXA4OBK72AMKA7XO6N:/var/lib/docker/overlay2/l/U4TO6HGWKZIPEO3TZLGSRFM3RZ:/var/lib/docker/overlay2/l/524BCNZJEGUX7SNXBQDNMUPSON:/var/lib/docker/overlay2/l/A3EXFEBPJ2BUEZXPJRZQR6NP64:/var/lib/docker/overlay2/l/6K5OUG4V4S7XIDA4YPCYYAI2ZH:/var/lib/docker/overlay2/l/6UEJFQNGYQE77PDPM2FHAYGXOU:/var/lib/docker/overlay2/l/KGLRUR4I7B4PB5HZ47SSLRZZKR:/var/lib/docker/overlay2/l/K4VKZY4JNRZ7Z2RAKPO2ENJONS:/var/lib/docker/overlay2/l/E4WZ7OBV4T7SX5FHRVEON3WY5C:/var/lib/docker/overlay2/l/YW3WHISABW6HOKQW5XD3ZP2TA5:/var/lib/docker/overlay2/l/DGFMBIJIULXCTE2CPAEMJIOYIE,upperdir=/var/lib/docker/overlay2/81d85e2358468765c33f9e21a2d7a05a4c2ef13e4b59565d4cc90a8740627255/diff,workdir=/var/lib/docker/overlay2/81d85e2358468765c33f9e21a2d7a05a4c2ef13e4b59565d4cc90a8740627255/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup type cgroup2 (ro,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,inode64)
/dev/mapper/ubuntu--vg-ubuntu--lv on /target type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /etc/resolv.conf type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /etc/hostname type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /etc/hosts type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/local/cargo type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /home/rustuser/.xargo type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /usr/local/cargo/bin type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/home/rustuser/rustuser/hello-world type ext4 (rw,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on /var/lib/docker/overlay2/kusnn1ksitcoorg62e3s43xo4/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu type ext4 (ro,relatime)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
proc on /proc/bus type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/fs type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/irq type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
tmpfs on /proc/asound type tmpfs (ro,relatime,inode64)
tmpfs on /proc/acpi type tmpfs (ro,relatime,inode64)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)
tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,mode=755,inode64)
tmpfs on /proc/scsi type tmpfs (ro,relatime,inode64)
tmpfs on /sys/firmware type tmpfs (ro,relatime,inode64)
$ ls /
base-runner.sh  bin  boot  common.sh  dev  etc  home  lib  lib.sh  lib32  lib64  libx32  linux-image.sh  linux-runner  media  mnt  opt  proc  qemu  root  run  sbin  srv  sys  target  tmp  usr  var
$ ls /usr/local/cargo
bin
$ ls /usr/local/cargo/bin
$ ls /usr/local/rustup
ls: cannot access '/usr/local/rustup': No such file or directory
$ ls /usr/local/
bin  cargo  doc  etc  games  include  lib  libexec  man  sbin  share  src

[nuket]

Ahh, let me add the docker-compose.yml I'm using for the outer Docker container:

version: '3.5'

services:
  rust-build:
    build:
      context: containers/rust-build
    environment:
      - CROSS_CONTAINER_IN_CONTAINER=true
    group_add:
      - docker
    hostname: rust-build
    image: rust-build:latest
    stdin_open: true
    tmpfs:
      - /tmp:exec
    user: rustuser
    volumes:
      - ${HOME}:/home/rustuser
      - ${HOME}/.ssh:/home/rustuser/.ssh
      - /var/run/docker.sock:/var/run/docker.sock  # needed to run 'cross' within container
    working_dir: /home/rustuser

And the Dockerfile for that:

FROM rust:latest

ARG USERNAME=rustuser
ARG GROUPNAME=rustuser
ARG USER_UID=1000
ARG USER_GID=1000

ARG DEBIAN_FRONTEND=noninteractive

ENV TZ=Europe/Berlin

RUN mkdir -p /usr/share/man/man1/ \
    # Configure apt and install packages
    && apt-get update -y \
    && apt-get -y --no-install-recommends install \
        docker.io \
        less \
        nano \
        sudo

# Create a non-root user to use
RUN groupadd --gid $USER_GID $GROUPNAME || true \
    && useradd -s /bin/bash --uid $USER_UID --gid $USER_GID -m $USERNAME

# Hacky solutions to make sure user is part of docker group
#
# You have to set the inner gid to the same integer as the host system docker gid
# otherwise inner docker can't talk to /var/run/docker.sock
RUN groupmod -g 113 docker \
    && adduser $USERNAME docker

[lumasepa]

I'm facing the same problem, any ideas on how to solve it? If there is a solution I can work on it.

[Emilgardis]

I'm facing the same problem, any ideas on how to solve it? If there is a solution I can work on it.

@lumasepa are you also setting CROSS_CONTAINER_IN_CONTAINER=1 ?

[lumasepa]

@lumasepa are you also setting CROSS_CONTAINER_IN_CONTAINER=1 ?

I tried with and without that env var, but in my case I have dind with two docker demons, one for the outer container and the other running inside the outer container to run the inner container. So host has the crate directory, outer mounts the crate directory and has the rust tooling + cross, cross runs in the outer with its container having access to rust tooling.

I changed the setup to use outer docker and inner podman and it works without problems, and I don't have to run docker demon inside of docker.

Sorry for the late response, I've been busy with other stuff, I came back to this today.