Cross fails to mount if docker daemon not running on same machine

[Lakier15]

I have been trying desperately to get the cross command working in a gitlab CI.

The problem is really what happens when cross tries to mount via '-v' when the docker daemon is not running on the same machine (so either by mounting '/var/run/docker.sock' or by setting 'DOCKER_HOST=tcp://something:2375').

It seems that when you pass a '-v' command to the docker binary, it mounts drives relative to where the docker daemon is running, and not relative to where the docker binary is running.

So, in a configuration like docker:dind, cross tries to mount non-existent directories outside of the container is running in, failing miserably and returning "cargo: not found".

The only alternative I have found is to run the daemon inside the same container.
Now, because the docker daemon is quite heavy, in terms of computing power, it is not efficient to do so.
Imagine spawning 10 containers running the 'cross' command, you would have to spawn 10 daemons just so that 'cross' can mount the folders properly.

I know that Travis works with cross. The fact is that I have no idea how Travis manages to do so, but there are cases where using, and being limited to, Travis, is not ideal.
In the commercial, enterprise world, there are usually strict requirements on hosting the code only on company servers on company locations. And because Travis is not something one can run locally, like Gitlab or Bamboo, one is stuck with the usual cargo commands without having the possibility to use 'cross'.

Is there an alternative, or a way to solve this issue?

[fenollp]

Me too! I'd love to be able to run

export DOCKER_HOST=ssh://beefy.machine
cross build --release --target=armv7-unknown-linux-musleabihf

[Alexhuszagh]

Currently working on this: unfortunately the overhead is fairly large so we can't enable this by default. It's likely going to require some persistent data storage as an option, because otherwise the alternate is to copy all files from the local machine to the docker host, including the project directory, target directory, cargo directory, xargo directory, and rust toolchain. This can take 5+ minutes easily with a large number of crates in the registry and for large projects, but I'm getting pretty close to full support.

Sorry for the delay, this was on our radar.

[fenollp]

Thanks @Alexhuszagh! Any in-dev branch I can take a look at?

[Alexhuszagh]

Thanks @Alexhuszagh! Any in-dev branch I can take a look at?

This is still very rudimentary and not yet complete (it does not work yet), but here's the branch I'm working off. I will frequently be squashing commits so these commit names and history won't stay. I'll update you when I have an actual, viable example.

https://github.com/Alexhuszagh/cross/tree/remote

The general logic is as follows, and this code does actually work, except with changes to use the actual toolchain paths, etc:

#!/bin/bash

target=arm-unknown-linux-gnueabihf

docker volume create data-volume

docker run \
    --userns host \
    -v data-volume:/cross \
    -e PKG_CONFIG_ALLOW_CROSS=1 \
    -e XARGO_HOME=/xargo \
    -e CARGO_HOME=/cargo \
    -e CARGO_TARGET_DIR=/target \
    -e USER=ahuszagh \
    -e CROSS_RUNNER= \
    -v /cross/cargo/bin \
    -td \
    --name helper \
    ghcr.io/cross-rs/"${target}":main
    sh -c "tail -f /dev/null"

docker cp -a /home/ahuszagh/.cargo helper:/cross/cargo
docker cp -a /home/ahuszagh/.xargo helper:/cross/xargo
docker cp -a /home/ahuszagh/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu helper:/cross/rust
docker cp -a "$PWD" helper:/cross/project
if [[ -d "$PWD/target" ]]; then
    docker cp -a "$PWD/target" helper:/cross/target
fi

docker exec helper sh -c "mkdir -p /cross/target && chown -R 1000:1000 /cross/* && ln -s /cross/cargo /cargo && ln -s /cross/xargo /xargo && ln -s /cross/rust /rust && ln -s /cross/project /project && ln -s /cross/target /target"

docker exec \
    --user 1000:1000 \
    -w /project helper \
    sh -c 'PATH=$PATH:/rust/bin cargo build --target '"${target}"' --verbose'

docker cp -a helper:/cross/target "$PWD"/target
docker stop helper
docker rm helper
docker volume rm data-volume

This is still very rudimentary, but I'm going to be adding persistent data volumes (just for the cargo registry, git, and parts of the toolchain) so those aren't copied every single time over. I'm also going to be limiting how much of the toolchain is copied over, just so this goes a lot faster. We only need one of the targets present, and for example with me and my 40+ targets, this reduces the amount of bytes to copy over from 5.0GB to 160MB. Same for the cargo registry and git: if I can pre-fetch them with a persistent data store, that's 3GB to nothing.

We also make sure the volumes/containers are uniquely labeled based on the path and package name, to avoid having conflicts with anything else on the system (not just helper and data-volume).

[Alexhuszagh]

Ok I've got this mostly working with a few caveats, @fenollp. It should be ready for basic use, however. It's probably still worth knowing how to prune local volumes.

The code has been updated at remote and the logic of how it works has been commented:

    // the logic is broken into the following steps
    // 1. create a data volume to store everything
    // 2. start our container with the data volume and all envvars
    // 3. copy all mounted volumes over
    // 4. create symlinks for all mounted data
    // 5. execute our cargo command inside the container
    // 6. copy data from target dir back to host
    // 7. stop container and delete data volume
    //
    // we use structs that wrap the resources to ensure they're dropped
    // in the correct order even on error, to ensure safe cleanup

A very verbose output is as follows, but the key points will be described below:

1. Create a data-volume if a persistent storage volume isn't available, with the name unique based on the path and package name. Here, the name is cross-cc-arm-unknown-linux-gnueabihf-5ff45, where the last 5 are the first 5 characters of a sha1 hexdigest.
   • A deleter is then added, so if any errors occur, this volume will be reliably cleaned up.
2. Start our container with all our environment variables. The container name is the same as the volume name (separate namespaces, easy unique naming).
   • A deleter is then added, so if any errors occur, this container will be stopped, removed, and then the volume cleaned up.
3. Copy all the data over (we do one redundant copy right now: we copy the target directory in most cases over twice. I can symlink it to /target later).
   • A lot of effort was put in to minimizing the amount of stuff copied, so very little of your active toolchain gets copied over, and more, leading to much faster build times.
4. Symlink everything in the data directory. Normally, we have our paths mounted at /cargo, /xargo, etc. But this doesn't work, because everything is a subdir of /cross. So we just ensure everything has user permissions, symlink it to the expected location, and go from there. We also do a recursive symlink check, so if we try to symlink /cross/home, it won't link to /home, but rather recurse into the users and only symlink stuff not present in the container (which should always be a non-issue here).
5. Execute our command, and store the status.
6. Copy the data in the target directory back, conserving our timestamps and user permissions.
7. Leave scope, running our DeleteContainer and DeleteVolume destructors in that order.

$ CROSS_REMOTE_SKIP_REGISTRY=true CROSS_REMOTE=true /home/ahuszagh/git/cross/target/debug/cross build --target arm-unknown-linux-gnueabihf --verbose
+ "cargo" "metadata" "--format-version=1" "--filter-platform" "arm-unknown-linux-gnueabihf"
+ "rustc" "--print" "sysroot"
+ "rustup" "toolchain" "list"
+ "rustup" "target" "list" "--toolchain" "stable-x86_64-unknown-linux-gnu"
+ "rustup" "component" "list" "--toolchain" "stable-x86_64-unknown-linux-gnu"
+ "/usr/bin/docker" "--help"
+ "/usr/bin/docker" "volume" "inspect" "cross-cc-arm-unknown-linux-gnueabihf-5ff45-keep"
+ "/usr/bin/docker" "volume" "create" "cross-cc-arm-unknown-linux-gnueabihf-5ff45"
cross-cc-arm-unknown-linux-gnueabihf-5ff45
+ "/usr/bin/docker" "run" "--userns" "host" "--name" "cross-cc-arm-unknown-linux-gnueabihf-5ff45" "-v" "cross-cc-arm-unknown-linux-gnueabihf-5ff45:/cross" "-e" "VAR=VALUE" "-e" "PKG_CONFIG_ALLOW_C
ROSS=1" "-e" "XARGO_HOME=/xargo" "-e" "CARGO_HOME=/cargo" "-e" "CARGO_TARGET_DIR=/target" "-e" "CROSS_RUNNER=" "-e" "USER=ahuszagh" "-e" "VOL=/home/ahuszagh/git/f128" "-v" "/cross/cargo/bin" "-d"
"-t" "ghcr.io/cross-rs/arm-unknown-linux-gnueabihf:main" "sh" "-c" "tail -f /dev/null"
f1451ec50ba6144c64e1ae4d66fb9ccd3953e7fddcd9ca7c7d3ccc2edfac5e21
+ "/usr/bin/docker" "exec" "cross-cc-arm-unknown-linux-gnueabihf-5ff45" "sh" "-c" "mkdir -p \'/cross/cargo\'"
+ "/usr/bin/docker" "cp" "-a" "/home/ahuszagh/.cargo/bin" "cross-cc-arm-unknown-linux-gnueabihf-5ff45:/cross/cargo"
+ "/usr/bin/docker" "cp" "-a" "/home/ahuszagh/.cargo/env" "cross-cc-arm-unknown-linux-gnueabihf-5ff45:/cross/cargo"
+ "/usr/bin/docker" "cp" "-a" "/home/ahuszagh/.cargo/credentials" "cross-cc-arm-unknown-linux-gnueabihf-5ff45:/cross/cargo"
+ "/usr/bin/docker" "cp" "-a" "/home/ahuszagh/.cargo/config" "cross-cc-arm-unknown-linux-gnueabihf-5ff45:/cross/cargo"
+ "/usr/bin/docker" "exec" "cross-cc-arm-unknown-linux-gnueabihf-5ff45" "sh" "-c" "mkdir -p \'/cross/rust\'"
+ "/usr/bin/docker" "cp" "-a" "/home/ahuszagh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin" "cross-cc-arm-unknown-linux-gnueabihf-5ff45:/cross/rust"
+ "/usr/bin/docker" "cp" "-a" "/home/ahuszagh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/libexec" "cross-cc-arm-unknown-linux-gnueabihf-5ff45:/cross/rust"
+ "/usr/bin/docker" "cp" "-a" "/home/ahuszagh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/etc" "cross-cc-arm-unknown-linux-gnueabihf-5ff45:/cross/rust
+ "/usr/bin/docker" "cp" "-a" "/tmp/.tmpxB6X65" "cross-cc-arm-unknown-linux-gnueabihf-5ff45:/cross/rust/lib"
+ "/usr/bin/docker" "exec" "cross-cc-arm-unknown-linux-gnueabihf-5ff45" "sh" "-c" "mkdir -p \'/cross/rust/lib/rustlib\'"
+ "/usr/bin/docker" "cp" "-a" "/home/ahuszagh/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/arm-unknown-linux-gnueabihf" "cross-cc-arm-unknown-linux-gnueabihf-5ff45:/cross/rust/li
b/rustlib"
+ "/usr/bin/docker" "cp" "-a" "/home/ahuszagh/Desktop/cross/rust-cpp-hello-word/target" "cross-cc-arm-unknown-linux-gnueabihf-5ff45:/cross/target"
+ "/usr/bin/docker" "exec" "cross-cc-arm-unknown-linux-gnueabihf-5ff45" "sh" "-c" "mkdir -p \'/cross/home/ahuszagh/Desktop/cross\'"
+ "/usr/bin/docker" "cp" "-a" "/home/ahuszagh/Desktop/cross/rust-cpp-hello-word" "cross-cc-arm-unknown-linux-gnueabihf-5ff45:/cross/home/ahuszagh/Desktop/cross/rust-cpp-hello-word"
+ "/usr/bin/docker" "exec" "cross-cc-arm-unknown-linux-gnueabihf-5ff45" "sh" "-c" "mkdir -p \'/cross/home/ahuszagh/git\'"
+ "/usr/bin/docker" "cp" "-a" "/home/ahuszagh/git/f128" "cross-cc-arm-unknown-linux-gnueabihf-5ff45:/cross/home/ahuszagh/git/f128"
+ "/usr/bin/docker" "exec" "cross-cc-arm-unknown-linux-gnueabihf-5ff45" "sh" "-c" "set -e pipefail\nset -x\nchown -R 1000:1000 /cross/*\nprefix=\"/cross\"\n\nsymlink_recurse() {\n    for f[50/992]
1}\"/*; do\n        dst=${f#\"$prefix\"}\n        if [ -f \"${dst}\" ]; then\n            echo \"invalid: got unexpected file at ${dst}\" 1>&2\n            exit 1\n        elif [ -d \"${dst}\" ];
then\n            symlink_recurse \"${f}\"\n        else\n            ln -s \"${f}\" \"${dst}\"\n        fi\n    done\n}\n\nsymlink_recurse \"${prefix}\"\n"
+ chown -R 1000:1000 /cross/cargo /cross/home /cross/rust /cross/target
+ prefix=/cross
+ symlink_recurse /cross
+ dst=/cargo
+ [ -f /cargo ]
+ [ -d /cargo ]
+ ln -s /cross/cargo /cargo
+ dst=/home
+ [ -f /home ]
+ [ -d /home ]
+ symlink_recurse /cross/home
+ dst=/home/ahuszagh
+ [ -f /home/ahuszagh ]
+ [ -d /home/ahuszagh ]
+ ln -s /cross/home/ahuszagh /home/ahuszagh
+ dst=/rust
+ [ -f /rust ]
+ [ -d /rust ]
+ ln -s /cross/rust /rust
+ dst=/target
+ [ -f /target ]
+ [ -d /target ]
+ ln -s /cross/target /target
+ "/usr/bin/docker" "exec" "--user" "1000:1000" "-w" "/home/ahuszagh/Desktop/cross/rust-cpp-hello-word" "cross-cc-arm-unknown-linux-gnueabihf-5ff45" "sh" "-c" "PATH=$PATH:/rust/bin cargo build --t
arget arm-unknown-linux-gnueabihf --verbose"
    Updating crates.io index
 Downloading crates ...
  Downloaded cc v1.0.73
       Fresh cc v1.0.73
       Fresh hellopp v0.1.0 (/cross/home/ahuszagh/Desktop/cross/rust-cpp-hello-word)
    Finished dev [unoptimized + debuginfo] target(s) in 1m 01s
+ "/usr/bin/docker" "cp" "-a" "cross-cc-arm-unknown-linux-gnueabihf-5ff45:/cross/target" "/home/ahuszagh/Desktop/cross/rust-cpp-hello-word/target"
+ "/usr/bin/docker" "stop" "cross-cc-arm-unknown-linux-gnueabihf-5ff45"
cross-cc-arm-unknown-linux-gnueabihf-5ff45
+ "/usr/bin/docker" "rm" "cross-cc-arm-unknown-linux-gnueabihf-5ff45"
cross-cc-arm-unknown-linux-gnueabihf-5ff45
+ "/usr/bin/docker" "volume" "rm" "cross-cc-arm-unknown-linux-gnueabihf-5ff45"
cross-cc-arm-unknown-linux-gnueabihf-5ff45

So yeah, this is likely ready for more testing (I've got an SSH cloud server I'm using), and I'll get back to you, but we should have everything (and the worst merge conflicts of my life) in a day.

[Alexhuszagh]

@fenollp If you'd like support now, there's a working branch that should be fully functional and support persistent data volumes, with most of the initial bugs ironed out.

To use it, first install from git:

cargo install --git https://github.com/Alexhuszagh/cross --branch remote

Then I'd highly recommend using persistent data volumes, since it will be much faster if you use more than 1 command. The create-crate-volume automatically creates a persistent data volume based off the name of the package (in the workspace root, if present), the target triple, a hash of the path to the package manifest path, and a hash for the current toolchain path, and the commit hash for the system cargo version. In short, the persistent volume aims to be unique only if you are compiling for the same package and the same toolchain, otherwise, it correctly changes. For example, in my project, the volume name is cross-hellopp-arm-unknown-linux-gnueabihf-c1f13-16b8c-fe5b13d681f25ee6474be29d748c65adcd91f69e-keep. Namespaced for cross, the toolchain, and unlikely to every have accidental conflicts, to say the least.

export DOCKER_HOST="tcp://docker:2375"
export CROSS_REMOTE=true
# setup our data volume
cross-util create-crate-volume --target arm-unknown-linux-gnueabihf

# run multiple commands
cross build --target arm-unknown-linux-gnueabihf
cross run --target arm-unknown-linux-gnueabihf
cross test --target arm-unknown-linux-gnueabihf

# cleanup our data volume (only when done)
cross-util remove-crate-volume --target arm-unknown-linux-gnueabihf

# just in case, remove any lingering data (don't do this normally)
cross-util remove-containers --execute  # execute means don't do a dry run, the default
cross-util remove-volumes --execute
cross-util prune-volumes

And... that should be all there is to it :D Let me know how it works. Any bug reports would be greatly appreciated. Just an FYI: we've added cross-util as another binary since we can't possibly introduce compatibility issues with cargo itself, so adding any subcommands or commands would be strictly prohibited. It will be automatically installed along with cross in a future release.