Support Remote Docker tcp://docker:2375 #449

schrieveslaach · 2020-08-02T18:52:33Z

This PR changes the behavior of cross so that this PR fixes #260. By using Docker data volumes such issues such not arise anymore and running the Docker client on a different host.

Maeda-Naoki · 2021-05-25T15:03:00Z

I think this PR is very effective.
Why don't you do a review or merge?

Alexhuszagh

If I'm counting, this means we need to invoke Docker 11 more times than normal, and we need to copy every file we use to a temporary data volume? Surely there must be a better solution than this?

I'll see if I can find a better solution, but this seems like it would make cross-compiling take forever, and we've also made numerous changes since then (#684) which would requiring copying even more files over I believe, as well as more Docker invocations.

Alexhuszagh · 2022-06-08T23:46:09Z

src/docker.rs

+    copy_files(&cargo_dir, "/cross", verbose)?;
+    copy_files(&xargo_dir, "/cross", verbose)?;
+    copy_files(&sysroot, "/cross", verbose)?;
+    copy_files(&mount_root, "/cross", verbose)?;


Is there a solution other than copying every single file in our cargo, xargo, sysroot, and mount root over?

Alexhuszagh · 2022-06-09T00:46:44Z

We might be able to add a remote flag or check if DOCKER_HOST is set, but this does not sound like an ideal option by default, just because of the sheer overhead of it. This would also have to support podman, so we'd need podman-remote instead of the podman executable, and we'd need to construct the --url likely, or use the value of CONTAINER_CONNECTION. See podman-remote.

schrieveslaach · 2022-06-09T05:54:37Z

@Alexhuszagh, thanks for looking into it.

If I'm counting, this means we need to invoke Docker 11 more times than normal, and we need to copy every file we use to a temporary data volume? Surely there must be a better solution than this?

I don't think so that there is another solution. As stated in Docker's best practices the docker build command does the same thing:

Regardless of where the Dockerfile actually lives, all recursive contents of files and directories in the current directory are sent to the Docker daemon as the build context.

Because Docker runs on a different system/host, which is the case in GitLab CI, the files have to be copied over there.

I could image that cross could determine if it is talking to a remote Docker daemon or to a remote podman daemon. If that is the case than cross can copy the files to the remote process, otherwise it will use the volumes.

Alexhuszagh · 2022-06-09T06:00:41Z

I could image that cross could determine if it is talking to a remote Docker daemon or to a remote podman daemon. If that is the case than cross can copy the files to the remote process, otherwise it will use the volumes.

I think this is doable, since there's environment variables you can set to do this (such as DOCKER_HOST, PODMAN_VARLINK_ADDRESS, etc.). I can do some quick tests to see how much overhead this adds if running on a local machine. If it's as I assume, and there is significant overhead, then we probably would need slightly different codepaths (and likely an environment variable for testing that would also force this codepath). If it doesn't add much overhead, we might be able to get lucky with the single code path.

I'll take a look and get back to you.

EDIT: We might also have to deal with system connection defaults (see here for podman), which I might document isn't supported.

Alexhuszagh · 2022-06-09T16:40:16Z

I've got a pretty good idea on how to support this, and I think it's better just to let the user set the CROSS_REMOTE environment variable if they want remote use, and also allow persistent volumes as an option since we now have cross-util, with the ability to cleanup these volumes when needed. For volumes that aren't read-only, we would have to copy data over on each build, and then I believe copy the target directory back to the host after the build (since otherwise we just have a compilation check).

I've done an extensive analysis on how to detect if we're using a remote client, and it's a bit complex. It's not impossible, but it's not fun, and should have some serious overhead. Which, considering the user should know if it's a remote or local environment, I think is irrelevant. Some factors that affect it with docker include:

DOCKER_CONTEXT (overrides DOCKER_HOST).
DOCKER_HOST (overrides the current default context).
docker context ls to grab the current context using a format option.

Once we have that, if we have a context, we need to get the host viadocker context inspect ... -f "{{.Endpoints.docker.Host}}". Once we have the host, then we need to determine if it's remote or local, by determining if it's an SSH or TCP or other connection (since the connection can still be local, such as a unix:// host.

Podman's different entirely, but is doable as well. I think the better option is a simple environment variable. This controls whether we use data volumes or not.

Alexhuszagh · 2022-06-10T19:10:42Z

This is glacial but I think I have a working solution that will require a config variable. Compiling a zero-dependency crate with a lot of toolchains and targets installed (to be fair, 40 targets and 40 toolchains) is taking 10+ minutes.

That being said, this should be doable and the above logic will work with any mounts, because the symlinks will allow us to ensure the paths are consistent with the normal ones. The basic logic (in bash) is as follows:

#!/bin/bash

target=arm-unknown-linux-gnueabihf

docker volume create data-volume

docker run \
    --userns host \
    -v data-volume:/cross \
    -e PKG_CONFIG_ALLOW_CROSS=1 \
    -e XARGO_HOME=/xargo \
    -e CARGO_HOME=/cargo \
    -e CARGO_TARGET_DIR=/target \
    -e USER=ahuszagh \
    -e CROSS_RUNNER= \
    -v /cross/cargo/bin \
    -td \
    --name helper \
    ghcr.io/cross-rs/"${target}":main

docker cp /home/ahuszagh/.cargo helper:/cross/cargo
docker cp /home/ahuszagh/.xargo helper:/cross/xargo
docker cp /home/ahuszagh/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu helper:/cross/rust
docker cp "$PWD" helper:/cross/project
if [[ -d "$PWD/target" ]]; then
    docker cp "$PWD/target" helper:/cross/target
fi

docker exec helper bash -c "mkdir -p /cross/target && chown -R 1000:1000 /cross/* && ln -s /cross/cargo /cargo && ln -s /cross/xargo /xargo && ln -s /cross/rust /rust && ln -s /cross/project /project && ln -s /cross/target /target"

docker exec \
    --user 1000:1000 \
    -w /project helper \
    bash -c 'PATH=$PATH:/rust/bin cargo build --target '"${target}"' --verbose'

docker cp helper:/cross/target "$PWD"/target
docker stop helper
docker rm helper
docker volume rm data-volume

docker start "${container}" unfortunately didn't work because the container stops immediately: the only solution seems to be docker run -td "${image}", then copy the data in, create the symlinks, execute the build command, and then go from there. The other thing we need to do is ensure if a command fails, we don't forget to cleanup, or else this will be quite problematic.

    docker
        .arg(&image(toml, target)?)
        .args(&["sh", "-c", &format!("PATH=$PATH:/cross/{}/bin {:?}", sysroot.file_name().unwrap().to_str().unwrap(), cmd)])
        .run_and_get_status(verbose)?;
    docker_command("cp")?
        .args(&["helper:/cross/target", &format!("{}", target_dir.parent().unwrap().display())])
        .run_and_get_status(verbose)?;

    docker_command("rm")?
        .arg("helper")
        .run_and_get_status(verbose)?;
    docker_command("volume")?
        .args(&["rm", "data-volume"])
        .run_and_get_status(verbose)

The cp, rm, and volume rm commands need to execute no matter if the build step succeeds or fails. The return status should be the result of the run command (which is now an exec command).

We'll also be changing helper and data-volume to a unique identifier based on the path of the host and the package name. The end result will look practically nothing like this PR, so I'll be submitting a PR that replaces this one and I will give you credit for your work.

I'm also planning on adding persistent volume support as a feature, since this is quite slow.

schrieveslaach · 2022-06-11T06:19:43Z

@Alexhuszagh, submitting a new PR sounds good to me because this PR is quite old. I appreciate that you'll give me credit for this. Let me know if you have something that I can test. I have a GitLab project that we can use for testing driving the new PR.

I'm also planning on adding persistent volume support as a feature, since this is quite slow.

I've one question: I'm not sure if that is desirable in the long term because it would be very nice to construct CI pipelines that can rely on a precompiled binary. If that volume feature is behind a Cargo feature, it is required to build cross from source in the CI pipeline and that takes time. For example, building cross in one my project's jobs takes about two minutes and I think that copying the files to the volume might be quicker.

So, I would like to propose that the volume code path should be enabled by a command line feature. For example cross build --copy-sources-to-volume … might be a viable approach that can serve CI and local development use cases.

Alexhuszagh · 2022-06-12T17:35:56Z

@Alexhuszagh, submitting a new PR sounds good to me because this PR is quite old. I appreciate that you'll give me credit for this. Let me know if you have something that I can test. I have a GitLab project that we can use for testing driving the new PR.

I'm also planning on adding persistent volume support as a feature, since this is quite slow.

I've one question: I'm not sure if that is desirable in the long term because it would be very nice to construct CI pipelines that can rely on a precompiled binary. If that volume feature is behind a Cargo feature, it is required to build cross from source in the CI pipeline and that takes time. For example, building cross in one my project's jobs takes about two minutes and I think that copying the files to the volume might be quicker.

So, I would like to propose that the volume code path should be enabled by a command line feature. For example cross build --copy-sources-to-volume … might be a viable approach that can serve CI and local development use cases.

We'd be using environment variables for this (currently CROSS_REMOTE, since to ensure we don't interfere with new cargo flags, we don't support any custom subcommands or command-line flags), but we wouldn't have it as a feature for sure. I've actually created a new binary, cross-util, just so we can support other features that require subcommands or command-line flags (such as listing cross container images and deleting them), just to ensure we never break compatibility.

See this comment for how the (incomplete) implementation is designed. I've also added another tentative flag (CROSS_REMOTE_SKIP_REGISTRY) to avoid copying over the cargo git and registry folders. Also just a notice: these commits aren't going to be the final ones, and will be entirely removed (and credit for your work will be added once I have a more solid foundation and not just "Dummy commit 1" and "plz work" as all the messages).

Also, if you would like to, I'll request your feedback just to ensure it works for your use-case (as well as one other), since you will be the initial testers and users of said functionality.

This supports the volume-based structure, and uses some nice optimizations to ensure that only the desired toolchain and cargo items are copied over. It also uses drops to ensure scoped deletion of resources, to avoid complex logic ensuring their cleanup. It also supports persistent data volumes, through `cross-util`. In order to setup a persistent data volume, use: ```bash cross-util create-crate-volume --target arm-unknown-linux-gnueabihf ``` Make sure you provide your `DOCKER_HOST` or correct engine type to ensure these are being made on the remote host. Then, run your command as before: ```bash CROSS_REMOTE=true cross build --target arm-unknown-linux-gnueabihf ``` Finally, you can clean up the generated volume using: ```bash cross-util remove-crate-volume --target arm-unknown-linux-gnueabihf ``` A few other utilities are present in `cross-util`: - `list-volumes`: list all volumes created by cross. - `remove-volumes`: remove all volumes created by cross. - `prune-volumes`: prune all volumes unassociated with a container. - `list-containers`: list all active containers created by cross. - `remove-containers`: remove all active containers created by cross. The initial implementation was done by Marc Schreiber, https://github.com/schrieveslaach. Fixes cross-rs#248. Fixes cross-rs#273. Closes cross-rs#449.

schrieveslaach · 2022-06-13T07:07:45Z

Sounds good to me and thanks for keeping me in the loop. 👍 I'm happy to test and provide feedback. Just let me know when you have something to test or review for me.

Emilgardis · 2022-06-13T13:56:29Z

closing this pr, see #785 and previous discussion

This supports the volume-based structure, and uses some nice optimizations to ensure that only the desired toolchain and cargo items are copied over. It also uses drops to ensure scoped deletion of resources, to avoid complex logic ensuring their cleanup. It also supports persistent data volumes, through `cross-util`. In order to setup a persistent data volume, use: ```bash cross-util create-crate-volume --target arm-unknown-linux-gnueabihf ``` Make sure you provide your `DOCKER_HOST` or correct engine type to ensure these are being made on the remote host. Then, run your command as before: ```bash CROSS_REMOTE=true cross build --target arm-unknown-linux-gnueabihf ``` Finally, you can clean up the generated volume using: ```bash cross-util remove-crate-volume --target arm-unknown-linux-gnueabihf ``` A few other utilities are present in `cross-util`: - `list-volumes`: list all volumes created by cross. - `remove-volumes`: remove all volumes created by cross. - `prune-volumes`: prune all volumes unassociated with a container. - `list-containers`: list all active containers created by cross. - `remove-containers`: remove all active containers created by cross. The initial implementation was done by Marc Schreiber, https://github.com/schrieveslaach. Fixes cross-rs#248. Fixes cross-rs#273. Closes cross-rs#449.

This supports the volume-based structure, and uses some nice optimizations to ensure that only the desired toolchain and cargo items are copied over. It also uses drops to ensure scoped deletion of resources, to avoid complex logic ensuring their cleanup. It also supports persistent data volumes, through `cross-util`. In order to setup a persistent data volume, use: ```bash cross-util create-crate-volume --target arm-unknown-linux-gnueabihf ``` Make sure you provide your `DOCKER_HOST` or correct engine type to ensure these are being made on the remote host. Then, run your command as before: ```bash CROSS_REMOTE=true cross build --target arm-unknown-linux-gnueabihf ``` Finally, you can clean up the generated volume using: ```bash cross-util remove-crate-volume --target arm-unknown-linux-gnueabihf ``` A few other utilities are present in `cross-util`: - `list-volumes`: list all volumes created by cross. - `remove-volumes`: remove all volumes created by cross. - `prune-volumes`: prune all volumes unassociated with a container. - `list-containers`: list all active containers created by cross. - `remove-containers`: remove all active containers created by cross. The initial implementation was done by Marc Schreiber, https://github.com/schrieveslaach. A few more environment variables exist to fine-tune performance, as well as handle private dependencies. - `CROSS_REMOTE_COPY_REGISTRY`: copy the cargo registry - `CROSS_REMOTE_COPY_CACHE`: copy cache directories, including the target directory. Fixes cross-rs#248. Fixes cross-rs#273. Closes cross-rs#449.

This supports the volume-based structure, and uses some nice optimizations to ensure that only the desired toolchain and cargo items are copied over. It also uses drops to ensure scoped deletion of resources, to avoid complex logic ensuring their cleanup. It also supports persistent data volumes, through `cross-util`. In order to setup a persistent data volume, use: ```bash cross-util create-crate-volume --target arm-unknown-linux-gnueabihf ``` Make sure you provide your `DOCKER_HOST` or correct engine type to ensure these are being made on the remote host. Then, run your command as before: ```bash CROSS_REMOTE=true cross build --target arm-unknown-linux-gnueabihf ``` Finally, you can clean up the generated volume using: ```bash cross-util remove-crate-volume --target arm-unknown-linux-gnueabihf ``` A few other utilities are present in `cross-util`: - `list-volumes`: list all volumes created by cross. - `remove-volumes`: remove all volumes created by cross. - `prune-volumes`: prune all volumes unassociated with a container. - `list-containers`: list all active containers created by cross. - `remove-containers`: remove all active containers created by cross. The initial implementation was done by Marc Schreiber, schrieveslaach. A few more environment variables exist to fine-tune performance, as well as handle private dependencies. - `CROSS_REMOTE_COPY_REGISTRY`: copy the cargo registry - `CROSS_REMOTE_COPY_CACHE`: copy cache directories, including the target directory. Fixes cross-rs#248. Fixes cross-rs#273. Closes cross-rs#449.

This supports the volume-based structure, and uses some nice optimizations to ensure that only the desired toolchain and cargo items are copied over. It also uses drops to ensure scoped deletion of resources, to avoid complex logic ensuring their cleanup. It also supports persistent data volumes, through `cross-util`. In order to setup a persistent data volume, use: ```bash cross-util volumes create ``` This will create a persistent data volume specific for your current toolchain, and will be shared by all targets. This volume is specific for the toolchain version and channel, so a new volume will be created for each toolchain. Make sure you provide your `DOCKER_HOST` or correct engine type to ensure these are being made on the remote host. Then, run your command as before: ```bash CROSS_REMOTE=1 cross build --target arm-unknown-linux-gnueabihf ``` Finally, you can clean up the generated volume using: ```bash cross-util volumes remove ``` A few other utilities are present in `cross-util`: - `volumes list`: list all volumes created by cross. - `volumes remove`: remove all volumes created by cross. - `volumes prune`: prune all volumes unassociated with a container. - `containers list`: list all active containers created by cross. - `containers remove`: remove all active containers created by cross. - `clean`: clean all temporary data (images, containers, volumes, temp data) created by cross. The initial implementation was done by Marc Schreiber, schrieveslaach. A few more environment variables exist to fine-tune performance, as well as handle private dependencies. - `CROSS_REMOTE_COPY_REGISTRY`: copy the cargo registry - `CROSS_REMOTE_COPY_CACHE`: copy cache directories, including the target directory. Both of these generally lead to substantial performance penalties, but can enable the use of private SSH dependencies. In either case, the use of persistent data volumes is highly recommended. Fixes cross-rs#248. Fixes cross-rs#273. Closes cross-rs#449.

785: Add comprehensive support for remote docker. r=Emilgardis a=Alexhuszagh Add comprehensive support for remote docker. This supports the volume-based structure, and uses some nice optimizations to ensure that only the desired toolchain and cargo items are copied over. It also uses drops to ensure scoped deletion of resources, to avoid complex logic ensuring their cleanup. It also supports persistent data volumes, through `cross-util`. In order to setup a persistent data volume, use: ```bash cross-util volumes create ``` This will create a persistent data volume specific for your current toolchain, and will be shared by all targets. This volume is specific for the toolchain version and channel, so a new volume will be created for each toolchain. Make sure you provide your `DOCKER_HOST` or correct engine type to ensure these are being made on the remote host. Then, run your command as before: ```bash CROSS_REMOTE=1 cross build --target arm-unknown-linux-gnueabihf ``` Finally, you can clean up the generated volume using: ```bash cross-util volumes remove ``` A few other utilities are present in `cross-util`: - `volumes list`: list all volumes created by cross. - `volumes remove`: remove all volumes created by cross. - `volumes prune`: prune all volumes unassociated with a container. - `containers list`: list all active containers created by cross. - `containers remove`: remove all active containers created by cross. - `clean`: clean all temporary data (images, containers, volumes, temp data) created by cross. The initial implementation was done by Marc Schreiber, schrieveslaach. A few more environment variables exist to fine-tune performance, as well as handle private dependencies. - `CROSS_REMOTE_COPY_REGISTRY`: copy the cargo registry - `CROSS_REMOTE_COPY_CACHE`: copy cache directories, including the target directory. Both of these generally lead to substantial performance penalties, but can enable the use of private SSH dependencies. In either case, the use of persistent data volumes is highly recommended. Fixes #248. Fixes #273. Closes #449. Co-authored-by: Alex Huszagh <[email protected]>

This supports the volume-based structure, and uses some nice optimizations to ensure that only the desired toolchain and cargo items are copied over. It also uses drops to ensure scoped deletion of resources, to avoid complex logic ensuring their cleanup. It also supports persistent data volumes, through `cross-util`. In order to setup a persistent data volume, use: ```bash cross-util volumes create ``` This will create a persistent data volume specific for your current toolchain, and will be shared by all targets. This volume is specific for the toolchain version and channel, so a new volume will be created for each toolchain. Make sure you provide your `DOCKER_HOST` or correct engine type to ensure these are being made on the remote host. Then, run your command as before: ```bash CROSS_REMOTE=1 cross build --target arm-unknown-linux-gnueabihf ``` Finally, you can clean up the generated volume using: ```bash cross-util volumes remove ``` A few other utilities are present in `cross-util`: - `volumes list`: list all volumes created by cross. - `volumes remove`: remove all volumes created by cross. - `volumes prune`: prune all volumes unassociated with a container. - `containers list`: list all active containers created by cross. - `containers remove`: remove all active containers created by cross. - `clean`: clean all temporary data (images, containers, volumes, temp data) created by cross. The initial implementation was done by Marc Schreiber, schrieveslaach. A few more environment variables exist to fine-tune performance, as well as handle private dependencies. - `CROSS_REMOTE_COPY_REGISTRY`: copy the cargo registry - `CROSS_REMOTE_COPY_CACHE`: copy cache directories, including the target directory. Both of these generally lead to substantial performance penalties, but can enable the use of private SSH dependencies. In either case, the use of persistent data volumes is highly recommended. Fixes cross-rs#248. Fixes cross-rs#273. Closes cross-rs#449.

schrieveslaach requested review from Dylan-DPC-zz and a team as code owners August 2, 2020 18:52

schrieveslaach force-pushed the docker-remote branch 2 times, most recently from cdd2351 to 7821fac Compare August 3, 2020 07:21

schrieveslaach mentioned this pull request Aug 12, 2020

Recommended GitLab usage #273

Closed

Support Remote Docker tcp://docker:2375

e641dd9

schrieveslaach force-pushed the docker-remote branch from 7821fac to e641dd9 Compare September 30, 2020 19:53

Alexhuszagh added the enhancement label May 29, 2022

Alexhuszagh requested changes Jun 8, 2022

View reviewed changes

Alexhuszagh mentioned this pull request Jun 9, 2022

Simplify config internals. #774

Merged

Alexhuszagh mentioned this pull request Jun 12, 2022

Add comprehensive support for remote docker. #785

Merged

Emilgardis closed this Jun 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Remote Docker tcp://docker:2375 #449

Support Remote Docker tcp://docker:2375 #449

schrieveslaach commented Aug 2, 2020

Maeda-Naoki commented May 25, 2021

Alexhuszagh left a comment •

edited

Loading

Alexhuszagh Jun 8, 2022

Alexhuszagh commented Jun 9, 2022 •

edited

Loading

schrieveslaach commented Jun 9, 2022

Alexhuszagh commented Jun 9, 2022 •

edited

Loading

Alexhuszagh commented Jun 9, 2022 •

edited

Loading

Alexhuszagh commented Jun 10, 2022

schrieveslaach commented Jun 11, 2022

Alexhuszagh commented Jun 12, 2022 •

edited

Loading

schrieveslaach commented Jun 13, 2022

Emilgardis commented Jun 13, 2022

Support Remote Docker tcp://docker:2375 #449

Support Remote Docker tcp://docker:2375 #449

Conversation

schrieveslaach commented Aug 2, 2020

Maeda-Naoki commented May 25, 2021

Alexhuszagh left a comment • edited Loading

Choose a reason for hiding this comment

Alexhuszagh Jun 8, 2022

Choose a reason for hiding this comment

Alexhuszagh commented Jun 9, 2022 • edited Loading

schrieveslaach commented Jun 9, 2022

Alexhuszagh commented Jun 9, 2022 • edited Loading

Alexhuszagh commented Jun 9, 2022 • edited Loading

Alexhuszagh commented Jun 10, 2022

schrieveslaach commented Jun 11, 2022

Alexhuszagh commented Jun 12, 2022 • edited Loading

schrieveslaach commented Jun 13, 2022

Emilgardis commented Jun 13, 2022

Alexhuszagh left a comment •

edited

Loading

Alexhuszagh commented Jun 9, 2022 •

edited

Loading

Alexhuszagh commented Jun 9, 2022 •

edited

Loading

Alexhuszagh commented Jun 9, 2022 •

edited

Loading

Alexhuszagh commented Jun 12, 2022 •

edited

Loading