Docker in docker #387

urso · 2020-03-08T18:20:20Z

When using docker-in-docker, for example in dev or CI containers, one
normally gives access to the host docker environment to the parent container.
If the container spins up another child container, and wishes
to share a directory, one needs to compute the correct paths from
view of the host system.

How it Works

If the environment variable CROSS_DOCKER_IN_DOCKER=true, we call
docker inspect $HOSTNAME, to learn about details of the current container.
The Mounts field contains information about all mounts from the host
into the current container. The GraphDriver field gives us information
about the storage driver and the location of the contains root directory on the host system. Based on these information we compute
a table of all container mounts (source->destination).

When starting a child container we adapt all paths to be mounted, that have
a prefix in the table, to start with source. For example when mounting a
dev containers /usr/local/cargo directory, we will actually mount
/var/lib/docker/overlay2/<parent container id>/merged/usr/local/cargo.

If the project itself is mounted into the parent container, we will not
use the overlay directory, but find the project path on the host system.

Limitations

Finding the mount point for the containers root directory is only
implemented for the overlayfs2 storage driver.

In order to access the parent containers rust setup, the child container
mounts the parents overlayfs. The parent must not be stopped before the
child container, as the overlayfs can not be unmounted correctly by
docker if the child container still accesses it.

src/cli.rs

reitermarkus · 2020-03-12T21:56:51Z

Please also add some tests for the parse functions.

urso · 2020-03-20T14:23:46Z

Please also add some tests for the parse functions.

I didn't really find any unit tests. It seams that all tests are driven by ci/test.sh. Plus I'm not really familiar with your CI setup. Any recommendations on testing?

reitermarkus · 2020-03-21T02:11:27Z

Add them as you would in any other Rust project, so they run when you run cargo test. Then we can add a cargo test before the cargo install in ci/test.sh.

urso · 2020-03-21T21:51:54Z

Added some tests + added cargo test the azure pipeline instead of test.sh.

I don't think there is need to run the unit tests with each target in the pipeline. Would it make sense to enable/use stages to the azure pipeline (needs to be explicitely enabled, as its a feature in preview)?

reitermarkus · 2020-03-23T01:12:04Z

src/cli.rs

+    }
+}
+
+fn parse_bool_env(val: Result<String, env::VarError>) -> bool {


I don't think this needs to be a function, given that it is only used once.

True. Given that there is no dedicated CI test for docker-in-docker, I at least wanted to test that the environment variable is parsed correctly.

@reitermarkus Would you prefer me to remove the test again, or keep it as is?

Let's remove it. It's mainly testing FromStr for bool, which should be tested plenty already.

TotalKrill · 2020-04-01T07:09:00Z

Is there any progress on this? I need it for our CI setup

urso · 2020-04-01T12:09:45Z

@TotalKrill I currently use cross like this in my development container: https://gist.github.com/urso/033a80a10a9b9656548134adeb373e8d#file-dockerfile-L46

README.md

reitermarkus · 2020-04-07T12:44:18Z

src/cli.rs

+    }
+}
+
+fn parse_bool_env(val: Result<String, env::VarError>) -> bool {


Let's remove it. It's mainly testing FromStr for bool, which should be tested plenty already.

reitermarkus · 2020-04-10T16:30:26Z

Thanks, @urso! Could you please squash your last two commits?

When using docker-in-docker, for example in dev or CI containers, one normally gives access to the host docker environment to the docker container. If the container spins up another child container, and wishes to share a directory, one needs to compute the pass the right paths from view of the host system. If the environment variable CROSS_DOCKER_IN_DOCKER=true, we call `docker inspect $HOSTNAME`, to learn about details of the current container. The `Mounts` field contains information about all mounts from the host into the current container. The `GraphDriver` field gives us information about the storage driver and the location in the host system of the current containers root directory. Based on these information we compute a table of all container mounts (`source->destination`). When starting a child container we adapt all paths to be mounted having a prefix in the table, to start with source. For example when mounting a dev containers `/usr/local/cargo` directory, we will actually mount `/var/lib/docker/overlay2/<parent container id>/merged/usr/local/cargo`. If the project itself is mounted into the parent container, we will not use the overlay directory, but find the project path on the host. In order to access the parent containers rust setup, the child container mounts the parents overlayfs. The parent must not be stopped before the child container, as the overlayfs can not be unmounted correctly by docker if the child container still accesses it.

urso · 2020-04-11T16:11:17Z

Commits have been squashed.

reitermarkus · 2020-04-13T00:18:58Z

bors r+

bors · 2020-04-13T01:27:25Z

Build succeeded

rust-embedded.cross

reitermarkus · 2020-04-13T02:16:17Z

Thanks!

romainreignier · 2020-04-14T16:07:24Z

src/docker.rs

+            for details in v {
+                let source = make_path(&details["Source"]);
+                let destination = make_path(&details["Destination"]);
+                if source != destination {


@urso I am not sure to understand why this check is needed?
Actually, I have tried to run latest cross from github with docker-in-docker and if I mount my rust project as -v $PWD:$PWD -w $PWD, source == destination so it is not mounted in the second stage docker and /project is empty.
I might open an issue to discuss it further if you want.
Thank you.

This is just a 'translation' table replacing the prefix that matches 'destination' with the prefix that matches 'source'. If source == destination, then this will be a noop.

If you open a shell in your development/CI container you can check the environment via $ echo $CROSS_DOCKER_IN_DOCKER; docker inspect $HOSTNAME. Running cross <command> -v will also print the docker commands as they will be executed. Very helpful for debugging.

Thanks, I did some debugging yesterday and by removing this check in my fork allowed me to use cross in a docker ran by Jenkins.

The issue here is that Jenkins mount the current workspace with the same path in Docker with

-v /path/to/my/workspace:/path/to/my/workspace

So source == destination but it is not a noop, it allows to mount files from host to the second docker stage.

I have managed to reproduce with:

# On host, create a file in current directory touch testfile # start a first docker stage docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:$PWD -w $PWD debian:buster bash # in the first docker stage, install docker apt update && apt install -y --no-install-recommends docker.io # check that the test file is there ls # it is here, OK # get the merged path merge_path=$(docker inspect `hostname` --format '{{.GraphDriver.Data.MergedDir}}') # start the second docker stage docker run -it --rm -v $merge_path/$PWD:/project debian:buster bash # in the second docker stage, check that the test file is here ls /project # there is no file, NOK # exit the second docker exit # new start with classic mount docker run -it --rm -v $PWD:/project debian:buster bash ls /project # the testfile is here, OK

Oh, I see. If source == destination for mounted dirs, then the replacement will swap out the root path / with the containers merge path. Thanks to default mounting rules the /project will be empty (mounts are not passed through by default). This is why we need the path translation rules in the first place.

You are right, we should remove the if condition and always add the mount infos. I'm no maintainer, nor regular contributor to cross, but would be nice if you can create a PR removing this check.

Ok, thanks for having a look to it. I will create a PR.

409: Allow to mount host path in docker-in-docker mode r=reitermarkus a=romainreignier As discussed here: #387 (comment) The check is not needed to allow to mount paths that are mounted in the first docker with the same path. Co-authored-by: Romain Reignier <[email protected]>

urso requested review from Dylan-DPC-zz and a team as code owners March 8, 2020 18:20

urso mentioned this pull request Mar 8, 2020

Build fails with sh: 1: cargo: not found #260

Closed

7 tasks

reitermarkus reviewed Mar 12, 2020

View reviewed changes

src/cli.rs Outdated Show resolved Hide resolved

urso force-pushed the docker-in-docker branch from bd85e47 to 7c3da09 Compare March 21, 2020 21:56

reitermarkus reviewed Mar 23, 2020

View reviewed changes

reitermarkus reviewed Apr 7, 2020

View reviewed changes

urso requested a review from reitermarkus April 10, 2020 12:23

urso force-pushed the docker-in-docker branch from 4102930 to f2c70b3 Compare April 11, 2020 16:10

reitermarkus approved these changes Apr 13, 2020

View reviewed changes

bors bot merged commit 8e8615b into cross-rs:master Apr 13, 2020

romainreignier reviewed Apr 14, 2020

View reviewed changes

romainreignier mentioned this pull request Apr 16, 2020

Allow to mount host path in docker-in-docker mode #409

Merged

bytedream mentioned this pull request Mar 18, 2022

Add dockerfile for cross itself #667

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker in docker #387

Docker in docker #387

urso commented Mar 8, 2020 •

edited

Loading

reitermarkus commented Mar 12, 2020

urso commented Mar 20, 2020

reitermarkus commented Mar 21, 2020

urso commented Mar 21, 2020

reitermarkus Mar 23, 2020

urso Mar 23, 2020

urso Apr 1, 2020

reitermarkus Apr 7, 2020

urso Apr 10, 2020

TotalKrill commented Apr 1, 2020

urso commented Apr 1, 2020

reitermarkus Apr 7, 2020

reitermarkus commented Apr 10, 2020

urso commented Apr 11, 2020

reitermarkus commented Apr 13, 2020

bors bot commented Apr 13, 2020

reitermarkus commented Apr 13, 2020

romainreignier Apr 14, 2020

urso Apr 15, 2020

romainreignier Apr 15, 2020

urso Apr 16, 2020

romainreignier Apr 16, 2020

Docker in docker #387

Docker in docker #387

Conversation

urso commented Mar 8, 2020 • edited Loading

How it Works

Limitations

reitermarkus commented Mar 12, 2020

urso commented Mar 20, 2020

reitermarkus commented Mar 21, 2020

urso commented Mar 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TotalKrill commented Apr 1, 2020

urso commented Apr 1, 2020

Choose a reason for hiding this comment

reitermarkus commented Apr 10, 2020

urso commented Apr 11, 2020

reitermarkus commented Apr 13, 2020

bors bot commented Apr 13, 2020

Build succeeded

reitermarkus commented Apr 13, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

urso commented Mar 8, 2020 •

edited

Loading