finishing up v2.4 docker to singularity conversion #27
Conversation
|
I am a bit busy right now - please give me a few days to thoroughly test
this PR.
Best,
Chris
…On Wed, Oct 18, 2017 at 6:54 PM, Vanessa Sochat ***@***.***> wrote:
hey @singularityware/singularity-admin
<https://github.com/orgs/singularityware/teams/singularity-admin>
@chrisfilo <https://github.com/chrisfilo> here is a first shot at
docker2singularity conversion for 2.4. We do the following:
1. use an empty build sandbox to construct the image from docker export
2. use the folder to build a squashfs image
I had some trouble with the check for busybox to determine if -executable
should be used or not. For an ubuntu test image, it triggered the case that
failed. If it's the case that not using executable works for both busybox
and other flavors, I just removed that step. But I'm not familiar with the
addition.
Chris - squashfs can be converted back to a writable thing, if needed. I
will leave it up for open discussion to decide how we should maintain the
branches (e.g., there are many benefits to squashfs that would make it an
appealing default). Take a look at the changes and let me know your
thoughts! I'm likely closing up shop soon, I think it's 10pm here, and I
haven't gone home yet... oups :P
------------------------------
You can view, comment on, or merge this pull request online at:
#27
Commit Summary
- finishing up v2.4 docker to singularity conversion
File Changes
- *M* Dockerfile
<https://github.com/singularityware/docker2singularity/pull/27/files#diff-0>
(10)
- *M* README.md
<https://github.com/singularityware/docker2singularity/pull/27/files#diff-1>
(12)
- *M* docker2singularity.sh
<https://github.com/singularityware/docker2singularity/pull/27/files#diff-2>
(81)
Patch Links:
- https://github.com/singularityware/docker2singularity/pull/27.patch
- https://github.com/singularityware/docker2singularity/pull/27.diff
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#27>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAOkp-xCJ2lW8KGMkXl6aiwe4VWtJ93Jks5stqvMgaJpZM4P-mAe>
.
|
|
No worries! Please take the time that you need. Happy Friday! 🍨 |
|
I tested it on the following images:
Conversion did not error for any of them, but the With old converter (using .img) with new converter |
|
Great news! I'll test out that last one this weekend. Thanks for testing these out, I'll keep you updated if I figure out the last one. |
|
Great!
With regards to image format I would at least provide an option for people
to chose between .img and .simg. Not all systems have updated singularity.
Best,
Chris
…On Fri, Oct 27, 2017 at 4:10 PM, Vanessa Sochat ***@***.***> wrote:
Great news! I'll test out that last one this weekend. Thanks for testing
these out, I'll keep you updated if I figure out the last one.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#27 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAOkp6VU0R3i2Oln5jnZJBN3AdBLoscnks5swmL6gaJpZM4P-mAe>
.
|
|
Good idea. I'll have the default build a folder (sandbox) to simg, and then the user can choose to either keep the folder (for development) OR to produce instead an ext3 image. I will think more about the specific commands and include them in my notes after testing. |
|
hey @chrisfilo ! A quick answer for you. The reason for the missed import is that and we can see yaml is where it should be, with miniconda and this all stems from the fact that for the new |
|
Okay, you can check out the updated README to see the usage, and I'll show examples here! First, some general comments:
The user has the option to build:
Build into Sandbox FolderI'll start with a folder, because we can easily peek inside to see some changes. By specifying let's go look at it... Note that it runs without error. Also note that it's a folder, and we can see that this updated docker2singularity has a proper metadata folder (.singularity.d) instead of having /environment we mirror the practice of docker import and add it to 10-docker.sh and the runscript Build SquashfsNow let's look at the default, which is to build a squashfs image: Look at the difference in size!! 3689MB down to 1338MB and more robust! Woot! Let's test it out... Legacy FormatIf the user wants the legacy format, that is available with and test... This image format is less ideal because it's larger, more prone to corruption over time, and likely to not have future support. ConversionImportantly, any format can be converted to another format easily, for example, let's convert the ext3 image to squashfs: and that would work for the sandbox folder too. Still not recommended, because if there were changes not recorded, this is not reproducible. |
|
huh, I wonder if I can get the labels from the image too? Let me try that. |
README.md
Outdated
| ubuntu:14.04 | ||
| ``` | ||
|
|
||
| Replace `D:\host\path\where\to\output\singularity\image` with a path on the host filesystem where your Singularity image will be created. Replace `ubuntu:14.04` with the docker image name you wish to convert (it will be pulled from Docker Hub if it does not exist on your host system). | ||
|
|
||
| `docker2singularity` uses the Docker daemon located on the host system. It will access the Docker image cache from the host system avoiding having to redownload images that are already present locally. | ||
|
|
||
| ## Build a Squashfs Image |
There was a problem hiding this comment.
Could you mention here what is the minimal version of Singularity required to run Squashfs images?
There was a problem hiding this comment.
I can find when support was added in the source code, but I don't recall if there were any issues. If they were they would have been fixed later. I will cite when support was added and if users have issues I will help them.
README.md
Outdated
| @@ -16,14 +58,66 @@ docker run \ | |||
| -v /var/run/docker.sock:/var/run/docker.sock \ | |||
| -v D:\host\path\where\to\output\singularity\image:/output \ | |||
| --privileged -t --rm \ | |||
| singularityware/docker2singularity \ | |||
| singularityware/docker2singularity:2.4 \ | |||
There was a problem hiding this comment.
I used tags so far to denote the version of docker client inside the container (see below - the client version needed to be compatible with the host version). I don't think we need to add tags here for singularity version. After this is merged I will make sure all branches will get updated.
|
To keep a record in our discussion, here is how the modified generation looks, showing an added step to generate metadata labels and now the container can be inspected in various ways, default shows labels: and then with different args ( |
|
ping! |
|
Sorry for the late reply. I'm running into the following problem: This happens with |
|
Thanks @chrisfilo ! A few quick questions to help debug, since I don't have windows.
Did these all have the same error as you reported above?
Do you get the same error and failure on a linux host? I think this might be related to the filesystem or overlay fs, see these issues: We also might want to consider the version of Docker running on the host: We are having a release of 2.4.1 soon, and we might have a solution for the whiteout files, although not sure yet. Huge thanks for helping with the testing, the most important/relevant environment is Windows, and I don't think I'd know where to find it if I tried. And again please take the time you need to properly test! |
|
Out of the list this is the only image I get this error for. I did not have a chance to try it on Linux. What makes you think this is a Windows related issue? Have you tried to replicate it on your machine? The two issue you linked to were reported for Linux hosts. As for .wh files I tried this but I run into another error: |
|
Is there any update on this? |
|
I think these bugs were present in earlier versions of 2.4 and may be resolved now (but in development branch) I think the best thing to do is to keep it in a branch for now (and @0xaf1f you should clone the branch and build the image freshly) to do your conversions (with the development version of singularity). When the next release goes out, we will make a corresponding branch (and only merge to be the "default" given no errors). It is my preference to only create the branches with official releases of some flavor, so having 2.4 here represent some stage of development doesn't seem ideal. |
|
hey @0xaf1f I just threw together a docker2singularity that has the updated 2.4 (development) installed if you want to wait for it to build, and then try. --> https://hub.docker.com/r/vanessa/singularity/builds/ and probably faster to build on your own --> https://github.com/singularityhub/docker/blob/master/docker2singularity/Dockerfile.dev I'm going to be afk a bit but will be back if you have questions or want to work together, I didn't put much time into writing docs or testing.... :) |
|
I really appreciate it, @vsoch. Though I went on the route of creating a local registry and pulling from that, and it looks like it produced a good image. |
|
hey, that works too! :) |
|
Version 2.5 is released, so this PR is being closed as it wouldn't be ideal to merge into master. For those wanting docker2singularity for version 2.4, use the 2.4 branch: https://github.com/singularityware/docker2singularity/tree/v2.4 I will prepare a v2.5 branch soon. |
hey @singularityware/singularity-admin @chrisfilo here is a first shot at docker2singularity conversion for 2.4. We do the following:
I had some trouble with the check for busybox to determine if
-executableshould be used or not. For an ubuntu test image, it triggered the case that failed. If it's the case that not using executable works for both busybox and other flavors, I just removed that step. But I'm not familiar with the addition.Chris - squashfs can be converted back to a writable thing, if needed. I will leave it up for open discussion to decide how we should maintain the branches (e.g., there are many benefits to squashfs that would make it an appealing default). Take a look at the changes and let me know your thoughts! I'm likely closing up shop soon, I think it's 10pm here, and I haven't gone home yet... oups :P