Future of community.docker

[felixfontein]

community.docker has been using the Docker SDK for Python ("docker-py") for almost everything (directly for most content, and indirectly for the docker_compose module; only the docker connection plugin and the docker_stack modules use the CLI tool).

Unfortunately, it looks like he Docker SDK for Python is pretty much dead (PR for adding unmaintained notice, activity since October 2021). Maybe a fork will come up, maybe they'll allow some community folks to take the project over under its current repository and PyPi package name, and maybe it will just die (as the Python docker-compose did).

Anyway, this is a moment to think about what we could and/or should do. Having a fork or the original project continued would obviously be the best solutions ;-) With a fork we might have to adjust the import statements (resp. allow to import both original docker-py and the fork), but (if they don't do something drastic) most things can just stay as they are. In case the project is dead and noone wants to maintain a fork, we do have a problem.

We have four different groups of contents:

1. Content that uses the Docker CLI program, or something else:

   • docker connection plugin
   • nsenter connection plugin (uses the nsenter CLI tool)
   • docker_machine inventory plugin (uses the docker-machine CLI tool)
   • current_container_facts module (inspects files)
   • docker_stack* modules

2. Docker content that uses the API part of the Docker SDK for Python (i.e. not the high-level functionality)

   • docker_api connection plugin
   • docker_containers inventory plugin
   • docker_swarm inventory plugin
   • docker_config module
   • docker_container_exec module
   • docker_container_info module
   • docker_container module
   • docker_host_info module
   • docker_image_info module
   • docker_image_load module
   • docker_image module
   • docker_login module
   • docker_network_info module
   • docker_network module
   • docker_node_info module
   • docker_node module
   • docker_prune module
   • docker_secret module
   • docker_swarm_info module
   • docker_swarm module
   • docker_swarm_service_info module
   • docker_swarm_service module
   • docker_volume_info module
   • docker_volume module

3. Docker content that uses the high-level functionality of the Docker SDK for Python

   • docker_plugin module

4. Docker content that uses the Python version of docker-compose

   • docker_compose module

[felixfontein]

I'm interested in at least keeping (most of) the Docker "plain" modules / plugins alive, probably by allowing them to work without the Docker SDK for Python. (I still don't use Swarm or Docker Compose, and I'm kind of glad especially about the latter.)

This will be a not be something easy, but I think it's worth it - also since it would allow to run the modules / plugins without installing anything.

[markuman]

Anyway, this is a moment to think about what we could and/or should do.

Another possibility is not to use (fork & maintain) the sdk but directly use the docker engine api* (what the sdk is also doing imo).

m@rocket:~$ docker run -d --rm alpine:3.15 tail -f /dev/null
WARNING: IPv4 forwarding is disabled. Networking will not work.
bc26fd52fb92e8d656eafc415d9866f016f1a383e6f7830d66692d4c36db16f0

m@rocket:~$ curl --silent -XGET --unix-socket /run/docker.sock http://localhost/containers/json | jq
[
  {
    "Id": "bc26fd52fb92e8d656eafc415d9866f016f1a383e6f7830d66692d4c36db16f0",
    "Names": [
      "/keen_chatterjee"
    ],
    "Image": "alpine:3.15",
    "ImageID": "sha256:0ac33e5f5afa79e084075e8698a22d574816eea8d7b7d480586835657c3e1c8b",
    "Command": "tail -f /dev/null",
    "Created": 1653381812,
    "Ports": [],
    "Labels": {},
    "State": "running",
    "Status": "Up 3 seconds",
    "HostConfig": {
      "NetworkMode": "default"
    },
    "NetworkSettings": {
      "Networks": {
        "bridge": {
          "IPAMConfig": null,
          "Links": null,
          "Aliases": null,
          "NetworkID": "b0e362e764dd2e8ed8452ab738cc6830b2002da9435396e746c2edb8d34e7b32",
          "EndpointID": "c922ea8969b376c1de6d15b509ac599cd7f8ed814c08998f00b9f8efaf7ed904",
          "Gateway": "172.17.0.1",
          "IPAddress": "172.17.0.2",
          "IPPrefixLen": 16,
          "IPv6Gateway": "",
          "GlobalIPv6Address": "",
          "GlobalIPv6PrefixLen": 0,
          "MacAddress": "02:42:ac:11:00:02",
          "DriverOpts": null
        }
      }
    },
    "Mounts": []
  }
]
m@rocket:~$ 

---

* https://docs.docker.com/engine/api/v1.41/

[felixfontein]

@markuman that's what I mean with "work without the Docker SDK for Python" :)

[ssbarnea]

Directly related to docker/compose#9510 - please comment there, maybe we can get few more months to perform a migration out of docker_compose if we can fix the too restrictive pinning of dependencies.

[ju2wheels]

If its going to be refactored completely, I have some bugs that I have yet to file on some of the commands where they are doing more than is necessary to complete the requested task based on the module arguments provided and in some cases that leads to failure where it really shouldnt.

I have to find my old tickets for the manual process on how to reproduce the issue, but basically one example I think was the docker_image task is unable to remove an image if you have overlayfs corruption issues (despite the docker-cli tools being able to do it perfectly fine) because it would collect info from the overlayfs that it didnt really need before actually using the API call to remove the image. I basically had to create a custom module that did nothing but call the API to remove the image to work around issues like this.

For us this corruption usually happens in our in field IOT devices while dowloading images that get interrupted due to bad network connectivity, so the corruption itself is a Docker side issue but the fact that we cant recover using the Ansible modules while we can recover with the docker-cli is a module implementation issue.

Food for thought.

[felixfontein]

@ju2wheels please file the bugs in new issues. This is not really related to this discussion.

[WojciechowskiPiotr]

I think we need to wait and see what the status of the Docker SDK will be. After all, it is maintained by the Docker team. Alternatives are to work directly with Docker API or use other SDK (probably not possible except for current SDK forks).

[ssbarnea]

That "need to wait"... until forever? ;) I doubt they are going to do anything TBH. Best bet would be to fork old compose and publish that fix under a different package name. After this work to migrate to use of cli for compose can be started.

[felixfontein]

I don't think anyone of us is interested in maintaining a fork of either Docker SDK for Python or Docker Compose.

[felixfontein]

One thing that could be interesting is vendoring the Docker SDK for Python into the collection, or at least parts of it. (That would also allow us to update it independently, and we could kill (some of) the code that handles different versions of it.) I've looked at how to implement some things yesterday and today, and especialy all the code that handles talking to the daemon (via Unix sockets, IP sockets, TLS, SSH, ...) will be painful to reimplement, while Docker SDK for Python already has pretty mature code for this. I created ansible-community/community-topics#109 to discuss the licensing issues for this.

[InCogNiTo124]

What about creating commands by hand and invoking them through os.system()? There is not much user input to validate and sanitize so there is an option of constructing a command that would be ran.

Example:

community.docker.docker_compose:
  build: true
  nocache: true
  up: true

would become something along the lines of os.system("docker compose build --no-cache && docker compose up -d")

[felixfontein]

For docker_compose that's likely what needs to be done, though no idea how easy/hard that is to do, and no idea how check mode can be done this way.

For other modules, I would really, really avoid that.

[felixfontein]

One thing that could be interesting is vendoring the Docker SDK for Python into the collection, or at least parts of it. (That would also allow us to update it independently, and we could kill (some of) the code that handles different versions of it.) I've looked at how to implement some things yesterday and today, and especialy all the code that handles talking to the daemon (via Unix sockets, IP sockets, TLS, SSH, ...) will be painful to reimplement, while Docker SDK for Python already has pretty mature code for this. I created ansible-community/community-topics#109 to discuss the licensing issues for this.

Turns out the licensing discussion was already resolved ... one year ago. Damn, I completely forgot :) So including Apache 2.0 licensed content is totally fine as long as it is in module_utils (or complete modules), and you avoid importing GPL licensed code in it.

Simply vendoring the Docker SDK for Python won't work well since it is Python 3.6+ by now, while this collection stlil supports Python down to 2.6. (Even though you'll have to use a very old Docker SDK for Python for 2.6 support...) But we can extract the basic connection handling and util code from Docker SDK for Python and do not have to reimplement that. (And we can mix code from several versions of it, to get something that supports 2.6+.) I'll probably look into that soon...

[ssbarnea]

@felixfontein Thanks for helping with this, especially as every day passing is hurting molecule-docker users. IMHO, i would just drop support for old python and make a new release with vendored version as soon as possible.

If anyone wants to fix problems related to py2, they are welcome to spend some time doing so. Based on what I seen so far, there are not many around willing to help.

I will be quite happy to bring back the plugin to the list of supported plugins. Replacing the docker_compose module with command line calls can come later, when the pressure is smaller.

[aehlke]

I got bit really nastily by adopting this without knowing better, while still learning the Docker ecosystem. Would be great to put up big disclaimers about these being outdated/unmaintained (for now) in the documentation.

[felixfontein]

I finished a PR that vendors some code from Docker SDK for Python (adjusted to work with Python 2.7 and 3.5 as well), and adjusted all plain docker modules (except docker_container and docker_plugin) to use that code instead of relying on the Docker SDK for Python: #387 Reviews/testing/feedback are welcome. I'm also planning to adjust docker_container, but that is going to be more complex. I'll also work on the docker_api connection and the docker_containers inventory plugin. If someone else wants to work on docker_plugin or (some of) the Docker Swarm modules and inventory plugin, feel free to create follow-up PRs.

My general idea would be:

1. Release one more minor community.docker release with Python 2.6 deprecation (Deprecate Python 2.6 #388) and some licensing adjustments ([WIP] Move PSF-license.txt to licenses/ #386 / Draft: Start adding SPDX-License-Identifier comments #393 / whatever it will be).
2. Then bump main branch to the next major version, remove deprecations, drop support for Python 2.6, merge [WILL BE SPLIT UP] Start adjusting some modules #387, and merge/work on follow-up PRs.
3. Once everyone is happy with the results, release 3.0.0 (maybe first one or more beta versions to gather more feedback).

Any opinions / comments / feedback?

[gotmax23]

+1 to deprecate Python 2.6 support. ansible-core 2.13 has already completely removed support for Python 2.6 targets. As the package maintainer for ansible-collection-community-docker in Fedora, I would appreciate at least some support for the stable-2 branch for a little while longer.

[felixfontein]

stable-2 will pretty surely get bugfixes for some time (easily backportable things will get backported anyway, more complicated things... depends on whether someone is willing to backport it manually - I guess it depends a lot on what exactly it is). I think so far we backported every bugfix to stable-1, so I would assume this continues to happen if possible.

[felixfontein]

CC @Ajpantuso @dariko @DBendit @hannseman @imjoseangel @jgoldschrafe @jwitko @morph027 @porshkevich @ushuz @WojciechowskiPiotr @ximon18 as you have contributed new modules/plugins or stepped up as maintainers in the past, and might have so far missed this discussion. (If you're no longer interested in this collection, sorry for bothering you, simply ignore this discussion / unsubscribe from it :) )

[imjoseangel]

Thanks for sharing @felixfontein!

Didn't know about this move.

I understand from the #387 that we have moved to our own API client and daemon maintenance and we are changing all the modules to work with these.

To avoid overlapping on changes, where do you need help?

[felixfontein]

@imjoseangel thanks for your message! What would be helpful:

1. Comments on the general situation

2. Reviews and testing of (parts of) [WILL BE SPLIT UP] Start adjusting some modules #387

3. If you're interested in the docker_compose module, it probably needs a complete rewrite to handle docker-compose >= 2.0.0. At the same time it needs to keep backwards compatibility with 1.x.y, so this is going to be a hard one. Maybe simply creating a new module for docker-compose >= 2.0.0 (say docker_compose2) would be better?

4. If someone wants to convert the docker_plugin module, that one is some more work to do

5. Then there are all the Docker Swarm modules:

   • docker_config module
   • docker_node_info module
   • docker_node module
   • docker_secret module
   • docker_swarm_info module
   • docker_swarm module
   • docker_swarm_service_info module
   • docker_swarm_service module

   I guess it would be great if these also would be converted (docker_swarm_service will probably be the hardest one though; the rest should be relatively easy). I don't use Swarm and don't plan on ever using it, so I won't work on these. If anyone is interested in converting them, feel free to work on them (either now based on my branch, or wait until my PR is merged and then work on them).

Fortunately all the other modules and plugins (especially the Docker Stack modules, that you @imjoseangel worked on so far) don't need any work, since they "only" work with the Docker CLI tool (or other CLI tools) that aren't affected by any of this.

[felixfontein]

A 3.0.0-a1 prerelease has been released with the current state of the main branch.

[felixfontein]

I think my rewrite of docker_container is ready (#422 for anyone who is interested). Is anyone interested in rewriting docker_plugin, or any of the Swarm modules? Or wants to work on docker_compose?

[ssbarnea]

@felixfontein How am I supposed to install the pre-release of the collection, the command I was assuming to be working was ansible-galaxy collection install 'community.docker:>=3.0.0-a1' and apparently it does not.

[ssbarnea]

I found it, I need to add --pre, and that is from inside ansible-compat. The bug is in ansible-galaxy cli which is not able to infer the fact that the range requires a pre-release, like pip does.

That prevents me from using a pre-release inside molecule-docker...

[felixfontein]

I've created another WIP PR #429 to adjust docker_plugin. Then all non-Swarm plugins and modules (that use the Docker SDK for Python) have been converted.

[felixfontein]

docker_container and docker_plugin have been converted, and docker_container has a few new features that weren't possible with Docker SDK for Python. I've created a new prerelease 3.0.0-a2 with all latest changes. Please test it thoroughly so we can weed out as many potential bugs before the 3.0.0 release as possible :)

[ssbarnea]

I plan to fix molecule-docker plugin over this weekend, so that should give a good testing ground for the new version.

[felixfontein]

I've ceated a 3.0.0-rc1 release with one bugfix (the only bug reported so far). If there are no reports in, say, ~two weeks, I'll create the final 3.0.0 release.

[felixfontein]

Looks like Docker SDK for Python is back to being actively maintained, judging from the list of commits to main in the last three days (one full page): https://github.com/docker/docker-py/commits/main

[felixfontein]

I created #435 to keep track of changes to Docker SDK for Python that might need to be cherry-picked to our vendored code.

[felixfontein]

I created a 3.0.0-rc2 release chandidate including the changes from Docker SDK for Python. If there are no further bug reports, 3.0.0 will be released in 1-2 weeks.

[gotmax23]

https://github.com/gabrieldemarmiesse/python-on-whales (python-on-whales on PyPI) provides a Python interface to compose v2 (the `docker compose` cli plugin), amongst other things. This is a community project, but it was featured on the Docker blog last year https://www.docker.com/blog/guest-post-calling-the-docker-cli-from-python-with-python-on-whales/. It shells out to the docker binary, but that's really the only option with the current `docker compose` interface.

[felixfontein]

I've seen that library before. There are two problems with it:

1. It's yet another library that users would have to install to use the module;
2. It requires Python 3.7+ (https://github.com/gabrieldemarmiesse/python-on-whales/blob/master/setup.py#L22).

So if the docker_compose module would start using it to talk to docker-compose 2.0+, it would still need to be able to support docker-compose < 2.0.0 without this library installed, and on Python 2.7, 3.5, and 3.6.

Alternatively, maybe it might be interesting to create a new module docker_compose2 which only talks to docker-compose 2.0+. Such a new module could have different requirements than the old module, and it wouldn't need to support both docker-compose < 2.0 and 2.0+ at the same time.

On the other hand, I'm not sure how useful the python-on-whales library actually is for this task, since composing a command line and calling an external CLI program isn't something that hard to do manually / with module.run_command(). But I guess that's something that someone has to decide who actually wants to work on docker-compose support. I'm not using it at all, so I probably won't work on it.

[ssbarnea]

Give what happened during the last years, I would say that probably the best bet is to just call "docker compose" cli. I am not concerned so much about backwards compatibility because that would be insanely hard to achieve and maintain, likely impossible in some cases too. If anyone need to use old dockers, they should stick with old libraries/modules/collections. Extra libraries in-between would only add another thing to maintain.

Usually I am in favour of using APIs, but when it comes to docker, it seems that only stable api is "cli".

I find it funny that it took docker team years to put some fixed to old docker python library, and that happens only after we decided to go this way.

[maica1]

It seens like the last days had a lot of news, first we have the python-on-whales project that uses the CLI tool under the hood.
And some activity on the docker-py repo, with new releases and the closing of the Add unmaintained notice issue
So maybe we can see some progress on the community.

[felixfontein]

I'm currently planning to do the 3.0.0 release this weekend, assuming nobody wants to veto that :)

[felixfontein]

I've just released 3.0.0 🎉 I'm closing this issue for now. If someone wants to discuss some aspects we haven't discussed / completed yet, please create a new issue for this. Thanks everyone!