community.docker.docker_compose does not work on debian 12 bookworm #672

lukaskaplan · 2023-07-21T10:08:57Z

SUMMARY

Community.docker.docker_compose does not work on debian 12 bookworm because of missing python module docker-compose.

If you try to install it by pip pip3 install docker-compose, it will fail because of adoption of PEP668.

I tried to install it via pipx, it will fail too. So I tried to install it in python venv, it fails too.

What is best practice to run this task on Debian 12?

ISSUE TYPE

Bug Report

COMPONENT NAME

community.docker.docker_compose

ANSIBLE VERSION

ansible [core 2.12.10]
  config file = /home/***/***/***/***/***/ansible.cfg
  configured module search path = ['/home/***/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/***/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110]
  jinja version = 2.11.3
  libyaml = True

COLLECTION VERSION

none

CONFIGURATION

none

OS / ENVIRONMENT

Debian 12 bookworm

STEPS TO REPRODUCE

Try to make docker-compose up with any compose file.

- name: "docker-compose up"
  community.docker.docker_compose:
    project_src: /path/to/any/docker-compose.yml

EXPECTED RESULTS

Expected behavior is docker-compose up -d with given compose file.

ACTUAL RESULTS

It will fail because of missing python module docker-compose. But it is (at this time) not possible to install it on Debian 12.

fatal: [my-host]: FAILED! => {"changed": false, "msg": "Unable to load docker-compose. Try `pip install docker-compose`. Error: Traceback (most recent call last):\n  File \"/tmp/ansible_community.docker.docker_compose_payload_cmcqwicq/ansible_community.docker.docker_compose_payload.zip/ansible_collections/community/docker/plugins/modules/docker_compose.py\", line 497, in <module>\nModuleNotFoundError: No module named 'compose'\n"}

The text was updated successfully, but these errors were encountered:

shantanoo-desai · 2023-07-21T12:00:55Z

Pretty sure Debian 12 won't have Docker Compose v1 since it is already deprecated. See #216

felixfontein · 2023-07-22T06:31:48Z

Why does installing in a venv not work?

Tardo · 2023-07-24T08:57:51Z

docker-compose depends on PyYAML<6 ... PyYAML fails to build.

pip seemed to fail to build package:
    PyYAML<6,>=3.10

Some possibly relevant errors from pip install:
    error: subprocess-exited-with-error
    AttributeError: cython_sources

Error installing docker-compose.

felixfontein · 2023-07-25T07:57:16Z

Force PyYAML==5.3.1 in the venv, that version should build.

lukaskaplan · 2023-07-25T12:15:02Z

Ok, understand. So, should I create venv for ansible on remote hosts? And then change ansible_python_interpereter for it?

felixfontein · 2023-07-26T06:08:49Z

You can create and use a venv for all docker_compose tasks only, whatever you prefer.

lukaskaplan · 2023-07-26T06:55:39Z

I did it this way now (fast and easy workaround):

- name: "docker-compose up"
  docker_compose:
    project_src: "{{ dir }}"
  when: ansible_distribution_major_version | int < 12

- name: docker-compose up
  shell:
    cmd: "docker-compose up -d"
    chdir: "{{ dir }}"
  when: ansible_distribution_major_version | int > 11

I am not sure, how to use venv for docker_compose task only.

felixfontein · 2023-07-26T07:33:20Z

You should be able to set ansible_python_interpereter in vars: for a task. If you point it to the Python script of the venv, it should use the venv.

N0zz · 2023-08-04T06:44:22Z

You should be able to set ansible_python_interpereter in vars: for a task. If you point it to the Python script of the venv, it should use the venv.

It doesn't help when I use default interpreter, install docker-compose via ansible pip module, and then try to run docker-compose via community.docker.docker_compose module. I would have to run first task with one interpreter, than change interpreter, then run second task with new interpreter. But you can't change interpreter on the run. You would have to run playbook two times separately.

Solution is simple. community.docker.docker_compose should use docker compose plugin instead of deprecated standalone docker-compose version.

felixfontein · 2023-08-04T06:59:31Z

But you can't change interpreter on the run.

Yes, you can. You can define the interpreter to use per task.

Solution is simple. community.docker.docker_compose should use docker compose plugin instead of deprecated standalone docker-compose version.

#216.

exhuma · 2023-08-07T07:13:13Z

I have run into the same issue today when I needed to re-create my ansible venv on a new machine. I have updated their open issue regarding this (See docker/compose#9510). They make it clear that they don't intend to fix this as they consider it EOL.

I just started writing all the ansible recipes using the community module and am a bit miffed that I need to rewrite everything now.

It would be helpful if the ansible documentation would add a "deprectaion" note. And provide or suggest an alternative.

I actually don't know what's the best alternative right now. Should I write compose-files, upload them to the remote and then run docker-compose as a shell command?

It would be nice if this would be abstracted away by ansible.

felixfontein · 2023-08-07T07:17:51Z

The documentation says "This module requires docker-compose < 2.0.0.". The module is not deprecated as there is no intention of removing it, it works totally fine with docker-compose v1.

I actually don't know what's the best alternative right now. Should I write compose-files, upload them to the remote and then run docker-compose as a shell command?

It would be nice if this would be abstracted away by ansible.

A module for docker compose v2 would solve this, but while there have been two PRs (#540, #586), none of them got completed. So contributions are still welcome.

lukaskaplan · 2023-08-07T22:18:29Z

I actually don't know what's the best alternative right now. Should I write compose-files, upload them to the remote and then run docker-compose as a shell command?

I do it this way for debian 12 hosts. Because there is not possible to install docker-compose python plugin by pip, pipx, apt.

Venv is too complicated workaround for me. I like straight, "clean" and easy to understand solutions.

exhuma · 2023-08-08T05:50:47Z

I have started to do this now as well as I ran into the same pyyaml issue when using ansible-lint. ansible-lint has pyyaml > 5.4.1 as dependency so as soon as that is included in the ansible install it breaks docker-compose. ansible-lint can be installed separately (either via pre-commit or pipx for example). But considering that a) docker-compose is deprecated and b) I have run into the same "pyyaml" issue 3 times in the last 2 days I decided to drop the ansible community.docker package and do the steps manually.

soakes · 2023-10-15T09:18:46Z

Force PyYAML==5.3.1 in the venv, that version should build.

Thank you so much @felixfontein for pointing this out. I have spent the past several hours trying to get docker-compose to compile with pip3 on Debian 12 and was having no success until I forced this exact PyYAML version which you mentioned.

Thanks again.

theko2fi · 2023-10-15T16:22:14Z

I just want to point out that I face the same issue on fedora 38

When I try to install docker-compose module with pip3 install docker-compose command, the error below occurs:

  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [68 lines of output]
      /tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!
      
              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.
      
              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
      
              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************
      
      !!
        parsed = self.parsers.get(option_name, lambda x: x)(value)
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/usr/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/usr/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 318, in run
          self.find_sources()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 326, in find_sources
          mm.run()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 548, in run
          self.add_defaults()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 586, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 113, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

soakes · 2023-10-15T16:27:10Z

I just want to point out that I face the same issue on fedora 38

When I try to install docker-compose module with pip3 install docker-compose command, the error below occurs:

  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [68 lines of output]
      /tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!
      
              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.
      
              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
      
              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************
      
      !!
        parsed = self.parsers.get(option_name, lambda x: x)(value)
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/usr/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/usr/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 318, in run
          self.find_sources()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 326, in find_sources
          mm.run()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 548, in run
          self.add_defaults()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 586, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 113, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-g2pvfmnd/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

If you install docker-compose via pip3 using virtualenv, this should resolve your issue.

- name: Install docker python modules
  pip:
    name:
      - PyYAML==5.3.1
      - docker>=1.8.0
      - docker-compose>==1.24.0
    virtualenv: "{{ docker_virtualenv_path }}"

The key/magic here is PyYAML==5.3.1 as without this, it wont compile sucessfully at least on Debian 12 and likely same as your Fedora.

theko2fi · 2023-10-15T17:45:22Z

@soakes Thanks for your help, I will give it a try

felixfontein added question Further information is requested docker-compose-v1 Docker Compose v1 labels Jul 22, 2023

felixfontein mentioned this issue Aug 7, 2023

Remove dependency timebomb from old v1 docker-compose package docker/compose#9510

Closed

soakes mentioned this issue Oct 15, 2023

docker-compose pip package installation fails on Debian 12. geerlingguy/ansible-role-pip#59

Closed

florianschroen mentioned this issue Nov 8, 2023

[BUG] "pip install docker-compose" fails on debian 12 docker/compose#11168

Closed

felixfontein closed this as completed Jan 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

community.docker.docker_compose does not work on debian 12 bookworm #672

community.docker.docker_compose does not work on debian 12 bookworm #672

lukaskaplan commented Jul 21, 2023

shantanoo-desai commented Jul 21, 2023

felixfontein commented Jul 22, 2023

Tardo commented Jul 24, 2023

felixfontein commented Jul 25, 2023

lukaskaplan commented Jul 25, 2023

felixfontein commented Jul 26, 2023

lukaskaplan commented Jul 26, 2023

felixfontein commented Jul 26, 2023

N0zz commented Aug 4, 2023 •

edited

Loading

felixfontein commented Aug 4, 2023

exhuma commented Aug 7, 2023

felixfontein commented Aug 7, 2023

lukaskaplan commented Aug 7, 2023

exhuma commented Aug 8, 2023

soakes commented Oct 15, 2023 •

edited

Loading

theko2fi commented Oct 15, 2023

soakes commented Oct 15, 2023

theko2fi commented Oct 15, 2023

community.docker.docker_compose does not work on debian 12 bookworm #672

community.docker.docker_compose does not work on debian 12 bookworm #672

Comments

lukaskaplan commented Jul 21, 2023

SUMMARY

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

COLLECTION VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

shantanoo-desai commented Jul 21, 2023

felixfontein commented Jul 22, 2023

Tardo commented Jul 24, 2023

felixfontein commented Jul 25, 2023

lukaskaplan commented Jul 25, 2023

felixfontein commented Jul 26, 2023

lukaskaplan commented Jul 26, 2023

felixfontein commented Jul 26, 2023

N0zz commented Aug 4, 2023 • edited Loading

felixfontein commented Aug 4, 2023

exhuma commented Aug 7, 2023

felixfontein commented Aug 7, 2023

lukaskaplan commented Aug 7, 2023

exhuma commented Aug 8, 2023

soakes commented Oct 15, 2023 • edited Loading

theko2fi commented Oct 15, 2023

soakes commented Oct 15, 2023

theko2fi commented Oct 15, 2023

N0zz commented Aug 4, 2023 •

edited

Loading

soakes commented Oct 15, 2023 •

edited

Loading