Rework how packages install on EXTERNALLY-MANAGED environment

[socheatsok78]

Add ability to control how "EXTERNALLY-MANAGED" python base environment via flag pip_externally_managed (boolean).

If pip_externally_managed: true, we preserver or create the EXTERNALLY-MANAGED file in it intended place. Otherwise remove it as original impl.

I'm not entirely sure if it is the right way to do this, but i'm open to suggestion.

Ref #57 (comment)

[gesinn-it-gea]

I tried the current v3.0.3 as well as this PR on Debian 12. In both cases, I'm getting "error: externally-managed-environment" during TASK [geerlingguy.pip : Ensure pip_install_packages are installed.]

[gesinn-it-gea]

That's what I get in the output. Could it be, that the EXTERNALLY-MANAGED get's restored during "Ensure Pip is installed."?

 default: TASK [geerlingguy.pip : Get installed python3 version] *************************
    default: ok: [192.168.56.10] => changed=false
    default:   cmd: python3 --version | awk '{split($2, a, "."); print a[1] "." a[2]}'
    default:   delta: '0:00:00.015955'
    default:   end: '2024-08-05 15:37:06.316133'
    default:   msg: ''
    default:   rc: 0
    default:   start: '2024-08-05 15:37:06.300178'
    default:   stderr: ''
    default:   stderr_lines: <omitted>
    default:   stdout: '3.11'
    default:   stdout_lines: <omitted>
    default:
    default: TASK [geerlingguy.pip : Ensure "absent" of EXTERNALLY-MANAGED's Python base environments] ***
    default: changed: [192.168.56.10] => changed=true
    default:   path: /usr/lib/python3.11/EXTERNALLY-MANAGED
    default:   state: absent
    default:
    default: TASK [geerlingguy.pip : Ensure Pip is installed.] ******************************
    default: changed: [192.168.56.10] => changed=true
    default:   cache_update_time: 1722871960
    default:   cache_updated: false
    default:   stderr: ''
    default:   stderr_lines: <omitted>
    default:   stdout: |-
    default:     Reading package lists...
    default:     Building dependency tree...
    default:     Reading state information...
    default:     The following additional packages will be installed:
    default:       build-essential cpp cpp-12 dpkg-dev fakeroot fontconfig-config
    default:       fonts-dejavu-core g++ g++-12 gcc gcc-12 javascript-common libabsl20220623
    default:       libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl
    default:       libaom3 libasan8 libatomic1 libavif15 libc-bin libc-dev-bin libc-devtools
    default:       libc-l10n libc6 libc6-dev libcc1-0 libcrypt-dev libdav1d6 libde265-0
    default:       libdeflate0 libdpkg-perl libexpat1-dev libfakeroot libfile-fcntllock-perl
    default:       libfontconfig1 libgav1-1 libgcc-12-dev libgd3 libgomp1 libheif1 libisl23
    default:       libitm1 libjbig0 libjpeg62-turbo libjs-jquery libjs-sphinxdoc
    default:       libjs-underscore liblerc4 liblsan0 libmpc3 libmpfr6 libnsl-dev libnuma1
    default:       libpython3-dev libpython3.11 libpython3.11-dev libpython3.11-minimal
    default:       libpython3.11-stdlib libquadmath0 librav1e0 libstdc++-12-dev libsvtav1enc1
    default:       libtiff6 libtirpc-dev libtsan2 libubsan1 libwebp7 libx11-6 libx11-data
    default:       libx265-199 libxau6 libxcb1 libxdmcp6 libxpm4 libyuv0 linux-libc-dev locales
    default:       make manpages-dev patch python3-dev python3-distutils python3-lib2to3
    default:       python3-setuptools python3-wheel python3.11 python3.11-dev
    default:       python3.11-minimal rpcsvc-proto zlib1g-dev
    default:     Suggested packages:
    default:       cpp-doc gcc-12-locales cpp-12-doc debian-keyring g++-multilib
    default:       g++-12-multilib gcc-12-doc gcc-multilib autoconf automake libtool flex bison
    default:       gdb gcc-doc gcc-12-multilib apache2 | lighttpd | httpd glibc-doc libnss-nis
    default:       libnss-nisplus git bzr libgd-tools libstdc++-12-doc make-doc ed
    default:       diffutils-doc python-setuptools-doc python3.11-venv python3.11-doc
    default:       binfmt-support
    default:     The following NEW packages will be installed:
    default:       build-essential cpp cpp-12 dpkg-dev fakeroot fontconfig-config
    default:       fonts-dejavu-core g++ g++-12 gcc gcc-12 javascript-common libabsl20220623
    default:       libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl
    default:       libaom3 libasan8 libatomic1 libavif15 libc-dev-bin libc-devtools libc6-dev
    default:       libcc1-0 libcrypt-dev libdav1d6 libde265-0 libdeflate0 libdpkg-perl
    default:       libexpat1-dev libfakeroot libfile-fcntllock-perl libfontconfig1 libgav1-1
    default:       libgcc-12-dev libgd3 libgomp1 libheif1 libisl23 libitm1 libjbig0
    default:       libjpeg62-turbo libjs-jquery libjs-sphinxdoc libjs-underscore liblerc4
    default:       liblsan0 libmpc3 libmpfr6 libnsl-dev libnuma1 libpython3-dev libpython3.11
    default:       libpython3.11-dev libquadmath0 librav1e0 libstdc++-12-dev libsvtav1enc1
    default:       libtiff6 libtirpc-dev libtsan2 libubsan1 libwebp7 libx11-6 libx11-data
    default:       libx265-199 libxau6 libxcb1 libxdmcp6 libxpm4 libyuv0 linux-libc-dev make
    default:       manpages-dev patch python3-dev python3-distutils python3-lib2to3 python3-pip
    default:       python3-setuptools python3-wheel python3.11-dev rpcsvc-proto zlib1g-dev
    default:     The following packages will be upgraded:
    default:       libc-bin libc-l10n libc6 libpython3.11-minimal libpython3.11-stdlib locales
    default:       python3.11 python3.11-minimal

default: TASK [geerlingguy.pip : Ensure pip_install_packages are installed.] ************
    default: failed: [192.168.56.10] (item={'name': 'docker'}) => changed=false
    default:   ansible_loop_var: item
    default:   cmd:
    default:   - /usr/bin/pip3
    default:   - install
    default:   - docker
    default:   item:
    default:     name: docker
    default:   msg: |2-
    default:
    default:     :stderr: error: externally-managed-environment
    default:
    default:     × This environment is externally managed
    default:     ╰─> To install Python packages system-wide, try apt install
    default:         python3-xyz, where xyz is the package you are trying to
    default:         install.

[gizero]

Could it be, that the EXTERNALLY-MANAGED get's restored during "Ensure Pip is installed."

@gesinn-it-gea yes and no... I mean... On a fresh Ubuntu 24.04 removing EXTERNALLY-MANAGED and THEN installing python3-pip package does not bring the file back to life BUT, what I noticed is that:

root@foo:~# dpkg -S /usr/lib/python3.12/EXTERNALLY-MANAGED
libpython3.12-stdlib:amd64: /usr/lib/python3.12/EXTERNALLY-MANAGED

Showing EXTERNALLY-MANAGED belongs to libpython3.12-stdlib package and from your logs:

default: The following packages will be upgraded:
default: libc-bin libc-l10n libc6 libpython3.11-minimal libpython3.11-stdlib locales
default: python3.11 python3.11-minimal

we can see libpython3.11-stdlib went through an update while installing python3-pip. This is likely the reason for EXTERNALLY-MANAGED being restored. To the contrary, on my Ubuntu 24.04 test, that library was not updated while installing python3-pip. We probably need a safer way to consistently get rid of that file, if that's really the way we want to go...

[socheatsok78]

We probably need a safer way to consistently get rid of that file, if that's really the way we want to go..

Totally agree on this, but removing the EXTERNALLY-MANAGED file seem not the play here. Maybe we can use the break_system_packages parameter on the ansible.builtin.pip module instead?

[skylenet]

Some nice investigations! Btw, you can also run it currently like this if you're fine with adding some extra_args to all the packages that you want to install:

pip_install_packages:
  - name: some-package
    extra_args: --break-system-packages

[mihalt]

Some nice investigations! Btw, you can also run it currently like this if you're fine with adding some extra_args to all the packages that you want to install:

pip_install_packages:
  - name: some-package
    extra_args: --break-system-packages

There is a problem in this case, that looks like pip3 does't support that

TASK [geerlingguy.pip : Ensure pip_install_packages are installed.] ****************************************************************************************************************
failed: [52.143.140.75] (item={'name': 'jmespath', 'extra_args': '--break-system-packages'}) => {"ansible_loop_var": "item", "changed": false, "cmd": ["/usr/bin/pip3", "install", "--break-system-packages", "jmespath"], "item": {"extra_args": "--break-system-packages", "name": "jmespath"}, "msg": "\n:stderr: \nUsage:   \n  pip3 install [options] <requirement specifier> [package-index-options] ...\n  pip3 install [options] -r <requirements file> [package-index-options] ...\n  pip3 install [options] [-e] <vcs project url> ...\n  pip3 install [options] [-e] <local project path> ...\n  pip3 install [options] <archive url/path> ...\n\nno such option: --break-system-packages\n"}

[socheatsok78]

Closing this due to complications with multiple different Python versions specifically the old Ansible and Python that doesn't support 'break-system-packages' flags.

Currently I've made an alternative role to test the implementation here https://github.com/socheatsok78-lab/ansible-role-unsafe-pip.