tree-wide: port debpython to termux

[robertkirkman]

• Fixes [Bug]: Conflicts: asciidoc and tinysparql #25921

• Fixes [Bug]: mesa & mesa-vulkan-icd-freedreno-dri3 build error #21658

• Fixes Auto update failing for python-onnxruntime #26687

• Fixes [Bug]: python-llvmlite conflict with libllvm #25621

What is debpython?

• debpython is the commands py3compile and py3clean from Debian. I
  am calling them that because a large chunk of their source code is
  found inside a folder inside Debian's source code named "debpython"

  • https://salsa.debian.org/cpython-team/python3-defaults/-/tree/5348f704668c0b6c360b6c6fb10153b9c2898af5

• rather than packaging .pyc files into packages, py3compile and
  py3clean can be called from postinst and prerm scripts to
  generate all .pyc for the .py files in the package immediately
  after the package is installed, and remove all .pyc files immediately
  before uninstalling the package, respectively

• fixes the error trying to overwrite '/data/data/com.termux/files/usr/lib/python3.12/__pycache__/cProfile.cpython-312.pyc' when packages were built on-device, but at the same time, also:

• prevents the warnings dpkg: warning: while removing python, directory '/data/data/com.termux/files/usr/lib/python3.12/site-packages' not empty so not removed as long as no packages were installed using pip

• The termux_step_create_python_debscripts.sh can configure work on debpython (i.e. its py3copile and py3clean commands) from the glibc package python-glibc, if some glibc package is being compiled.

• New variables have been implemented:

  • TERMUX_PYTHON_CROSSENV_BUILDHOME - location of crossenv's python build libraries.
  • TERMUX_PKG_PYTHON_RUNTIME_DEPS - configures the installation of the python modules via pip3 in the pkg's debscripts. If not configured in the package, it will use the value from TERMUX_PKG_PYTHON_TARGET_DEPS. If the variable is set to false, then the customization of installing python modules will be disabled, even if the TERMUX_PKG_PYTHON_TARGET_DEPS variable is set in the package.
  • TERMUX_SUBPKG_PYTHON_RUNTIME_DEPS - configures the installation of the python modules via pip3 in the subpkg's debscripts.

• Implemented reconfiguration of prefixes in python module sysconfig and setting in TERMUX_PYTHON_CROSSENV_BUILDHOME, so that python modules from crossenv building can specify system paths of termux for correct compilation.

• Added automatic addition of python-glibc{-glibc} dependency when using the TERMUX_PKG_PYTHON_RUNTIME_DEPS (for pkg; will be disabled, i.e. will not be added, if the variable is set to false) or TERMUX_SUBPKG_PYTHON_RUNTIME_DEPS (for subpkg) value.

How to add a new Python package after this?

Everything is the same, except, now, this block is no longer necessary in build.sh.

termux_step_create_debscripts() {
	cat <<- EOF > ./postinst
	#!$TERMUX_PREFIX/bin/sh
	echo "Installing dependencies through pip..."
	pip3 install ${TERMUX_PKG_PYTHON_TARGET_DEPS//, / }
	EOF
}

• Instead, scripts/build/termux_step_create_python_debscripts.sh can now detect the presence of pip package lists in $TERMUX_PKG_PYTHON_TARGET_DEPS, $TERMUX_SUBPKG_PYTHON_TARGET_DEPS, and the METADATA file of the Python package if it exists, and automatically insert them as a block into the postinst script for all relevant packages.
• $TERMUX_PKG_PYTHON_TARGET_DEPS is used for pip dependencies that are both on-device build-time and on-device run-time dependencies, and $TERMUX_PKG_PYTHON_RUNTIME_DEPS is used for runtime-only pip dependencies. $TERMUX_PKG_PYTHON_RUNTIME_DEPS overrides $TERMUX_PKG_PYTHON_TARGET_DEPS for runtime dependencies,
  • i.e. if TERMUX_PKG_PYTHON_RUNTIME_DEPS is not specified, but TERMUX_PKG_PYTHON_TARGET_DEPS is, then TERMUX_PKG_PYTHON_TARGET_DEPS will be used as both on-device build and on-device runtime dependencies,
  • but if TERMUX_PKG_PYTHON_RUNTIME_DEPS is specified, then TERMUX_PKG_PYTHON_TARGET_DEPS, if specified, is used only for on-device build-time dependencies.
• If python-pip is not already in the dependencies of any package that needs it, the build will fail with an error instructing maintainers to add python-pip to the dependencies of the package that needs it.

[licy183]

This folder should only be excluded only when performing on-device build. In some packages, like python, this content in this folder is managed by apt. It is quite annoying to see some warnings like xxx directory is not empty.

[robertkirkman]

This folder should only be excluded only when performing on-device build. In some packages, like python, this content in this folder is managed by apt

Should the content in these folders remain managed by apt in any case?

In Debian, I believe these folders are not managed by apt.

https://packages.debian.org/sid/amd64/libpython3.12-minimal/filelist

In Arch Linux, I believe these folders are managed by pacman.

https://archlinux.org/packages/core/x86_64/python/ (file list)

Fedora also ships them

https://packages.fedoraproject.org/pkgs/python3.12/python3-libs/fedora-40.html

it seems to me like it is optional whether to include them in the package, because I think Python can automatically generate them, but this problem is very difficult because, I think Arch Linux and Fedora might ship them for performance.

At least this does not break Python, I think, because a python package built from this change and installed in Termux after rm -rf $PREFIX/lib/python3.12 still works and runs.

It is quite annoying to see some warnings like xxx directory is not empty.

I do not think there is any way to get rid of this warning, because on Termux python-pip generates and stores the __pycache__ folders into the site-packages folder in the same $PREFIX/lib/python3.12/ folder where the $PREFIX/lib/python3.12/zoneinfo/_zoneinfo.py files are, etc.

Example: warning always reproducible with current packages

pkg install python-pip
pip install setuptools
pkg remove python

Removing python-pip (25.0.1) ...
Removing python-ensurepip-wheels (3.12.9) ...
Removing python (3.12.9) ...
dpkg: warning: while removing python, directory '/data/data/com.termux/files/usr/lib/python3.12/site-packages' not empty so not removed
Processing triggers for man (1.14.6-1) ...

That happens without this change.

Can you think of any way to:

• prevent the __pycache__ folders from being incorrectly included in non-Python packages during on-device build-package.sh, to fix trying to overwrite '/data/data/com.termux/files/usr/lib/python3.12/__pycache__/cProfile.cpython-312.pyc', which is also in package python 3.12.9
• but still make sure __pycache__ folders are included in the package only when they actually belong to the package currently being built, so they can be shipped for a Python startup performance boost?

I do not actually notice the performance difference at the startup of Python, when I try to test for it, but I believe someone else might.

[licy183]

In Debain, they have a postinst script to compile the modules when new packages are installed, and a prerm script to clean up __pycache__ dir and *.pyc files. I have done the same cleanup logic in pypy3 on Termux.

See /usr/share/python3/runtime.d/public_modules.rt{install,remove} in Debain.

[robertkirkman]

In Debain, they have a postinst script to compile the modules when new packages are installed, and a prerm script to clean up __pycache__ dir and *.pyc files. I have done the same cleanup logic in pypy3 on Termux.

See /usr/share/python3/runtime.d/public_modules.rt{install,remove} in Debain.

ok I did it, is this what you meant?

[TomJo2000]

Thanks for putting in the effort.
Looks like I accidentally created a merge conflict for you with one of my earlier PRs, sorry about that.

[robertkirkman]

I have finished final testing and it is working, so I will merge it in 24 hours if there are no more reviews.

I historically wrote a great amount of words discussing this message in this thread:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.

To recap concisely: thorough testing indicates that the reason this message occurs is because, when multiple packages are being installed at once, their postinst scripts run one at a time in order,

and each time pip install runs during a postinst script, other Python packages whose postinst scripts have not run yet, but whose files have already been unpacked, are detected as not having their dependencies from PyPi installed yet, which is a temporary state.

Once all the postinst scripts finish running and the apt install command completes, manual pip install commands from the shell after that do not show that message anymore, confirming that all of the necessary PyPi dependencies of all of the installed Python packages have been automatically installed.