Skip to content

[Packaging] Bump Python to 3.12 on RHEL and CentOS Stream #31264

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bebound merged 3 commits into from
Apr 24, 2025
Merged

[Packaging] Bump Python to 3.12 on RHEL and CentOS Stream #31264

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5251ae2
Init
bebound Apr 14, 2025
a64cea1
Install pip
bebound Apr 18, 2025
8ae6d82
Remove python3.12-devel, gcc
bebound Apr 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -816,13 +816,13 @@ jobs:
dockerfile: ubi
image: registry.access.redhat.com/ubi8/ubi:8.4
artifact: rpm-ubi8-${{ arch.value }}
python_package: python39
python_package: python3.12
pool: ${{ arch.pool }}
Red Hat Universal Base Image 9 ${{ arch.name }}:
dockerfile: ubi
image: registry.access.redhat.com/ubi9/ubi:9.0.0
artifact: rpm-ubi9-${{ arch.value }}
python_package: python3.9
python_package: python3.12
pool: ${{ arch.pool }}
steps:
- bash: ./scripts/ci/install_docker.sh
Expand Down Expand Up @@ -864,17 +864,17 @@ jobs:
artifact: rpm-ubi8-${{ arch.value }}
distro: el8
image: registry.access.redhat.com/ubi8/ubi:8.4
python_package: python39
python_cmd: python3.9
pip_cmd: pip3.9
python_package: python3.12
python_cmd: python3.12
pip_cmd: pip3.12
pool: ${{ arch.pool }}
Red Hat Universal Base Image 9 ${{ arch.name }}:
artifact: rpm-ubi9-${{ arch.value }}
distro: el9
image: registry.access.redhat.com/ubi9/ubi:9.0.0
python_package: python3.9
python_cmd: python3.9
pip_cmd: pip3.9
python_package: python3.12
python_cmd: python3.12
pip_cmd: pip3.12
pool: ${{ arch.pool }}
steps:
- task: DownloadPipelineArtifact@1
Expand Down
1 change: 1 addition & 0 deletions scripts/release/rpm/azure-cli.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ deactivate

# Fix up %{buildroot} appearing in some files...
for d in %{buildroot}%{cli_lib_dir}/bin/*; do perl -p -i -e "s#%{buildroot}##g" $d; done;
rm %{buildroot}%{cli_lib_dir}/pyvenv.cfg

@bebound bebound Apr 18, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el9.x86_64/usr/lib64/az/pyvenv.cfg content is

home = /usr/bin
include-system-site-packages = false
version = 3.12.5
executable = /usr/bin/python3.12
command = /usr/bin/python3.12 -m venv /root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el9.x86_64/usr/lib64/az

It raises this error during build.

#10 83.39 RPM build errors:
#10 83.39 error: Bad exit status from /var/tmp/rpm-tmp.ZBlRUB (%install)
#10 83.39     Bad exit status from /var/tmp/rpm-tmp.ZBlRUB (%install)
#10 ERROR: process "/bin/sh -c export PIP_INDEX_URL=$(cat /run/secrets/PIP_INDEX_URL) &&     dos2unix ./scripts/release/rpm/azure-cli.spec &&     REPO_PATH=$(pwd) CLI_VERSION=$cli_version PYTHON_PACKAGE=$python_package PYTHON_CMD=$python_package     rpmbuild -v -bb --clean scripts/release/rpm/azure-cli.spec &&     cp /root/rpmbuild/RPMS/*/azure-cli-${cli_version}-1.*.rpm /azure-cli-dev.rpm &&     mkdir /out && cp /root/rpmbuild/RPMS/*/azure-cli-${cli_version}-1.*.rpm /out/" did not complete successfully: exit code: 1
------
 > [build-env 6/6] RUN --mount=type=secret,id=PIP_INDEX_URL export PIP_INDEX_URL=$(cat /run/secrets/PIP_INDEX_URL) &&     dos2unix ./scripts/release/rpm/azure-cli.spec &&     REPO_PATH=$(pwd) CLI_VERSION=2.71.0 PYTHON_PACKAGE=python3.12 PYTHON_CMD=python3.12     rpmbuild -v -bb --clean scripts/release/rpm/azure-cli.spec &&     cp /root/rpmbuild/RPMS/*/azure-cli-2.71.0-1.*.rpm /azure-cli-dev.rpm &&     mkdir /out && cp /root/rpmbuild/RPMS/*/azure-cli-2.71.0-1.*.rpm /out/:
82.82 + mkdir -p /root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el8.x86_64/etc/bash_completion.d/
82.82 + cat /azure-cli/az.completion
82.82 + /usr/lib/rpm/check-buildroot
83.38 /root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el8.x86_64/usr/lib64/az/pyvenv.cfg:command = /usr/bin/python3.12 -m venv /root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el8.x86_64/usr/lib64/az
83.38 Found '/root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el8.x86_64' in installed files; aborting
83.39 
83.39 
83.39 RPM build errors:
83.39 error: Bad exit status from /var/tmp/rpm-tmp.ZBlRUB (%install)
83.39     Bad exit status from /var/tmp/rpm-tmp.ZBlRUB (%install)

In 3.9, pyvenv.cfg does not include rpmbuild root:

home = /usr/bin
include-system-site-packages = false
version = 3.9.20

@jiasli jiasli Apr 23, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/usr/lib/rpm/check-buildroot wants to make sure the RPM build path /root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el9.x86_64 does not exist in any of the installed file, but pyvenv.cfg contains command = /usr/bin/python3.12 -m venv /root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el9.x86_64/usr/lib64/az which makes /usr/lib/rpm/check-buildroot fail.

  1. Will deleting pyvenv.cfg cause any side effect?
  2. Can we use sed or perl (L60) to only remove command = /usr/bin/python3.12 -m venv /root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el9.x86_64/usr/lib64/az?

@bebound bebound Apr 24, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pyvenv.cfg contains some metadata of the venv. We only need the site-pacakges folder in the venv.
This is the RPM entry script:

azure-cli/scripts/release/rpm/azure-cli.spec

Line 79 in 81de68b

AZ_INSTALLER=RPM PYTHONPATH=\"\$bin_dir/../lib64/az/lib/${python_version}/site-packages\" \$python_cmd -sm azure.cli \"\$@\"

I thinks it's fine to remove it. As a proof, we have explicitly excluded the bin folder created by venv:

azure-cli/scripts/release/rpm/azure-cli.spec

Line 88 in 81de68b

%exclude %{cli_lib_dir}/bin/

PS: Although {cli_lib_dir}/bin/ is excluded in RPM, it still checks whether this path contains hard-coded buildroot during the build. Line 60 is still necessary.


# Create executable (entry script)

Expand Down
2 changes: 1 addition & 1 deletion scripts/release/rpm/test_rpm_in_docker.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ export USERNAME=azureuser

dnf --nogpgcheck install /mnt/rpm/$RPM_NAME -y

dnf install git gcc $PYTHON_PACKAGE-devel findutils -y
dnf install git gcc $PYTHON_PACKAGE-devel findutils $PYTHON_PACKAGE-pip -y

@bebound bebound Apr 18, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

python3.12-devel installs python3.12-pip-wheel, which is only a pip wheel used in venv and does not install pip module in python3.12.

I need to install python3.12-pip explicitly as it's used in line 18

PS: In 3.9, python39-devel installs python39-pip.

Ref: https://centos.pkgs.org/9-stream/centos-appstream-aarch64/python3.12-pip-wheel-23.2.1-4.el9.noarch.rpm.html

@jiasli jiasli Apr 23, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the content of https://mirror.stream.centos.org/9-stream/AppStream/aarch64/os/Packages/python3.12-pip-wheel-23.2.1-4.el9.noarch.rpm and it contains python3.12-pip-wheel-23.2.1-4.el9.noarch.cpio\.\usr\share\python3.12-wheels\pip-23.2.1-py3-none-any.whl.

https://docs.redhat.com/de/documentation/red_hat_enterprise_linux/9/html-single/installing_and_using_dynamic_programming_languages/index#proc_installing-additional-python-3-packages_assembly_installing-and-using-python contains instruction for installing python3.12-pip:

dnf install python3.12-pip

Actually I am wondering if we really need python3.12-devel at all:

https://centos.pkgs.org/9-stream/centos-appstream-aarch64/python3.12-devel-3.12.9-1.el9.aarch64.rpm.html

This package contains the header files and configuration needed to compile
Python extension modules (typically written in C or C++), to embed Python
into other programs, and to make binary distributions for Python libraries.
It also contains the necessary macros to build RPM packages with Python modules
and 2to3 tool, an automatic source converter from Python 2.X.

@bebound bebound Apr 23, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gcc and python3-devel is introduced in #14806.
It seems fine to remove them.


ln -s -f /usr/bin/$PYTHON_CMD /usr/bin/python
ln -s -f /usr/bin/$PIP_CMD /usr/bin/pip
Expand Down
4 changes: 2 additions & 2 deletions scripts/release/rpm/ubi.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ ARG image=registry.access.redhat.com/ubi8/ubi:8.4

FROM ${image} AS build-env
ARG cli_version=dev
ARG python_package=python39
ARG python_package=python3.12

RUN yum update -y
RUN yum install -y wget rpm-build gcc libffi-devel ${python_package}-devel openssl-devel make bash diffutils patch dos2unix perl
Expand All @@ -18,7 +18,7 @@ COPY . .
# We have to explicitly specify 'python39' to install Python 3.9.
RUN --mount=type=secret,id=PIP_INDEX_URL export PIP_INDEX_URL=$(cat /run/secrets/PIP_INDEX_URL) && \
dos2unix ./scripts/release/rpm/azure-cli.spec && \
REPO_PATH=$(pwd) CLI_VERSION=$cli_version PYTHON_PACKAGE=$python_package PYTHON_CMD=python3.9 \
REPO_PATH=$(pwd) CLI_VERSION=$cli_version PYTHON_PACKAGE=$python_package PYTHON_CMD=$python_package \

@bebound bebound Apr 23, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In RHEL8, the PYTHON_PACKAGE is python39, PYTHON_CMD is python3.9.
In RHEL9, both are unified as python3.9. Ref: https://github.com/Azure/azure-cli/pull/23556/files#r952101959

In python3.12, both distros use python3.12.

I'm unsure if these values will differ in the future; keep these variables for now.

rpmbuild -v -bb --clean scripts/release/rpm/azure-cli.spec && \
cp /root/rpmbuild/RPMS/*/azure-cli-${cli_version}-1.*.rpm /azure-cli-dev.rpm && \
mkdir /out && cp /root/rpmbuild/RPMS/*/azure-cli-${cli_version}-1.*.rpm /out/
Expand Down