Skip to content

GH-33804: [Python] Add support for manylinux_2_28 wheel #33805

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sjperkins wants to merge 27 commits into from
Closed

GH-33804: [Python] Add support for manylinux_2_28 wheel #33805

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
f8fc629
Add manylinux_2_28 wheel
sjperkins Jan 19, 2023
bb9268d
Address review comments
sjperkins Jan 21, 2023
424fa08
Fix manylinux variable discrepancies
sjperkins Jan 21, 2023
bf6f3b1
Fix incorrect arch causing duplicate yaml keys
sjperkins Jan 21, 2023
26cc4c0
Avoid duplicate keys in task definition
sjperkins Jan 21, 2023
b490f15
fix arch and remove ci from python wheel key
sjperkins Jan 22, 2023
ee4d79d
underscores to dashes in manylinux 2_28 ccache name
sjperkins Jan 22, 2023
d196e09
Update docker-compose.yml
sjperkins Jan 22, 2023
79661a1
Update dev/tasks/tasks.yml
sjperkins Jan 22, 2023
3363331
Create unified manylinux Dockerfile
sjperkins Jan 22, 2023
1171e32
Remove unused install_glibc.sh
sjperkins Jan 22, 2023
59a0fb2
Add comments explaining placement of Python 3.8 on the PATH
sjperkins Jan 23, 2023
46579ae
Improve comment
sjperkins Jan 23, 2023
3924533
Distinguish artifact naming schema produced by auditwheel for differe…
sjperkins Jan 23, 2023
a26c20d
Correct jinja2 equality operator
sjperkins Jan 23, 2023
cc083e0
Fix jinja2 templating
sjperkins Jan 23, 2023
7bf98e2
openssl wants linux-headers on arm64 platform
sjperkins Jan 25, 2023
3c90df8
spelling
sjperkins Jan 25, 2023
d0d6894
Install basic dependencies with dnf, rather than yum
sjperkins Jan 27, 2023
3d640ca
Introduce TEST_WHEEL_PLATFORM_TAGS environment variable
sjperkins Jan 29, 2023
21a7ff1
Fix task.yml errors and lint
sjperkins Jan 29, 2023
0cc575c
Fix typo in artifact string
sjperkins Jan 30, 2023
f10be24
Merge branch 'master' into manylinux-2-28
sjperkins Feb 2, 2023
999a927
Merge branch 'master' into manylinux-2-28
sjperkins Feb 16, 2023
dbb4473
Merge branch 'main' into manylinux-2-28
sjperkins Feb 22, 2023
9c1a3dc
Merge branch 'main' into manylinux-2-28
sjperkins Feb 24, 2023
95d6827
Add --cross-compile-prefix option to openssl CMakeList.txt
sjperkins Feb 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions ...er/python-wheel-manylinux-201x.dockerfile → ci/docker/python-wheel-manylinux.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,19 @@ ARG manylinux

ENV MANYLINUX_VERSION=${manylinux}

# Ensure dnf is installed, especially for the manylinux2014 base
RUN yum install -y dnf

# Install basic dependencies
RUN yum install -y git flex curl autoconf zip perl-IPC-Cmd wget
RUN dnf install -y git flex curl autoconf zip perl-IPC-Cmd wget kernel-headers

# A system Python is required for ninja and vcpkg in this Dockerfile.
# On manylinux2014 base images, system Python is 2.7.5, while
# on manylinux_2_28, no system python is installed.
# We therefore override the PATH with Python 3.8 in /opt/python
# so that we have a consistent Python version across base images.
ENV CPYTHON_VERSION=cp38
ENV PATH=/opt/python/${CPYTHON_VERSION}-${CPYTHON_VERSION}/bin:${PATH}

# Install CMake
# AWS SDK doesn't work with CMake=3.22 due to https://gitlab.kitware.com/cmake/cmake/-/issues/22524
Expand All @@ -45,12 +56,10 @@ RUN /arrow/ci/scripts/install_ccache.sh ${ccache} /usr/local

# Install vcpkg
ARG vcpkg
ARG glibc=2.18
COPY ci/vcpkg/*.patch \
ci/vcpkg/*linux*.cmake \
arrow/ci/vcpkg/
COPY ci/scripts/install_vcpkg.sh \
ci/scripts/install_glibc.sh \
arrow/ci/scripts/
ENV VCPKG_ROOT=/opt/vcpkg
RUN arrow/ci/scripts/install_vcpkg.sh ${VCPKG_ROOT} ${vcpkg}
Expand All @@ -77,6 +86,7 @@ RUN vcpkg install \
--x-feature=json \
--x-feature=parquet

# Configure Python for applications running in the bash shell of this Dockerfile
ARG python=3.8
ENV PYTHON_VERSION=${python}
RUN PYTHON_ROOT=$(find /opt/python -name cp${PYTHON_VERSION/./}-*) && \
Expand Down
42 changes: 0 additions & 42 deletions ci/scripts/install_glibc.sh
View file
Open in desktop

This file was deleted.

12 changes: 12 additions & 0 deletions ci/vcpkg/ports.patch
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,3 +64,15 @@ index 4611af6..d11936f 100644
set(brew_package_name "python")
set(apt_package_name "python3")
endif()
diff --git a/ports/openssl/unix/CMakeLists.txt b/ports/openssl/unix/CMakeLists.txt
index 75c21805a..f95792a74 100644
--- a/ports/openssl/unix/CMakeLists.txt
+++ b/ports/openssl/unix/CMakeLists.txt
@@ -224,6 +224,7 @@ else()
"--prefix=${CMAKE_INSTALL_PREFIX}"
"--libdir=${CMAKE_INSTALL_PREFIX}/lib"
"--openssldir=/etc/ssl"
+ "--cross-compile-prefix=\"\""
${CFLAGS}
VERBATIM
APPEND
2 changes: 1 addition & 1 deletion dev/release/verify-release-candidate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1028,7 +1028,7 @@ test_linux_wheels() {
fi

local python_versions="${TEST_PYTHON_VERSIONS:-3.7m 3.8 3.9 3.10 3.11}"
local platform_tags="manylinux_2_17_${arch}.manylinux2014_${arch}"
local platform_tags="${TEST_WHEEL_PLATFORM_TAGS:-manylinux_2_17_${arch}.manylinux2014_${arch} manylinux_2_28_${arch}}"

for python in ${python_versions}; do
local pyver=${python/m}
Expand Down
1 change: 1 addition & 0 deletions dev/tasks/python-wheels/github.linux.amd64.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,7 @@ jobs:
-e TEST_DEFAULT=0 \
-e TEST_PYARROW_VERSION={{ arrow.no_rc_version }} \
-e TEST_PYTHON_VERSIONS={{ python_version }} \
-e TEST_WHEEL_PLATFORM_TAGS={{ wheel_platform_tag }} \
-e TEST_WHEELS=1 \
ubuntu-verify-rc

Expand Down
13 changes: 8 additions & 5 deletions dev/tasks/tasks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -452,16 +452,19 @@ tasks:

{############################## Wheel Linux ##################################}

{% for ci, arch, arch_alias, x_y, manylinux in [("github", "amd64", "x86_64", "2_17", "2014"),
("travis", "arm64", "aarch64", "2_17", "2014")] %}
wheel-manylinux{{ manylinux }}-{{ python_tag }}-{{ arch }}:
{% for ci, arch, manylinux_version, platform_tag in [("github", "amd64", "2014", "manylinux_2_17_x86_64.manylinux2014_x86_64"),
("github", "amd64", "2-28", "manylinux_2_28_x86_64"),
("travis", "arm64", "2014", "manylinux_2_17_aarch64.manylinux2014_aarch64"),
("travis", "arm64", "2-28", "manylinux_2_28_aarch64")] %}
wheel-manylinux-{{ manylinux_version }}-{{ python_tag }}-{{ arch }}:
ci: "{{ ci }}"
template: python-wheels/{{ ci }}.linux.{{ arch }}.yml
params:
python_version: "{{ python_version }}"
manylinux_version: "{{ manylinux }}"
manylinux_version: "{{ manylinux_version }}"
wheel_platform_tag: "{{ platform_tag }}"
artifacts:
- pyarrow-{no_rc_version}-{{ python_tag }}-{{ abi_tag }}-manylinux_{{ x_y }}_{{ arch_alias }}.manylinux{{ manylinux }}_{{ arch_alias }}.whl
- pyarrow-{no_rc_version}-{{ python_tag }}-{{ abi_tag }}-{{ platform_tag }}.whl
{% endfor %}

{############################## Wheel OSX ####################################}
Expand Down
28 changes: 27 additions & 1 deletion docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,7 @@ x-hierarchy:
- postgres
- python-wheel-manylinux-2014:
- java-jni-manylinux-2014
- python-wheel-manylinux-2-28
- python-wheel-manylinux-test-imports
- python-wheel-manylinux-test-unittests
- python-wheel-windows-vs2017
Expand All @@ -182,6 +183,8 @@ volumes:
name: maven-cache
python-wheel-manylinux2014-ccache:
name: python-wheel-manylinux2014-ccache
python-wheel-manylinux-2-28-ccache:
name: python-wheel-manylinux-2-28-ccache
python-wheel-windows-clcache:
name: python-wheel-windows-clcache
ubuntu-ccache:
Expand Down Expand Up @@ -976,7 +979,7 @@ services:
python: ${PYTHON}
manylinux: 2014
context: .
dockerfile: ci/docker/python-wheel-manylinux-201x.dockerfile
dockerfile: ci/docker/python-wheel-manylinux.dockerfile
cache_from:
- ${REPO}:${ARCH}-python-${PYTHON}-wheel-manylinux-2014-vcpkg-${VCPKG}
environment:
Expand All @@ -986,6 +989,29 @@ services:
- ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux2014-ccache:/ccache:delegated
command: /arrow/ci/scripts/python_wheel_manylinux_build.sh

# See available versions at:
# https://quay.io/repository/pypa/manylinux_2_28_x86_64?tab=tags
python-wheel-manylinux-2-28:
image: ${REPO}:${ARCH}-python-${PYTHON}-wheel-manylinux-2-28-vcpkg-${VCPKG}
build:
args:
arch: ${ARCH}
arch_short: ${ARCH_SHORT}
base: quay.io/pypa/manylinux_2_28_${ARCH_ALIAS}:2023-01-14-103cb93
vcpkg: ${VCPKG}
python: ${PYTHON}
manylinux: 2_28
context: .
dockerfile: ci/docker/python-wheel-manylinux.dockerfile
cache_from:
- ${REPO}:${ARCH}-python-${PYTHON}-wheel-manylinux-2-28-vcpkg-${VCPKG}
environment:
<<: *ccache
volumes:
- .:/arrow:delegated
- ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux-2-28-ccache:/ccache:delegated
command: /arrow/ci/scripts/python_wheel_manylinux_build.sh

python-wheel-manylinux-test-imports:
image: ${ARCH}/python:${PYTHON}
shm_size: 2G
Expand Down