Skip to content

GH-43519: [Python] Set up wheel building for Python 3.13 #43539

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jorisvandenbossche merged 35 commits into from
Aug 22, 2024
Merged

GH-43519: [Python] Set up wheel building for Python 3.13 #43539

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
ed4e9d1
Set up CI for Python 3.13
lysnikolaou Aug 2, 2024
58617a6
Address feedback
lysnikolaou Aug 5, 2024
cca8c65
Update env
lysnikolaou Aug 6, 2024
b6cabf0
Handle two cp313-cp313* directories in /opt/python in manylinux
lysnikolaou Aug 6, 2024
b5d172b
Fix build issues
lysnikolaou Aug 7, 2024
7040788
Update dev/tasks/python-wheels/github.linux.yml
lysnikolaou Aug 7, 2024
1d03354
Address feedback; remove 3.6-7 checks and wrap long bash line
lysnikolaou Aug 7, 2024
511f31d
Remove wrong semicolon
lysnikolaou Aug 7, 2024
f33cf7d
Merge branch 'main' into setup-ci-313
lysnikolaou Aug 7, 2024
7685c96
Address feedback; use python_image_tag and install numpy nightly
lysnikolaou Aug 8, 2024
6b36d3f
Fix CI; Add python_image_tag arg & install numpy>2 in 3.13
lysnikolaou Aug 8, 2024
14185ab
Add pre and extra index url in requirements files; pin to pandas>=3 o…
lysnikolaou Aug 9, 2024
3ed915c
Update python/requirements-wheel-test.txt
jorisvandenbossche Aug 9, 2024
13c28c3
Merge branch 'main' into setup-ci-313
lysnikolaou Aug 9, 2024
975d5ec
Only use testbench when not in 3.13
lysnikolaou Aug 9, 2024
b662379
Update ci/scripts/python_wheel_macos_build.sh
lysnikolaou Aug 9, 2024
1b836c7
Fix link in install_python.sh
lysnikolaou Aug 9, 2024
4f5c3f6
Set env correctly in macOS
lysnikolaou Aug 12, 2024
93b06a3
Remove --platform when running pip intall
lysnikolaou Aug 12, 2024
89965c1
Fix pip install setuptools in install_gcs_testbench
lysnikolaou Aug 12, 2024
eab3cfd
Use PYTHON_VERSION environment variable in install_gcs_testbench
lysnikolaou Aug 12, 2024
489bae6
Update ci/docker/python-wheel-manylinux-test.dockerfile
jorisvandenbossche Aug 13, 2024
00efbd9
Debug python version
lysnikolaou Aug 13, 2024
62d27d3
more debugging
jorisvandenbossche Aug 13, 2024
4332b6e
fix and undo debugging
jorisvandenbossche Aug 13, 2024
0b5b2aa
Fix manylinux installing testbench
lysnikolaou Aug 13, 2024
825bb98
Fix typo
lysnikolaou Aug 13, 2024
d2e4347
Correctly set python build arg and PYTHON_VERSION env var
lysnikolaou Aug 13, 2024
7ddac07
Update python/requirements-wheel-test.txt
jorisvandenbossche Aug 13, 2024
09fd530
Merge branch 'main' into setup-ci-313
lysnikolaou Aug 15, 2024
e12e2f5
Update ci/docker/python-wheel-manylinux-test.dockerfile
lysnikolaou Aug 15, 2024
3e3e5cc
Merge remote-tracking branch 'upstream/main' into setup-ci-313
jorisvandenbossche Aug 20, 2024
1e9e387
pin to numpy 2.1.0 now it is released
jorisvandenbossche Aug 20, 2024
ef520ba
Merge remote-tracking branch 'upstream/main' into setup-ci-313
jorisvandenbossche Aug 22, 2024
747a630
Merge remote-tracking branch 'upstream/main' into setup-ci-313
jorisvandenbossche Aug 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions ci/docker/python-wheel-windows-test-vs2019.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,9 @@ ARG python=3.8
RUN (if "%python%"=="3.8" setx PYTHON_VERSION "3.8.10" && setx PATH "%PATH%;C:\Python38;C:\Python38\Scripts") & \
(if "%python%"=="3.9" setx PYTHON_VERSION "3.9.13" && setx PATH "%PATH%;C:\Python39;C:\Python39\Scripts") & \
(if "%python%"=="3.10" setx PYTHON_VERSION "3.10.11" && setx PATH "%PATH%;C:\Python310;C:\Python310\Scripts") & \
(if "%python%"=="3.11" setx PYTHON_VERSION "3.11.5" && setx PATH "%PATH%;C:\Python311;C:\Python311\Scripts") & \
(if "%python%"=="3.12" setx PYTHON_VERSION "3.12.0" && setx PATH "%PATH%;C:\Python312;C:\Python312\Scripts")
(if "%python%"=="3.11" setx PYTHON_VERSION "3.11.9" && setx PATH "%PATH%;C:\Python311;C:\Python311\Scripts") & \
(if "%python%"=="3.12" setx PYTHON_VERSION "3.12.4" && setx PATH "%PATH%;C:\Python312;C:\Python312\Scripts") & \
(if "%python%"=="3.13" setx PYTHON_VERSION "3.13.0-rc1" && setx PATH "%PATH%;C:\Python313;C:\Python313\Scripts")

# Install archiver to extract xz archives
RUN choco install -r -y --no-progress python --version=%PYTHON_VERSION% & \
Comment thread
lysnikolaou marked this conversation as resolved.
Outdated
Expand Down
5 changes: 3 additions & 2 deletions ci/docker/python-wheel-windows-vs2019.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,8 +83,9 @@ ARG python=3.8
RUN (if "%python%"=="3.8" setx PYTHON_VERSION "3.8.10" && setx PATH "%PATH%;C:\Python38;C:\Python38\Scripts") & \
(if "%python%"=="3.9" setx PYTHON_VERSION "3.9.13" && setx PATH "%PATH%;C:\Python39;C:\Python39\Scripts") & \
(if "%python%"=="3.10" setx PYTHON_VERSION "3.10.11" && setx PATH "%PATH%;C:\Python310;C:\Python310\Scripts") & \
(if "%python%"=="3.11" setx PYTHON_VERSION "3.11.5" && setx PATH "%PATH%;C:\Python311;C:\Python311\Scripts") & \
(if "%python%"=="3.12" setx PYTHON_VERSION "3.12.0" && setx PATH "%PATH%;C:\Python312;C:\Python312\Scripts")
(if "%python%"=="3.11" setx PYTHON_VERSION "3.11.9" && setx PATH "%PATH%;C:\Python311;C:\Python311\Scripts") & \
(if "%python%"=="3.12" setx PYTHON_VERSION "3.12.4" && setx PATH "%PATH%;C:\Python312;C:\Python312\Scripts") & \
(if "%python%"=="3.13" setx PYTHON_VERSION "3.13.0-rc1" && setx PATH "%PATH%;C:\Python313;C:\Python313\Scripts")
RUN choco install -r -y --no-progress python --version=%PYTHON_VERSION%
RUN python -m pip install -U pip setuptools

Expand Down
7 changes: 4 additions & 3 deletions ci/scripts/install_python.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,9 @@ declare -A versions
versions=([3.8]=3.8.10
[3.9]=3.9.13
[3.10]=3.10.11
[3.11]=3.11.5
[3.12]=3.12.0)
[3.11]=3.11.9
[3.12]=3.12.4
[3.13]=3.13.0rc1)

if [ "$#" -ne 2 ]; then
echo "Usage: $0 <platform> <version>"
Expand All @@ -46,7 +47,7 @@ full_version=${versions[$2]}
if [ $platform = "macOS" ]; then
echo "Downloading Python installer..."

if [ "$(uname -m)" = "arm64" ] || [ "$version" = "3.10" ] || [ "$version" = "3.11" ] || [ "$version" = "3.12" ]; then
if [ "$(uname -m)" = "arm64" ] || [ "$version" = "3.10" ] || [ "$version" = "3.11" ] || [ "$version" = "3.12" ] || [ "$version" = "3.13" ]; then
Comment thread
lysnikolaou marked this conversation as resolved.
Outdated
fname="python-${full_version}-macos11.pkg"
else
fname="python-${full_version}-macosx10.9.pkg"
Comment thread
lysnikolaou marked this conversation as resolved.
Expand Down
6 changes: 3 additions & 3 deletions dev/release/verify-release-candidate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1146,7 +1146,7 @@ test_linux_wheels() {
local arch="x86_64"
fi

local python_versions="${TEST_PYTHON_VERSIONS:-3.8 3.9 3.10 3.11 3.12}"
local python_versions="${TEST_PYTHON_VERSIONS:-3.8 3.9 3.10 3.11 3.12 3.13}"
local platform_tags="${TEST_WHEEL_PLATFORM_TAGS:-manylinux_2_17_${arch}.manylinux2014_${arch} manylinux_2_28_${arch}}"

for python in ${python_versions}; do
Expand All @@ -1170,11 +1170,11 @@ test_macos_wheels() {

# apple silicon processor
if [ "$(uname -m)" = "arm64" ]; then
local python_versions="3.8 3.9 3.10 3.11 3.12"
local python_versions="3.8 3.9 3.10 3.11 3.12 3.13"
local platform_tags="macosx_11_0_arm64"
local check_flight=OFF
else
local python_versions="3.8 3.9 3.10 3.11 3.12"
local python_versions="3.8 3.9 3.10 3.11 3.12 3.13"
local platform_tags="macosx_10_15_x86_64"
fi

Expand Down
5 changes: 3 additions & 2 deletions dev/tasks/tasks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -387,7 +387,8 @@ tasks:
("3.9", "cp39", "cp39"),
("3.10", "cp310", "cp310"),
("3.11", "cp311", "cp311"),
("3.12", "cp312", "cp312")] %}
("3.12", "cp312", "cp312"),
("3.13", "cp313", "cp313")] %}

{############################## Wheel Linux ##################################}

Expand Down Expand Up @@ -1191,7 +1192,7 @@ tasks:
UBUNTU: 22.04
image: ubuntu-cpp-emscripten

{% for python_version in ["3.8", "3.9", "3.10", "3.11", "3.12"] %}
{% for python_version in ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"] %}
test-conda-python-{{ python_version }}:

@jorisvandenbossche jorisvandenbossche Aug 5, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume a conda build testing 3.13 will not yet work since conda-forge does not yet have Python 3.13. So maybe this change can be left out for now.

Also for a free-threaded variant, setting this up would not just work like this because then we need to install python from a custom channel? (https://py-free-threading.github.io/installing_cpython/#conda) But even then I am not sure if all other dependencies would then already have packages compatible with that python version?

@lysnikolaou lysnikolaou Aug 5, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's true. I removed 3.13 for now. Maybe it can be added later, when conda-forge does have a 3.13 version.

ci: github
template: docker-tests/github.linux.yml
Expand Down
1 change: 1 addition & 0 deletions python/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ classifiers = [
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
'Programming Language :: Python :: 3.12',
'Programming Language :: Python :: 3.13',
]
maintainers = [
{name = "Apache Arrow Developers", email = "dev@arrow.apache.org"}
Expand Down
14 changes: 7 additions & 7 deletions python/requirements-wheel-test.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,16 @@ pytest
pytz
tzdata; sys_platform == 'win32'

numpy==1.21.3; platform_system == "Linux" and platform_machine == "aarch64" and python_version < "3.11"
numpy==1.23.4; python_version == "3.11"
numpy==1.26.0; python_version >= "3.12"
numpy==1.21.6; platform_system == "Linux" and platform_machine == "aarch64" and python_version < "3.11"
numpy==1.23.5; python_version == "3.11"
numpy==1.26.4; python_version >= "3.12"
Comment thread
lysnikolaou marked this conversation as resolved.
Outdated
numpy==1.19.5; platform_system == "Linux" and platform_machine != "aarch64" and python_version < "3.9"
numpy==1.21.3; platform_system == "Linux" and platform_machine != "aarch64" and python_version >= "3.9" and python_version < "3.11"
numpy==1.21.3; platform_system == "Darwin" and platform_machine == "arm64" and python_version < "3.11"
numpy==1.21.6; platform_system == "Linux" and platform_machine != "aarch64" and python_version >= "3.9" and python_version < "3.11"
numpy==1.21.6; platform_system == "Darwin" and platform_machine == "arm64" and python_version < "3.11"
numpy==1.19.5; platform_system == "Darwin" and platform_machine != "arm64" and python_version < "3.9"
numpy==1.21.3; platform_system == "Darwin" and platform_machine != "arm64" and python_version >= "3.9" and python_version < "3.11"
numpy==1.21.6; platform_system == "Darwin" and platform_machine != "arm64" and python_version >= "3.9" and python_version < "3.11"
numpy==1.19.5; platform_system == "Windows" and python_version < "3.9"
numpy==1.21.3; platform_system == "Windows" and python_version >= "3.9" and python_version < "3.11"
numpy==1.21.6; platform_system == "Windows" and python_version >= "3.9" and python_version < "3.11"

pandas<1.1.0; platform_system == "Linux" and platform_machine != "aarch64" and python_version < "3.8"
pandas; platform_system == "Linux" and platform_machine != "aarch64" and python_version >= "3.8"
Expand Down