Add support for Python 3.14 by gforsyth · Pull Request #597 · rapidsai/ucxx

gforsyth · 2026-02-24T20:04:23Z

Description

Contributes to rapidsai/build-planning#205

This PR adds support for Python 3.14.

Notes for Reviewers

This is part of ongoing work to add Python 3.14 support across RAPIDS.
It temporarily introduces a build/test matrix including Python 3.14, from rapidsai/shared-workflows#508.

A follow-up PR will revert back to pointing at the main branch of shared-workflows once all
RAPIDS repos have added Python 3.14 support.

This will fail until all dependencies have been updated to Python 3.14

CI here is expected to fail until all of this project's upstream dependencies support Python 3.14.

This can be merged whenever all CI jobs are passing.

copy-pr-bot · 2026-02-24T20:04:27Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

gforsyth · 2026-02-24T20:09:39Z

/ok to test

gforsyth · 2026-02-24T21:16:09Z

/ok to test

gforsyth · 2026-02-24T21:37:27Z

/ok to test

gforsyth · 2026-02-26T17:10:38Z

On Python 3.13, ucxx async tests run fine:

🐚 pytest --sw --import-mode=append -vs python/ucxx/ucxx/_lib_async/tests --runslow
========================================================================== test session starts ===========================================================================
platform linux -- Python 3.13.12, pytest-8.4.2, pluggy-1.6.0 -- /home/gforsyth/miniforge3/envs/ucxxdev/bin/python3.13
cachedir: .pytest_cache
rootdir: /home/gforsyth/github.com/rapidsai/ucxx/python/ucxx
configfile: pyproject.toml
plugins: asyncio-1.3.0, rerunfailures-16.1
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 579 items
stepwise: no previously failed tests, not skipping.

python/ucxx/ucxx/_lib_async/tests/test_benchmark_cluster.py::test_benchmark_cluster [1772122774.873342] [0e080ca-lcedt:606109:0]     ucp_context.c:2339 UCX  WARN  UCP API version is incompatible: required >= 1.20, actual 1.19.0 (loaded from /home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.13/site-packages/libucx/lib/libucp.so)
[1772122775.098271] [0e080ca-lcedt:606109:0]          parser.c:2359 UCX  WARN  unused environment variable: UCX_MEMTYPE_CACHE (maybe: UCX_MEMTYPE_CACHE?)
[1772122775.098271] [0e080ca-lcedt:606109:0]          parser.c:2359 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
[1772122775.126022] [0e080ca-lcedt:606287:0]     ucp_context.c:2339 UCX  WARN  UCP API version is incompatible: required >= 1.20, actual 1.19.0 (loaded from /home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.13/site-packages/libucx/lib/libucp.so)
[1772122775.131814] [0e080ca-lcedt:606292:0]     ucp_context.c:2339 UCX  WARN  UCP API version is incompatible: required >= 1.20, actual 1.19.0 (loaded from /home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.13/site-packages/libucx/lib/libucp.so)
[1772122775.137594] [0e080ca-lcedt:606299:0]     ucp_context.c:2339 UCX  WARN  UCP API version is incompatible: required >= 1.20, actual 1.19.0 (loaded from /home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.13/site-packages/libucx/lib/libucp.so)
[1772122775.143577] [0e080ca-lcedt:606304:0]     ucp_context.c:2339 UCX  WARN  UCP API version is incompatible: required >= 1.20, actual 1.19.0 (loaded from /home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.13/site-packages/libucx/lib/libucp.so)
[1772122775.298053] [0e080ca-lcedt:606287:0]          parser.c:2359 UCX  WARN  unused environment variable: UCX_MEMTYPE_CACHE (maybe: UCX_MEMTYPE_CACHE?)
[1772122775.298053] [0e080ca-lcedt:606287:0]          parser.c:2359 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
[1772122775.320073] [0e080ca-lcedt:606304:0]          parser.c:2359 UCX  WARN  unused environment variable: UCX_MEMTYPE_CACHE (maybe: UCX_MEMTYPE_CACHE?)
[1772122775.320073] [0e080ca-lcedt:606304:0]          parser.c:2359 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
[1772122775.320493] [0e080ca-lcedt:606292:0]          parser.c:2359 UCX  WARN  unused environment variable: UCX_MEMTYPE_CACHE (maybe: UCX_MEMTYPE_CACHE?)
[1772122775.320493] [0e080ca-lcedt:606292:0]          parser.c:2359 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
[1772122775.331453] [0e080ca-lcedt:606299:0]          parser.c:2359 UCX  WARN  unused environment variable: UCX_MEMTYPE_CACHE (maybe: UCX_MEMTYPE_CACHE?)
[1772122775.331453] [0e080ca-lcedt:606299:0]          parser.c:2359 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
PASSED
python/ucxx/ucxx/_lib_async/tests/test_config.py::test_get_config PASSED
python/ucxx/ucxx/_lib_async/tests/test_config.py::test_set_env PASSED
python/ucxx/ucxx/_lib_async/tests/test_config.py::test_init_options [1772122776.238999] [0e080ca-lcedt:605539:0]     ucp_context.c:2339 UCX  WARN  UCP API version is incompatible: required >= 1.20, actual 1.19.0 (loaded from /home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.13/site-packages/libucx/lib/libucp.so)
[1772122776.353658] [0e080ca-lcedt:605539:0]          parser.c:2359 UCX  WARN  unused environment variable: UCX_MEMTYPE_CACHE (maybe: UCX_MEMTYPE_CACHE?)
[1772122776.353658] [0e080ca-lcedt:605539:0]          parser.c:2359 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
PASSED
python/ucxx/ucxx/_lib_async/tests/test_config.py::test_init_options_and_env [1772122776.539047] [0e080ca-lcedt:605539:0]     ucp_context.c:2339 UCX  WARN  UCP API version is incompatible: required >= 1.20, actual 1.19.0 (loaded from /home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.13/site-packages/libucx/lib/libucp.so)
PASSED
python/ucxx/ucxx/_lib_async/tests/test_config.py::test_init_unknown_option SKIPPED (Beginning with UCX >= 1.12, it's only possible to validate UCP options but...)
python/ucxx/ucxx/_lib_async/tests/test_config.py::test_init_invalid_option [1772122776.705415] [0e080ca-lcedt:605539:0]          parser.c:1340 UCX  ERROR Invalid value for SEG_SIZE: 'invalid-size'. Expected: memory units: <number>[b|kb|mb|gb], "inf", or "auto"
PASSED
python/ucxx/ucxx/_lib_async/tests/test_config.py::test_logging [1772122776.741599] [0e080ca-lcedt:605539:0]     ucp_context.c:2339 UCX  WARN  UCP API version is incompatible: required >= 1.20, actual 1.19.0 (loaded from /home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.13/site-packages/libucx/lib/libucp.so)
[1772122776.898959] [0e080ca-lcedt:605539:0]     ucp_context.c:2339 UCX  WARN  UCP API version is incompatible: required >= 1.20, actual 1.19.0 (loaded from /home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.13/site-packages/libucx/lib/libucp.so)

However when trying this on Python 3.14:

🐚 pytest --sw --import-mode=append -vs python/ucxx/ucxx/_lib_async/tests --runslow
========================================================================== test session starts ===========================================================================
platform linux -- Python 3.14.3, pytest-8.4.2, pluggy-1.6.0 -- /home/gforsyth/miniforge3/envs/ucxxdev/bin/python3.14
cachedir: .pytest_cache
rootdir: /home/gforsyth/github.com/rapidsai/ucxx/python/ucxx
configfile: pyproject.toml
plugins: asyncio-1.3.0, rerunfailures-16.1
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 579 items
stepwise: no previously failed tests, not skipping.

python/ucxx/ucxx/_lib_async/tests/test_benchmark_cluster.py::test_benchmark_cluster Traceback (most recent call last):
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/forkserver.py", line 344, in main
    code = _serve_one(child_r, fds,
                      unused_fds,
                      old_handlers)
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/forkserver.py", line 384, in _serve_one
    code = spawn._main(child_r, parent_sentinel)
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/synchronize.py", line 117, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/forkserver.py", line 344, in main
    code = _serve_one(child_r, fds,
                      unused_fds,
                      old_handlers)
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/forkserver.py", line 384, in _serve_one
    code = spawn._main(child_r, parent_sentinel)
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/synchronize.py", line 117, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/forkserver.py", line 344, in main
    code = _serve_one(child_r, fds,
                      unused_fds,
                      old_handlers)
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/forkserver.py", line 344, in main
    code = _serve_one(child_r, fds,
                      unused_fds,
                      old_handlers)
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/forkserver.py", line 384, in _serve_one
    code = spawn._main(child_r, parent_sentinel)
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/forkserver.py", line 384, in _serve_one
    code = spawn._main(child_r, parent_sentinel)
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/synchronize.py", line 117, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/multiprocessing/synchronize.py", line 117, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory
FileNotFoundError: [Errno 2] No such file or directory

These failures originate in calls to ucxx._lib_async.tests.test_benchmark_cluster.py:

@pytest.mark.asyncio
async def test_benchmark_cluster(n_chunks=1, n_nodes=2, n_workers=2):
    server_file = tempfile.NamedTemporaryFile()

    server, server_ret = _run_cluster_server(server_file.name, n_nodes * n_workers)

    # Wait for server to become available
    with open(server_file.name, "r") as f:
        while len(f.read()) == 0:
            pass

    workers = list(
        chain.from_iterable(
            _run_cluster_workers(server_file.name, n_chunks, n_workers, i, _worker)
            for i in range(n_nodes)
        )
    )

    join_processes(workers + [server], timeout=30)
    for worker in workers:
        terminate_process(worker)
    terminate_process(server)

Given the errors raise from multiprocessing, I strongly suspect that this is due to changes in Python 3.14 where multiprocessing now defaults to forkserver (https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods)

Print debugging at: file:///home/gforsyth/github.com/rapidsai/ucxx/python/ucxx/ucxx/benchmarks/utils.py

    q = mp.Queue()
    p = mp.Process(
        target=_server_process,
        args=(
            q,
            server_file,
            n_workers,
            ucx_options_list,
        ),
    )

    p.start()
    return p, q

On Python 3.14, The context at test time is python/ucxx/ucxx/_lib_async/tests/test_benchmark_cluster.py::test_benchmark_cluster <multiprocessing.context.ForkServerContext object at 0x7b25987ee120>

On Python 3.13, python/ucxx/ucxx/_lib_async/tests/test_benchmark_cluster.py::test_benchmark_cluster <multiprocessing.context.ForkContext object at 0x7d2a170ff4d0>

This does indicate that the issue stems from the Fork -> ForkServer switch.

Changing multiprocessing in utils.py to use an explicit ctx = mp.get_context("fork") at every instance of mp usage fixes the initial error, but then the tests fail at teardown.

diff --git a/python/ucxx/ucxx/benchmarks/utils.py b/python/ucxx/ucxx/benchmarks/utils.py
index 5ff72b9..cb77a54 100644
--- a/python/ucxx/ucxx/benchmarks/utils.py
+++ b/python/ucxx/ucxx/benchmarks/utils.py
@@ -163,8 +163,9 @@ def _run_cluster_server(
         A tuple with two elements: the process spawned and a queue where results
         will eventually be stored.
     """
-    q = mp.Queue()
-    p = mp.Process(
+    ctx = mp.get_context("fork")
+    q = ctx.Queue()
+    p = ctx.Process(
         target=_server_process,
         args=(
             q,
@@ -173,6 +174,7 @@ def _run_cluster_server(
             ucx_options_list,
         ),
     )
+
     p.start()
     return p, q

@@ -343,10 +345,11 @@ def _run_cluster_workers(
         )

     processes = []
+    ctx = mp.get_context("fork")
     for worker_num in range(num_node_workers):
         rank = node_idx * num_node_workers + worker_num
-        q = mp.Queue()
-        p = mp.Process(
+        q = ctx.Queue()
+        p = ctx.Process(
             target=_worker_process,
             args=(
                 q,

  File "/home/gforsyth/github.com/rapidsai/ucxx/python/ucxx/ucxx/_lib_async/tests/conftest.py", line 70, in ucxx_setup_teardown
    loop = asyncio.get_event_loop()
  File "/home/gforsyth/miniforge3/envs/ucxxdev/lib/python3.14/asyncio/events.py", line 715, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
                       % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'MainThread'.

file:///home/gforsyth/github.com/rapidsai/ucxx/python/ucxx/ucxx/benchmarks/utils.py

gforsyth · 2026-02-26T18:04:31Z

Ah, the event loop teardown issue is also because of Python 3.14:
https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.get_event_loop

Python 3.14 changed default context from `fork` to `forkserver` -- it was unset here and working, so changing it to explicit `fork`

Python 3.14 raises a `RuntimeError` for `get_event_loop`, ignoring that

TomAugspurger

Thanks.

Everything I've read suggests that correctly and safely using fork is basically impossible for us mortals. But given that that change is just in a benchmark and it's worked up until this point, I think that it's OK to not investigate this any further for now.

gforsyth · 2026-02-26T20:04:13Z

/merge

wence- · 2026-02-27T09:36:38Z

I think that it's OK to not investigate this any further for now.

#598

wence- · 2026-02-27T09:37:39Z

conda/environments/all_cuda-131_arch-x86_64.yaml

 - pytest-rerunfailures!=16.0.0
 - pytest<9.0.0
- python>=3.11,<3.14
+- python>=3.11


As usual, please to have upper-bounded by the version we haven't tested (i.e. 3.15).

feat: add support for Python 3.14

a12fd19

gforsyth added non-breaking Introduces a non-breaking change improvement Improves an existing functionality labels Feb 24, 2026

gforsyth mentioned this pull request Feb 24, 2026

Python 3.14 support rapidsai/build-planning#205

Open

38 tasks

gforsyth marked this pull request as ready for review February 24, 2026 21:54

gforsyth requested review from a team as code owners February 24, 2026 21:54

gforsyth requested a review from msarahan February 24, 2026 21:54

fix(conda): add 3.14 to conda build config

46d8e4d

jameslamb requested review from jameslamb and removed request for msarahan February 25, 2026 21:53

gforsyth added 2 commits February 26, 2026 13:05

fix(multiprocessing): set context to fork explicitly

d611fa4

Python 3.14 changed default context from `fork` to `forkserver` -- it was unset here and working, so changing it to explicit `fork`

fix(teardown): ignore runtime error when no event loop

69e5cbf

Python 3.14 raises a `RuntimeError` for `get_event_loop`, ignoring that

gforsyth requested a review from a team as a code owner February 26, 2026 18:07

TomAugspurger approved these changes Feb 26, 2026

View reviewed changes

bdice approved these changes Feb 26, 2026

View reviewed changes

jameslamb approved these changes Feb 26, 2026

View reviewed changes

rapids-bot bot merged commit 8926b3c into rapidsai:main Feb 26, 2026
175 of 177 checks passed

gforsyth deleted the python-3.14 branch February 26, 2026 20:05

wence- mentioned this pull request Feb 27, 2026

Remove necessity for fork process-start mechanism in multiprocess benchmarks #598

Open

wence- reviewed Feb 27, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Python 3.14#597

Add support for Python 3.14#597
rapids-bot[bot] merged 4 commits intorapidsai:mainfrom
gforsyth:python-3.14

gforsyth commented Feb 24, 2026

Uh oh!

copy-pr-bot bot commented Feb 24, 2026

Uh oh!

gforsyth commented Feb 24, 2026

Uh oh!

gforsyth commented Feb 24, 2026

Uh oh!

gforsyth commented Feb 24, 2026

Uh oh!

gforsyth commented Feb 26, 2026

Uh oh!

gforsyth commented Feb 26, 2026

Uh oh!

TomAugspurger left a comment

Uh oh!

gforsyth commented Feb 26, 2026

Uh oh!

Uh oh!

wence- commented Feb 27, 2026

Uh oh!

wence- Feb 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

gforsyth commented Feb 24, 2026

Description

Notes for Reviewers

This will fail until all dependencies have been updated to Python 3.14

Uh oh!

copy-pr-bot bot commented Feb 24, 2026

Uh oh!

gforsyth commented Feb 24, 2026

Uh oh!

gforsyth commented Feb 24, 2026

Uh oh!

gforsyth commented Feb 24, 2026

Uh oh!

gforsyth commented Feb 26, 2026

Uh oh!

gforsyth commented Feb 26, 2026

Uh oh!

TomAugspurger left a comment

Choose a reason for hiding this comment

Uh oh!

gforsyth commented Feb 26, 2026

Uh oh!

Uh oh!

wence- commented Feb 27, 2026

Uh oh!

wence- Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants