Flaky timeout in cudf-polars Ray worker setup

[TomAugspurger]

We're seeing some timeouts while setting up a Ray engine for cudf-polars tests. The exact test varies, but the traceback looks like

___________________ ERROR at setup of test_decimal_aggs[ray] ___________________
[gw1] linux -- Python 3.14.6 /opt/conda/envs/test/bin/python
Traceback (most recent call last):
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/runner.py", line 353, in from_call
    result: TResult | None = func()
                             ~~~~^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/runner.py", line 245, in <lambda>
    lambda: runtest_hook(item=item, **kwds),
            ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/logging.py", line 843, in pytest_runtest_setup
    yield
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup
    return (yield)
            ^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup
    item.session._setupstate.setup(item)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/runner.py", line 523, in setup
    col.setup()
    ~~~~~~~~~^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/python.py", line 1723, in setup
    self._request._fillfixtures()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures
    item.funcargs[argname] = self.getfixturevalue(argname)
                             ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue
    fixturedef = self._get_active_fixturedef(argname)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/fixtures.py", line 1110, in execute
    result: FixtureValue = ihook.pytest_fixture_setup(
                           ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        fixturedef=self, request=request
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup
    return (yield)
            ^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup
    result = call_fixture_func(fixturefunc, request, kwargs)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/fixtures.py", line 908, in call_fixture_func
    fixture_result = next(generator)
  File "/__w/cudf/cudf/python/cudf_polars/tests/conftest.py", line 156, in _unconfigured_engine
    engine = RayEngine(
        num_ranks=ray_num_ranks,
        engine_options={"allow_gpu_sharing": True},
        ray_init_options=ray_init_options,
    )
  File "/__w/cudf/cudf/python/cudf_polars/cudf_polars/engine/ray.py", line 706, in __init__
    root_ucxx_address_as_bytes = ray.get(rank_actors[0].setup_root.remote())
  File "/opt/conda/envs/test/lib/python3.14/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/ray/_private/client_mode_hook.py", line 107, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/ray/_private/worker.py", line 2980, in get
    values, debugger_breakpoint = worker.get_objects(
                                  ~~~~~~~~~~~~~~~~~~^
        object_refs, timeout, use_object_store=_use_object_store
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/ray/_private/worker.py", line 993, in get_objects
    ] = self.core_worker.get_objects(
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        object_refs,
        ^^^^^^^^^^^^
        timeout_ms,
        ^^^^^^^^^^^
    )
    ^
  File "python/ray/_raylet.pyx", line 3014, in ray._raylet.CoreWorker.get_objects
  File "python/ray/_raylet.pyx", line 352, in ray._raylet.RayObjectsToSerializedRayObjects
  File "<string>", line 1, in <lambda>
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pytest_timeout.py", line 317, in handler
    timeout_sigalrm(item, settings)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/pytest_timeout.py", line 502, in timeout_sigalrm
    pytest.fail(PYTEST_FAILURE_MESSAGE % settings.timeout)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/outcomes.py", line 163, in __call__
    raise Failed(msg=reason, pytrace=pytrace)
Failed: Timeout (>45.0s) from pytest-timeout.
---------------------------- Captured stdout setup -----------------------------
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~ Stack of ray_print_logs (134722940278336) ~~~~~~~~~~~~~~~~~~~
  File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1044, in _bootstrap
    self._bootstrap_inner()
  File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1082, in _bootstrap_inner
    self._context.run(self.run)
  File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1024, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/ray/_private/worker.py", line 1061, in print_logs
    data = subscriber.poll()
~~~~~~~~~~~~~ Stack of ray_listen_error_messages (134722948671040) ~~~~~~~~~~~~~
  File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1044, in _bootstrap
    self._bootstrap_inner()
  File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1082, in _bootstrap_inner
    self._context.run(self.run)
  File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1024, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/ray/_private/worker.py", line 2452, in listen_error_messages
    _, error_data = worker.gcs_error_subscriber.poll()
~~~~~~~~~~~~~~~~~~ Stack of PythonGCThread (134722965456448) ~~~~~~~~~~~~~~~~~~~
  File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1044, in _bootstrap
    self._bootstrap_inner()
  File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1082, in _bootstrap_inner
    self._context.run(self.run)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/ray/_private/gc_collect_manager.py", line 29, in run
    self._gc_event.wait()
  File "/opt/conda/envs/test/lib/python3.14/threading.py", line 670, in wait
    signaled = self._cond.wait(timeout)
  File "/opt/conda/envs/test/lib/python3.14/threading.py", line 369, in wait
    waiter.acquire()
~~~~~~~~~~~~~~~~~~~~~ Stack of <unknown> (134740844639808) ~~~~~~~~~~~~~~~~~~~~~
  File "/opt/conda/envs/test/lib/python3.14/site-packages/execnet/gateway_base.py", line 411, in _perform_spawn
    reply.run()
  File "/opt/conda/envs/test/lib/python3.14/site-packages/execnet/gateway_base.py", line 341, in run
    self._result = func(*args, **kwargs)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/execnet/gateway_base.py", line 1160, in _thread_receiver
    msg = Message.from_io(io)
  File "/opt/conda/envs/test/lib/python3.14/site-packages/execnet/gateway_base.py", line 567, in from_io
    header = io.read(9)  # type 1, channel 4, payload 4
  File "/opt/conda/envs/test/lib/python3.14/site-packages/execnet/gateway_base.py", line 534, in read
    data = self._read(numbytes - len(buf))

Observations:

• https://github.com/rapidsai/cudf/actions/runs/29395567159/job/87288199355 (conda-python-other-tests / 12.9.2, 3.13, amd64, ubuntu24.04, h100, latest-driver, latest-deps)
• https://github.com/rapidsai/cudf/actions/runs/29464129599/job/87518630068?pr=23284 (conda-python-other-tests / 12.9.2, 3.14, amd64, ubuntu22.04, h100, latest-driver, latest-deps)
• https://github.com/rapidsai/cudf/actions/runs/29616909593/job/88009272906?pr=23273

[TomAugspurger]

• https://github.com/rapidsai/cudf/actions/runs/29478723667/job/87557395708 (cudf-polars-polars-tests / 13.3.0, 3.13, amd64, ubuntu26.04, h100, latest-driver, latest-deps) is a failure during a test.

_______________________ test_collapse_joins_combinations _______________________
[gw1] linux -- Python 3.13.14 /pyenv/versions/3.13.14/bin/python
Traceback (most recent call last):
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/_pytest/runner.py", line 353, in from_call
    result: TResult | None = func()
                             ~~~~^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/_pytest/runner.py", line 245, in <lambda>
    lambda: runtest_hook(item=item, **kwds),
            ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/_pytest/logging.py", line 850, in pytest_runtest_call
    yield
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
    return result.get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_result.py", line 103, in get_result
    raise exc.with_traceback(tb)
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
    res = yield
          ^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
    return result.get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_result.py", line 103, in get_result
    raise exc.with_traceback(tb)
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
    res = yield
          ^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
    teardown.throw(exception)
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/_pytest/runner.py", line 179, in pytest_runtest_call
    item.runtest()
    ~~~~~~~~~~~~^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/_pytest/python.py", line 1720, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/_pytest/python.py", line 166, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/__w/cudf/cudf/polars/py-polars/tests/unit/lazyframe/test_optimizations.py", line 375, in test_collapse_joins_combinations
    optimized = cross.filter(e).sort(pl.all()).collect()
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/polars/_utils/deprecation.py", line 97, in wrapper
    return function(*args, **kwargs)
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/polars/lazyframe/opt_flags.py", line 344, in wrapper
    return function(*args, **kwargs)
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/polars/lazyframe/frame.py", line 2630, in collect
    return wrap_df(ldf.collect(engine, callback))
                   ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/cudf_polars/callback.py", line 306, in _callback
    df = ir.evaluate(cache={}, timer=timer, context=context).to_polars()
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/cudf_polars/containers/dataframe.py", line 140, in to_polars
    df = pl.DataFrame(table_with_metadata)
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/polars/dataframe/frame.py", line 479, in __init__
    self._df = pycapsule_to_frame(
               ~~~~~~~~~~~~~~~~~~^
        data,
        ^^^^^
        schema=schema,
        ^^^^^^^^^^^^^^
        schema_overrides=schema_overrides,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )._df
    ^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/polars/_utils/pycapsule.py", line 38, in pycapsule_to_frame
    df = s.to_frame(tmp_col_name).unnest(tmp_col_name)
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/polars/dataframe/frame.py", line 12533, in unnest
    self.lazy()
    ~~~~~~~~~~~
    .unnest(columns, *more_columns, separator=separator)
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/polars/lazyframe/frame.py", line 9085, in unnest
    subset = (
             ~
        cs.empty() if columns is None else parse_list_into_selector(columns)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    ) | parse_list_into_selector(more_columns)
    ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/polars/selectors.py", line 494, in __or__
    if is_column(other):  # @2.0: remove
       ~~~~~~~~~^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/polars/_utils/various.py", line 187, in is_column
    from polars.expr import Expr
  File "<frozen importlib._bootstrap>", line 1420, in _handle_fromlist
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pytest_timeout.py", line 317, in handler
    timeout_sigalrm(item, settings)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/pytest_timeout.py", line 502, in timeout_sigalrm
    pytest.fail(PYTEST_FAILURE_MESSAGE % settings.timeout)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/_pytest/outcomes.py", line 163, in __call__
    raise Failed(msg=reason, pytrace=pytrace)
Failed: Timeout (>240.0s) from pytest-timeout.
----------------------------- Captured stdout call -----------------------------
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~ Stack of spmd-executor_0 (128994460538560) ~~~~~~~~~~~~~~~~~~
  File "/pyenv/versions/3.13.14/lib/python3.13/threading.py", line 1015, in _bootstrap
    self._bootstrap_inner()
  File "/pyenv/versions/3.13.14/lib/python3.13/threading.py", line 1044, in _bootstrap_inner
    self.run()
  File "/pyenv/versions/3.13.14/lib/python3.13/threading.py", line 995, in run
    self._target(*self._args, **self._kwargs)
  File "/pyenv/versions/3.13.14/lib/python3.13/concurrent/futures/thread.py", line 90, in _worker
    work_item = work_queue.get(block=True)
~~~~~~~~~~~~~ Stack of default-singleton-engine (128994595825344) ~~~~~~~~~~~~~~
  File "/pyenv/versions/3.13.14/lib/python3.13/threading.py", line 1015, in _bootstrap
    self._bootstrap_inner()
  File "/pyenv/versions/3.13.14/lib/python3.13/threading.py", line 1044, in _bootstrap_inner
    self.run()
  File "/pyenv/versions/3.13.14/lib/python3.13/threading.py", line 995, in run
    self._target(*self._args, **self._kwargs)
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/cudf_polars/engine/default_singleton_engine.py", line 131, in _run
    self._shutdown_signal.wait()
  File "/pyenv/versions/3.13.14/lib/python3.13/threading.py", line 660, in wait
    signaled = self._cond.wait(timeout)
  File "/pyenv/versions/3.13.14/lib/python3.13/threading.py", line 359, in wait
    waiter.acquire()
~~~~~~~~~~~~~~~~~~~~~ Stack of <unknown> (129097493386944) ~~~~~~~~~~~~~~~~~~~~~
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/execnet/gateway_base.py", line 411, in _perform_spawn
    reply.run()
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/execnet/gateway_base.py", line 341, in run
    self._result = func(*args, **kwargs)
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/execnet/gateway_base.py", line 1160, in _thread_receiver
    msg = Message.from_io(io)
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/execnet/gateway_base.py", line 567, in from_io
    header = io.read(9)  # type 1, channel 4, payload 4
  File "/pyenv/versions/3.13.14/lib/python3.13/site-packages/execnet/gateway_base.py", line 534, in read
    data = self._read(numbytes - len(buf))

The durations shows a handful of very slow tests:

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
============================= slowest 10 durations =============================
240.01s call     tests/unit/lazyframe/test_optimizations.py::test_collapse_joins_combinations
210.68s call     tests/unit/operations/test_join_asof.py::test_asof_join_nearest_reference[True]
204.71s call     tests/unit/operations/test_join_asof.py::test_asof_join_nearest_reference[False]
129.50s call     tests/unit/operations/test_index_of.py::test_randomized
52.89s call     tests/unit/io/test_parquet.py::test_delta_encoding_roundtrip
45.39s call     tests/unit/operations/test_group_by.py::test_grouped_agg_parametric[unique-False-False-False-False]
45.30s call     tests/unit/io/test_parquet.py::test_parametric_small_page_mask_filtering
41.19s call     tests/unit/io/test_parquet.py::test_byte_stream_split_encoding_roundtrip
41.00s call     tests/unit/io/test_parquet.py::test_roundtrip_parametric
36.19s call     tests/unit/io/test_ipc.py::test_ipc_roundtrip_pyarrow_parametric[lz4]

[wence-]

test_collapse_joins_combinations runs, iirc, ~10000 different combinations inside a single test, so it plausibly is just slow on an overloaded runner (locally it takes 77s for me on an unloaded system).

[madsbk]

Same issue, I think:
https://github.com/rapidsai/cudf/actions/runs/29498114100/job/87625723967?pr=23234
https://github.com/rapidsai/cudf/actions/runs/29498114100/job/87626285198?pr=23234

[wence-]

https://github.com/rapidsai/cudf/actions/runs/29546734388/attempts/3

[TomAugspurger]

The class of errors that manifested as an error in tests/streaming/test_select.py::test_select_with_mixed_fusable_non_fusable_exprs[ray] after a XFAIL tests/streaming/test_select.py::test_select_with_len[ray], sometimes with a warning about a rapidsmpf AllGather, is probably fixed now.

• Support zero-column operands in cross_join #23235 (main)
• Avoid leaving multi-rank execution in a bad state in cudf-polars tests #23340 (release/26.08)

I'll leave this open for a bit to see if we have any more.

[mroeschke]

I encountered this in #23033 after merging in main post #23235

https://github.com/rapidsai/cudf/actions/runs/29767668768/job/88448527910?pr=23033

2026-07-20T21:43:22.7659416Z ==================================== ERRORS ====================================
2026-07-20T21:43:22.7664830Z ___________________ ERROR at setup of test_decimal_aggs[ray] ___________________
2026-07-20T21:43:22.7679253Z [gw2] linux -- Python 3.14.6 /opt/conda/envs/test/bin/python
2026-07-20T21:43:22.7681522Z Traceback (most recent call last):
2026-07-20T21:43:22.7683575Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/runner.py", line 353, in from_call
2026-07-20T21:43:22.7685464Z     result: TResult | None = func()
2026-07-20T21:43:22.7688931Z                              ~~~~^^
2026-07-20T21:43:22.7689679Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/runner.py", line 245, in <lambda>
2026-07-20T21:43:22.7691615Z     lambda: runtest_hook(item=item, **kwds),
2026-07-20T21:43:22.7692635Z             ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
2026-07-20T21:43:22.7694333Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__
2026-07-20T21:43:22.7702181Z     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
2026-07-20T21:43:22.7703522Z            ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-20T21:43:22.7704793Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec
2026-07-20T21:43:22.7712987Z     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
2026-07-20T21:43:22.7715300Z            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-20T21:43:22.7721415Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall
2026-07-20T21:43:22.7722376Z     raise exception
2026-07-20T21:43:22.7734179Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
2026-07-20T21:43:22.7745516Z     teardown.throw(exception)
2026-07-20T21:43:22.7749994Z     ~~~~~~~~~~~~~~^^^^^^^^^^^
2026-07-20T21:43:22.7751631Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/logging.py", line 843, in pytest_runtest_setup
2026-07-20T21:43:22.7752416Z     yield
2026-07-20T21:43:22.7753922Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
2026-07-20T21:43:22.7755313Z     teardown.throw(exception)
2026-07-20T21:43:22.7764703Z     ~~~~~~~~~~~~~~^^^^^^^^^^^
2026-07-20T21:43:22.7782666Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/capture.py", line 895, in pytest_runtest_setup
2026-07-20T21:43:22.7793684Z     return (yield)
2026-07-20T21:43:22.7794517Z             ^^^^^
2026-07-20T21:43:22.7795346Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall
2026-07-20T21:43:22.7816587Z     res = hook_impl.function(*args)
2026-07-20T21:43:22.7821540Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/runner.py", line 165, in pytest_runtest_setup
2026-07-20T21:43:22.7823794Z     item.session._setupstate.setup(item)
2026-07-20T21:43:22.7824945Z     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
2026-07-20T21:43:22.7833376Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/runner.py", line 523, in setup
2026-07-20T21:43:22.7843434Z     col.setup()
2026-07-20T21:43:22.7843967Z     ~~~~~~~~~^^
2026-07-20T21:43:22.7844964Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/python.py", line 1723, in setup
2026-07-20T21:43:22.7853013Z     self._request._fillfixtures()
2026-07-20T21:43:22.7853963Z     ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
2026-07-20T21:43:22.7855293Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/fixtures.py", line 707, in _fillfixtures
2026-07-20T21:43:22.7858178Z     item.funcargs[argname] = self.getfixturevalue(argname)
2026-07-20T21:43:22.7860458Z                              ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
2026-07-20T21:43:22.7862548Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/fixtures.py", line 539, in getfixturevalue
2026-07-20T21:43:22.7864459Z     fixturedef = self._get_active_fixturedef(argname)
2026-07-20T21:43:22.7873946Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/fixtures.py", line 627, in _get_active_fixturedef
2026-07-20T21:43:22.7880014Z     fixturedef.execute(request=subrequest)
2026-07-20T21:43:22.7882173Z     ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
2026-07-20T21:43:22.7883372Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/fixtures.py", line 1110, in execute
2026-07-20T21:43:22.7884537Z     result: FixtureValue = ihook.pytest_fixture_setup(
2026-07-20T21:43:22.7891577Z                            ~~~~~~~~~~~~~~~~~~~~~~~~~~^
2026-07-20T21:43:22.7892205Z         fixturedef=self, request=request
2026-07-20T21:43:22.7894747Z         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-20T21:43:22.7895496Z     )
2026-07-20T21:43:22.7930746Z     ^
2026-07-20T21:43:22.7934711Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_hooks.py", line 512, in __call__
2026-07-20T21:43:22.7935591Z     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
2026-07-20T21:43:22.7942094Z            ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-20T21:43:22.7943248Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_manager.py", line 120, in _hookexec
2026-07-20T21:43:22.7944910Z     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
2026-07-20T21:43:22.7951808Z            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-20T21:43:22.7953149Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 167, in _multicall
2026-07-20T21:43:22.7954004Z     raise exception
2026-07-20T21:43:22.7955643Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 139, in _multicall
2026-07-20T21:43:22.7959239Z     teardown.throw(exception)
2026-07-20T21:43:22.7963703Z     ~~~~~~~~~~~~~~^^^^^^^^^^^
2026-07-20T21:43:22.7970224Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup
2026-07-20T21:43:22.7971148Z     return (yield)
2026-07-20T21:43:22.7972277Z             ^^^^^
2026-07-20T21:43:22.7973866Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/pluggy/_callers.py", line 121, in _multicall
2026-07-20T21:43:22.7975126Z     res = hook_impl.function(*args)
2026-07-20T21:43:22.7983032Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/fixtures.py", line 1202, in pytest_fixture_setup
2026-07-20T21:43:22.7985348Z     result = call_fixture_func(fixturefunc, request, kwargs)
2026-07-20T21:43:22.7992673Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/fixtures.py", line 915, in call_fixture_func
2026-07-20T21:43:22.7994204Z     fixture_result = fixturefunc(**kwargs)
2026-07-20T21:43:22.8000023Z   File "/__w/cudf/cudf/python/cudf_polars/tests/conftest.py", line 283, in streaming_engine
2026-07-20T21:43:22.8001437Z     return configure_streaming_engine(engine, options)
2026-07-20T21:43:22.8002577Z   File "/__w/cudf/cudf/python/cudf_polars/cudf_polars/testing/engine_utils.py", line 192, in configure_streaming_engine
2026-07-20T21:43:22.8003384Z     engine._reset(
2026-07-20T21:43:22.8003907Z     ~~~~~~~~~~~~~^
2026-07-20T21:43:22.8004463Z         rapidsmpf_options=options.to_rapidsmpf_options(),
2026-07-20T21:43:22.8005306Z         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-20T21:43:22.8010757Z         executor_options=options.to_executor_options(),
2026-07-20T21:43:22.8012061Z         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-20T21:43:22.8013073Z         engine_options=options.to_engine_options(),
2026-07-20T21:43:22.8013880Z         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-20T21:43:22.8014699Z     )
2026-07-20T21:43:22.8015210Z     ^
2026-07-20T21:43:22.8019020Z   File "/__w/cudf/cudf/python/cudf_polars/cudf_polars/engine/ray.py", line 862, in _reset
2026-07-20T21:43:22.8021395Z     ray.get(
2026-07-20T21:43:22.8022666Z     ~~~~~~~^
2026-07-20T21:43:22.8023686Z         [
2026-07-20T21:43:22.8024482Z         ^
2026-07-20T21:43:22.8024993Z     ...<4 lines>...
2026-07-20T21:43:22.8029125Z         ]
2026-07-20T21:43:22.8030217Z         ^
2026-07-20T21:43:22.8031891Z     )
2026-07-20T21:43:22.8032435Z     ^
2026-07-20T21:43:22.8033790Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
2026-07-20T21:43:22.8034609Z     return fn(*args, **kwargs)
2026-07-20T21:43:22.8035559Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/ray/_private/client_mode_hook.py", line 107, in wrapper
2026-07-20T21:43:22.8038931Z     return func(*args, **kwargs)
2026-07-20T21:43:22.8040973Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/ray/_private/worker.py", line 2980, in get
2026-07-20T21:43:22.8041791Z     values, debugger_breakpoint = worker.get_objects(
2026-07-20T21:43:22.8042599Z                                   ~~~~~~~~~~~~~~~~~~^
2026-07-20T21:43:22.8043859Z         object_refs, timeout, use_object_store=_use_object_store
2026-07-20T21:43:22.8044677Z         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-20T21:43:22.8045258Z     )
2026-07-20T21:43:22.8047644Z     ^
2026-07-20T21:43:22.8048703Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/ray/_private/worker.py", line 993, in get_objects
2026-07-20T21:43:22.8049496Z     ] = self.core_worker.get_objects(
2026-07-20T21:43:22.8050577Z         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
2026-07-20T21:43:22.8051136Z         object_refs,
2026-07-20T21:43:22.8051670Z         ^^^^^^^^^^^^
2026-07-20T21:43:22.8052312Z         timeout_ms,
2026-07-20T21:43:22.8053106Z         ^^^^^^^^^^^
2026-07-20T21:43:22.8053642Z     )
2026-07-20T21:43:22.8054117Z     ^
2026-07-20T21:43:22.8054734Z   File "python/ray/_raylet.pyx", line 3014, in ray._raylet.CoreWorker.get_objects
2026-07-20T21:43:22.8057559Z   File "python/ray/_raylet.pyx", line 352, in ray._raylet.RayObjectsToSerializedRayObjects
2026-07-20T21:43:22.8059056Z   File "<string>", line 1, in <lambda>
2026-07-20T21:43:22.8059945Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/pytest_timeout.py", line 317, in handler
2026-07-20T21:43:22.8060646Z     timeout_sigalrm(item, settings)
2026-07-20T21:43:22.8061489Z     ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
2026-07-20T21:43:22.8062640Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/pytest_timeout.py", line 502, in timeout_sigalrm
2026-07-20T21:43:22.8063601Z     pytest.fail(PYTEST_FAILURE_MESSAGE % settings.timeout)
2026-07-20T21:43:22.8064219Z     ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-07-20T21:43:22.8064956Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/_pytest/outcomes.py", line 163, in __call__
2026-07-20T21:43:22.8067152Z     raise Failed(msg=reason, pytrace=pytrace)
2026-07-20T21:43:22.8068222Z Failed: Timeout (>45.0s) from pytest-timeout.
2026-07-20T21:43:22.8069954Z ---------------------------- Captured stdout setup -----------------------------
2026-07-20T21:43:22.8070692Z +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
2026-07-20T21:43:22.8071521Z ~~~~~~~~~~~~~~~~~~ Stack of ray_print_logs (130640523540032) ~~~~~~~~~~~~~~~~~~~
2026-07-20T21:43:22.8072259Z   File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1044, in _bootstrap
2026-07-20T21:43:22.8072924Z     self._bootstrap_inner()
2026-07-20T21:43:22.8073600Z   File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1082, in _bootstrap_inner
2026-07-20T21:43:22.8074287Z     self._context.run(self.run)
2026-07-20T21:43:22.8074930Z   File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1024, in run
2026-07-20T21:43:22.8075723Z     self._target(*self._args, **self._kwargs)
2026-07-20T21:43:22.8080567Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/ray/_private/worker.py", line 1061, in print_logs
2026-07-20T21:43:22.8081458Z     data = subscriber.poll()
2026-07-20T21:43:22.8082101Z ~~~~~~~~~~~~~ Stack of ray_listen_error_messages (130640531932736) ~~~~~~~~~~~~~
2026-07-20T21:43:22.8082836Z   File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1044, in _bootstrap
2026-07-20T21:43:22.8083501Z     self._bootstrap_inner()
2026-07-20T21:43:22.8084143Z   File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1082, in _bootstrap_inner
2026-07-20T21:43:22.8084819Z     self._context.run(self.run)
2026-07-20T21:43:22.8085473Z   File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1024, in run
2026-07-20T21:43:22.8088831Z     self._target(*self._args, **self._kwargs)
2026-07-20T21:43:22.8090186Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/ray/_private/worker.py", line 2452, in listen_error_messages
2026-07-20T21:43:22.8091599Z     _, error_data = worker.gcs_error_subscriber.poll()
2026-07-20T21:43:22.8092298Z ~~~~~~~~~~~~~~~~~~ Stack of PythonGCThread (130640540325440) ~~~~~~~~~~~~~~~~~~~
2026-07-20T21:43:22.8093024Z   File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1044, in _bootstrap
2026-07-20T21:43:22.8093689Z     self._bootstrap_inner()
2026-07-20T21:43:22.8094508Z   File "/opt/conda/envs/test/lib/python3.14/threading.py", line 1082, in _bootstrap_inner
2026-07-20T21:43:22.8095184Z     self._context.run(self.run)
2026-07-20T21:43:22.8097916Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/ray/_private/gc_collect_manager.py", line 29, in run
2026-07-20T21:43:22.8098936Z     self._gc_event.wait()
2026-07-20T21:43:22.8099564Z   File "/opt/conda/envs/test/lib/python3.14/threading.py", line 670, in wait
2026-07-20T21:43:22.8100360Z     signaled = self._cond.wait(timeout)
2026-07-20T21:43:22.8101003Z   File "/opt/conda/envs/test/lib/python3.14/threading.py", line 369, in wait
2026-07-20T21:43:22.8101654Z     waiter.acquire()
2026-07-20T21:43:22.8102224Z ~~~~~~~~~~~~~~~~~~~~~ Stack of <unknown> (130658901624384) ~~~~~~~~~~~~~~~~~~~~~
2026-07-20T21:43:22.8103045Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/execnet/gateway_base.py", line 411, in _perform_spawn
2026-07-20T21:43:22.8103791Z     reply.run()
2026-07-20T21:43:22.8104445Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/execnet/gateway_base.py", line 341, in run
2026-07-20T21:43:22.8105193Z     self._result = func(*args, **kwargs)
2026-07-20T21:43:22.8107557Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/execnet/gateway_base.py", line 1160, in _thread_receiver
2026-07-20T21:43:22.8108445Z     msg = Message.from_io(io)
2026-07-20T21:43:22.8109931Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/execnet/gateway_base.py", line 567, in from_io
2026-07-20T21:43:22.8111225Z     header = io.read(9)  # type 1, channel 4, payload 4
2026-07-20T21:43:22.8112007Z   File "/opt/conda/envs/test/lib/python3.14/site-packages/execnet/gateway_base.py", line 534, in read
2026-07-20T21:43:22.8112792Z     data = self._read(numbytes - len(buf))
2026-07-20T21:43:22.8113541Z +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

[TomAugspurger]

Thanks. Looks like the first failure in time was

2026-07-20T20:44:07.8004432Z [gw2] [ 56%] FAILED tests/streaming/test_join.py::test_join[ray-options0-True-left]
2026-07-20T21:43:34.3917392Z FAILED tests/streaming/test_join.py::test_join[ray-options0-True-left] - SystemError: <built-in method __new__ of type object at 0x6292404d7a00> returned a result with an exception set

Edit: nope, wrong. The ERROR here ocurred first:

2026-07-20T20:34:29.2410635Z [gw2] [ 53%] ERROR tests/streaming/test_agg.py::test_decimal_aggs[ray] 

Looking at the logs (https://grafana.ops.gha-runners.nvidia.com/d/runner-metrics/gha-runners-metrics?orgId=1&from=2026-07-20T18:49:47.000Z&to=2026-07-21T01:49:47.000Z&timezone=UTC&var-host=609e-l-amd-g-h100-l-1-b5lz8-runner-glsxt&var-datasource=P8058B42239793B94&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B%7Cmmcblk%5B0-9%5D%2B&var-node=4u8g-gen-0430.ipp6a1.colossus.nvidia.com&refresh=5m; nvidia VPN required) the CPU utilization was relatively high. "Busy System" was between 60-80% for most of the test run.

[TomAugspurger]

Another hang at https://github.com/rapidsai/cudf/actions/runs/29784003473/job/88497349909 on main.

Edit: I might have been too quick to call that a hang...

2026-07-21T01:20:43.1583335Z [gw3] [ 58%] PASSED tests/test_sort.py::test_sort[ray-stable-True-sort_keys2] 
2026-07-21T01:20:43.7793681Z tests/test_sort.py::test_sort[ray-stable-True-sort_keys3] 
2026-07-21T01:20:43.8995899Z ##[error]The operation was canceled.

So there was a log message just before I manually canceled it.