TEST: arrow v22.0.0 (end-to-end)

[h-vetinari]

Some changes require testing across both libarrow and pyarrow; since we split off the pyarrow builds in a66c7c6, we'd need to publish a changed libarrow, test the changes in pyarrow, and then potentially mark the builds as broken. In order to simplify this process, reintroduce some end-to-end testing here that can be used to do this all in one PR.

Continued from #1432

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe/meta.yaml:

• ℹ️ The recipe is not parsable by parser conda-souschef (grayskull). This parser is not currently used by conda-forge, but may be in the future. We are collecting information to see which recipes are compatible with grayskull.
• ℹ️ The recipe is not parsable by parser conda-recipe-manager. The recipe can only be automatically migrated to the new v1 format if it is parseable by conda-recipe-manager.

_{This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/12749814205. Examine the logs at this URL for more detail.}

[raulcd]

This seems deprecated (as has already been done). I am closing it, feel free to reopen if necessary.

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-linting service.

I was trying to look for recipes to lint for you, but it appears we have a merge conflict. Please try to merge or rebase with the base branch to resolve this conflict.

Please ping the 'conda-forge/core' team (using the @ notation in a comment) if you believe this is a bug.

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe/meta.yaml:

• ℹ️ The recipe is not parsable by parser conda-souschef (grayskull). This parser is not currently used by conda-forge, but may be in the future. We are collecting information to see which recipes are compatible with grayskull.
• ℹ️ The recipe is not parsable by parser conda-recipe-manager. The recipe can only be automatically migrated to the new v1 format if it is parseable by conda-recipe-manager.

_{This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/16213744741. Examine the logs at this URL for more detail.}

[h-vetinari]

arrow v20.0.0-rc0 is looking good from the POV of conda-forge! 🥳 @raulcd @kou @pitrou @jorisvandenbossche @assignUser

[raulcd]

@h-vetinari we have split libarrow-compute to its own shared library instead of being bundled with libarrow. We will have to do some work to adapt that and the dependencies:

• GH-25025: [C++] Move non core compute kernels into separate shared library apache/arrow#46261

I was planning to do some work there but I wanted to give you a heads up in case you start working on it

[h-vetinari]

Thanks for the heads-up @raulcd, feel free to either push into this PR or open a new one that adapts to the new schema.

I just saw the notifications of the rc tags and decided to take them for a spin. :)

[h-vetinari]

@h-vetinari we have split libarrow-compute to its own shared library instead of being bundled with libarrow.

Is it intentional that libarrow-compute is still necessary even for the most basic import pyarrow?

import: 'pyarrow'
Traceback (most recent call last):
  File "/home/conda/feedstock_root/build_artifacts/apache-arrow_1752217099854/test_tmp/run_test.py", line 2, in <module>
    import pyarrow
  File "$PREFIX/lib/python3.9/site-packages/pyarrow/__init__.py", line 61, in <module>
    import pyarrow.lib as _lib
ImportError: libarrow_compute.so.2100: cannot open shared object file: No such file or directory

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-linting service.

I was trying to look for recipes to lint for you, but it appears we have a merge conflict. Please try to merge or rebase with the base branch to resolve this conflict.

Please ping the 'conda-forge/core' team (using the @ notation in a comment) if you believe this is a bug.

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe/meta.yaml:

• ℹ️ The recipe is not parsable by parser conda-souschef (grayskull). This parser is not currently used by conda-forge, but may be in the future. We are collecting information to see which recipes are compatible with grayskull.
• ℹ️ The recipe is not parsable by parser conda-recipe-manager. The recipe can only be automatically migrated to the new v1 format if it is parseable by conda-recipe-manager.

_{This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/18916120236. Examine the logs at this URL for more detail.}

[raulcd]

Thanks for working on this! I was at EuroPython last week but I am back now.

libarrow-compute is required for Acero and Dataset, we should add them to their host/run dependencies too.

I am +1 on adding it to pyarrow-core, we haven't done the work for it to be an optional dependency on pyarrow yet.

[h-vetinari]

We do have some genuine pyarrow test failures on windows:

= 23 failed, 7660 passed, 276 skipped, 21 deselected, 23 xfailed, 2 xpassed, 1 warning in 389.96s (0:06:29) =

the all look like

E           AssertionError: DataFrame.iloc[:, 0] (column name="uint8") are different
E           
E           DataFrame.iloc[:, 0] (column name="uint8") values are different (8.0 %)
E           [index]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]
E           [left]:  [88, 94, 107, 145, 47, 170, 196, 151, 117, 166, 22, 183, 192, 204, 33, 216, 67, 179, 78, 154, 251, 82, 162, 219, 195, 118, 125, 139, 103, 125, 229, 216, 9, 164, 116, 108, 211, 222, 161, 159, 21, 81, 89, 165, 242, 214, 102, 98, 36, 183, 5, 112, 87, 58, 43, 76, 70, 60, 75, 228, 216, 189, 132, 14, 88, 154, 178, 246, 140, 205, 204, 69, 58, 57, 41, 98, 193, 66, 72, 122, 230, 125, 174, 202, 39, 74, 234, 207, 87, 168, 101, 135, 174, 200, 223, 122, 223, 129, 44, 118]
E           [right]: [172, 10, 127, 140, 47, 170, 196, 151, 117, 166, 22, 183, 192, 204, 33, 216, 67, 179, 78, 154, 251, 82, 162, 219, 195, 118, 125, 139, 103, 125, 229, 216, 9, 164, 116, 108, 211, 222, 161, 159, 21, 81, 89, 165, 242, 214, 102, 98, 36, 183, 5, 112, 87, 58, 43, 76, 70, 60, 75, 228, 216, 189, 132, 14, 88, 154, 178, 246, 140, 205, 204, 69, 58, 57, 41, 98, 193, 66, 72, 122, 230, 125, 174, 202, 39, 74, 234, 207, 87, 168, 101, 135, 174, 200, 223, 122, 88, 94, 107, 145]

[raulcd]

This seems related to this MSVC issue:

• [Python][Release] verify-rc-source-windows Python tests are failing due to MSVC compiler bug apache/arrow#47051

Yes, from the logs this is using:
Visual Studio 2022 Developer Command Prompt v17.14.8
the SIMD compiler fix should be on v17.14.9

[raulcd]

The current failures seem related to the fix we had to apply to arrow-cpp, right?

FAILED: [code=1] src/arrow/CMakeFiles/arrow_util.dir/util/byte_stream_split_internal.cc.o 
$BUILD_PREFIX/bin/x86_64-conda-linux-gnu-c++ -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_AVX512 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DGFLAGS_IS_A_DLL=0 -DGLOG_USE_GFLAGS -DGLOG_USE_GLOG_EXPORT -DOPENTELEMETRY_ABI_VERSION_NO=1 -DPROTOBUF_USE_DLLS -DURI_STATIC_BUILD -I$SRC_DIR/cpp/build/src -I$SRC_DIR/cpp/src -I$SRC_DIR/cpp/src/generated -Wno-noexcept-type -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/apache-arrow-22.0.0.rc1 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -isystem /usr/local/cuda/include -fdiagnostics-color=always -fuse-ld=gold  -Wall -fno-semantic-interposition -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/apache-arrow-22.0.0.rc1 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -isystem /usr/local/cuda/include -O3 -DNDEBUG -O2 -ftree-vectorize  -std=c++17 -flto=auto -fno-fat-lto-objects -fPIC -pthread -MD -MT src/arrow/CMakeFiles/arrow_util.dir/util/byte_stream_split_internal.cc.o -MF src/arrow/CMakeFiles/arrow_util.dir/util/byte_stream_split_internal.cc.o.d -o src/arrow/CMakeFiles/arrow_util.dir/util/byte_stream_split_internal.cc.o -c $SRC_DIR/cpp/src/arrow/util/byte_stream_split_internal.cc
$SRC_DIR/cpp/src/arrow/util/byte_stream_split_internal.cc: In static member function 'static constexpr auto arrow::util::internal::ByteStreamSplitDecodeDynamic<kNumStreams>::implementations()':
$SRC_DIR/cpp/src/arrow/util/byte_stream_split_internal.cc:55:16: error: 'ByteStreamSplitDecodeSimd' was not declared in this scope; did you mean 'ByteStreamSplitDecode'?
   55 |               &ByteStreamSplitDecodeSimd<xsimd::avx2, kNumStreams>,

[h-vetinari]

Yeah, CI here is outdated; I'll rebase and pick the changes from #1886 once CI there is through