ARROW-17016: [C++][Python] Move Arrow Python C++ tests into Cython #14117

AlenkaF · 2022-09-14T10:29:23Z

This PR tries to connect the PyArrow C++ tests with PyArrow tests so they can all be run from pytest. This will remove GoogleTest as a dependency for PyArrow and therefore a change is needed in the C++ tests so they return a Status which can then be checked through Cython/Python.

Example of pytest run:

pyarrow/tests/test_cpp_internals.py::test_owned_ref_moves PASSED
pyarrow/tests/test_cpp_internals.py::test_owned_ref_nogil_moves PASSED
pyarrow/tests/test_cpp_internals.py::test_check_pyerror_status PASSED
pyarrow/tests/test_cpp_internals.py::test_check_pyerror_status_nogil PASSED
pyarrow/tests/test_cpp_internals.py::test_restore_pyerror_basics PASSED
pyarrow/tests/test_cpp_internals.py::test_pybuffer_invalid_input_object PASSED
pyarrow/tests/test_cpp_internals.py::test_pybuffer_numpy_array PASSED
pyarrow/tests/test_cpp_internals.py::test_numpybuffer_numpy_array PASSED
pyarrow/tests/test_cpp_internals.py::test_python_decimal_to_string PASSED
pyarrow/tests/test_cpp_internals.py::test_infer_precision_and_scale PASSED
pyarrow/tests/test_cpp_internals.py::test_infer_precision_and_negative_scale PASSED
pyarrow/tests/test_cpp_internals.py::test_infer_all_leading_zeros PASSED
pyarrow/tests/test_cpp_internals.py::test_infer_all_leading_zeros_exponential_notation_positive PASSED
pyarrow/tests/test_cpp_internals.py::test_infer_all_leading_zeros_exponential_notation_negative PASSED
pyarrow/tests/test_cpp_internals.py::test_object_block_write_fails PASSED
pyarrow/tests/test_cpp_internals.py::test_mixed_type_fails PASSED
pyarrow/tests/test_cpp_internals.py::test_from_python_decimal_rescale_not_truncateable PASSED
pyarrow/tests/test_cpp_internals.py::test_from_python_decimal_rescale_truncateable PASSED
pyarrow/tests/test_cpp_internals.py::test_from_python_negative_decimal_rescale PASSED
pyarrow/tests/test_cpp_internals.py::test_decimal128_from_python_integer PASSED
pyarrow/tests/test_cpp_internals.py::test_decimal256_from_python_integer PASSED
pyarrow/tests/test_cpp_internals.py::test_decimal128_overflow_fails PASSED
pyarrow/tests/test_cpp_internals.py::test_decimal256_overflow_fails PASSED
pyarrow/tests/test_cpp_internals.py::test_none_and_nan PASSED
pyarrow/tests/test_cpp_internals.py::test_mixed_precision_and_scale PASSED
pyarrow/tests/test_cpp_internals.py::test_mixed_precision_and_scale_sequence_convert PASSED
pyarrow/tests/test_cpp_internals.py::test_simple_inference PASSED
pyarrow/tests/test_cpp_internals.py::test_update_with_nan PASSED

github-actions · 2022-09-14T10:31:11Z

https://issues.apache.org/jira/browse/ARROW-17016

github-actions · 2022-09-14T10:31:12Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

python/pyarrow/src/python_test.cc

pitrou

Just some general comments. I will probably want to make some slight changes on the C++ side as well.

python/pyarrow/_pyarrow_cpp_tests.pxd

python/setup.cfg

python/pyarrow/tests/test_python.py

AlenkaF · 2022-09-27T08:47:36Z

Should we wait for ARROW-17843 so that the PyArrow C++ code is checked or we can correct that within the linter issue itself?

pitrou · 2022-09-27T08:50:38Z

We can correct the C++ linting later IMHO.

AlenkaF · 2022-09-27T15:18:27Z

The failures on the CI are not new, have seen them on the nightly builds.

I think this PR is ready, totally approved from my side :)
@pitrou did you have any other corrections in mind?

jorisvandenbossche · 2022-09-27T15:40:19Z

@AlenkaF can you merge in or rebase on latest master? That should resolve those CI failures

pitrou

+1, thank you very much for doing this @AlenkaF !

pitrou · 2022-09-28T15:00:19Z

@github-actions crossbow submit -g python -g wheel

github-actions · 2022-09-28T17:48:59Z

Revision: 7e4e2e653d8ce24cb3d8961fb386881114fda126

Submitted crossbow builds: ursacomputing/crossbow @ actions-6dfbce193c

Task	Status
test-conda-python-3.10
test-conda-python-3.7
test-conda-python-3.7-hdfs-2.9.2
test-conda-python-3.7-hdfs-3.2.1
test-conda-python-3.7-kartothek-latest
test-conda-python-3.7-kartothek-master
test-conda-python-3.7-pandas-0.24
test-conda-python-3.7-pandas-latest
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8
test-conda-python-3.8-hypothesis
test-conda-python-3.8-pandas-latest
test-conda-python-3.8-pandas-nightly
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9
test-conda-python-3.9-dask-latest
test-conda-python-3.9-dask-master
test-conda-python-3.9-pandas-master
test-conda-python-3.9-spark-master
test-debian-11-python-3
test-fedora-35-python-3
test-ubuntu-20.04-python-3
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp310-universal2
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-big-sur-cp39-universal2
wheel-macos-mojave-cp310-amd64
wheel-macos-mojave-cp37-amd64
wheel-macos-mojave-cp38-amd64
wheel-macos-mojave-cp39-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp37-arm64
wheel-manylinux2014-cp38-amd64
wheel-manylinux2014-cp38-arm64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

AlenkaF · 2022-09-29T07:51:25Z

I think the failures of the wheels are related to the fact that ARROW_PYTHON is not set anymore and so for example csv module is not build and not found by PyArrow (ARROW_CSV=OFF).

pitrou · 2022-09-29T07:53:25Z

Yes, we will probably need to enable all desired components manually.

jorisvandenbossche · 2022-09-29T07:53:42Z

Yes, that is getting fixed in #14273

AlenkaF · 2022-09-29T07:56:01Z

You think I should wait for #14273 to get merged and then rebase?

…orking yet

…d InferPrecisionAndNegativeScale

…return a Status and run them from Pytest

AlenkaF · 2022-10-03T06:16:28Z

@pitrou @jorisvandenbossche this PR is now ready for final round of review/merge.

jorisvandenbossche · 2022-10-03T14:02:33Z

@github-actions crossbow submit -g python -g wheel

github-actions · 2022-10-03T14:04:11Z

Revision: be21035

Submitted crossbow builds: ursacomputing/crossbow @ actions-f48ddf2e66

Task	Status
test-conda-python-3.10
test-conda-python-3.7
test-conda-python-3.7-hdfs-2.9.2
test-conda-python-3.7-hdfs-3.2.1
test-conda-python-3.7-pandas-0.24
test-conda-python-3.7-pandas-latest
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8
test-conda-python-3.8-hypothesis
test-conda-python-3.8-pandas-latest
test-conda-python-3.8-pandas-nightly
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9
test-conda-python-3.9-dask-latest
test-conda-python-3.9-dask-master
test-conda-python-3.9-pandas-master
test-conda-python-3.9-spark-master
test-debian-11-python-3
test-fedora-35-python-3
test-ubuntu-20.04-python-3
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp310-universal2
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-big-sur-cp39-universal2
wheel-macos-mojave-cp310-amd64
wheel-macos-mojave-cp37-amd64
wheel-macos-mojave-cp38-amd64
wheel-macos-mojave-cp39-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp37-arm64
wheel-manylinux2014-cp38-amd64
wheel-manylinux2014-cp38-arm64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

pitrou · 2022-10-03T16:33:56Z

Nothing seems to have changed since my last review, so I'm simply gonna merge!

AlenkaF · 2022-10-03T16:43:36Z

Thank you for running the wheels build - totally forgot that was what I was waiting to check :)

ursabot · 2022-10-04T02:22:40Z

Benchmark runs are scheduled for baseline = 441af34 and contender = 5fd54bc. 5fd54bc is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.0% ⬆️0.0%] test-mac-arm
[Failed ⬇️0.82% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.46% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 5fd54bc6 ec2-t3-xlarge-us-east-2
[Failed] 5fd54bc6 test-mac-arm
[Failed] 5fd54bc6 ursa-i9-9960x
[Finished] 5fd54bc6 ursa-thinkcentre-m75q
[Finished] 441af348 ec2-t3-xlarge-us-east-2
[Failed] 441af348 test-mac-arm
[Failed] 441af348 ursa-i9-9960x
[Finished] 441af348 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ursabot · 2022-10-04T02:22:57Z

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

…14435) As #14117 is merged separate run of `ctest` for `PyArrow C++` tests is not needed anymore and is removed in this PR. Authored-by: Alenka Frim <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

@wjones127

…pers/python page (#14350) This PR updates Windows section of the Python Development page. Main changes: - use Python 3.10 (also in instructions for Linux/MacOs) - definition of `PATH` not needed as Python doesn't search in `PATH` for dlls anymore ([3.8 +](https://bugs.python.org/issue43173)) - use `CONDA_PREFIX` to define `ARROW_HOME` as in other parts of the docs - remove **Running C++ unit tests for Python integration** section (C++ unit tests are part of `pytest`-based test module as of #14117) cc @wjones127 @jorisvandenbossche Authored-by: Alenka Frim <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

…14435) As #14117 is merged separate run of `ctest` for `PyArrow C++` tests is not needed anymore and is removed in this PR. Authored-by: Alenka Frim <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

@wjones127

…pers/python page (#14350) This PR updates Windows section of the Python Development page. Main changes: - use Python 3.10 (also in instructions for Linux/MacOs) - definition of `PATH` not needed as Python doesn't search in `PATH` for dlls anymore ([3.8 +](https://bugs.python.org/issue43173)) - use `CONDA_PREFIX` to define `ARROW_HOME` as in other parts of the docs - remove **Running C++ unit tests for Python integration** section (C++ unit tests are part of `pytest`-based test module as of #14117) cc @wjones127 @jorisvandenbossche Authored-by: Alenka Frim <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

…ke build system for the MATLAB interface (#37784) ### Rationale for this change This pull request removes `GoogleTest` support from the CMake build system for the MATLAB interface. 1. `GoogleTest` support adds a lot of additional complexity to the CMake build system for the MATLAB interface, and we currently don't have any standalone C++ tests for the MATLAB interface code. 2. In order to use `GoogleTest` in the MATLAB CI workflows, we are currently relying on building the tests for the Arrow C++ libraries in order to "re-use" the `GoogleTest binaries. This adds additional overhead to the MATLAB CI workflows. 3. If we want to test some internal C++ code for the MATLAB interface in the future, we can instead use a MEX function to call the code from a MATLAB test as suggested by @ kou in #37532 (comment). 4. There is [precedent for testing internal C++ code without GoogleTest for the Python bindings](#14117). 5. On a somewhat related note - removing `GoogleTest` support will help unblock #37773 as discussed in #37773 (comment). ### What changes are included in this PR? 1. Removed the `MATLAB_BUILD_TESTS` flag from the CMake build system for the MATLAB interface since there are no longer any C++ tests for the MATLAB interface to build. 2. Updated the `matlab_build.sh` CI workflow script to avoid building the tests for the Arrow C++ libraries and to no longer call `ctest`. 3. Updated the `README.md` for the MATLAB interface to no longer mention building or running C++ tests. 4. Updated the design document for the MATLAB Interface to no longer mention `GoogleTest` since we may end up testing internal C++ code using MEX function calls from MATLAB instead. 5. Removed placeholder C++ test (i.e. `placeholder_test.cc`). ### Are these changes tested? Yes. The MATLAB CI workflow is passing on all platforms. ### Are there any user-facing changes? Yes. There are no longer any C++ tests for the MATLAB interface. The `MATLAB_BUILD_TESTS` flag has been removed from the CMake build system to reflect this change. If a user supplies a value for `MATLAB_BUILD_TESTS` when building the MATLAB interface, the flag will be ignored by CMake. ### Future Directions 1. Add more developer-focused documentation on how to test C++ code via MEX function calls from MATLAB. ### Notes 1. In the future, we can consider testing internal C++ code using MEX function calls from MATLAB tests as suggested by @ kou in #37532 (comment). Currently, we don't have any C++ tests that need to be adapted to use this approach. 2. Thank you @ sgilmore10 for your help with this pull request! * Closes: #37532 Lead-authored-by: Kevin Gurney <[email protected]> Co-authored-by: Sarah Gilmore <[email protected]> Signed-off-by: Kevin Gurney <[email protected]>

…he CMake build system for the MATLAB interface (apache#37784) ### Rationale for this change This pull request removes `GoogleTest` support from the CMake build system for the MATLAB interface. 1. `GoogleTest` support adds a lot of additional complexity to the CMake build system for the MATLAB interface, and we currently don't have any standalone C++ tests for the MATLAB interface code. 2. In order to use `GoogleTest` in the MATLAB CI workflows, we are currently relying on building the tests for the Arrow C++ libraries in order to "re-use" the `GoogleTest binaries. This adds additional overhead to the MATLAB CI workflows. 3. If we want to test some internal C++ code for the MATLAB interface in the future, we can instead use a MEX function to call the code from a MATLAB test as suggested by @ kou in apache#37532 (comment). 4. There is [precedent for testing internal C++ code without GoogleTest for the Python bindings](apache#14117). 5. On a somewhat related note - removing `GoogleTest` support will help unblock apache#37773 as discussed in apache#37773 (comment). ### What changes are included in this PR? 1. Removed the `MATLAB_BUILD_TESTS` flag from the CMake build system for the MATLAB interface since there are no longer any C++ tests for the MATLAB interface to build. 2. Updated the `matlab_build.sh` CI workflow script to avoid building the tests for the Arrow C++ libraries and to no longer call `ctest`. 3. Updated the `README.md` for the MATLAB interface to no longer mention building or running C++ tests. 4. Updated the design document for the MATLAB Interface to no longer mention `GoogleTest` since we may end up testing internal C++ code using MEX function calls from MATLAB instead. 5. Removed placeholder C++ test (i.e. `placeholder_test.cc`). ### Are these changes tested? Yes. The MATLAB CI workflow is passing on all platforms. ### Are there any user-facing changes? Yes. There are no longer any C++ tests for the MATLAB interface. The `MATLAB_BUILD_TESTS` flag has been removed from the CMake build system to reflect this change. If a user supplies a value for `MATLAB_BUILD_TESTS` when building the MATLAB interface, the flag will be ignored by CMake. ### Future Directions 1. Add more developer-focused documentation on how to test C++ code via MEX function calls from MATLAB. ### Notes 1. In the future, we can consider testing internal C++ code using MEX function calls from MATLAB tests as suggested by @ kou in apache#37532 (comment). Currently, we don't have any C++ tests that need to be adapted to use this approach. 2. Thank you @ sgilmore10 for your help with this pull request! * Closes: apache#37532 Lead-authored-by: Kevin Gurney <[email protected]> Co-authored-by: Sarah Gilmore <[email protected]> Signed-off-by: Kevin Gurney <[email protected]>

github-actions bot added the Component: Python label Sep 14, 2022

AlenkaF force-pushed the cython_test_example branch from 8ff5341 to afe5866 Compare September 20, 2022 12:05

jorisvandenbossche reviewed Sep 21, 2022

View reviewed changes

python/pyarrow/src/python_test.cc Outdated Show resolved Hide resolved

pitrou reviewed Sep 26, 2022

View reviewed changes

python/pyarrow/_pyarrow_cpp_tests.pxd Outdated Show resolved Hide resolved

python/setup.cfg Outdated Show resolved Hide resolved

jorisvandenbossche reviewed Sep 27, 2022

View reviewed changes

python/pyarrow/tests/test_python.py Outdated Show resolved Hide resolved

AlenkaF marked this pull request as ready for review September 27, 2022 10:10

AlenkaF force-pushed the cython_test_example branch from b8bffc2 to ebeee04 Compare September 27, 2022 16:50

AlenkaF requested review from jorisvandenbossche and pitrou September 28, 2022 12:23

pitrou approved these changes Sep 28, 2022

View reviewed changes

AlenkaF added 8 commits October 3, 2022 06:03

First try at adding cython test example to ve run from pytest - not w…

869b352

…orking yet

Make the test simpler - segfaults currently

a6081c8

Add working test for PythonDecimalToString, InferPrecisionAndScale an…

28c926b

…d InferPrecisionAndNegativeScale

Linter corrections

7a0352d

Add other test from group TestInfer*

728158f

Remove GTest dependency from PyArrow, change tests staying in C++ to …

d192a45

…return a Status and run them from Pytest

Linter corrections

03e8f3b

Make the check_error inline lambda function working

1d187e9

pitrou and others added 5 commits October 3, 2022 06:03

Automate C++ test collection

ba5081a

Improve assertions

002bb71

Remove redundant ignore warning from setup.cfg

9204b5c

Change test_python.py to test_cpp_internals.py

1c2d8d6

Remove python/pyarrow/src/util/ folder as it is not needed anymore

be21035

AlenkaF force-pushed the cython_test_example branch from 7e4e2e6 to be21035 Compare October 3, 2022 04:04

pitrou merged commit 5fd54bc into apache:master Oct 3, 2022

AlenkaF deleted the cython_test_example branch October 3, 2022 16:45

AlenkaF mentioned this pull request Oct 8, 2022

ARROW-17891: [Docs][Python] Update and sync Win section of the developers/python page #14350

Merged

This was referenced Oct 17, 2022

ARROW-18054: [Python][CI] Enable Cython tests on windows wheels #13552

Merged

ARROW-18074: [CI] Running ctest for PyArrow C++ not needed anymore #14435

Merged

kou mentioned this pull request Sep 4, 2023

[CI][Docs][MATLAB] Remove GoogleTest support from the CMake build system for the MATLAB interface #37532

Closed

This was referenced Sep 18, 2023

GH-37770: [MATLAB] Add CSV TableReader and TableWriter MATLAB classes #37773

Merged

GH-37532: [CI][Docs][MATLAB] Remove GoogleTest support from the CMake build system for the MATLAB interface #37784

Merged

kou mentioned this pull request Sep 23, 2023

GH-23221: [C++] Add support for building with Emscripten #37821

Merged

ARROW-17016: [C++][Python] Move Arrow Python C++ tests into Cython #14117

ARROW-17016: [C++][Python] Move Arrow Python C++ tests into Cython #14117

Uh oh!

Conversation

AlenkaF commented Sep 14, 2022 • edited by pitrou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 14, 2022

Uh oh!

github-actions bot commented Sep 14, 2022

Uh oh!

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlenkaF commented Sep 27, 2022

Uh oh!

pitrou commented Sep 27, 2022

Uh oh!

AlenkaF commented Sep 27, 2022

Uh oh!

jorisvandenbossche commented Sep 27, 2022

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou commented Sep 28, 2022

Uh oh!

github-actions bot commented Sep 28, 2022

Uh oh!

AlenkaF commented Sep 29, 2022

Uh oh!

pitrou commented Sep 29, 2022

Uh oh!

jorisvandenbossche commented Sep 29, 2022

Uh oh!

AlenkaF commented Sep 29, 2022

Uh oh!

AlenkaF commented Oct 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jorisvandenbossche commented Oct 3, 2022

Uh oh!

github-actions bot commented Oct 3, 2022

Uh oh!

pitrou commented Oct 3, 2022

Uh oh!

AlenkaF commented Oct 3, 2022

Uh oh!

ursabot commented Oct 4, 2022

Uh oh!

ursabot commented Oct 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AlenkaF commented Sep 14, 2022 •

edited by pitrou

Loading

AlenkaF commented Oct 3, 2022 •

edited

Loading