TST: run array-api-tests in CI

[lucascolley]

No description provided.

[lucascolley]

This is a bit hacky. I wonder whether I could create a GitHub Action to automate the patching for a given array-api-tests version.

[mdhaber]

That would be very cool!
I'll make you a maintainer so CI runs for you.
Will there be a way to skip tests? There will probably be a few that are hard to get working (e.g. if there are some with dlpack).

[lucascolley]

yep, https://github.com/numpy/numpy/blob/main/tools/ci/array-api-xfails.txt

[mdhaber]

OK, ran CI and made you a collaborator so you can play around with this. Thanks!

[lucascolley]

thanks!

[lucascolley]

alright, modulo a few hiccups in array-api-tests, only 19 failures currently!

FAILED array_api_tests/test_array_object.py::test_getitem_masking - TypeError: The array_api_strict namespace does not support the dtype 'object'
FAILED array_api_tests/test_array_object.py::test_setitem_masking - TypeError: The array_api_strict namespace does not support the dtype 'object'
FAILED array_api_tests/test_linalg.py::test_tensordot - ExceptionGroup: Hypothesis found 3 distinct failures. (3 sub-exceptions)
FAILED array_api_tests/test_manipulation_functions.py::test_repeat - TypeError: repeats must be an int or array
FAILED array_api_tests/test_manipulation_functions.py::test_unstack - assert False
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_sqrt - AssertionError: out=(inf+0j), but should be roughly sqrt(x)=(3968032412.8978596+1643612645.4244113j) [sqrt()]
FAILED array_api_tests/test_set_functions.py::test_unique_all - ExceptionGroup: Hypothesis found 4 distinct failures. (4 sub-exceptions)
FAILED array_api_tests/test_set_functions.py::test_unique_counts - ExceptionGroup: Hypothesis found 4 distinct failures. (4 sub-exceptions)
FAILED array_api_tests/test_set_functions.py::test_unique_inverse - ExceptionGroup: Hypothesis found 4 distinct failures. (4 sub-exceptions)
FAILED array_api_tests/test_set_functions.py::test_unique_values - TypeError: The array_api_strict namespace does not support the dtype 'object'
FAILED array_api_tests/test_signatures.py::test_array_method_signature[__dlpack__] - AssertionError: Argument 'stream' missing from signature
FAILED array_api_tests/test_special_cases.py::test_empty_arrays[mean] - TypeError: Only floating-point dtypes are allowed in __truediv__
FAILED array_api_tests/test_special_cases.py::test_empty_arrays[std] - TypeError: Only floating-point dtypes are allowed in __truediv__
FAILED array_api_tests/test_special_cases.py::test_empty_arrays[var] - TypeError: Only floating-point dtypes are allowed in __truediv__
FAILED array_api_tests/test_special_cases.py::test_nan_propagation[mean] - TypeError: Only floating-point dtypes are allowed in __truediv__
FAILED array_api_tests/test_special_cases.py::test_nan_propagation[std] - TypeError: Only floating-point dtypes are allowed in __truediv__
FAILED array_api_tests/test_special_cases.py::test_nan_propagation[var] - TypeError: Only floating-point dtypes are allowed in __truediv__
FAILED array_api_tests/test_statistical_functions.py::test_cumulative_sum - ValueError: operands could not be broadcast together with remapped shapes [original->remapped]: (0,)  and requested shape (1,)
FAILED array_api_tests/test_statistical_functions.py::test_mean - TypeError: Only floating-point dtypes are allowed in __truediv__

[mdhaber]

What's the easiest way to run these locally? I'd like to be able to run them in my IDE and use the debugger.

[lucascolley]

What's the easiest way to run these locally? I'd like to be able to run them in my IDE and use the debugger.

wait for the pixi PR to merge, then I'll set it up so you can do pixi run xp-tests

[mdhaber]

> (marray) C:\Users\matth\Desktop\marray\array-api-tests>pip install -e .
ERROR: file:///C:/Users/matth/Desktop/marray/array-api-tests does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

[lucascolley]

array-api-tests does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

I suppose that is a feature request for array-api-tests then. Can you workaround it for now by changing to relative imports?

I also installed some other required packages like nditer using mamba to get other tests running.)

Which packages did you add? We should add them to the xp-tests feature (and they should probably also be listed in https://github.com/data-apis/array-api-tests/blob/master/requirements.txt).

[mdhaber]

hypothesis, pytest-json-report, and ndindex

[lucascolley]

they are already part of the xp-tests feature which is included in the dev environment - which environment were you in?

[mdhaber]

I created my own marray mamba environment before there was a proper environment.yml, I think. So that one.

[lucascolley]

Sorry, I might not be understanding the problem. For pixi run -e=dev xp-tests, all of those dependencies should be there. If you want to run other commands in that environment you should first run pixi shell -e dev. There shouldn't be any need to use a mamba env.

[mdhaber]

This is all about running the tests in my IDE, not using pixi from the command line. Was that the missing piece?
My IDE needs to know what environment to be in, and for a long time, it's been my own marray environment.

[lucascolley]

My IDE needs to know what environment to be in, and for a long time, it's been my own marray environment.

Right - can you point your IDE to .pixi/envs/dev/?

[mdhaber]

It doesn't list pixi specifically as a type of environment I can add. Should one of the other options work?

Also, I tried adding these to array-api-strict-skips.txt, but I don't think they were skipped:

# marray does not support `linalg` extension
array_api_tests/test_linalg.py::test_tensordot

# marray does not correctly define `unique_` functions or `__dlpack__ yet`
array_api_tests/test_set_functions.py::test_unique_all
array_api_tests/test_set_functions.py::test_unique_counts
array_api_tests/test_set_functions.py::test_unique_inverse
array_api_tests/test_set_functions.py::test_unique_values
array_api_tests/test_signatures.py::test_array_method_signature[__dlpack__]

Where do these go? (Could you add them, and I'll follow in the future?)

[lucascolley]

sounds like you are looking for https://pixi.sh/dev/ide_integration/pycharm/

[lucascolley]

marray does not support linalg extension
array_api_tests/test_linalg.py::test_tensordot

tensordot is actually in both the main API standard and the linalg extension: https://data-apis.org/array-api/2023.12/API_specification/linear_algebra_functions.html

[mdhaber]

Oops, and yeah, that is in the code now and should work. I'll check it out.

[lucascolley]

just 6 failures now!

[mdhaber]

Hah yeah well skipping tests will do that : ) But yeah fixing mean/var/std helped.

[mdhaber]

Merging gh-56 should get all the tests to pass.

[lucascolley]

1 failure left for sqrt

EDIT: on macOS only? I can reproduce it locally on macOS

[lucascolley]

E                   AssertionError: out=(inf+0j), but should be roughly sqrt(x)=(3968032412.8978596+1643612645.4244113j) [sqrt()]
E                     x=(1.3043818701608976e+19+1.304381650258572e+19j)
E                   assert False
E                    +  where False = isclose_complex((inf+0j), (3968032412.8978596+1643612645.4244113j), 3.4028234663852886e+38)
E                   Falsifying example: test_sqrt(
E                       x=MArray(
E                           Array(1.3043819e+19+1.3043817e+19j, dtype=array_api_strict.complex64),
E                           Array(False, dtype=array_api_strict.bool)
E                       ),
E                   )

the reference implementation is cmath.sqrt

[lucascolley]

hmm, 3.4028234663852886e+38 seems close to the limit for single precision, so maybe NumPy (via array-api-strict) is overflowing where cmath isn't? EDIT: that's just the maximum value. Anyway, seems unlikely to be an marray bug

[lucascolley]

okay, in it goes, nice work!