TEST: 1.8.x + blas variants by h-vetinari · Pull Request #199 · conda-forge/scipy-feedstock

h-vetinari · 2022-02-06T03:36:23Z

Following the same scheme as #172, #153 & #130. Should not be merged for the same reasons as #153.

conda-forge-linter · 2022-02-06T03:36:26Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

h-vetinari · 2022-02-06T11:56:43Z

Scipy 1.8.0

TL;DR: A quarter** of relevant combinations is failing in some way, and that's not even counting pypy (or the propack segfaults)

From 30 failures out of 96 runs for 1.7.3, we're now at 6 segfaults and 15 failures out of 66 CI runs (dropping cpython 3.7 & pypy 3.7, adding accelerate for osx, switching split of avx512 presence/absence from blis to mkl)

** discounting the 6 PPC failures, which are due to emulation

Notable:

OSX now has a libblas=*=*accelerate flavour
pythran on windows was deactivated for 1.8.0 (but looks to be fixed soonish by re-enable pythran on windows; test skip hyghiene #201)
status of pypy unclear because pypy 3.8 not yet migrated in conda-forge

The good news:

Aside from the completely new accelerate flavour, all failures are pre-existing
All the blis failures are gone

The bad news:

segfaults with accelerate - these only show up on x86, but it's almost certain that they're relevant for arm as well
propack tests were disabled upstream (i.e. the underlying issues haven't been solved yet), see WIP: Re-enable PROPACK tests #200.
new failure for MKL+AVX512; this was skipped for 1.8.0 but I've opened BUG: test failure in test_x0_equals_Mb with bicgstab scipy/scipy#15533 because it should be fixed IMO, especially as such processors are becoming more and more common (in fact, it's getting harder and harder to purposefully catch a non-AVX512 windows CI agent on azure)

Details

lib	for 1.7.3	now	updated version	updated build
`scipy`	`1.7.3`	`1.8.0`	X
`numpy`	`1.21.4`	`1.22.2`	X
`libblas`	`3.9.0-12`	`3.9.0-13`		X
`blis`	`0.8.1-1`	`0.8.1-1`
`openblas`	`0.3.18-pthreads-0`	`0.3.18-pthreads-0`
`mkl`	`2021.4.0-729`	`2022.0.1-803`	X
`netlib`	`3.9.0-5`	`3.9.0-5`
`pypy`	`7.3.7-3`	not used
`pythran`	`0.11.0-0`	`0.11.0-0`

variant	1.7.3	1.8.0
linux + x86 + blis	1 test failure for non-avx512	passes 🥳
linux + x86 + mkl + avx512	✔️	1 test failure; scipy/scipy#15533
linux + ppc	test suite disabled due to emulation problems	same
osx + accelerate	-	segfault 💥
osx + x86 + blis + cpython	1 test failure	passes 🥳
osx + x86 + mkl + pypy	2 test failures	not built, no pypy 3.8
win + mkl + avx512	✔️	1 test failure; scipy/scipy#15533
win + openblas	3 failures	same 3 failures remain
win + pypy	hangs indefinitely	not built, no pypy 3.8

variant	accelerate	blis	mkl w/ avx512	mkl w/o avx512	netlib	openblas	sum**
linux / x86	➖	✔️	❌	✔️	✔️	✔️	3F
linux / aarch	➖	➖	➖	➖	✔️	✔️	-
linux / ppc64le	➖	➖	➖	➖	✖️	✖️	6F
osx / arm	💥	➖	➖	➖	✔️	✔️	3S
osx / x86	💥	✔️	➖	✔️	✔️	✔️	3S
win / x86	➖	✔️	❌	✔️	✔️	❌	6F
sum**	6S	-	6F	-	3F	6F	6S/15F

** sum of Segfaults (S), Failures (F), resp. Timeouts (T); out of a total of 66 CI combinations being tested

Build logs:
Azure

(linux / windows) + mkl + avx512: 1 failure

=================================== FAILURES ===================================
_________________________ test_x0_equals_Mb[bicgstab] __________________________
[...]/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/tests/test_iterative.py:538: in test_x0_equals_Mb
    assert_equal(info, 0)
E   AssertionError: 
E   Items are not equal:
E    ACTUAL: -11
E    DESIRED: 0
        A          = <10x10 sparse matrix of type '<class 'numpy.complex64'>'
	with 19 stored elements in Compressed Sparse Row format>
        b          = array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])
        case       = <nonsymposdef>
        info       = -11
        solver     = <function bicgstab at 0x7fc6545a4ee0>
        sup        = <numpy.testing._private.utils.suppress_warnings object at 0x7fc62555d040>
        tol        = 1e-08
        x          = array([0.        +0.j, 0.50000358+0.j, 1.25016941+0.j, 2.12653797+0.j,
       3.0674186 +0.j, 4.03716976+0.j, 5.0158146 +0.j, 6.00288608+0.j,
       7.        +0.j, 8.        +0.j])
        x0         = 'Mb'

Also, at the end of the test suite, some (seemingly delayed) log output appears that is perhaps relevant:


   Normal return from subroutine COBYLA

   NFVALS =   50   F = 2.485185E+01    MAXCV = 1.999965E-10
   X = 4.955358E+00   6.666553E-01

 NNLS quitting on iteration count.

windows + openblas: 3 failures

================================== FAILURES ===================================
________________________ TestFBLAS2Simple.test_syr_her ________________________
..\_test_env\lib\site-packages\scipy\linalg\tests\test_blas.py:316: in test_syr_her
    assert_allclose(f(1.0, z), resz, rtol=rtol)
E   AssertionError:
E   Not equal to tolerance rtol=1e-07, atol=0
E
E   Mismatched elements: 13 / 16 (81.2%)
E   Max absolute difference: 113.
E   Max relative difference: 3.79473319
E    x: array([[ -3.  +4.j,  -7. +16.j, -13. +82.j,   0.  +0.j],
E          [  0.  +0.j,  -9. +38.j, -15.+112.j,   0.  +0.j],
E          [ -5. +10.j, -20. +82.j,   0.  +0.j,   0.  +0.j],
E          [ -7. +24.j, -11. +52.j,   0.  +0.j,   0.  +0.j]], dtype=complex64)
E    y: array([[ -3.  +4.j,  -5. +10.j,  -7. +16.j,  -9. +22.j],
E          [  0.  +0.j,  -7. +24.j,  -9. +38.j, -11. +52.j],
E          [  0.  +0.j,   0.  +0.j, -11. +60.j, -13. +82.j],
E          [  0.  +0.j,   0.  +0.j,   0.  +0.j, -15.+112.j]])
        a          = array([[ 1.,  2.,  3.,  4.],
       [ 0.,  4.,  6.,  8.],
       [ 0.,  0.,  9., 12.],
       [ 0.,  0.,  0., 16.]])
        b          = array([[ 3.,  6.,  9., 12.],
       [ 0., 12., 18., 24.],
       [ 0.,  0., 27., 36.],
       [ 0.,  0.,  0., 48.]])
        f          = <fortran object>
        p          = 'c'
        rehz       = array([[  5.+0.j,  11.+2.j,  17.+4.j,  23.+6.j],
       [  0.+0.j,  25.+0.j,  39.+2.j,  53.+4.j],
       [  0.+0.j,   0.+0.j,  61.+0.j,  83.+2.j],
       [  0.+0.j,   0.+0.j,   0.+0.j, 113.+0.j]])
        rehz_reverse = array([[113.+0.j,  83.-2.j,  53.-4.j,  23.-6.j],
       [  0.+0.j,  61.+0.j,  39.-2.j,  17.-4.j],
       [  0.+0.j,   0.+0.j,  25.+0.j,  11.-2.j],
       [  0.+0.j,   0.+0.j,   0.+0.j,   5.+0.j]])
        resx       = array([[ 1.,  2.,  3.,  4.],
       [ 0.,  4.,  6.,  8.],
       [ 0.,  0.,  9., 12.],
       [ 0.,  0.,  0., 16.]])
        resx_reverse = array([[16., 12.,  8.,  4.],
       [ 0.,  9.,  6.,  3.],
       [ 0.,  0.,  4.,  2.],
       [ 0.,  0.,  0.,  1.]])
        resz       = array([[ -3.  +4.j,  -5. +10.j,  -7. +16.j,  -9. +22.j],
       [  0.  +0.j,  -7. +24.j,  -9. +38.j, -11. +52.j],
       [  0.  +0.j,   0.  +0.j, -11. +60.j, -13. +82.j],
       [  0.  +0.j,   0.  +0.j,   0.  +0.j, -15.+112.j]])
        resz_reverse = array([[-15.+112.j, -13. +82.j, -11. +52.j,  -9. +22.j],
       [  0.  +0.j, -11. +60.j,  -9. +38.j,  -7. +16.j],
       [  0.  +0.j,   0.  +0.j,  -7. +24.j,  -5. +10.j],
       [  0.  +0.j,   0.  +0.j,   0.  +0.j,  -3.  +4.j]])
        rtol       = 1e-07
        self       = <scipy.linalg.tests.test_blas.TestFBLAS2Simple object at 0x000002B8402EAF70>
        w          = array([0.+0.j, 1.+2.j, 0.+0.j, 0.+0.j, 3.+4.j, 0.+0.j, 0.+0.j, 5.+6.j,
       0.+0.j, 0.+0.j, 7.+8.j, 0.+0.j])
        x          = array([1., 2., 3., 4.])
        y          = array([0. , 0.5, 1. , 1.5, 2. , 2.5, 3. , 3.5, 4. , 4.5, 5. , 5.5, 6. ,
       6.5, 7. , 7.5, 8. ])
        z          = array([1.+2.j, 3.+4.j, 5.+6.j, 7.+8.j])
_________________________________ test_simple _________________________________
..\_test_env\lib\site-packages\scipy\linalg\tests\test_decomp_ldl.py:64: in test_simple
    assert_allclose(u.dot(d).dot(u.T), a, atol=spacing(1000.), rtol=0)
E   AssertionError:
E   Not equal to tolerance rtol=0, atol=1.13687e-13
E
E   Mismatched elements: 8 / 16 (50%)
E   Max absolute difference: 5.58902689
E   Max relative difference: 9.75886426
E    x: array([[-0.39    -0.71j    ,  5.14    -0.64j    , -9.664201-2.000677j,
E            8.200375-1.876397j],
E          [ 5.14    -0.64j    ,  8.86    +1.81j    , -3.52    +0.58j    ,...
E    y: array([[-0.39-0.71j,  5.14-0.64j, -7.86-2.96j,  3.8 +0.92j],
E          [ 5.14-0.64j,  8.86+1.81j, -3.52+0.58j,  5.32-1.59j],
E          [-7.86-2.96j, -3.52+0.58j, -2.83-0.03j, -1.54-2.86j],
E          [ 3.8 +0.92j,  5.32-1.59j, -1.54-2.86j, -0.56+0.12j]])
        a          = array([[-0.39-0.71j,  5.14-0.64j, -7.86-2.96j,  3.8 +0.92j],
       [ 5.14-0.64j,  8.86+1.81j, -3.52+0.58j,  5.32-1.59...    [-7.86-2.96j, -3.52+0.58j, -2.83-0.03j, -1.54-2.86j],
       [ 3.8 +0.92j,  5.32-1.59j, -1.54-2.86j, -0.56+0.12j]])
        b          = array([[ 5., 10.,  1., 18.],
       [10.,  2., 11.,  1.],
       [ 1., 11., 19.,  9.],
       [18.,  1.,  9.,  0.]])
        c          = array([[ 52.,  97., 112., 107.,  50.],
       [ 97., 114.,  89.,  98.,  13.],
       [112.,  89.,  64.,  33.,   6.],
       [107.,  98.,  33.,  60.,  73.],
       [ 50.,  13.,   6.,  73.,  77.]])
        d          = array([[-3.7989066 -3.32560972j,  0.        +0.j        ,
         0.        +0.j        ,  0.        +0.j        ],
 ...,
       [ 0.        +0.j        ,  0.        +0.j        ,
         0.        +0.j        ,  8.86      +1.81j      ]])
        e          = array([[-1.36+0.j  ,  0.  +0.j  ,  0.  +0.j  ,  0.  +0.j  ],
       [ 1.58-0.9j , -8.87+0.j  ,  0.  +0.j  ,  0.  +0.j ...    [ 2.21+0.21j, -1.84+0.03j, -4.63+0.j  ,  0.  +0.j  ],
       [ 3.91-1.5j , -1.78-1.18j,  0.11-0.11j, -1.84+0.j  ]])
        l          = array([[ 1.        +0.j        ,  0.        +0.j        ,
         0.        +0.j        ,  0.        +0.j        ],
 ...,
       [ 0.44255824+0.19364837j, -0.4822823 +0.01498936j,
        -0.10708219-0.31567809j,  1.        +0.j        ]])
        p          = array([3, 0, 2, 1], dtype=int64)
        u          = array([[ 0.        +0.j        ,  1.        +0.j        ,
         0.        +0.j        ,  0.54272846-0.18310818j],
 ...,
       [ 1.        +0.j        ,  0.50671191+0.06166567j,
        -0.79529754+0.31777367j,  0.54120356-0.29002014j]])
        x          = array([[ 2.,  2., -4.,  0.,  4.],
       [ 2., -2., -2., 10., -8.],
       [-4., -2.,  6., -8., -4.],
       [ 0., 10., -8.,  6., -6.],
       [ 4., -8., -4., -6., 10.]])
_______________________ test_ldl_type_size_combinations _______________________
..\_test_env\lib\site-packages\scipy\linalg\tests\test_decomp_ldl.py:134: in test_ldl_type_size_combinations
    assert_allclose(l.dot(d1).dot(l.T), x, rtol=rtol, err_msg=msg2)
E   AssertionError:
E   Not equal to tolerance rtol=0.0001, atol=0
E   Sym failed for size: 30, dtype: <class 'numpy.complex64'>
E   Mismatched elements: 414 / 900 (46%)
E   Max absolute difference: 0.00153339
E   Max relative difference: 0.00138451
E    x: array([[4.286979e+04+8.179955e-01j, 6.061730e-01+1.105522e+00j,
E           1.237040e+00+1.046835e+00j, 7.228262e-01+9.566016e-01j,
E           4.196392e-01+5.735543e-01j, 1.478078e+00+9.985576e-01j,...
E    y: array([[4.286979e+04+8.179955e-01j, 6.061730e-01+1.105522e+00j,
E           1.237040e+00+1.046835e+00j, 7.228262e-01+9.566016e-01j,
E           4.196392e-01+5.735544e-01j, 1.478078e+00+9.985576e-01j,...
        complex_dtypes = [<class 'numpy.complex64'>, <class 'numpy.complex128'>]
        d1         = array([[42869.793+8.1799555e-01j,     0.   +0.0000000e+00j,
            0.   +0.0000000e+00j,     0.   +0.0000000e+00j...00e+00j,     0.   +0.0000000e+00j,
            0.   +0.0000000e+00j, 42870.035+1.4342686e+00j]],
      dtype=complex64)
        d2         = array([[42869.793+8.1678456e-01j,     0.   +0.0000000e+00j,
            0.   +0.0000000e+00j,     0.   +0.0000000e+00j...00e+00j,     0.   +0.0000000e+00j,
            0.   +0.0000000e+00j, 42870.035+1.4343451e+00j]],
      dtype=complex64)
        dtype      = <class 'numpy.complex64'>
        l          = array([[1.00000000e+00+0.00000000e+00j, 0.00000000e+00+0.00000000e+00j,
        0.00000000e+00+0.00000000e+00j, 0.0000...5e-05+1.72517612e-05j,
        2.61453479e-05+3.41489576e-05j, 1.00000000e+00+0.00000000e+00j]],
      dtype=complex64)
        msg        = "Failed for size: 750, dtype: <class 'numpy.float64'>"
        msg1       = "Her failed for size: 30, dtype: <class 'numpy.complex64'>"
        msg2       = "Sym failed for size: 30, dtype: <class 'numpy.complex64'>"
        n          = 30
        p          = array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
       17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29], dtype=int64)
        real_dtypes = [<class 'numpy.float32'>, <class 'numpy.float64'>]
        rtol       = 0.0001
        sizes      = [30, 750]
        u          = array([[1.00000000e+00+0.00000000e+00j, 1.41365481e-05+2.57585070e-05j,
        2.88502779e-05+2.43929080e-05j, 1.6857...0e+00+0.00000000e+00j,
        0.00000000e+00+0.00000000e+00j, 1.00000000e+00+0.00000000e+00j]],
      dtype=complex64)
        x          = array([[4.2869793e+04+8.1799555e-01j, 6.0617304e-01+1.1055222e+00j,
        1.2370398e+00+1.0468354e+00j, 7.2282618e-0...100193e-01+7.3975587e-01j,
        1.1208339e+00+1.4640008e+00j, 4.2870035e+04+1.4343451e+00j]],
      dtype=complex64)

rgommers · 2022-02-07T18:38:14Z

Is there someone who cares about the re-enabled Accelerate who can isolate the problems and file new issues?

Given that everything gets built with the same g77 ABI and Netlib/OpenBLAS work fine it seems clear that there's an Accelerate issue here. I'm happy to forward the bug reports to Apple to see if they want to prioritize fixes.

h-vetinari · 2022-02-08T00:21:01Z

Is there someone who cares about the re-enabled Accelerate who can isolate the problems and file new issues?

Given that everything gets built with the same g77 ABI and Netlib/OpenBLAS work fine it seems clear that there's an Accelerate issue here. I'm happy to forward the bug reports to Apple to see if they want to prioritize fixes.

The accelerate stuff in conda-forge is very recent, and not just plain vanilla either - we're using BLAS from accelerate and LAPACK from netlib (because LAPACK from accelerate is too old). I think it's possible that there are still issues on our side after this relatively short incubation time, and maybe not yet the time to bug Apple.

CC @isuruf

ngam · 2022-02-27T22:49:14Z

Is there someone who cares about the re-enabled Accelerate who can isolate the problems and file new issues?

Given that everything gets built with the same g77 ABI and Netlib/OpenBLAS work fine it seems clear that there's an Accelerate issue here. I'm happy to forward the bug reports to Apple to see if they want to prioritize fixes.

@rgommers I am happy to help with this if you give me a brief overview (additional context besides the thread from previous convos) on things to look for. I was pushing for wider adoption of Accelerate a few weeks ago, but I have since concluded that it would likely not be worth it (largely in alignment with the scipy maintainers' argument). I recently started using scipy for a side project and I have been super happy/impressed by some accelerations I got via FITPACK, so I am happy to do what I can in return and learn more about the workings of scipy anyway :)

Let me know what would be most helpful here

ngam · 2022-02-27T22:54:20Z

@h-vetinari my impression (via @isuruf) was that the segfaults were only on osx-arm. Just to confirm: are they also present on osx-64?

h-vetinari · 2022-02-27T23:56:49Z

my impression was that the segfaults were only on osx-arm. Just to confirm: are they also present on osx-64?

It's the other way around: osx-64 builds are segfaulting, but that's very likely because that's the only osx flavour we have CI for. I expect a very high likelihood that the osx-arm64 builds are also segfaulting with the current scipy-builds.

…nda-forge-pinning 2023.01.26.16.59.55

conda-forge-webservices · 2023-01-27T04:20:38Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

h-vetinari · 2023-01-27T08:05:01Z

Scipy 1.8.1

I had originally picked this up in mid 2022, but despaired based on the number of different failures. Now picking this up again to have a reasonable chain record of the changes from here to 1.10.

TL;DR: Some persistent failures, but less segfaults; cautious improvements overall

From 6 segfaults and 15 failures out of 66 runs for 1.8.0, we're now at 0 segfaults and 40 failures out of 110 CI runs (re-added PyPy)

The good news:

No more segfaults for osx + accelerate
Even though I didn't keep a record, the results are not as bad as ~half a year ago.

The bad news:

uniform test failures for 7 tests related to TestCOO
additional test failures appeared

ALL builds fail the following tests:

sparse/tests/test_base.py::TestCOO::test_reshape_copy
sparse/tests/test_base.py::TestCOONonCanonical::test_reshape_copy
sparse/tests/test_base.py::Test64Bit::test_resiliency_all_32[TestCOO-test_reshape_copy]
sparse/tests/test_base.py::Test64Bit::test_resiliency_limit_10[TestCOO-test_reshape_copy]
sparse/tests/test_base.py::Test64Bit::test_resiliency_all_64[TestCOO-test_reshape_copy]
sparse/tests/test_base.py::Test64Bit::test_resiliency_random[TestCOO-test_reshape_copy]
sparse/tests/test_base.py::Test64Bit::test_no_64[TestCOO-test_reshape_copy]

To avoid a completely red matrix below, I'm discounting those failures from each build

detailed stacktrace

=================================== FAILURES ===================================
__________________________ TestCOO.test_reshape_copy ___________________________
[gw1] linux -- Python 3.10.8 $PREFIX/bin/python
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4304: in test_reshape_copy
    assert_(y.data is x.data)
E   AssertionError
        arr        = [[0, 10, 0, 0], [0, 0, 0, 0], [0, 20, 30, 40]]
        new_shape  = (2, 6)
        self       = <scipy.sparse.tests.test_base.TestCOO object at 0x7fdc18c6dc60>
        x          = <3x4 sparse matrix of type '<class 'numpy.int64'>'
with 4 stored elements in COOrdinate format>
        y          = <2x6 sparse matrix of type '<class 'numpy.int64'>'
with 4 stored elements in COOrdinate format>
_________ Test64Bit.test_resiliency_all_32[TestCOO-test_reshape_copy] __________
[gw0] linux -- Python 3.10.8 $PREFIX/bin/python
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4927: in test_resiliency_all_32
    self._check_resiliency(cls, method_name, fixed_dtype=np.int32)
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        method_name = 'test_reshape_copy'
        self       = <scipy.sparse.tests.test_base.Test64Bit object at 0x7f3589f45330>
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4912: in _check_resiliency
    check(cls, method_name)
        check      = <function Test64Bit._check_resiliency.<locals>.check at 0x7f357c49beb0>
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        kw         = {'fixed_dtype': <class 'numpy.int32'>}
        method_name = 'test_reshape_copy'
        self       = <scipy.sparse.tests.test_base.Test64Bit object at 0x7f3589f45330>
<decorator-gen-1873>:2: in check
    ???
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        method_name = 'test_reshape_copy'
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:160: in deco
    return func(*a, **kw)
        a          = (<class 'scipy.sparse.tests.test_base.TestCOO'>, 'test_reshape_copy')
        backup     = [(<module 'scipy.sparse._bsr' from '[...]/lib/python3.10/site-packages/scipy/sparse/_dok.py'>, 'get_index_dtype', <function get_index_dtype at 0x7f3595f4b760>), ...]
        downcast_maxval = None
        func       = <function Test64Bit._check_resiliency.<locals>.check at 0x7f357c49bd00>
        kw         = {}
        mod        = <module 'scipy.sparse._construct' from '[...]/lib/python3.10/site-packages/scipy/sparse/_construct.py'>
        modules    = [<module 'scipy.sparse._bsr' from '[...]/lib/python3.10/site-packages/scipy/sparse/_dok.py'>, ...]
        name       = 'get_index_dtype'
        new_get_index_dtype = <function with_64bit_maxval_limit.<locals>.new_get_index_dtype at 0x7f357c49b880>
        oldfunc    = <function get_index_dtype at 0x7f3595f4b760>
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4907: in check
    getattr(instance, method_name)()
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        instance   = <scipy.sparse.tests.test_base.TestCOO object at 0x7f357c347670>
        method_name = 'test_reshape_copy'
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4304: in test_reshape_copy
    assert_(y.data is x.data)
E   AssertionError
        arr        = [[0, 10, 0, 0], [0, 0, 0, 0], [0, 20, 30, 40]]
        new_shape  = (2, 6)
        self       = <scipy.sparse.tests.test_base.TestCOO object at 0x7f357c347670>
        x          = <3x4 sparse matrix of type '<class 'numpy.int64'>'
with 4 stored elements in COOrdinate format>
        y          = <2x6 sparse matrix of type '<class 'numpy.int64'>'
with 4 stored elements in COOrdinate format>
____________________ TestCOONonCanonical.test_reshape_copy _____________________
[gw1] linux -- Python 3.10.8 $PREFIX/bin/python
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4304: in test_reshape_copy
    assert_(y.data is x.data)
E   AssertionError
        arr        = [[0, 10, 0, 0], [0, 0, 0, 0], [0, 20, 30, 40]]
        new_shape  = (2, 6)
        self       = <scipy.sparse.tests.test_base.TestCOONonCanonical object at 0x7fdc1896d6f0>
        x          = <3x4 sparse matrix of type '<class 'numpy.int64'>'
with 4 stored elements in COOrdinate format>
        y          = <2x6 sparse matrix of type '<class 'numpy.int64'>'
with 4 stored elements in COOrdinate format>
________ Test64Bit.test_resiliency_limit_10[TestCOO-test_reshape_copy] _________
[gw1] linux -- Python 3.10.8 $PREFIX/bin/python
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4916: in test_resiliency_limit_10
    self._check_resiliency(cls, method_name, maxval_limit=10)
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        method_name = 'test_reshape_copy'
        self       = <scipy.sparse.tests.test_base.Test64Bit object at 0x7fdc185baf80>
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4912: in _check_resiliency
    check(cls, method_name)
        check      = <function Test64Bit._check_resiliency.<locals>.check at 0x7fdc1139ba30>
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        kw         = {'maxval_limit': 10}
        method_name = 'test_reshape_copy'
        self       = <scipy.sparse.tests.test_base.Test64Bit object at 0x7fdc185baf80>
<decorator-gen-512>:2: in check
    ???
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        method_name = 'test_reshape_copy'
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:160: in deco
    return func(*a, **kw)
        a          = (<class 'scipy.sparse.tests.test_base.TestCOO'>, 'test_reshape_copy')
        backup     = [(<module 'scipy.sparse._bsr' from '[...]/lib/python3.10/site-packages/scipy/sparse/_dok.py'>, 'get_index_dtype', <function get_index_dtype at 0x7fdc2813b640>), ...]
        downcast_maxval = None
        func       = <function Test64Bit._check_resiliency.<locals>.check at 0x7fdc1139b880>
        kw         = {}
        mod        = <module 'scipy.sparse._construct' from '[...]/lib/python3.10/site-packages/scipy/sparse/_construct.py'>
        modules    = [<module 'scipy.sparse._bsr' from '[...]/lib/python3.10/site-packages/scipy/sparse/_dok.py'>, ...]
        name       = 'get_index_dtype'
        new_get_index_dtype = <function with_64bit_maxval_limit.<locals>.new_get_index_dtype at 0x7fdc1139b250>
        oldfunc    = <function get_index_dtype at 0x7fdc2813b640>
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4907: in check
    getattr(instance, method_name)()
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        instance   = <scipy.sparse.tests.test_base.TestCOO object at 0x7fdc119b4970>
        method_name = 'test_reshape_copy'
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4304: in test_reshape_copy
    assert_(y.data is x.data)
E   AssertionError
        arr        = [[0, 10, 0, 0], [0, 0, 0, 0], [0, 20, 30, 40]]
        new_shape  = (2, 6)
        self       = <scipy.sparse.tests.test_base.TestCOO object at 0x7fdc119b4970>
        x          = <3x4 sparse matrix of type '<class 'numpy.int64'>'
with 4 stored elements in COOrdinate format>
        y          = <2x6 sparse matrix of type '<class 'numpy.int64'>'
with 4 stored elements in COOrdinate format>
_________ Test64Bit.test_resiliency_all_64[TestCOO-test_reshape_copy] __________
[gw0] linux -- Python 3.10.8 $PREFIX/bin/python
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4931: in test_resiliency_all_64
    self._check_resiliency(cls, method_name, fixed_dtype=np.int64)
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        method_name = 'test_reshape_copy'
        self       = <scipy.sparse.tests.test_base.Test64Bit object at 0x7f3589c623b0>
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4912: in _check_resiliency
    check(cls, method_name)
        check      = <function Test64Bit._check_resiliency.<locals>.check at 0x7f356da8c310>
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        kw         = {'fixed_dtype': <class 'numpy.int64'>}
        method_name = 'test_reshape_copy'
        self       = <scipy.sparse.tests.test_base.Test64Bit object at 0x7f3589c623b0>
<decorator-gen-3865>:2: in check
    ???
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        method_name = 'test_reshape_copy'
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:160: in deco
    return func(*a, **kw)
        a          = (<class 'scipy.sparse.tests.test_base.TestCOO'>, 'test_reshape_copy')
        backup     = [(<module 'scipy.sparse._bsr' from '[...]/lib/python3.10/site-packages/scipy/sparse/_dok.py'>, 'get_index_dtype', <function get_index_dtype at 0x7f3595f4b760>), ...]
        downcast_maxval = None
        func       = <function Test64Bit._check_resiliency.<locals>.check at 0x7f356da8c160>
        kw         = {}
        mod        = <module 'scipy.sparse._construct' from '[...]/lib/python3.10/site-packages/scipy/sparse/_construct.py'>
        modules    = [<module 'scipy.sparse._bsr' from '[...]/lib/python3.10/site-packages/scipy/sparse/_dok.py'>, ...]
        name       = 'get_index_dtype'
        new_get_index_dtype = <function with_64bit_maxval_limit.<locals>.new_get_index_dtype at 0x7f356daebc70>
        oldfunc    = <function get_index_dtype at 0x7f3595f4b760>
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4907: in check
    getattr(instance, method_name)()
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        instance   = <scipy.sparse.tests.test_base.TestCOO object at 0x7f356da11ed0>
        method_name = 'test_reshape_copy'
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4304: in test_reshape_copy
    assert_(y.data is x.data)
E   AssertionError
        arr        = [[0, 10, 0, 0], [0, 0, 0, 0], [0, 20, 30, 40]]
        new_shape  = (2, 6)
        self       = <scipy.sparse.tests.test_base.TestCOO object at 0x7f356da11ed0>
        x          = <3x4 sparse matrix of type '<class 'numpy.int64'>'
with  4 stored elements in COOrdinate format>
        y          = <2x6 sparse matrix of type '<class 'numpy.int64'>'
with 4 stored elements in COOrdinate format>
_________ Test64Bit.test_resiliency_random[TestCOO-test_reshape_copy] __________
[gw1] linux -- Python 3.10.8 $PREFIX/bin/python
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4923: in test_resiliency_random
    self._check_resiliency(cls, method_name, random=True)
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        method_name = 'test_reshape_copy'
        self       = <scipy.sparse.tests.test_base.Test64Bit object at 0x7fdc18738040>
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4912: in _check_resiliency
    check(cls, method_name)
        check      = <function Test64Bit._check_resiliency.<locals>.check at 0x7fdc112de200>
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        kw         = {'random': True}
        method_name = 'test_reshape_copy'
        self       = <scipy.sparse.tests.test_base.Test64Bit object at 0x7fdc18738040>
<decorator-gen-2504>:2: in check
    ???
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        method_name = 'test_reshape_copy'
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:160: in deco
    return func(*a, **kw)
        a          = (<class 'scipy.sparse.tests.test_base.TestCOO'>, 'test_reshape_copy')
        backup     = [(<module 'scipy.sparse._bsr' from '[...]/lib/python3.10/site-packages/scipy/sparse/_dok.py'>, 'get_index_dtype', <function get_index_dtype at 0x7fdc2813b640>), ...]
        downcast_maxval = None
        func       = <function Test64Bit._check_resiliency.<locals>.check at 0x7fdc112de050>
        kw         = {}
        mod        = <module 'scipy.sparse._construct' from '[...]/lib/python3.10/site-packages/scipy/sparse/_construct.py'>
        modules    = [<module 'scipy.sparse._bsr' from '[...]/lib/python3.10/site-packages/scipy/sparse/_dok.py'>, ...]
        name       = 'get_index_dtype'
        new_get_index_dtype = <function with_64bit_maxval_limit.<locals>.new_get_index_dtype at 0x7fdc112ddbd0>
        oldfunc    = <function get_index_dtype at 0x7fdc2813b640>
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4907: in check
    getattr(instance, method_name)()
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        instance   = <scipy.sparse.tests.test_base.TestCOO object at 0x7fdc112acdc0>
        method_name = 'test_reshape_copy'
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4304: in test_reshape_copy
    assert_(y.data is x.data)
E   AssertionError
        arr        = [[0, 10, 0, 0], [0, 0, 0, 0], [0, 20, 30, 40]]
        new_shape  = (2, 6)
        self       = <scipy.sparse.tests.test_base.TestCOO object at 0x7fdc112acdc0>
        x          = <3x4 sparse matrix of type '<class 'numpy.int64'>'
with 4 stored elements in COOrdinate format>
        y          = <2x6 sparse matrix of type '<class 'numpy.int64'>'
with 4 stored elements in COOrdinate format>
_______________ Test64Bit.test_no_64[TestCOO-test_reshape_copy] ________________
[gw0] linux -- Python 3.10.8 $PREFIX/bin/python
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4935: in test_no_64
    self._check_resiliency(cls, method_name, assert_32bit=True)
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        method_name = 'test_reshape_copy'
        self       = <scipy.sparse.tests.test_base.Test64Bit object at 0x7f358998b430>
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4912: in _check_resiliency
    check(cls, method_name)
        check      = <function Test64Bit._check_resiliency.<locals>.check at 0x7f356d0f0700>
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        kw         = {'assert_32bit': True}
        method_name = 'test_reshape_copy'
        self       = <scipy.sparse.tests.test_base.Test64Bit object at 0x7f358998b430>
<decorator-gen-5857>:2: in check
    ???
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        method_name = 'test_reshape_copy'
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:160: in deco
    return func(*a, **kw)
        a          = (<class 'scipy.sparse.tests.test_base.TestCOO'>, 'test_reshape_copy')
        backup     = [(<module 'scipy.sparse._bsr' from '[...]/lib/python3.10/site-packages/scipy/sparse/_dok.py'>, 'get_index_dtype', <function get_index_dtype at 0x7f3595f4b760>), ...]
        downcast_maxval = None
        func       = <function Test64Bit._check_resiliency.<locals>.check at 0x7f356d0f0550>
        kw         = {}
        mod        = <module 'scipy.sparse._construct' from '[...]/lib/python3.10/site-packages/scipy/sparse/_construct.py'>
        modules    = [<module 'scipy.sparse._bsr' from '[...]/lib/python3.10/site-packages/scipy/sparse/_dok.py'>, ...]
        name       = 'get_index_dtype'
        new_get_index_dtype = <function with_64bit_maxval_limit.<locals>.new_get_index_dtype at 0x7f356d0884c0>
        oldfunc    = <function get_index_dtype at 0x7f3595f4b760>
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4907: in check
    getattr(instance, method_name)()
        cls        = <class 'scipy.sparse.tests.test_base.TestCOO'>
        instance   = <scipy.sparse.tests.test_base.TestCOO object at 0x7f356d186140>
        method_name = 'test_reshape_copy'
[...]/lib/python3.10/site-packages/scipy/sparse/tests/test_base.py:4304: in test_reshape_copy
    assert_(y.data is x.data)
E   AssertionError
        arr        = [[0, 10, 0, 0], [0, 0, 0, 0], [0, 20, 30, 40]]
        new_shape  = (2, 6)
        self       = <scipy.sparse.tests.test_base.TestCOO object at 0x7f356d186140>
        x          = <3x4 sparse matrix of type '<class 'numpy.int64'>'
with 4 stored elements in COOrdinate format>
        y          = <2x6 sparse matrix of type '<class 'numpy.int64'>'
with 4 stored elements in COOrdinate format>
__________________________ TestTruncnorm.test_moments __________________________
[gw1] linux -- Python 3.10.8 $PREFIX/bin/python
[...]/lib/python3.10/site-packages/scipy/stats/tests/test_distributions.py:951: in test_moments
    self._test_moments_one_range(-20, -19, [-19.0523439459766628,
        self       = <scipy.stats.tests.test_distributions.TestTruncnorm object at 0x7fdc1646bee0>
[...]/lib/python3.10/site-packages/scipy/stats/tests/test_distributions.py:911: in _test_moments_one_range
    assert_almost_equal(k, k0)
        a          = -20
        b          = -19
        decimal_s  = 7
        expected   = [-19.052343945976663, 0.0027250730180314, -1.983869402262929, 5.871785002828759]
        k          = array(5.8718046)
        k0         = 5.871785002828759
        m          = array(-19.05234395)
        m0         = -19.052343945976663
        s          = array(-1.98386936)
        s0         = -1.983869402262929
        self       = <scipy.stats.tests.test_distributions.TestTruncnorm object at 0x7fdc1646bee0>
        v          = array(0.00272507)
        v0         = 0.0027250730180314
[...]/lib/python3.10/contextlib.py:79: in inner
    return func(*args, **kwds)
        args       = (array(5.8718046), 5.871785002828759)
        func       = <function assert_almost_equal at 0x7fdc285b1a20>
        kwds       = {}
        self       = <contextlib._GeneratorContextManager object at 0x7fdc285be230>
[...]/lib/python3.10/contextlib.py:79: in inner
    return func(*args, **kwds)
        args       = (array(5.8718046), 5.871785002828759, 7, '')
        func       = <function assert_array_almost_equal at 0x7fdc285b1e10>
        kwds       = {}
        self       = <contextlib._GeneratorContextManager object at 0x7fdc285bdb70>
[...]/lib/python3.10/contextlib.py:79: in inner
    return func(*args, **kwds)
E   AssertionError:
E   Arrays are not almost equal to 7 decimals
E
E   Mismatched elements: 1 / 1 (100%)
E   Max absolute difference: 1.9596281e-05
E   Max relative difference: 3.3373635e-06
E    x: array(5.8718046)
E    y: array(5.871785)
        args       = (<function assert_array_almost_equal.<locals>.compare at 0x7fdc10b80700>, array(5.8718046), 5.871785002828759)
        func       = <function assert_array_compare at 0x7fdc285b1c60>
        kwds       = {'err_msg': '', 'header': 'Arrays are not almost equal to 7 decimals', 'precision': 7, 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x7fdc285bdcf0>

Details

lib	for 1.8.0	now	updated version	updated build
`scipy`	`1.8.0`	`1.8.1`	X
`numpy`	`1.22.2`	`1.24.1`	X
`libblas`	`3.9.0-13`	`3.9.0-16`		X
`blis`	`0.8.1-1`	`0.9.0-0`	X
`openblas`	`0.3.18-pthreads-0`	`0.3.21-pthreads-3`	X
`mkl`	`2022.0.1-803`	`2022.1.0-915`	X
`netlib`	`3.9.0-5`	`3.9.0-5`
`pypy`	not used (`7.3.7-3` for 1.7.3)	`7.3.9-7`	X
`pythran`	`0.11.0-0`	`0.12.1-0`	X

variant	1.8.0	1.8.1
linux + x86 + mkl + avx512	1 test failure; scipy/scipy#15533	remains; 1 additional failure
linux + x86 + netlib	✔️	1-2 test failures depending on absence/presence of avx512
linux + x86 + openblas	✔️	2-3 test failures depending on absence/presence of avx512
linux + aarch + openblas	✔️	2 test failures
linux + ppc	test suite disabled due to emulation problems	cautious signs that this might finally get solved 🥳
osx + accelerate	segfault 💥	8 test failures
osx + x86 + mkl + pypy	2 test failures for 1.7.3 (may have been avx512 actually?!)	same as CPython
osx + x86 + mkl	✔️	same two failures as for 1.7.3
osx + x86 + openblas	✔️	3 failures
win + mkl + avx512	1 test failure; scipy/scipy#15533	remains
win + openblas	3 failures	8-9 test failures depending on absence/presence of avx512
win + pypy	hung indefinitely for 1.7.3	same as CPython

Matrix below is discounting the above-mentioned 7 failures present in all builds

variant	accelerate	blis	mkl w/ avx512	mkl w/o avx512	netlib	openblas	sum**
linux / x86	➖	✔️	❌	✔️	❌	❌	15F
linux / aarch	➖	➖	➖	➖	✔️	❌	5F
linux / ppc64le	➖	➖	➖	➖	❔	❔	-
osx / arm	❔	➖	➖	➖	❔	❔	-
osx / x86	❌	✔️	➖	❌	✔️	❌	15F
win / x86	➖	✔️	❌	✔️	✔️	❌	10F
sum**	5F	-	10F	-	5F	20F	40F

** sum of Segfaults (S), Failures (F), resp. Timeouts (T); out of a total of 110 CI combinations being tested

Build logs:
Azure

Failures in addition to the 7 mentioned above

linux + mkl + avx512: 2 additional failures

=========================== short test summary info ============================
FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab] - AssertionError: 
Items are not equal:
 ACTUAL: -11
 DESIRED: 0

FAILED stats/tests/test_distributions.py::TestTruncnorm::test_moments - AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 1 / 1 (100%)
Max absolute difference: 1.9596281e-05
Max relative difference: 3.3373635e-06
 x: array(5.8718046)
 y: array(5.871785)
= 9 failed, 35473 passed, 2739 skipped, 109 xfailed, 7 xpassed, 535 warnings in 1284.82s (0:21:24) =

linux + netlib + avx512: 1 additional failure

=========================== short test summary info ============================
FAILED stats/tests/test_distributions.py::TestTruncnorm::test_moments - AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 1 / 1 (100%)
Max absolute difference: 1.9596281e-05
Max relative difference: 3.3373635e-06
 x: array(5.8718046)
 y: array(5.871785)
= 8 failed, 35474 passed, 2739 skipped, 109 xfailed, 7 xpassed, 535 warnings in 1064.78s (0:17:44) =

linux + netlib + non-avx512: 2 additional failures

=========================== short test summary info ============================
FAILED linalg/tests/test_decomp.py::TestSchur::test_sort - AssertionError: 
Arrays are not almost equal to 3 decimals

Mismatched elements: 8 / 16 (50%)
Max absolute difference: 1.64897635
Max relative difference: 2.00019569
 x: array([[ 0.113,  0.544,  0.832,  0.   ],
       [-0.113, -0.825,  0.554,  0.   ],
       [-0.821,  0.131,  0.026, -0.555],
       [-0.547,  0.087,  0.018,  0.832]])
 y: array([[-1.134e-01, -5.436e-01,  8.316e-01, -8.918e-15],
       [ 1.134e-01,  8.245e-01,  5.544e-01, -6.509e-15],
       [ 8.213e-01, -1.308e-01,  2.650e-02, -5.547e-01],
       [ 5.475e-01, -8.718e-02,  1.767e-02,  8.321e-01]])

FAILED linalg/tests/test_solvers.py::test_solve_discrete_are - AssertionError: 
Arrays are not almost equal to 5 decimals

Mismatched elements: 1 / 9 (11.1%)
Max absolute difference: 1.77845359e-05
Max relative difference: inf
 x: array([[ 1.77845e-05,  1.13063e-05, -2.25008e-06],
       [ 1.13025e-05,  7.33137e-06, -1.53296e-06],
       [-2.24821e-06, -1.53482e-06,  3.44589e-07]])
 y: array([[0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.]])
= 9 failed, 35473 passed, 2739 skipped, 104 xfailed, 12 xpassed, 535 warnings in 2046.34s (0:34:06) =

linux + openblas + avx512: 3 additional failures

=========================== short test summary info ============================
FAILED linalg/tests/test_decomp.py::TestSchur::test_sort - AssertionError: 
Arrays are not almost equal to 3 decimals

Mismatched elements: 8 / 16 (50%)
Max absolute difference: 1.64897635
Max relative difference: 2.00019569
 x: array([[ 0.113,  0.544,  0.832,  0.   ],
       [-0.113, -0.825,  0.554,  0.   ],
       [-0.821,  0.131,  0.026, -0.555],
       [-0.547,  0.087,  0.018,  0.832]])
 y: array([[-1.134e-01, -5.436e-01,  8.316e-01, -6.360e-15],
       [ 1.134e-01,  8.245e-01,  5.544e-01, -5.195e-15],
       [ 8.213e-01, -1.308e-01,  2.650e-02, -5.547e-01],
       [ 5.475e-01, -8.718e-02,  1.767e-02,  8.321e-01]])

FAILED linalg/tests/test_solvers.py::test_solve_discrete_are - AssertionError: 
Arrays are not almost equal to 5 decimals

Mismatched elements: 1 / 9 (11.1%)
Max absolute difference: 1.78702176e-05
Max relative difference: inf
 x: array([[ 1.78702e-05,  1.10194e-05, -2.05636e-06],
       [ 1.10194e-05,  6.89924e-06, -1.35228e-06],
       [-2.05636e-06, -1.35601e-06,  2.92435e-07]])
 y: array([[0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.]])

FAILED stats/tests/test_distributions.py::TestTruncnorm::test_moments - AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 1 / 1 (100%)
Max absolute difference: 1.9596281e-05
Max relative difference: 3.3373635e-06
 x: array(5.8718046)
 y: array(5.871785)
= 10 failed, 35472 passed, 2739 skipped, 109 xfailed, 7 xpassed, 535 warnings in 1176.14s (0:19:36) =

linux + {x86, aarch} + openblas + non-avx512: 2 additional failures

=========================== short test summary info ============================
FAILED linalg/tests/test_decomp.py::TestSchur::test_sort - AssertionError: 
Arrays are not almost equal to 3 decimals

Mismatched elements: 8 / 16 (50%)
Max absolute difference: 1.64897635
Max relative difference: 2.00019569
 x: array([[ 0.113,  0.544,  0.832,  0.   ],
       [-0.113, -0.825,  0.554,  0.   ],
       [-0.821,  0.131,  0.026, -0.555],
       [-0.547,  0.087,  0.018,  0.832]])
 y: array([[-1.134e-01, -5.436e-01,  8.316e-01, -8.918e-15],
       [ 1.134e-01,  8.245e-01,  5.544e-01, -6.509e-15],
       [ 8.213e-01, -1.308e-01,  2.650e-02, -5.547e-01],
       [ 5.475e-01, -8.718e-02,  1.767e-02,  8.321e-01]])

FAILED linalg/tests/test_solvers.py::test_solve_discrete_are - AssertionError: 
Arrays are not almost equal to 5 decimals

Mismatched elements: 1 / 9 (11.1%)
Max absolute difference: 1.77845359e-05
Max relative difference: inf
 x: array([[ 1.77845e-05,  1.13063e-05, -2.25008e-06],
       [ 1.13025e-05,  7.33137e-06, -1.53296e-06],
       [-2.24821e-06, -1.53482e-06,  3.44589e-07]])
 y: array([[0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.]])
= 9 failed, 46756 passed, 2792 skipped, 219 xfailed, 13 xpassed, 535 warnings in 1038.56s (0:17:18) =

windows + mkl + avx512: 1 additional failure

=========================== short test summary info ===========================
FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab]
= 8 failed, 34996 passed, 3217 skipped, 105 xfailed, 11 xpassed, 534 warnings in 1909.54s (0:31:49) =

windows + openblas + avx512: 9 additional failures

=========================== short test summary info ===========================
FAILED linalg/tests/test_blas.py::TestFBLAS2Simple::test_syr_her - AssertionE...
FAILED linalg/tests/test_decomp.py::TestEig::test_bad_geneig - AssertionError: 
FAILED linalg/tests/test_decomp.py::TestSchur::test_sort - AssertionError: 
FAILED linalg/tests/test_decomp_ldl.py::test_simple - AssertionError: 
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex64]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex128]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex64]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex128]
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_ARPACK::test_svds_parameter_tol
= 16 failed, 34988 passed, 3217 skipped, 105 xfailed, 11 xpassed, 536 warnings in 2361.68s (0:39:21) =

windows + openblas + non-avx512: 8 additional failures

=========================== short test summary info ===========================
FAILED linalg/tests/test_blas.py::TestFBLAS2Simple::test_syr_her - AssertionE...
FAILED linalg/tests/test_decomp.py::TestEig::test_bad_geneig - AssertionError: 
FAILED linalg/tests/test_decomp.py::TestSchur::test_sort - AssertionError: 
FAILED linalg/tests/test_decomp_ldl.py::test_simple - AssertionError: 
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex64]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex128]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex64]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex128]
= 15 failed, 34989 passed, 3217 skipped, 105 xfailed, 11 xpassed, 534 warnings in 1848.24s (0:30:48) =

osx + accelerate: 8 additional failures

=========================== short test summary info ============================
FAILED linalg/tests/test_cython_blas.py::TestWfuncPointers::test_float_args
FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_tolerance_float32 - AssertionError: 
Not equal to tolerance rtol=1e-07, atol=1.5e-05

Mismatched elements: 1 / 3 (33.3%)
Max absolute difference: 1.71661377e-05
Max relative difference: 5.7220459e-06
 x: array([-1.      , -1.999998, -2.999983], dtype=float32)
 y: array([-1, -2, -3])

FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_v0 - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 51 / 100 (51%)
Max absolute difference: 2.77555756e-17
Max relative difference: 2.92153795e-16
 x: array([[0.109858],
       [0.106207],
       [0.108685],...
 y: array([[0.109858],
       [0.106207],
       [0.108685],...

FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 61 / 100 (61%)
Max absolute difference: 4.16333634e-17
Max relative difference: 4.18263199e-16
 x: array([[0.109858],
       [0.106207],
       [0.108685],...
 y: array([[0.109858],
       [0.106207],
       [0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[0] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 71 / 100 (71%)
Max absolute difference: 4.16333634e-17
Max relative difference: 4.06316107e-16
 x: array([[0.109858],
       [0.106207],
       [0.108685],...
 y: array([[0.109858],
       [0.106207],
       [0.108685],...

FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[1] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 60 / 100 (60%)
Max absolute difference: 4.16333634e-17
Max relative difference: 4.02567023e-16
 x: array([[0.109858],
       [0.106207],
       [0.108685],...
 y: array([[0.109858],
       [0.106207],
       [0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state2] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 95 / 100 (95%)
Max absolute difference: 4.16333634e-17
Max relative difference: 4.30865517e-16
 x: array([[0.109858],
       [0.106207],
       [0.108685],...
 y: array([[0.109858],
       [0.106207],
       [0.108685],...

FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state3] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 58 / 100 (58%)
Max absolute difference: 4.16333634e-17
Max relative difference: 4.06316106e-16
 x: array([[0.109858],
       [0.106207],
       [0.108685],...
 y: array([[0.109858],
       [0.106207],
       [0.108685],...
= 15 failed, 35455 passed, 2752 skipped, 105 xfailed, 10 xpassed, 534 warnings in 1778.20s (0:29:38) =

osx + mkl: 2 additional failures

=========================== short test summary info ============================
FAILED linalg/tests/test_decomp_cossin.py::test_cossin[True-40-12-20-float32] - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-06

Mismatched elements: 164 / 1600 (10.2%)
Max absolute difference: 1.317905
Max relative difference: 2.0001829
 x: array([[-0.081348, -0.407225, -0.130576, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.300051, -0.301593,  0.239466, ...,  0.      ,  0.      ,...
 y: array([[-0.081348, -0.407225, -0.130575, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.300052, -0.301593,  0.239465, ...,  0.      ,  0.      ,...
FAILED linalg/tests/test_decomp_cossin.py::test_cossin[False-40-12-20-float32] - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-06

Mismatched elements: 82 / 1600 (5.12%)
Max absolute difference: 1.1718636
Max relative difference: 2.0002658
 x: array([[ 0.296194, -0.129601,  0.106824, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.148379, -0.196432,  0.578848, ...,  0.      ,  0.      ,...
 y: array([[ 0.296194, -0.129602,  0.106824, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.148379, -0.196431,  0.578847, ...,  0.      ,  0.      ,...
= 9 failed, 35461 passed, 2752 skipped, 104 xfailed, 11 xpassed, 533 warnings in 2065.87s (0:34:25) =

osx + openblas: 3 additional failures

=========================== short test summary info ============================
FAILED linalg/tests/test_decomp.py::TestSchur::test_sort - AssertionError: 
Arrays are not almost equal to 3 decimals

Mismatched elements: 8 / 16 (50%)
Max absolute difference: 1.64897635
Max relative difference: 2.00019569
 x: array([[ 0.113,  0.544,  0.832,  0.   ],
       [-0.113, -0.825,  0.554,  0.   ],
       [-0.821,  0.131,  0.026, -0.555],
       [-0.547,  0.087,  0.018,  0.832]])
 y: array([[-1.134e-01, -5.436e-01,  8.316e-01, -1.046e-14],
       [ 1.134e-01,  8.245e-01,  5.544e-01, -7.997e-15],
       [ 8.213e-01, -1.308e-01,  2.650e-02, -5.547e-01],
       [ 5.475e-01, -8.718e-02,  1.767e-02,  8.321e-01]])

FAILED linalg/tests/test_decomp.py::TestQZ::test_qz_single - AssertionError: 
Arrays are not almost equal to 5 decimals

Mismatched elements: 25 / 25 (100%)
Max absolute difference: 1.9282217
Max relative difference: 12.652091
 x: array([[ 0.25942,  0.42247,  0.29118,  1.3622 ,  0.07058],
       [ 1.54704,  1.04254,  0.65235, -1.17932,  1.4969 ],
       [ 0.37006,  0.76962, -0.09774,  1.07189, -0.13269],...
 y: array([[0.92962, 0.31638, 0.18392, 0.20456, 0.56773],
       [0.59554, 0.96451, 0.65318, 0.74891, 0.65357],
       [0.74771, 0.96131, 0.00839, 0.10644, 0.2987 ],...

FAILED linalg/tests/test_lapack.py::test_gges_tgexc[float32] - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-05

Mismatched elements: 100 / 100 (100%)
Max absolute difference: 0.5596407
Max relative difference: 45.19812
 x: array([[ 0.269565,  0.510163,  0.307816,  0.720287,  0.713599,  0.25576 ,
         0.131274,  0.843207,  0.835774,  1.005878],
       [ 0.364995, -0.002645,  0.148027,  0.409446,  0.200136,  0.596843,...
 y: array([[0.191519, 0.622109, 0.437728, 0.785359, 0.779976, 0.272593,
        0.276464, 0.801872, 0.958139, 0.875933],
       [0.357817, 0.500995, 0.683463, 0.712702, 0.370251, 0.561196,...
= 10 failed, 46743 passed, 2805 skipped, 219 xfailed, 12 xpassed, 536 warnings in 1139.12s (0:18:59) =

rgommers · 2023-01-27T12:07:21Z

I see we have this now for 1.8.x, 1.9.x and 1.10.x - that's a bit much to look at. Should we close this and focus on 1.10.x?

h-vetinari · 2023-01-27T13:12:15Z

I see we have this now for 1.8.x, 1.9.x and 1.10.x - that's a bit much to look at. Should we close this and focus on 1.10.x?

I tend to close the old ones as they become obsolete, but in this case, I wanted to establish the progression from 1.8 to 1.10, and wanted to have a baseline for that (especially as I had missed redoing this for anything after 1.8.0).

It's more for historical reference and interested - feel free to skip ahead to 1.10. In any case, this PR can be closed indeed.

h-vetinari requested review from jakirkham, msarahan, ocefpaf and rgommers as code owners February 6, 2022 03:36

h-vetinari marked this pull request as draft February 6, 2022 03:36

h-vetinari force-pushed the 1.8_blas_vars branch 3 times, most recently from b986b4b to b2f9c58 Compare February 6, 2022 06:13

h-vetinari mentioned this pull request Feb 6, 2022

BUG: test failure in test_x0_equals_Mb with bicgstab scipy/scipy#15533

Closed

rgommers mentioned this pull request Feb 9, 2022

ENH, BLD: Add configure option for blas/lapack libraries scipy/scipy#15553

Merged

h-vetinari force-pushed the 1.8_blas_vars branch from b2f9c58 to de4847e Compare July 26, 2022 23:29

h-vetinari changed the base branch from main to v1.8 August 1, 2022 15:30

h-vetinari force-pushed the 1.8_blas_vars branch 2 times, most recently from b43b18e to 076b309 Compare August 1, 2022 16:33

rgommers mentioned this pull request Oct 17, 2022

Improve BLAS/LAPACK support: dependency detection, UX, and ILP64 scipy/scipy#17244

Open

18 tasks

h-vetinari added 6 commits January 27, 2023 15:11

try CI with all blas variants

012f567

distinguish blis runs for avx512 presence/absence

d5d4b40

enable setting pagefile size on windows

600de5f

unskip some tests

c25704f

update pypy patch

ea9c8ca

MNT: Re-rendered with conda-build 3.23.3, conda-smithy 3.22.1, and co…

1dee1bb

…nda-forge-pinning 2023.01.26.16.59.55

h-vetinari force-pushed the 1.8_blas_vars branch from 076b309 to 1dee1bb Compare January 27, 2023 04:20

This was referenced Jan 27, 2023

TEST: 1.9.x + blas variants #223

Closed

TEST: 1.10.x + blas variants #224

Closed

h-vetinari mentioned this pull request Jan 27, 2023

BUG: Seg. fault in scipy.sparse.linalg tests in PROPACK scipy/scipy#15108

Closed

h-vetinari closed this Jan 27, 2023

h-vetinari deleted the 1.8_blas_vars branch January 27, 2023 13:12

h-vetinari mentioned this pull request Jun 26, 2023

TEST: 1.11.x + blas variants #242

Closed

h-vetinari mentioned this pull request Jan 23, 2024

TEST: 1.12.x + blas variants #267

Closed

h-vetinari mentioned this pull request Apr 12, 2024

TEST: 1.13.x + blas variants #273

Closed

h-vetinari mentioned this pull request Jun 25, 2024

TEST: 1.14.x + blas variants #280

Closed

h-vetinari mentioned this pull request Jan 8, 2025

TEST: v1.15.x + blas variants #295

Closed

h-vetinari mentioned this pull request Jun 30, 2025

TEST: v1.16.x + blas variants #304

Closed

h-vetinari mentioned this pull request May 27, 2026

TEST: v1.17.x + blas variants #323

Draft

h-vetinari mentioned this pull request Jun 22, 2026

TEST: v1.18.x + blas variants #326

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

TEST: 1.8.x + blas variants#199

TEST: 1.8.x + blas variants#199
h-vetinari wants to merge 6 commits into
conda-forge:v1.8from
h-vetinari:1.8_blas_vars

h-vetinari commented Feb 6, 2022

Uh oh!

conda-forge-linter commented Feb 6, 2022

Uh oh!

h-vetinari commented Feb 6, 2022 •

edited

Loading

Uh oh!

rgommers commented Feb 7, 2022

Uh oh!

h-vetinari commented Feb 8, 2022

Uh oh!

ngam commented Feb 27, 2022

Uh oh!

ngam commented Feb 27, 2022

Uh oh!

h-vetinari commented Feb 27, 2022 •

edited

Loading

Uh oh!

conda-forge-webservices Bot commented Jan 27, 2023

Uh oh!

h-vetinari commented Jan 27, 2023 •

edited

Loading

Uh oh!

rgommers commented Jan 27, 2023

Uh oh!

h-vetinari commented Jan 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

h-vetinari commented Feb 6, 2022

Uh oh!

conda-forge-linter commented Feb 6, 2022

Uh oh!

h-vetinari commented Feb 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Scipy 1.8.0

TL;DR: A quarter** of relevant combinations is failing in some way, and that's not even counting pypy (or the propack segfaults)

Notable:

The good news:

The bad news:

Details

Uh oh!

rgommers commented Feb 7, 2022

Uh oh!

h-vetinari commented Feb 8, 2022

Uh oh!

ngam commented Feb 27, 2022

Uh oh!

ngam commented Feb 27, 2022

Uh oh!

h-vetinari commented Feb 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

conda-forge-webservices Bot commented Jan 27, 2023

Uh oh!

h-vetinari commented Jan 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Scipy 1.8.1

TL;DR: Some persistent failures, but less segfaults; cautious improvements overall

The good news:

The bad news:

Details

Matrix below is discounting the above-mentioned 7 failures present in all builds

Failures in addition to the 7 mentioned above

Uh oh!

rgommers commented Jan 27, 2023

Uh oh!

h-vetinari commented Jan 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

h-vetinari commented Feb 6, 2022 •

edited

Loading

h-vetinari commented Feb 27, 2022 •

edited

Loading

h-vetinari commented Jan 27, 2023 •

edited

Loading