Scipy 1.9.0, with distutils

[h-vetinari]

Minimal alternative to #205, fully on distutils to be able to run test suite.

Ideally, we'll be able to move the unixes to meson already for 1.9.0 GA

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[h-vetinari]

@rgommers @tylerjereddy
RC3 looks good (with the distutils builds) 👍

[tylerjereddy]

woot

[rgommers]

excellent, thanks for testing!

[h-vetinari]

@mattip, now that we have pypy builds again, I reran this. It looks like there are still some small issues to work out. However, if it's not easily fixable, I don't think we'll have to hold up the release for that. We've made do without pypy the last few months, so if 1.9.0 is released without initially, I don't think it'll be a big problem.

In more detail, it looks like there's e.g. some file handle probably not being closed "enough" and causing import errors on windows:

Details

Traceback (most recent call last):
  File "messagestream.pyx", line 91, in scipy._lib.messagestream.MessageStream.close
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: b'C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\scipy-nd4qm5tg'
Exception ignored in: 'scipy._lib.messagestream.MessageStream.__dealloc__'
Traceback (most recent call last):
  File "messagestream.pyx", line 91, in scipy._lib.messagestream.MessageStream.close
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: b'C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\scipy-nd4qm5tg'
Traceback (most recent call last):
  File "C:\bld\scipy_1658816029856\test_tmp\run_test.py", line 71, in <module>
    import scipy.signal
  File "C:\bld\scipy_1658816029856\_test_env\lib\site-packages\scipy\signal\__init__.py", line 331, in <module>
    from ._peak_finding import *
  File "C:\bld\scipy_1658816029856\_test_env\lib\site-packages\scipy\signal\_peak_finding.py", line 8, in <module>
    from scipy.stats import scoreatpercentile
  File "C:\bld\scipy_1658816029856\_test_env\lib\site-packages\scipy\stats\__init__.py", line 482, in <module>
    from ._rvs_sampling import rvs_ratio_uniforms, NumericalInverseHermite
  File "C:\bld\scipy_1658816029856\_test_env\lib\site-packages\scipy\stats\_rvs_sampling.py", line 3, in <module>
    from ._unuran import unuran_wrapper
  File "unuran_wrapper.pyx", line 223, in init scipy.stats._unuran.unuran_wrapper
  File "unuran_wrapper.pyx", line 202, in scipy.stats._unuran.unuran_wrapper._setup_unuran
  File "messagestream.pyx", line 33, in scipy._lib.messagestream.MessageStream.__cinit__
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: b'C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\scipy-nd4qm5tg'

Linux has some apparent segfault/crash at least in

optimize/tests/test_direct.py::TestDIRECT::test_segmentation_fault[False]

and additionally there are two failing tests on linux/osx, one of them refcount-related, and one with (AFAICT) another too-specific error message regex.

Details

________________________ test_sf_error_special_refcount ________________________
[gw1] darwin -- Python 3.9.12 $PREFIX/bin/python
../[...]/lib/pypy3.9/site-packages/scipy/special/tests/test_sf_error.py:74: in test_sf_error_special_refcount
    refcount_before = sys.getrefcount(sc)
E   AttributeError: module 'sys' has no attribute 'getrefcount'
____________________________ test_rvs_no_size_error ____________________________
[gw0] darwin -- Python 3.9.12 $PREFIX/bin/python
../[...]/lib/pypy3.9/site-packages/scipy/stats/tests/test_distributions.py:6985: in test_rvs_no_size_error
    rvs_no_size.rvs()
        rvs_no_size = <scipy.stats.tests.test_distributions.test_rvs_no_size_error.<locals>.rvs_no_size_gen object at 0x00007fc93c78a560>
        rvs_no_size_gen = <class 'scipy.stats.tests.test_distributions.test_rvs_no_size_error.<locals>.rvs_no_size_gen'>
../[...]/lib/pypy3.9/site-packages/scipy/stats/_distn_infrastructure.py:1078: in rvs
    vals = self._rvs(*args, size=size, random_state=random_state)
E   TypeError: _rvs() got 2 unexpected keyword arguments
        args       = []
        cond       = True
        discrete   = None
        kwds       = {}
        loc        = array(0)
        random_state = RandomState(MT19937) at 0x7FC975836540
        rndm       = None
        scale      = array(1)
        self       = <scipy.stats.tests.test_distributions.test_rvs_no_size_error.<locals>.rvs_no_size_gen object at 0x00007fc93c78a560>
        size       = ()

During handling of the above exception, another exception occurred:
../[...]/lib/pypy3.9/site-packages/scipy/stats/tests/test_distributions.py:6985: in test_rvs_no_size_error
    rvs_no_size.rvs()
E   AssertionError: Regex pattern '_rvs\\(\\)\\ got\\ an\\ unexpected' does not match '_rvs() got 2 unexpected keyword arguments'.
        rvs_no_size = <scipy.stats.tests.test_distributions.test_rvs_no_size_error.<locals>.rvs_no_size_gen object at 0x00007fc93c78a560>
        rvs_no_size_gen = <class 'scipy.stats.tests.test_distributions.test_rvs_no_size_error.<locals>.rvs_no_size_gen'>

[mattip]

I don't think we'll have to hold up the release for that

Agreed.

Linux has some apparent segfault/crash

Can we skip that test on PyPy? The test generates a segfault, which is tricky to control.

test_sf_error.py:74: in test_sf_error_special_refcount should be skipped on PyPy like all the refcount tests.

Is this enough to fix the regex in the error message?

diff --git a/scipy/stats/tests/test_distributions.py b/scipy/stats/tests/test_distributions.py
index b01cf452a..d1ccfda8d 100644
--- a/scipy/stats/tests/test_distributions.py
+++ b/scipy/stats/tests/test_distributions.py
@@ -6981,7 +6981,7 @@ def test_rvs_no_size_error():
 
     rvs_no_size = rvs_no_size_gen(name='rvs_no_size')
 
-    with assert_raises(TypeError, match=re.escape("_rvs() got an unexpected")):
+    with assert_raises(TypeError, match=re.escape("_rvs() got .* unexpected")):
         rvs_no_size.rvs()

I guess these fixes plus the ones already in the patches should be pushed to upstream for the 1.9.0 release...

[mattip]

The windows error is going to be more involved, since there is a mixture of cython, deallocators, and temp files. Is there something obvious that changed between 1.8 and 1.9?

[h-vetinari]

I'm leaving out the pypy builds for now, but will open a PR after merging this one where we can readd them and figure out the remaining issues.