SciPy builds for Windows

[FSund]

Is it possible to add OpenBLAS builds of scipy for Windows? As noted by @tadeu there is no scipy package for Windows in conda-forge, so the proper request to make here the addition of (OpenBLAS-enabled) Windows builds of SciPy to conda-forge. This would help a lot with the size of executables made from Python programs on Windows.

The MKL libraries installed by Anaconda on Windows when installing scipy are very large, so if making an executable (using PyInstaller or similar), even a simple import scipy; print("Hello, world!") ends up at around 200 MB.

A similar issue for numpy can be found here: conda-forge/numpy-feedstock#84

[tadeu]

conda-forge currently do not even have scipy packages for Windows :(

I'd recommend to try to use the nomkl feature (i.e., install the nomkl package together), but note that this do not have to do with conda-forge because conda-forge do not have packages that depend on MKL. But it also seems that there's no nomkl scipy package for Windows on defaults.

Perhaps this issue should be changed to SciPy builds for Windows, it should probably already be using OpenBLAS as it is on other platforms ;).

[jakirkham]

My understand has been nomkl is not an option on Windows. Has that changed? If so, what BLAS does it use?

[FSund]

You are both completely correct. There is no nomkl package for Windows, and conda-forge doesn't have a scipy package for Windows. I thought I installed scipy from conda-forge, but now see that the package comes from the regular channel.

So SciPy builds for Windows is probably a much better title for this issue.

[FSund]

I thought I'd try my hand at getting a Windows version up, but I think I'm in way over my head. Anyway, here's what I have learned so far:

• There seems to be a OpenBLAS that has recipies for Windows, contrary to what this comment says
  • I might be wrong about this, I have never read a yaml-file or any kind of feedstock before
• The conda-forge version of numpy uses OpenBLAS on Windows.

This indicates to me that it shouldn't be too hard to get SciPy+OpenBLAS working on Windows. I might be horribly wrong here though. Has there been any attempts at this?

I have tried looking at the recipies for numpy to figure out how they do it, but I have never set up a feedstock before, so I'm having some trouble figuring how all these files work. I've tried looking at the conda-forge docs, but that dives right into the details. Are there any shorter quick-start guides that give a more overarching view of what a feedstock is, and how it is used to create a conda package?

[jakirkham]

Work on this front is in PR ( #78 ). If you want to take a look, am sure help would be appreciated.

[carlkl]

@jakirkham, are there any news from FLANG concerning building scipy on windows? There is a one month old windows FLANG package on conda-forge. Are there any attempts to build scipy with FLANG on windows right now?

[jakirkham]

I have not been active in this work. Maybe @isuruf would know.

[isuruf]

@carlkl, current status with the testsuite is

34 failed, 12057 passed, 1361 skipped, 577 deselected, 75 xfailed, 6 xpassed, 16 warnings in 394.89 seconds 

#78

[carlkl]

The appveyor builds linked at #78 are three months old as far I can see. I could not find other more recent informations about windows scipy builds with flang. Mayby I missed something?

[rth]

Has anyone tried to use the f18 compiler frontend from the flang project? Since it's a new re-implementation one might hope that it is more portable than flang itself, though I'm not sure how production ready it is.

[isuruf]

@rth, f18 is currently a parser only. It parses the Fortran code to an AST, convert the AST back into Fortran and compile it using pgf90 to check that the parsing is correct. It doesn't produce LLVM IR yet.

See flang-compiler/f18#290, flang-compiler/f18#300

[rth]

Thanks @isuruf !

From flang-compiler/f18#300 (comment)

I predict that f18 will start to generate llvm ir in May 2019.

I suppose we'll have to wait and see..

[h-vetinari]

Any update on this? :)

[scopatz]

This is kinda crazy to me..

[rth]

It does looks like ARM builds will be available before Windows ones... While we are listing Fortran compilers here, there is also https://lfortran.org/ that says it supports Windows (doesn't look too production ready though).

On a related topic, somehow cross-compiling on Linux for Windows and then linking those objects in the normal MSVC build is really not possible?

[scopatz]

I think the anaconda builds use the intel compiler https://github.com/AnacondaRecipes/scipy-feedstock/blob/master/recipe/meta.yaml

[rgommers]

@scopatz just mentioned that an Intel Fortran license is available, so this is much closer to being solved now:) SciPy builds are pretty fast, so it seems the only issue would be to get the license working on CI or some VM. Is anyone looking at this? If not I could ask around in the SciPy project for a volunteer.

[scopatz]

@mariusvniekerk was looking into it, but I think he stalled out on building the image and would love some help. If there is a volunteer who wants to take that on, please let us know!

[scopatz]

It really would be to everyone's benefit and super high impact!

[rgommers]

Awesome, thanks @scopatz. @mariusvniekerk if you could post a short summary of current status/issue and what skills are needed to be able to help out here, that would be awesome.

[mariusvniekerk]

So the main thing that needs to happen is we need to go provision a windows / linux build host somewhere and install the azure pipelines agent on it and the relevant compilers. Microsoft has some build images available here that can be built with packer. This unfortunately takes HOURS to complete.

Once we have an image up we need to set up a small instance that will function as a licence server and have the image communicate with that. The build agent will communicate with the instance and that will be firewalled to all hell so that nothing other than the azure workers nodes we provision can talk to it.

My plan for this was to build some terraform to manage provisioning of all the things.

I would probably say that this infrastructure is going to be turned off most of the time since the number of things that actually need the intel compilers is very small.

[rgommers]

Thanks Marius! I'll go ask for a volunteer with good devops/sysadmin skills on the SciPy mailing list.

[msarahan]

There's probably at least 4 different asks here, right?

1. Windows hardware
2. linux hardware
3. license server? Is this really necessary? We haven't needed it at Anaconda.
4. devops to set all of this up

Anaconda has build images for Windows with Intel compilers installed for our purposes. Maybe these can be repurposed. I'm not sure about Windows licenses for these images.

I can offer some of my home computer stuff for hardware, but that's not exactly convenient for remote on/off and such, nor for if I have a conflicting need. I'm sure most of us are in the same position. @ocefpaf has mentioned that Kyle Wilcox of Axiom computers may have resources we could use. What about Azure instances?

Once Anaconda gets the GPU build machine(s) set up with NVIDIA, those machine(s) can probably also be used for this purpose. Unfortunately, I don't have any time estimate on that.

[msarahan]

Oh, I forgot, @sodre also offered some compute resources in the past

[sodre]

I still have 12 nodes sitting in a data center that I can offer as compute. Let's find time to workout the details that @mariusvniekerk brought up.

[rgommers]

2. linux hardware

Is this needed? The painful problem is Windows-specific, on Linux gfortran is fine.

[msarahan]

I don't know. I was going by this line in Marius' message:

we need to go provision a windows / linux build host

[ocefpaf]

I don't know. I was going by this line in Marius' message:

we need to go provision a windows / linux build host

Probably not for scipy linux but for the arm cross-compiled version maybe?!

PS: I believe we could also look into a Windows machine for a developer that is willing to do work on things like scipy, qt, qgis, etc. While CIs services are crucial for to move forward we still rely on 2-3 people Windows machines to iterate on the changes to these packages. I'll add a discussion item to the agenda b/c this is tangential to the issue here.

[isuruf]

I believe we could also look into a Windows machine for a developer that is willing to do work on things like scipy, qt, qgis, etc.

💯

[rth]

I believe we could also look into a Windows machine for a developer that is willing to do work on things like scipy, qt, qgis, etc.

Maybe it's common knowledge, but openssh can now run on recent versions of Windows, nvim also (kind of) works in cmd. So it doesn't necessarily need to be a physical machine (though one would still need an initial physical or RDP access to set thing up).

[jakirkham]

@isuruf, is it possible to link against Netlib during the build for Windows like is done on other platforms? If so, this would eliminate one complication.

[isuruf]

@jakirkham, that's possible, but scipy needs a fortran compiler as well.

[hmaarrfk]

i know it is blasphemy, but can we repackage the pip builds?

[jakirkham]

Would just wait for defaults then.

[jakirkham]

@mariusvniekerk built it for Python 3.8. So we should be good to as far as the migration is concerned. 🙂

[jakirkham]

FWIW Fortran support recently landed in LLVM. Though it might still be a bit before we see an LLVM release with Fortran support.