#809 Switch from Boost Python to PyBind11 #950

rmjarvis · 2018-02-27T07:07:28Z

Here is the final PR for issue #809. The last few things needed to finally make GalSim pip installable are:

Switch the wrapping code from Boost Python to PyBind11.
Switch the linear algebra back end from TMV to Eigen.
Write a setup.py installation script rather than use SCons.

Now the only non-pip-installable dependency is fftw, and if you are using conda, then even that one is conda-installable, so for most users it's not too hard.

I'd like as many users as possible to try out the new installation procedure and let me know any problems you have with it. I fully expect there to be some issues as we try this on a range of systems. So far, it works for me on my laptop (OSX 10.10), my wife's laptop (OSX 10.12), nersc (cori and edison), the BNL astro cluster, Travis, and a unix cluster at UPenn. But there are lots of other systems out there that you all use, so I expect there to be issues still... :)

Full explanation of the new installation process is in the INSTALL.md file. But here is a summary.

For conda users, the installation process is (from the GalSim root directory):

conda install -c conda-forge --file conda_requirements.txt
pip install -r requirements.txt
python setup.py install

If you don't use conda, then you are required to have a modern enough compiler that it can handle the c++11 standard (or newer). gcc 4.8 or later will work. You also need fftw installed somewhere appropriate (cf. instructions in INSTALL.md for details). Then

pip install -r requirements.txt
python setup.py install

(You might need to either precede these with sudo or follow them with --user.)

Then to run the tests, do

python setup.py test

A few further comments:

I haven't uploaded the tarball to pypi yet (which is what makes pip install galsim work), but I will do that soon. Probably after whatever early rounds of trouble-shooting are required from people trying the above methods.
You can still use SCons with the non-boost installation if you want. scons install now by default uses Eigen and PyBind11. That may be a more productive workflow than using setup.py during development, so I didn't get rid of it. We'll have to see.
You can still use Boost Python if you really want. But you need to use SCons: scons USE_BOOST=true.
You can still use TMV if you want, also using SCons: scons USE_TMV=true.
Depending on your setup, the pip installation location might be lower priority than where SCons installed it for you. If so, you may need to manually remove various GalSim files and directories from the old SCons installation. Checking import galsim; galsim.__file__ will tell you which file is being imported in python.
The setup.py installation isn't smart about only recompiling changed files, so it is highly recommended for developers to use ccache to speed up repeating installations with setup.py.
I'd like to release 2.0 from the noboost branch at the same time as 1.6 from master. So they will have the same functionality, just 2.0 will use this installation method, and all deprecated features removed. Then this can become master and the old one will become releases/1.6.
This PR would resolve or make obsolete the following issues: Switch from SCons to setup.py #809, installation FAQ could use some cleanup #897, and Consider updating install docs with anaconda #914. And I suspect it makes Conda install of galsim #794 far easier (although I don't actually know what's involved in that).

(#809-pybind11)

…t work in python 3 (#809-pybind11)

(#809-pybind11)

…le than manifest

(#809-pybind11)

rmjarvis · 2018-02-28T21:07:33Z

@jmeyers314 I think I fixed the issues you were having. Could you try again with the current version?

jmeyers314 · 2018-03-01T00:10:33Z

Thanks @rmjarvis . Unfortunately, that still didn't work; I got the same no member named 'tgamma' in namespace 'std' errors.

I went ahead and tried (and committed) making libc++ the default stdlib for clang, and that seems to work for my system (can build and all tests pass). Definitely feel free to revert or clean this up though if you see a more elegant solution or if I broke anything for other systems.

jmeyers314

Looks good to me. There are a few things broken in the markdown files that I noted, and a couple other questions. I managed to build and test with setup.py, and also with scons, using all 4 variations of EIGEN/TMV + BOOST/PYBIND11 (with one minor change to the PyBind11Helper.h file), so that's nice.

I think it would be nice if we could add some of these variations to our Travis-CI build matrix. Is that possible?

Also, since this is the precursor to our next release, I also ran all the lsst_wcs tests that normally get skipped. It looks like these have bitrotted, at least with respect to the master branch of the LSST stack. (Actually, I think they may have been broken since Sep 2016). But I'll file that in a separate issue; no reason to hold this up.

jmeyers314 · 2018-03-14T16:56:29Z

INSTALL.md



-0. Overall summary


Something's broken with the internal links and headers here (See https://github.com/GalSim-developers/GalSim/blob/%23809-pybind11/INSTALL.md). (True on master branch too).

jmeyers314 · 2018-03-14T17:01:30Z

INSTALL.md

-root, of course: `sudo ./b2`...  Also, you should be aware that if you are
-running `b2` a second time, you should use `b2 -a` to tell boost to
-recompile everything rather than use the existing libraries.
+- Eigen (3.2.8)  (via eigency 1.77)


Eigen 3.2.5 works for me. Maybe reduce this version requirement to that?

I tried to say that these weren't a minimum required version, but sure, I'll downgrade the requirements to the earlier versions that you tested with.

jmeyers314 · 2018-03-14T17:02:21Z

INSTALL.md

-running `b2` a second time, you should use `b2 -a` to tell boost to
-recompile everything rather than use the existing libraries.
+- Eigen (3.2.8)  (via eigency 1.77)
+- NumPy (1.14.0)


1.13.1 works for me.

jmeyers314 · 2018-03-14T17:05:15Z

INSTALL.md

-to them during the build. We require version 3 (or greater presumably), which
-is often distributed as fftw3.  See Section 5 for some suggestions about
-installing this on your platform.
+    python setup.py install


Maybe add

git clone [email protected]:GalSim-developers/GalSim.git cd GalSim

jmeyers314 · 2018-03-14T17:08:34Z

INSTALL.md

+- Eigen (3.2.8)  (via eigency 1.77)
+- NumPy (1.14.0)
+- Future (0.16.0)
+- Astropy (2.0.3)


jmeyers314 · 2018-03-14T17:12:48Z

INSTALL.md


-* `FFTW_DIR`: Explicitly give the FFTW prefix
+Probably, you should put this into your .bash_profile file so it always gets


"shell login file" maybe?

jmeyers314 · 2018-03-14T17:37:12Z

INSTALL.md

-   code information in the compiled library (-g3 for g++ compiler).  This
-   increases the size of the compiled library, but makes debugging with things
-   like gdb easier.  Probably end users will never need to use this.
+By default, the python tests will use the pytest plugins `pytest-xdist` (for 


Should probably mention pip install -r test_requirements.txt here.

jmeyers314 · 2018-03-14T17:40:16Z

INSTALL_SCONS.md

+
+
+1. Software required before building GalSim
+===========================================


===== sectioning in markdown seems to be broken here too (https://github.com/GalSim-developers/GalSim/blob/%23809-pybind11/INSTALL_SCONS.md)

I think the numbering is what messed it up. It looks like it tried to make a numbered list rather than treat it as a heading. Probably an ambiguity in Markdown's specs and GitHub's implementation changed at some point since we first wrote this.

jmeyers314 · 2018-03-14T17:48:47Z

INSTALL_SCONS.md

+* `TMV_DIR`: Explicitly give the TMV prefix
+* `EIGEN_DIR`: Explicitly give the Eigen prefix
+
+# `USE_BOOST`: Specify that you want to use Boost rather than PyBind11.


jmeyers314 · 2018-03-14T21:45:09Z

pysrc/PyBind11Helper.h

+// macros allow us to write code that works for either boost python or pybind11.
+
+// First some things where the boost equivalent of some pybind11 function is different:
+#define PYBIND11_MODULE(x,x) BOOST_PYTHON_MODULE(x)


My compiler didn't like this macro (duplicate variable). Changing it to

PYBIND11_MODULE(x,y) BOOST_PYTHON_MODULE(x)

seemed to work. But this is well outside my expertise so I'll let you decide...

(#809-pybind11)

rmjarvis · 2018-03-15T17:28:07Z

Thanks Josh. I fixed up the markdown. The links are all working now. Just had to remove the numbering from the section headings. Also made the other adjustments you recommended.

rmjarvis · 2018-03-15T18:12:16Z

I think it would be nice if we could add some of these variations to our Travis-CI build matrix. Is that possible?

I looked into this a bit, and yes, it's possible. But I think I'd rather not. I'd rather have us just run a few of the scons installations before each release to make sure things still work rather than have travis run these for every PR.

One nice feature of the new installation is that it is faster, especially when the cache isn't populated. So if we keep the boost/tmv tests on travis, we'll keep having to do those annoying "disable tests to repopulate cache" "git revert HEAD" cycles, which I don't think will be required for the setup.py installations.

(#809-pybind11)

rmjarvis · 2018-04-09T18:34:11Z

Let's merge this Friday if there are no objections. We'll want to ask more people to try installing using pip, but we can wait until 1.6 is ready, so we can have 2.0 released at the same time.

(#809-pybind11)

rmjarvis added 30 commits December 31, 2017 12:48

First pass at implementing pybind11 wrapping

83b95d9

(#809-pybind11)

Remove TMV from places that don't really need it

be6c282

(#809-pybind11)

Add check for Eigen in SConstruct

c4d8246

(#809-pybind11)

Fix a stray use of installed boost header file

eab89ae

(#809-pybind11)

Use Eigen rather than TMV in WCS.cpp, RealGalaxy.cpp

714ba52

(#809-pybind11)

Use Thomas algorithm for LookupTable spline solver

8162304

(#809-pybind11)

Use Eigen rather than TMV for Shapelet class

e808425

(#809-pybind11)

Use Eigen in PSFCorr rather than TMV

eced2a1

(#809-pybind11)

Make TMV optional in SConstruct

baf65fa

(#809-pybind11)

Automatically get Eigen directory from eigency if possible

2674501

(#809-pybind11)

Automatically get fftw3 library from pyfftw3 if possible

2948102

(#809-pybind11)

Don't do tgamma, lgamma if already in scope

d221e96

(#809-pybind11)

First pass at setup.py

ed4d247

(#809-pybind11)

Add requirements file

a04d582

(#809-pybind11)

Remove mmgr.cpp file

471225e

(#809-pybind11)

typo

10960dc

(#809-pybind11)

Make meta_data.py file using actual install_dir

ea38cdf

(#809-pybind11)

Switch to using entry_points rather than scripts for the executables

93b7b1f

(#809-pybind11)

Add conda_requirements for those so inclined

1205657

(#809-pybind11)

Use a custom function to find fftw3 rather than pyfftw3, which doesn'…

38dd0db

…t work in python 3 (#809-pybind11)

Add conda_requirements.txt to use conda for whatever it can do.

f5cd94d

(#809-pybind11)

gcc is not a pip package.

eefc527

(#809-pybind11)

Write Version.h file

d6451ab

(#809-pybind11)

Fix share symlink, which wasn't working properly on linux

e525e2c

Switch to using package_data for share, since seems to be more reliab…

0c8e64e

…le than manifest

Fix gcc compiler warnings

e6c644d

(#809-pybind11)

Fix making meta_data.py file for pip installation

d8a8c42

(#809-pybind11)

Fix script_install_dir check for pip installation

d32b077

(#809-pybind11)

Merge branch '#809f' into #809-pybind11

6abd2c2

Use -O2 to avoid numerical inaccuracies from over optimization

a79281c

(#809-pybind11)

rmjarvis added 4 commits February 28, 2018 13:08

Check EIGEN_DIR/include

f4705ad

(#809-pybind11)

Check if clang needs the -stdlib=libc++ flag

06e0c09

(#809-pybind11)

Fix -j flag to work even with older distutils

ed2b677

(#809-pybind11)

Fix -jN option to control number of jobs for compiling

990b643

(#809-pybind11)

Prefer libc++ for clang compiling and linking

9563445

rmjarvis and others added 2 commits March 7, 2018 11:49

Merge branch 'noboost' into #809-pybind11

287ae0f

typos

b783237

jmeyers314 approved these changes Mar 15, 2018

View reviewed changes

rmjarvis added 7 commits March 15, 2018 12:20

Fix boost version of PYBIND11_MODULE definition

d95730f

(#809-pybind11)

Apply Josh's suggested changes in INSTALL files

7f62305

(#809-pybind11)

Fix links

0663061

(#809-pybind11)

Try to fix aside spec

0f6d236

(#809-pybind11)

Fix indenting of toc

1600998

(#809-pybind11)

Try .note rather than aside

6ec0fa0

(#809-pybind11)

I guess .note isn't supported. Remove it.

2b94689

(#809-pybind11)

rmjarvis added 3 commits March 21, 2018 23:18

Remove some files that got accidentally added in the merge

d56431b

(#809-pybind11)

Merge branch 'noboost' into #809-pybind11

14018ac

Update codecov requirements to be a bit more strict

375dd3c

(#809-pybind11)

rmjarvis added 4 commits April 12, 2018 21:49

Merge branch 'noboost' into #809-pybind11

223a400

Merge branch 'noboost' into #809-pybind11

d1fac7f

Give better progress info about pybind11 header dirs

9867ce2

(#809-pybind11)

Limit to pip version 9.0, not 10.0

0b271ce

(#809-pybind11)

rmjarvis merged commit 424cada into noboost Apr 16, 2018

rmjarvis deleted the #809-pybind11 branch April 16, 2018 02:35

rmjarvis mentioned this pull request Apr 16, 2018

Switch from SCons to setup.py #809

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#809 Switch from Boost Python to PyBind11 #950

#809 Switch from Boost Python to PyBind11 #950

rmjarvis commented Feb 27, 2018

rmjarvis commented Feb 28, 2018

jmeyers314 commented Mar 1, 2018

jmeyers314 left a comment

jmeyers314 Mar 14, 2018

jmeyers314 Mar 14, 2018

rmjarvis Mar 15, 2018

jmeyers314 Mar 14, 2018

jmeyers314 Mar 14, 2018

jmeyers314 Mar 14, 2018

jmeyers314 Mar 14, 2018

jmeyers314 Mar 14, 2018

jmeyers314 Mar 14, 2018

rmjarvis Mar 15, 2018

jmeyers314 Mar 14, 2018

jmeyers314 Mar 14, 2018

rmjarvis commented Mar 15, 2018

rmjarvis commented Mar 15, 2018 •

edited

Loading

rmjarvis commented Apr 9, 2018


		* `FFTW_DIR`: Explicitly give the FFTW prefix
		Probably, you should put this into your .bash_profile file so it always gets



		1. Software required before building GalSim
		===========================================

#809 Switch from Boost Python to PyBind11 #950

#809 Switch from Boost Python to PyBind11 #950

Conversation

rmjarvis commented Feb 27, 2018

rmjarvis commented Feb 28, 2018

jmeyers314 commented Mar 1, 2018

jmeyers314 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmjarvis commented Mar 15, 2018

rmjarvis commented Mar 15, 2018 • edited Loading

rmjarvis commented Apr 9, 2018

rmjarvis commented Mar 15, 2018 •

edited

Loading