Py3k installable

[QuLogic]

Following up on #1699 and #1700, this one allows you to install on Python 3, i.e., you can run python3 setup.py install and there won't be errors or warnings. Now, it doesn't actually run, but that's a bigger problem.

Again, this looks huge since it's based on a couple previous ones. It's a little bit smaller than it looks right now.

[QuLogic]

This should be review-able now.

[rhattersley]

Could we avoid using six here and just pass thumbnails?

[rhattersley]

To be honest, I'm not really bothered either way.

[rhattersley]

Thanks @QuLogic. I've only one real request which is that we keep the complexities of six out of the example code. Other than that ... I'm 👍

[QuLogic]

The examples are run as tests, so we'd need to be extra careful about their cross-compatibility. six may be unavoidable sometimes (though not in the case pointed out above.)

[QuLogic]

OK, your request has been handled.

[QuLogic]

Wow, it took 10.5 minutes for that "Solving package specifications" step to finish. Is it because there are Python3 packages available now?

[pelson]

Wow, it took 10.5 minutes for that "Solving package specifications" step to finish.

Does that include the time to actually download all 200MB of distributions? Solving and installing took 590s (~10mn) according to the travis log:

Fetching package metadata: ......
Solving package specifications: .....................................
Package plan for installation in environment /home/travis/miniconda/envs/test-environment:
The following packages will be downloaded:
    package                    |            build
    ---------------------------|-----------------
    freetype-2.4.10            |                0         2.0 MB  defaults
    geos-3.3.3                 |                0        14.5 MB  defaults
    hdf5-1.8.9                 |                1         1.7 MB  defaults
    jpeg-8d                    |                0         699 KB  defaults
    lcms-1.19                  |                0         595 KB  defaults
    libmo_unpack-3.0           |                1          29 KB  scitools
    libpng-1.5.13              |                1         152 KB  defaults
    pixman-0.26.2              |                0         1.9 MB  defaults
    proj4-4.9.1                |                1         1.0 MB  scitools
    qt-4.8.5                   |                0        32.8 MB  defaults
    udunits-2.2.17             |                5         136 KB  scitools
    cairo-1.12.18              |                0         593 KB  defaults
    jasper-1.900.1             |                3         287 KB  scitools
    libnetcdf-4.2.1.1          |                0         637 KB  defaults
    alabaster-0.7.3            |           py27_0          13 KB  defaults
    docutils-0.12              |           py27_0         636 KB  defaults
    funcsigs-0.4               |           py27_0          19 KB  defaults
    markupsafe-0.23            |           py27_0          30 KB  defaults
    nose-1.3.7                 |           py27_0         194 KB  defaults
    numpy-1.8.2                |           py27_0         7.3 MB  defaults
    biggus-0.11.0              |         nppy27_0          84 KB  scitools
    gdal-1.10.1                |       np18py27_2        33.8 MB  defaults
    mo_pack-0.2.0              |       np18py27_1         468 KB  scitools
    netcdf4-1.1.1              |       np18py27_0         1.0 MB  defaults
    pep8-1.5.7                 |           py27_0          43 KB  defaults
    pil-1.1.7                  |           py27_1         587 KB  defaults
    py2cairo-1.10.0            |           py27_2          36 KB  defaults
    pygments-2.0.2             |           py27_0         1.0 MB  defaults
    pyke-1.1.1                 |           py27_1         192 KB  scitools
    pyparsing-2.0.1            |           py27_0          62 KB  defaults
    pytz-2015.4                |           py27_0         174 KB  defaults
    pyugrid-0.1.1              |         nppy27_1          33 KB  scitools
    scipy-0.14.0               |       np18py27_0        29.6 MB  defaults
    shapely-1.5.11             |           py27_0         252 KB  defaults
    shiboken-1.2.1             |           py27_0         883 KB  defaults
    six-1.9.0                  |           py27_0          17 KB  defaults
    snowballstemmer-1.2.0      |           py27_0          73 KB  defaults
    sphinx_rtd_theme-0.1.7     |           py27_0         209 KB  defaults
    babel-1.3                  |           py27_0         2.3 MB  defaults
    dateutil-2.4.1             |           py27_0         218 KB  defaults
    ecmwf_grib-1.12.1          |       np18py27_4         1.5 MB  scitools
    jinja2-2.8                 |           py27_0         263 KB  defaults
    pyepsg-0.2.0               |           py27_0           7 KB  scitools
    pyshp-1.2.1                |           py27_0          28 KB  scitools
    pyside-1.2.1               |           py27_1         5.7 MB  defaults
    python-dateutil-2.4.2      |           py27_0         219 KB  defaults
    matplotlib-1.3.1           |       np18py27_0        37.1 MB  defaults
    owslib-0.9.0               |           py27_0         179 KB  scitools
    pandas-0.14.1              |       np18py27_0         9.0 MB  defaults
    pbr-1.3.0                  |           py27_0          81 KB  defaults
    sphinx-1.3.1               |           py27_0         1.1 MB  defaults
    mock-1.3.0                 |           py27_0          95 KB  defaults
    cartopy-0.13.0             |       np18py27_0         7.6 MB  scitools
    ------------------------------------------------------------
                                           Total:       199.1 MB

[ajdawson]

In general I've found that the solving stage takes at least several minutes on my own machine when using the SciTools channel. This is way in excess of anything from the anaconda (default) channel. Is it because we only have packages for e.g. one version of numpy so it is more work for the solver to find a dependency solution? If so, can we do something about this?

[QuLogic]

No, I was watching it. At 1min 1s, it had the first 6 or so dots, and at 11m 30s or so it moved on to "Package plan". (Note, I was referring to an older build there where the build+install phase is 712.82 s = 11.88 min)

[pelson]

No, I was watching it. At 1min 1s, it had the first 6 or so dots, and at 11m 30s or so it moved on to "Package plan". (Note, I was referring to an older build there where the build+install phase is 712.82 s = 11.88 min)

Interesting. Given the number of dependencies, we may be choking conda. Might be worth looking at splitting our install phase into multiple commands to reduce the resolve step.

e.g.

conda install gdal
conda install cartopy
conda install <the rest>

[pelson]

In general I've found that the solving stage takes at least several minutes on my own machine when using the SciTools channel. This is way in excess of anything from the anaconda (default) channel. Is it because we only have packages for e.g. one version of numpy so it is more work for the solver to find a dependency solution? If so, can we do something about this?

We have added the full matrix of possibilities to anaconda.org/scitools now though, so that wouldn't explain the performance found in the last week.

@asmeurer - do you have any idea what might be slowing the conda resolution down on the scitools channel? Is it because some of our packages have a large number of dependencies, and the conda solver is choking?

[rhattersley]

Might be worth looking at splitting our install phase into multiple commands to reduce the resolve step.

We deliberately moved away from multi-command installation because it was causing trouble with inconsistent numpy-version "ping-ponging". For example, conda install package_a would install numpy 1.7 as a dependency. Then conda install package_b would bump numpy to 1.8 (but as I recall it didn't modify the installation of package_a, so nasty compatibility issues can occur). Then conda install package_c would drop numpy to 1.7 again (leaving package_b alone). Perhaps some of that has been fixed in conda?

[pelson]

Perhaps some of that has been fixed in conda?

And our recipes. Conda no longer bakes the numpy version into the run dependencies by default.
You'll see that scitools' Iris http://anaconda.org/SciTools/iris/files no longer has a baked numpy version.

Instead I'm proposing giving the solver less work todo by giving it an existing environment that it needs to fit into, but I have no idea if that would be of benefit to conda.solve.

[rhattersley]

Installing sequentially is faster (e.g. 85 s) but still results in the "ping-pong" issue (e.g. https://travis-ci.org/rhattersley/iris/jobs/79175852#L334) which leads to https://travis-ci.org/rhattersley/iris/jobs/79175852#L804.

[pelson]

The ping pong in this case looks like it is caused by us using a matplotlib which doesn't have a np18 build (we need to be aware that Continuum only build against the latest numpy and python 2/3 versions). Essentially, we just need to not do this naively and we will get the results we want (e.g. export CONDA_NPY=18 until we can use a later mpl).

[rhattersley]

NB. Fixing either matplotlib to 1.3.1 or fixing numpy to 1.8 makes the dependency resolution painfully slow. If you have neither of those constraints the resolution is pretty much instant.