Installation of sdist is (silently) failing #249

brechtm · 2021-01-31T15:48:17Z

Thank you for this very useful package! Unfortunately, I'm unable to test sdists.

My noxfile.py:

import nox
from nox_poetry import installroot, install, SDIST, WHEEL

PYTHONS = ['3.6', '3.7', '3.8', '3.9', '3.10', 'pypy3.6', 'pypy3.7']
DEPENDENCIES = ['pytest', 'pytest-xdist', 'pytest-cov', 'coverage', 'pygments',
                'Sphinx']

@nox.session(python=PYTHONS)
def unit(session):
    installroot(session, distribution_format=SDIST)
    install(session, *DEPENDENCIES)
    session.run('python', 'run_tests.py', *session.posargs, 'tests')

$ nox -v -rs unit-3.8

nox > Running session unit-3.8
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/unit-3-8
created virtual environment CPython3.8.7.final.0-64 in 576ms
  creator CPython3Posix(dest=/Users/brechtm/Documents/Code/rinohtype/.nox/unit-3-8, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/brechtm/Library/Application Support/virtualenv)
    added seed packages: pip==20.3.3, setuptools==51.0.0, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

nox > poetry build --format=sdist
Building rinohtype (0.5.0-dev.1)
  - Building sdist
  - Built rinohtype-0.5.0.dev1.tar.gz

nox > poetry export --format=requirements.txt --output=.nox/unit-3-8/tmp/requirements.txt --dev --without-hashes
nox > pip uninstall --yes file:///Users/brechtm/Documents/Code/rinohtype/dist/rinohtype-0.5.0.dev1.tar.gz#sha256=0ce6a74e1004f4576c87e0a931a2a9f849a852f592ced55ab05aac26c8acf2e1&egg=rinohtype
WARNING: Skipping rinohtype as it is not installed.

nox > python -m pip install --constraint=.nox/unit-3-8/tmp/requirements.txt file:///Users/brechtm/Documents/Code/rinohtype/dist/rinohtype-0.5.0.dev1.tar.gz#sha256=0ce6a74e1004f4576c87e0a931a2a9f849a852f592ced55ab05aac26c8acf2e1&egg=rinohtype
Processing ./dist/rinohtype-0.5.0.dev1.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting appdirs<2.0.0,>=1.4.4
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting docutils<0.17,>=0.16
  Using cached docutils-0.16-py2.py3-none-any.whl (548 kB)
Collecting recommonmark<0.7.0,>=0.6.0
  Using cached recommonmark-0.6.0-py2.py3-none-any.whl (10 kB)
Collecting commonmark>=0.8.1
  Using cached commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
Collecting rinoh-typeface-dejavuserif<0.2.0,>=0.1.3
  Using cached rinoh_typeface_dejavuserif-0.1.3-py3-none-any.whl (1.7 MB)
Collecting rinoh-typeface-texgyrecursor<0.2.0,>=0.1.1
  Using cached rinoh_typeface_texgyrecursor-0.1.1-py3-none-any.whl (242 kB)
Collecting rinoh-typeface-texgyreheros<0.2.0,>=0.1.1
  Using cached rinoh_typeface_texgyreheros-0.1.1-py3-none-any.whl (523 kB)
Collecting rinoh-typeface-texgyrepagella<0.2.0,>=0.1.1
  Using cached rinoh_typeface_texgyrepagella-0.1.1-py3-none-any.whl (321 kB)
Collecting sphinx>=1.3.1
  Using cached Sphinx-3.4.3-py3-none-any.whl (2.9 MB)
Requirement already satisfied: setuptools in ./.nox/unit-3-8/lib/python3.8/site-packages (from sphinx>=1.3.1->recommonmark<0.7.0,>=0.6.0->rinohtype) (51.0.0)
Collecting alabaster<0.8,>=0.7
  Using cached alabaster-0.7.12-py2.py3-none-any.whl (14 kB)
Collecting babel>=1.3
  Using cached Babel-2.9.0-py2.py3-none-any.whl (8.8 MB)
Collecting Jinja2>=2.3
  Using cached Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting MarkupSafe>=0.23
  Using cached MarkupSafe-1.1.1-cp38-cp38-macosx_10_9_x86_64.whl (16 kB)
Collecting Pygments>=2.0
  Using cached Pygments-2.7.4-py3-none-any.whl (950 kB)
Collecting pytz>=2015.7
  Using cached pytz-2020.5-py2.py3-none-any.whl (510 kB)
Collecting requests>=2.5.0
  Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting chardet<5,>=3.0.2
  Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting snowballstemmer>=1.1
  Using cached snowballstemmer-2.0.0-py2.py3-none-any.whl (97 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.2-py2.py3-none-any.whl (136 kB)
Collecting imagesize
  Using cached imagesize-1.2.0-py2.py3-none-any.whl (4.8 kB)
Collecting packaging
  Using cached packaging-20.8-py2.py3-none-any.whl (39 kB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting sphinxcontrib-applehelp
  Using cached sphinxcontrib_applehelp-1.0.2-py2.py3-none-any.whl (121 kB)
Collecting sphinxcontrib-devhelp
  Using cached sphinxcontrib_devhelp-1.0.2-py2.py3-none-any.whl (84 kB)
Collecting sphinxcontrib-htmlhelp
  Using cached sphinxcontrib_htmlhelp-1.0.3-py2.py3-none-any.whl (96 kB)
Collecting sphinxcontrib-jsmath
  Using cached sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl (5.1 kB)
Collecting sphinxcontrib-qthelp
  Using cached sphinxcontrib_qthelp-1.0.3-py2.py3-none-any.whl (90 kB)
Collecting sphinxcontrib-serializinghtml
  Using cached sphinxcontrib_serializinghtml-1.1.4-py2.py3-none-any.whl (89 kB)
Building wheels for collected packages: rinohtype
  Building wheel for rinohtype (PEP 517): started
  Building wheel for rinohtype (PEP 517): finished with status 'done'
  Created wheel for rinohtype: filename=rinohtype-0.5.0.dev1-py3-none-any.whl size=16382 sha256=f06b9496e25e6317da97eb9b5b756b9d947ffec6fbc4a68156d70042d35e6200
  Stored in directory: /Users/brechtm/Library/Caches/pip/wheels/b9/48/00/5eea4e606f55ae9a66c4e7a51180b04385eed8c74f71ab901b
Successfully built rinohtype
Installing collected packages: urllib3, pytz, pyparsing, MarkupSafe, idna, chardet, certifi, sphinxcontrib-serializinghtml, sphinxcontrib-qthelp, sphinxcontrib-jsmath, sphinxcontrib-htmlhelp, sphinxcontrib-devhelp, sphinxcontrib-applehelp, snowballstemmer, requests, Pygments, packaging, Jinja2, imagesize, docutils, babel, alabaster, sphinx, commonmark, rinoh-typeface-texgyrepagella, rinoh-typeface-texgyreheros, rinoh-typeface-texgyrecursor, rinoh-typeface-dejavuserif, recommonmark, appdirs, rinohtype
Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 Pygments-2.7.4 alabaster-0.7.12 appdirs-1.4.4 babel-2.9.0 certifi-2020.12.5 chardet-4.0.0 commonmark-0.9.1 docutils-0.16 idna-2.10 imagesize-1.2.0 packaging-20.8 pyparsing-2.4.7 pytz-2020.5 recommonmark-0.6.0 requests-2.25.1 rinoh-typeface-dejavuserif-0.1.3 rinoh-typeface-texgyrecursor-0.1.1 rinoh-typeface-texgyreheros-0.1.1 rinoh-typeface-texgyrepagella-0.1.1 rinohtype-0.5.0.dev1 snowballstemmer-2.0.0 sphinx-3.4.3 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-1.0.3 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.4 urllib3-1.26.2
WARNING: You are using pip version 20.3.3; however, version 21.0.1 is available.
You should consider upgrading via the '/Users/brechtm/Documents/Code/rinohtype/.nox/unit-3-8/bin/python -m pip install --upgrade pip' command.

nox > python -m pip install --constraint=.nox/unit-3-8/tmp/requirements.txt pytest pytest-xdist pytest-cov coverage pygments Sphinx
<SNIPPED>

nox > python run_tests.py tests
====================================== test session starts =======================================
platform darwin -- Python 3.8.7, pytest-6.1.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/brechtm/Documents/Code/rinohtype, configfile: setup.cfg
plugins: cov-2.11.1, xdist-2.2.0, forked-1.3.0
gw0 [0] / gw1 [0] / gw2 [0] / gw3 [0] / gw4 [0] / gw5 [0] / gw6 [0] / gw7 [0]

============================================= ERRORS =============================================
____________________________ ERROR collecting tests/test_attribute.py ____________________________
ImportError while importing test module '/Users/brechtm/Documents/Code/rinohtype/tests/test_attribute.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../.pyenv/versions/3.8.7/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_attribute.py:11: in <module>
    from rinoh.attribute import Attribute, Bool
E   ModuleNotFoundError: No module named 'rinoh'
<SNIPPED>

For some reason the rinoh package is missing, but the dist-info file is present:

$ ls -d .nox/unit-3-8/lib/python3.8/site-packages/rinoh*
.nox/unit-3-8/lib/python3.8/site-packages/rinoh_typeface_dejavuserif
.nox/unit-3-8/lib/python3.8/site-packages/rinoh_typeface_dejavuserif-0.1.3.dist-info
.nox/unit-3-8/lib/python3.8/site-packages/rinoh_typeface_texgyrecursor
.nox/unit-3-8/lib/python3.8/site-packages/rinoh_typeface_texgyrecursor-0.1.1.dist-info
.nox/unit-3-8/lib/python3.8/site-packages/rinoh_typeface_texgyreheros
.nox/unit-3-8/lib/python3.8/site-packages/rinoh_typeface_texgyreheros-0.1.1.dist-info
.nox/unit-3-8/lib/python3.8/site-packages/rinoh_typeface_texgyrepagella
.nox/unit-3-8/lib/python3.8/site-packages/rinoh_typeface_texgyrepagella-0.1.1.dist-info
.nox/unit-3-8/lib/python3.8/site-packages/rinohtype-0.5.0.dev1.dist-info

Testing the wheel works fine, though. Using tox, I can test both the sdist and wheel.

I tried to install the sdist by using the same command as executed by nox-poetry (quotes added):

.nox/unit-3-8/bin/python -m pip install --constraint=.nox/unit-3-8/tmp/requirements.txt \
    "file:///Users/brechtm/Documents/Code/rinohtype/dist/rinohtype-0.5.0.dev1.tar.gz#sha256=0ce6a74e1004f4576c87e0a931a2a9f849a852f592ced55ab05aac26c8acf2e1&egg=rinohtype"

This neither yields the rinoh directory. However, when I drop the sha256 bit, it does appear:

.nox/unit-3-8/bin/python -m pip install --constraint=.nox/unit-3-8/tmp/requirements.txt \
    "file:///Users/brechtm/Documents/Code/rinohtype/dist/rinohtype-0.5.0.dev1.tar.gz#egg=rinohtype"

I'm not sure what's going on here. Possibly this is an issue in pip instead.

The text was updated successfully, but these errors were encountered:

cjolowicz · 2021-01-31T18:08:43Z

Thanks for reporting the issue!

I would need to take a look at your branch to see what's going on here.

Originally, the #sha256 bit was required because the constraints file contained hashes. Due to changes in pip's resolver, we had to drop support for hashes in constraints files. So this isn't actually required anymore. I understand that pip's maintainer team is open to bringing back hash support to constraints files. But it's not clear when that would happen. So I think we could simply drop the #sha256 from package URLs for now, if that solves a problem with sdist support.

cjolowicz · 2021-02-02T12:20:17Z

@brechtm Does #260 solve the issue for you?

brechtm · 2021-02-06T11:10:31Z

I was only able to test now. I did so with 0.8.0 now and eventually it worked.

When I first tried with my original noxfile, it was still failing to produce the rinoh directory.

At that point, I tried some things (expand)

Manually executing the pip install command didn't work either this time. Possibly I messed something up last time...

I did some more tests that do produce the rinoh directory:

Create a separate venv, and execute pip install "file:///Users/brechtm/Documents/Code/rinohtype/dist/rinohtype-0.5.1.tar.gz#egg=rinohtype"

Create another venv, and execute python -m pip install --constraint=/Users/brechtm/Documents/Code/rinohtype/.nox/unit-3-8/tmp/requirements.txt "file:///Users/brechtm/Documents/Code/rinohtype/dist/rinohtype-0.5.1.tar.gz#egg=rinohtype"

Use a path instead of a file:// URL: .nox/unit-3-8/bin/python -m pip install --constraint=.nox/unit-3-8/tmp/requirements.txt /Users/brechtm/Documents/Code/rinohtype/dist/rinohtype-0.5.1.tar.gz

I'm still clueless. Note that I didn't test this with another project. Perhaps there's something odd about my package?

Giving up, I decided to have a look at the other changes in 0.8.0. After swapping nox.session with nox_poetry.session however (and installroot(session, ... with session.poetry.installroot ) I'm no longer seeing the issue! I can't explain, but hey 🤷‍♂️. Thanks! 😄

I'll report back in case I bump into this again.

brechtm · 2021-02-07T23:13:05Z

I ran into this again, when testing get_versions() from #268 (on 0.8.1) 😭
I'll need to do further testing...

brechtm · 2021-03-04T13:11:55Z

@cjolowicz I was wondering whether you were able to reproduce this issue yourself. I want to have another look at it soon, so any extra information you might have could be useful.

cjolowicz · 2021-03-04T13:54:24Z

I haven't. Please do let me know whatever you find out about this issue.

brechtm · 2021-03-12T10:03:48Z

My earlier confusion was caused by pip creating and caching wheels when installing from sdist. To be able to debug this issue, one needs to delete the cached wheel file after every install (or disable caching somehow).

The wheel file create by pip contains next to nothing:

      658  01-01-2016 00:00   rinohtype-0.5.3.dist-info/entry_points.txt
    34520  01-01-1980 00:00   rinohtype-0.5.3.dist-info/LICENSE
       83  01-01-2016 00:00   rinohtype-0.5.3.dist-info/WHEEL
     8750  01-01-2016 00:00   rinohtype-0.5.3.dist-info/METADATA
      401  01-01-2016 00:00   rinohtype-0.5.3.dist-info/RECORD

When I install the sdist into a venv (after deleting the cached wheel) from the command line using the same arguments to pip as used by nox-poetry, the wheel does include the rinoh directory.

By clearing env in nox's Session._run(), a proper wheel is created. It turns out the PATH set in the environment causes the difference. In Session._run(), this includes the test venv's bin path at the front. Removing that (so that the PATH is identical to that of the environment in which nox was started) fixes the issue.

It's not yet clear to me why the added search path affects creation of the wheel. I was hoping you have a clue? Even with the test venv bin path removed, its python is used for running pip.

brechtm · 2021-03-12T10:33:44Z

Running the pip install command from a terminal after activating the test environment manually, a proper wheel is produced. This revealed that a difference in the TMPPATH environment variable is also involved:

regular: /var/folders/cm/h_fnh5lx0b5b1b6wvnlc7qlw0000gn/T/ (= typical temp dir on macOS)
in Session._run(): .nox/unit-3-6-sdist/tmp

So, instead of adjusting the PATH, setting TMPDIR to the regular temp dir or an empty directory also fixes the issue. The presence of the requirements.txt in .nox/unit-3-6-sdist/tmp must be causing the issue.

I think the wheel building step is executed in TMPDIR, which is affected by the presence of a requirements.txt file. In that case, storing the requirements.txt in a subdirectory would fix this issue.

cjolowicz · 2021-03-12T11:14:38Z

Thank you for looking into this!

It would be good to know which version of pip you have in that Nox session? This looks a little like pypa/pip#7808 which was fixed in pip 20.2.

We set TMPDIR indirectly by invoking session.create_tmp from export_requirements. While it looks like an upstream bug, I've felt for a while that setting TMPDIR at unpredictable points in the build process is asking for trouble. I'm not quite sure which location would be best, though. We could just continue to use <venv>/tmp but create it ourselves instead of using session.create_tmp.

brechtm · 2021-03-12T15:04:29Z

It would be good to know which version of pip you have in that Nox session? This looks a little like pypa/pip#7808 which was fixed in pip 20.2.

The pip version in the test env was 21.0.1. But testing with pip 20.0.1, 20.3 and 20.3.3 shows from the same issue.

We set TMPDIR indirectly by invoking session.create_tmp from export_requirements. While it looks like an upstream bug, I've felt for a while that setting TMPDIR at unpredictable points in the build process is asking for trouble. I'm not quite sure which location would be best, though. We could just continue to use /tmp but create it ourselves instead of using session.create_tmp.

Not calling session.create_tmp() is indeed the key to success here! When calling session.create_tmp() and exporting requirements.txt to another directory (e.g. /tmp), the wheel produced is still missing my package. I still don't understand how this could affect pip's wheel creation though. For what it's worth, running pip wheel on my sdist with another TMPDIR still produces a proper wheel.

Perhaps you can export requirements.txt to .nox/ or .nox/.nox-poetry?

cjolowicz · 2021-03-13T13:18:19Z

Thanks for the additional info! Can you see if #298 works for you?

Perhaps you can export requirements.txt to .nox/ or .nox/.nox-poetry?

Yes, it looks like we no longer need to have separate requirements.txt per session. I'd like to do this in a separate PR though from the one dealing with TMPDIR.

cjolowicz · 2021-03-14T01:45:35Z

Released in 0.8.2

brechtm · 2021-03-14T13:14:36Z

Thanks for the additional info! Can you see if #298 works for you?

I tested 0.8.2, which works. Now I can finally migrate from tox to nox :-)

cjolowicz mentioned this issue Feb 2, 2021

Omit SHA256 hash for the locally built package #260

Merged

cjolowicz closed this as completed in #260 Feb 5, 2021

cjolowicz reopened this Mar 4, 2021

cjolowicz mentioned this issue Mar 13, 2021

Do not set TMPDIR when exporting to requirements.txt #298

Merged

cjolowicz closed this as completed in #298 Mar 14, 2021

This was referenced Apr 25, 2021

Crash when using PassthroughEnv #347

Closed

Use a single requirements file for all sessions #350

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installation of sdist is (silently) failing #249

Installation of sdist is (silently) failing #249

brechtm commented Jan 31, 2021 •

edited

Loading

cjolowicz commented Jan 31, 2021

cjolowicz commented Feb 2, 2021

brechtm commented Feb 6, 2021 •

edited

Loading

brechtm commented Feb 7, 2021 •

edited

Loading

brechtm commented Mar 4, 2021

cjolowicz commented Mar 4, 2021

brechtm commented Mar 12, 2021 •

edited

Loading

brechtm commented Mar 12, 2021 •

edited

Loading

cjolowicz commented Mar 12, 2021

brechtm commented Mar 12, 2021 •

edited

Loading

cjolowicz commented Mar 13, 2021

cjolowicz commented Mar 14, 2021

brechtm commented Mar 14, 2021

Installation of sdist is (silently) failing #249

Installation of sdist is (silently) failing #249

Comments

brechtm commented Jan 31, 2021 • edited Loading

cjolowicz commented Jan 31, 2021

cjolowicz commented Feb 2, 2021

brechtm commented Feb 6, 2021 • edited Loading

brechtm commented Feb 7, 2021 • edited Loading

brechtm commented Mar 4, 2021

cjolowicz commented Mar 4, 2021

brechtm commented Mar 12, 2021 • edited Loading

brechtm commented Mar 12, 2021 • edited Loading

cjolowicz commented Mar 12, 2021

brechtm commented Mar 12, 2021 • edited Loading

cjolowicz commented Mar 13, 2021

cjolowicz commented Mar 14, 2021

brechtm commented Mar 14, 2021

brechtm commented Jan 31, 2021 •

edited

Loading

brechtm commented Feb 6, 2021 •

edited

Loading

brechtm commented Feb 7, 2021 •

edited

Loading

brechtm commented Mar 12, 2021 •

edited

Loading

brechtm commented Mar 12, 2021 •

edited

Loading

brechtm commented Mar 12, 2021 •

edited

Loading