isolated build env should ignore deps #1239

webknjaz · 2019-04-01T22:20:39Z

So it looks like when you combine isolated_build = true with deps = -rrequirements.txt in combines deps from pyproject.toml and deps entries expecting that those would be direct dist specs and tries to parse them as such.

Python 3.7.1 + tox 3.8.4.

$ tox -e metadata-validation --notest -vv 
using tox.ini: ~/src/github/ansible/molecule/tox.ini (pid 8030)
  removing ~/src/github/ansible/molecule/.tox/log
using tox-3.8.4 from ~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/__init__.py (pid 8030)
.package start: getenv ~/src/github/ansible/molecule/.tox/.package
.package cannot reuse: no previous config ~/src/github/ansible/molecule/.tox/.package/.tox-config1
.package create: ~/src/github/ansible/molecule/.tox/.package
  removing ~/src/github/ansible/molecule/.tox/.package
setting PATH=~/src/github/ansible/molecule/.tox/.package/bin:~/.pyenv/versions/3.7.1/bin:~/.pyenv/libexec:~/.pyenv/plugins/python-build/bin:~/.pyenv/plugins/pyenv-virtualenv/bin:~/.pyenv/plugins/pyenv-update/bin:~/.pyenv/plugins/pyenv-installer/bin:~/.pyenv/plugins/pyenv-doctor/bin:~/.nvm/versions/node/v9.8.0/bin:~/.pyenv/shims:/usr/lib/llvm/7/bin:/usr/lib/llvm/6/bin:/usr/lib/llvm/5/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:~/.pyenv/bin
[8077] ~/src/github/ansible/molecule/.tox$ ~/.pyenv/versions/3.7.1/bin/python3.7 -m virtualenv --system-site-packages --python ~/.pyenv/versions/3.7.1/bin/python3.7 .package
Already using interpreter ~/.pyenv/versions/3.7.1/bin/python3.7
Using base prefix '~/.pyenv/versions/3.7.1'
~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/virtualenv.py:1041: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
New python executable in ~/src/github/ansible/molecule/.tox/.package/bin/python3.7
Also creating executable in ~/src/github/ansible/molecule/.tox/.package/bin/python
Installing setuptools, pip, wheel...done.
.package installdeps: setuptools >= 40.8.0, setuptools_scm >= 1.15.0, setuptools_scm_git_archive >= 1.0, wheel, -rlint-requirements.txt, -rtest-requirements.txt
setting PATH=~/src/github/ansible/molecule/.tox/.package/bin:~/.pyenv/versions/3.7.1/bin:~/.pyenv/libexec:~/.pyenv/plugins/python-build/bin:~/.pyenv/plugins/pyenv-virtualenv/bin:~/.pyenv/plugins/pyenv-update/bin:~/.pyenv/plugins/pyenv-installer/bin:~/.pyenv/plugins/pyenv-doctor/bin:~/.nvm/versions/node/v9.8.0/bin:~/.pyenv/shims:/usr/lib/llvm/7/bin:/usr/lib/llvm/6/bin:/usr/lib/llvm/5/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:~/.pyenv/bin
[8144] ~/src/github/ansible/molecule$ ~/src/github/ansible/molecule/.tox/.package/bin/python -m pip install 'setuptools >= 40.8.0' 'setuptools_scm >= 1.15.0' 'setuptools_scm_git_archive >= 1.0' wheel -rlint-requirements.txt -rtest-requirements.txt
Double requirement given: wheel==0.30.0 (from -r test-requirements.txt (line 12)) (already in wheel, name='wheel')
ERROR: invocation failed (exit code 1)
ERROR: could not install deps [setuptools >= 40.8.0, setuptools_scm >= 1.15.0, setuptools_scm_git_archive >= 1.0, wheel, -rlint-requirements.txt, -rtest-requirements.txt]; v = InvocationError("~/src/github/ansible/molecule/.tox/.package/bin/python -m pip install 'setuptools >= 40.8.0' 'setuptools_scm >= 1.15.0' 'setuptools_scm_git_archive >= 1.0' wheel -rlint-requirements.txt -rtest-requirements.txt", 1)
.package finish: getenv ~/src/github/ansible/molecule/.tox/.package after 3.18 seconds
.package start: get-build-requires ~/src/github/ansible/molecule/.tox/.package
setting PATH=~/src/github/ansible/molecule/.tox/.package/bin:~/.pyenv/versions/3.7.1/bin:~/.pyenv/libexec:~/.pyenv/plugins/python-build/bin:~/.pyenv/plugins/pyenv-virtualenv/bin:~/.pyenv/plugins/pyenv-update/bin:~/.pyenv/plugins/pyenv-installer/bin:~/.pyenv/plugins/pyenv-doctor/bin:~/.nvm/versions/node/v9.8.0/bin:~/.pyenv/shims:/usr/lib/llvm/7/bin:/usr/lib/llvm/6/bin:/usr/lib/llvm/5/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:~/.pyenv/bin
[8151] ~/src/github/ansible/molecule$ ~/src/github/ansible/molecule/.tox/.package/bin/python ~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/helper/build_requires.py setuptools.build_meta '' >.tox/.package/log/.package-0.log
.package finish: get-build-requires ~/src/github/ansible/molecule/.tox/.package after 0.30 seconds
Traceback (most recent call last):
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 90, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1654, in parseString
    raise exc
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1644, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 3417, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 3739, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 3400, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1406, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 2711, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pkg_resources._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3042, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 94, in __init__
    requirement_string[e.loc:e.loc + 8]))
pkg_resources.extern.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'-rlint-r'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/.pyenv/versions/3.7.1/bin/tox", line 10, in <module>
    sys.exit(cmdline())
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/session/__init__.py", line 43, in cmdline
    main(args)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/session/__init__.py", line 68, in main
    exit_code = session.runcommand()
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/session/__init__.py", line 192, in runcommand
    return self.subcommand_test()
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/session/__init__.py", line 208, in subcommand_test
    venv.package = self.hook.tox_package(session=self, venv=venv)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pluggy/__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pluggy/callers.py", line 201, in _multicall
    return outcome.get_result()
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pluggy/callers.py", line 76, in get_result
    raise ex[1].with_traceback(ex[2])
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pluggy/callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/package/__init__.py", line 16, in tox_package
    session.package, session.dist = get_package(session)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/package/__init__.py", line 29, in get_package
    package = acquire_package(config, session)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/package/__init__.py", line 40, in acquire_package
    path = build_package(config, session)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/package/builder/__init__.py", line 9, in build_package
    return build(config, session)
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/package/builder/isolated.py", line 34, in build
    base_build_deps = {pkg_resources.Requirement(r.name).key for r in package_venv.envconfig.deps}
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/tox/package/builder/isolated.py", line 34, in <setcomp>
    base_build_deps = {pkg_resources.Requirement(r.name).key for r in package_venv.envconfig.deps}
  File "~/.pyenv/versions/3.7.1/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3044, in __init__
    raise RequirementParseError(str(e))
pkg_resources.RequirementParseError: Invalid requirement, parse error at "'-rlint-r'"

The text was updated successfully, but these errors were encountered:

gaborbernat · 2019-04-01T22:26:29Z

Although the error is kinda cryptic this works as expected. If you define deps at base level (testenv) you're expected to clear it for the isolated build env:

[testenv:.package]
deps =

Will work.

webknjaz · 2019-04-01T22:28:16Z

So why can't tox do it automatically?

gaborbernat · 2019-04-01T22:29:55Z

We could... to be fair. Would make isolated build kinda special and not follow other env override logic, but I assume we already differ enough...

webknjaz · 2019-04-01T22:31:23Z

Yeah, at least make it more obvious what's happening and maybe yell at -rrequirements.txt in deps when first seeing it instead of making some internal parse failure leak...

gaborbernat · 2019-04-01T22:31:28Z

https://github.com/tox-dev/tox/blob/master/src/tox/config/__init__.py#L1138 beside description we should probably also ignore deps (aka overwrite to an empty array) 👍Want to do a PR with a test?

GitHub
tox-dev/tox
Command line driven CI frontend and development task automation tool - tox-dev/tox

webknjaz · 2019-04-01T22:32:44Z

Maybe, but not right now :) Trying to refactor packaging in another project...

webknjaz · 2019-04-02T21:08:20Z

@gaborbernat after hitting pypa/pip#6264 (comment) I thought that maybe this env could also disable system-site-packages for isolated venv as well?

gaborbernat · 2019-04-02T21:14:44Z

Agreed. Though I would personally remove and banish that option entirely. I don't think it ever is a good idea to have it.

gaborbernat · 2019-04-02T21:49:42Z

What's the reason your project uses it @webknjaz?

webknjaz · 2019-04-03T08:56:53Z

@gaborbernat the reason is that it uses libselinux python wrapper, which is not distributed via PyPI and linked against a lot of system things and only distributed via OS package managers meaning that it's either this or symlinking things into tox's venvs. It also speeds up some things when envs are heavy. It wasn't my idea but I couldn't come up with a better solution... I was trying to come up with a repackaging of selinux as manylinux1 wheels but it doesn't seem possible at all.

Ref: ansible/molecule#1823 (comment)

gaborbernat · 2019-04-03T13:25:22Z

I've done the fix yesterday and will release today 🤔

webknjaz · 2019-04-03T13:26:19Z

Nice, thanks!

webknjaz added the bug:normal affects many people or has quite an impact label Apr 1, 2019

gaborbernat changed the title ~~[BUG] isolated_build broken with indirect deps~~ isolated build env should ignore deps Apr 1, 2019

gaborbernat added the help:wanted Issues that have been acknowledged, a solution determined and a PR might likely be accepted. label Apr 1, 2019

gaborbernat mentioned this issue Apr 2, 2019

fix #1243

Merged

helpr bot added the pr-available label Apr 2, 2019

gaborbernat closed this as completed in #1243 Apr 2, 2019

helpr bot added pr-merged and removed pr-available labels Apr 2, 2019

tox-dev locked and limited conversation to collaborators Jan 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

isolated build env should ignore deps #1239

isolated build env should ignore deps #1239

webknjaz commented Apr 1, 2019

gaborbernat commented Apr 1, 2019

webknjaz commented Apr 1, 2019

gaborbernat commented Apr 1, 2019

webknjaz commented Apr 1, 2019

gaborbernat commented Apr 1, 2019 •

edited by unfurl-links bot

Loading

webknjaz commented Apr 1, 2019

webknjaz commented Apr 2, 2019

gaborbernat commented Apr 2, 2019

gaborbernat commented Apr 2, 2019

webknjaz commented Apr 3, 2019

gaborbernat commented Apr 3, 2019

webknjaz commented Apr 3, 2019

isolated build env should ignore deps #1239

isolated build env should ignore deps #1239

Comments

webknjaz commented Apr 1, 2019

gaborbernat commented Apr 1, 2019

webknjaz commented Apr 1, 2019

gaborbernat commented Apr 1, 2019

webknjaz commented Apr 1, 2019

gaborbernat commented Apr 1, 2019 • edited by unfurl-links bot Loading

webknjaz commented Apr 1, 2019

webknjaz commented Apr 2, 2019

gaborbernat commented Apr 2, 2019

gaborbernat commented Apr 2, 2019

webknjaz commented Apr 3, 2019

gaborbernat commented Apr 3, 2019

webknjaz commented Apr 3, 2019

gaborbernat commented Apr 1, 2019 •

edited by unfurl-links bot

Loading