Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create virtualenv failed via eclipse pydev debug while succeeded via run #1423

Closed
4 tasks done
guyingzhao opened this issue Oct 9, 2019 · 4 comments
Closed
4 tasks done

Comments

@guyingzhao
Copy link

guyingzhao commented Oct 9, 2019

  • Minimal reproducible example or detailed descriptions

Reproducible script:

import shutil
import virtualenv

try:
    virtualenv.create_environment("venv", site_packages=True)
finally:
    shutil.rmtree("venv", True)

I've add some print info to clearly address the issue.
If you execute above script using eclipse pydev debug, you will get:

pydev debugger: starting (pid: 64520)
cmd=['/Users/guying/workspace/Trial/venv/bin/python', '-', 'setuptools', 'pip', 'wheel']
virtualenv script args: ['install', '--ignore-installed', '-v', '--cert', '/var/folders/_b/k1ny77b976zcrghc8n8qly580000gp/T/tmp6pS_wr', '/Users/guying/.p2/pool/plugins/org.python.pydev.core_7.3.0.201908161924/pysrc/pydevd.py', '--port', '56449', '--vm_type', 'python', '--client', '127.0.0.1', '--multiprocess', '--print-in-debugger-startup', '--file', 'setuptools', 'pip', 'wheel']

Usage:   
  - install [options] <requirement specifier> [package-index-options] ...
  - install [options] -r <requirements file> [package-index-options] ...
  - install [options] [-e] <vcs project url> ...
  - install [options] [-e] <local project path> ...
  - install [options] <archive url/path> ...

no such option: --port
Traceback (most recent call last):
  File "/Users/guying/.p2/pool/plugins/org.python.pydev.core_7.3.0.201908161924/pysrc/pydevd.py", line 2643, in <module>
    main()
  File "/Users/guying/.p2/pool/plugins/org.python.pydev.core_7.3.0.201908161924/pysrc/pydevd.py", line 2636, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Users/guying/.p2/pool/plugins/org.python.pydev.core_7.3.0.201908161924/pysrc/pydevd.py", line 1920, in run
    return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
  File "/Users/guying/.p2/pool/plugins/org.python.pydev.core_7.3.0.201908161924/pysrc/pydevd.py", line 1927, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Users/guying/workspace/Trial/bbb.py", line 6, in <module>
    virtualenv.create_environment("venv", site_packages=True)
  File "/Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv.py", line 1179, in create_environment
    install_wheel(to_install, py_executable, search_dirs, download=download)
  File "/Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv.py", line 1019, in install_wheel
    _install_wheel_with_search_dir(download, project_names, py_executable, search_dirs)
  File "/Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv.py", line 1113, in _install_wheel_with_search_dir
    out = call_subprocess(cmd, extra_env=env, stdin=script)
  File "/Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv.py", line 963, in call_subprocess
    raise OSError("Command {} failed with error code {}".format(cmd_desc, proc.returncode))
OSError: Command /Users/guying/workspace/Trial/venv/bin/python - setuptools pip wheel failed with error code 2

And if you execute above script using eclipse pydev run, no error would be raised, and you will get:

cmd=['/Users/guying/workspace/Trial/venv/bin/python', '-', 'setuptools', 'pip', 'wheel']
virtualenv script args: ['install', '--ignore-installed', '-v', '--cert', '/var/folders/_b/k1ny77b976zcrghc8n8qly580000gp/T/tmpdkROGM', 'setuptools', 'pip', 'wheel']
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Ignoring indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Created temporary directory: /private/var/folders/_b/k1ny77b976zcrghc8n8qly580000gp/T/pip-ephem-wheel-cache-BAWcdz
Created temporary directory: /private/var/folders/_b/k1ny77b976zcrghc8n8qly580000gp/T/pip-req-tracker-lpWB7N
Created requirements tracker '/private/var/folders/_b/k1ny77b976zcrghc8n8qly580000gp/T/pip-req-tracker-lpWB7N'
Created temporary directory: /private/var/folders/_b/k1ny77b976zcrghc8n8qly580000gp/T/pip-install-NqjNvV
Looking in links: /Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support
Collecting setuptools
  0 location(s) to search for versions of setuptools:
  Skipping link /Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support (from -f); not a file
  Skipping link file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/pip-19.2.3-py2.py3-none-any.whl; wrong project name (not setuptools)
  Found link file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/setuptools-41.2.0-py2.py3-none-any.whl, version: 41.2.0
  Skipping link file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/__init__.py; unsupported archive format: .py
  Skipping link file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/__init__.pyc; unsupported archive format: .pyc
  Skipping link file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl; wrong project name (not setuptools)
  Skipping link file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/wheel-0.33.6-py2.py3-none-any.whl; wrong project name (not setuptools)
  Local files found: /Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/setuptools-41.2.0-py2.py3-none-any.whl
  Using version 41.2.0 (newest of versions: 41.2.0)
  Added setuptools from file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/setuptools-41.2.0-py2.py3-none-any.whl to build tracker '/private/var/folders/_b/k1ny77b976zcrghc8n8qly580000gp/T/pip-req-tracker-lpWB7N'
  Removed setuptools from file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/setuptools-41.2.0-py2.py3-none-any.whl from build tracker '/private/var/folders/_b/k1ny77b976zcrghc8n8qly580000gp/T/pip-req-tracker-lpWB7N'
Collecting pip
  0 location(s) to search for versions of pip:
  Found link file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/pip-19.2.3-py2.py3-none-any.whl, version: 19.2.3
  Skipping link file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/setuptools-41.2.0-py2.py3-none-any.whl; wrong project name (not pip)
  Found link file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl, version: 19.1.1
  Local files found: /Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/pip-19.2.3-py2.py3-none-any.whl, /Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl
  Using version 19.2.3 (newest of versions: 19.1.1, 19.2.3)
  Added pip from file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/pip-19.2.3-py2.py3-none-any.whl to build tracker '/private/var/folders/_b/k1ny77b976zcrghc8n8qly580000gp/T/pip-req-tracker-lpWB7N'
  Removed pip from file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/pip-19.2.3-py2.py3-none-any.whl from build tracker '/private/var/folders/_b/k1ny77b976zcrghc8n8qly580000gp/T/pip-req-tracker-lpWB7N'
Collecting wheel
  0 location(s) to search for versions of wheel:
  Found link file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/wheel-0.33.6-py2.py3-none-any.whl, version: 0.33.6
  Local files found: /Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/wheel-0.33.6-py2.py3-none-any.whl
  Using version 0.33.6 (newest of versions: 0.33.6)
  Added wheel from file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/wheel-0.33.6-py2.py3-none-any.whl to build tracker '/private/var/folders/_b/k1ny77b976zcrghc8n8qly580000gp/T/pip-req-tracker-lpWB7N'
  Removed wheel from file:///Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/wheel-0.33.6-py2.py3-none-any.whl from build tracker '/private/var/folders/_b/k1ny77b976zcrghc8n8qly580000gp/T/pip-req-tracker-lpWB7N'
Installing collected packages: setuptools, pip, wheel

  changing mode of /Users/guying/workspace/Trial/venv/bin/easy_install to 755
  changing mode of /Users/guying/workspace/Trial/venv/bin/easy_install-2.7 to 755

  changing mode of /Users/guying/workspace/Trial/venv/bin/pip to 755
  changing mode of /Users/guying/workspace/Trial/venv/bin/pip2 to 755
  changing mode of /Users/guying/workspace/Trial/venv/bin/pip2.7 to 755

  changing mode of /Users/guying/workspace/Trial/venv/bin/wheel to 755
Successfully installed pip-19.2.3 setuptools-41.2.0 wheel-0.33.6
Cleaning up...
Removed build tracker '/private/var/folders/_b/k1ny77b976zcrghc8n8qly580000gp/T/pip-req-tracker-lpWB7N'
{'PIP_FIND_LINKS': '/Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support', 'PIP_ONLY_BINARY': ':all:', 'PYTHONPATH': '/Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/setuptools-41.2.0-py2.py3-none-any.whl:/Users/guying/Library/Python/2.7/lib/python/site-packages/virtualenv_support/pip-19.1.1-py2.py3-none-any.whl', 'PIP_NO_INDEX': '1', 'PIP_USER': '0', 'PIP_USE_WHEEL': '1', 'PIP_NO_INPUT': '1'}

import sys
import pkgutil
import tempfile
import os

defined_cert = False

try:
    from pip._internal import main as _main
    cert_data = pkgutil.get_data("pip._vendor.certifi", "cacert.pem")
except ImportError:
    from pip import main as _main
    cert_data = pkgutil.get_data("pip._vendor.requests", "cacert.pem")
except IOError:
    cert_data = None

if not defined_cert and cert_data is not None:
    cert_file = tempfile.NamedTemporaryFile(delete=False)
    cert_file.write(cert_data)
    cert_file.close()
else:
    cert_file = None

try:
    args = ["install"] + ['--ignore-installed', '-v']
    if cert_file is not None:
        args += ["--cert", cert_file.name]
    args += sys.argv[1:]
    print("virtualenv script args: %s" % str(args))

    sys.exit(_main(args))
finally:
    if cert_file is not None:
        os.remove(cert_file.name)
  • OS and pip list output
    macos
Package                                Version
-------------------------------------- -----------
altgraph                               0.10.2
Appium-Python-Client                   0.46
astor                                  0.8.0
atomicwrites                           1.3.0
attrs                                  19.1.0
backports-abc                          0.5
bdist-mpkg                             0.5.0
bleach                                 3.1.0
bonjour-py                             0.3
certifi                                2019.6.16
chardet                                3.0.4
configparser                           4.0.1
contextlib2                            0.5.5
decorator                              4.4.0
dis3                                   0.1.3
docopt                                 0.6.2
docutils                               0.15.2
enum                                   0.4.7
facebook-wda                           0.3.4
fire                                   0.1.3
funcsigs                               1.0.2
functools32                            3.2.3.post2
futures                                3.3.0
humanize                               0.5.1
idna                                   2.8
importlib-metadata                     0.21
jsonschema                             3.0.2
logzero                                1.5.0
lxml                                   4.3.4
macholib                               1.11
matplotlib                             1.3.1
modulegraph                            0.10.4
more-itertools                         5.0.0
numpy                                  1.16.5
opencv-python                          4.1.1.26
packaging                              19.1
pathlib2                               2.3.4
Pillow                                 6.1.0
pip                                    19.2.1
pkginfo                                1.5.0.1
pluggy                                 0.13.0
ply                                    3.11
progress                               1.5
py                                     1.8.0
py2app                                 0.7.3
Pygments                               2.4.2
PyInstaller                            3.5
pyobjc-core                            2.5.1
pyobjc-framework-Accounts              2.5.1
pyobjc-framework-AddressBook           2.5.1
pyobjc-framework-AppleScriptKit        2.5.1
pyobjc-framework-AppleScriptObjC       2.5.1
pyobjc-framework-Automator             2.5.1
pyobjc-framework-CFNetwork             2.5.1
pyobjc-framework-Cocoa                 2.5.1
pyobjc-framework-Collaboration         2.5.1
pyobjc-framework-CoreData              2.5.1
pyobjc-framework-CoreLocation          2.5.1
pyobjc-framework-CoreText              2.5.1
pyobjc-framework-DictionaryServices    2.5.1
pyobjc-framework-EventKit              2.5.1
pyobjc-framework-ExceptionHandling     2.5.1
pyobjc-framework-FSEvents              2.5.1
pyobjc-framework-InputMethodKit        2.5.1
pyobjc-framework-InstallerPlugins      2.5.1
pyobjc-framework-InstantMessage        2.5.1
pyobjc-framework-LatentSemanticMapping 2.5.1
pyobjc-framework-LaunchServices        2.5.1
pyobjc-framework-Message               2.5.1
pyobjc-framework-OpenDirectory         2.5.1
pyobjc-framework-PreferencePanes       2.5.1
pyobjc-framework-PubSub                2.5.1
pyobjc-framework-QTKit                 2.5.1
pyobjc-framework-Quartz                2.5.1
pyobjc-framework-ScreenSaver           2.5.1
pyobjc-framework-ScriptingBridge       2.5.1
pyobjc-framework-SearchKit             2.5.1
pyobjc-framework-ServiceManagement     2.5.1
pyobjc-framework-Social                2.5.1
pyobjc-framework-SyncServices          2.5.1
pyobjc-framework-SystemConfiguration   2.5.1
pyobjc-framework-WebKit                2.5.1
pyOpenSSL                              0.13.1
pyparsing                              2.4.2
PyPubSub                               3.3.0
pyrsistent                             0.15.4
pytest                                 4.6.5
python-dateutil                        1.5
pytz                                   2013.7
readme-renderer                        24.0
requests                               2.22.0
requests-toolbelt                      0.9.1
retry                                  0.9.2
scandir                                1.10.0
scipy                                  0.13.0b1
selenium                               3.141.0
setuptools                             41.0.1
shoots                                 1.0.0
singledispatch                         3.4.0.3
six                                    1.12.0
tornado                                5.1.1
tqdm                                   4.32.2
twine                                  1.13.0
uiautomator2                           0.2.4.dev5
urllib3                                1.25.3
virtualenv                             16.7.5
wcwidth                                0.1.7
webencodings                           0.5.1
weditor                                0.1.1
whichcraft                             0.6.0
wxPython                               4.0.3
xattr                                  0.6.4
zipp                                   0.6.0
  • Reason:
    pydev debugger will rewrite command line and virtualenv cannot recognize the corresponding modification, it simply cascade the sys.argv[1:] to pre-defined args.

  • Fix:
    using sys.argv[-3:] instead of sys.argv[1:]

@guyingzhao
Copy link
Author

Fixed code as follows:
replace following script at virtualenv.py at line 1049

script = textwrap.dedent(
        """
        import sys
        import pkgutil
        import tempfile
        import os

        defined_cert = {defined_cert}

        try:
            from pip._internal import main as _main
            cert_data = pkgutil.get_data("pip._vendor.certifi", "cacert.pem")
        except ImportError:
            from pip import main as _main
            cert_data = pkgutil.get_data("pip._vendor.requests", "cacert.pem")
        except IOError:
            cert_data = None

        if not defined_cert and cert_data is not None:
            cert_file = tempfile.NamedTemporaryFile(delete=False)
            cert_file.write(cert_data)
            cert_file.close()
        else:
            cert_file = None

        try:
            args = ["install"] + [{extra_args}]
            if cert_file is not None:
                args += ["--cert", cert_file.name]
            args += sys.argv[-3:]

            sys.exit(_main(args))
        finally:
            if cert_file is not None:
                os.remove(cert_file.name)
    """.format(
            defined_cert=defined_cert, extra_args=", ".join(repr(i) for i in extra_args)
        )
    ).encode("utf8")

@guyingzhao
Copy link
Author

I've seen the same issue on Pycharm and it's supposed to be on vscode either.

@gaborbernat
Copy link
Contributor

@guyingzhao technically this is a pydev BUG, as it does not correctly handles rewrites when the input script is feed via the stdin (which is a perfectly valid use case). Will not manifest with the next generation virtualenv (WIP) inside #1366 #1377. Will be automatically solved by that by side-stepping.

@guyingzhao
Copy link
Author

@gaborbernat ok, I will temporarily resolve this by myself and wait for official update.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants