pipenv install with private source doesn't use provided environment variables #1906

AndreasPresthammer · 2018-04-04T13:56:05Z

Expected result

Have configured a private source which require username and password to authenticate. (It's a privately hosted https://jfrog.com/artifactory/ pypi repo.)

[[source]]
url = "http://${ARTIFACTORY_USERNAME}:${ARTIFACTORY_PASSWORD}@my-private-repo"
verify_ssl = false
name = "artifactory"

When running ARTIFACTORY_USERNAME=myuser ARTIFACTORY_PASSWORD=mypass pipenv install from the same directory as the pipfile on a project, I would expect the env-variables in the private source to use the values I provided.

I've also tried to provide env-variables in other way:

Setting env variable inline on the command (as shown in the example command above)
Exporting env variables
Putting env variables into a '.env' file that look like this:

ARTIFACTORY_USERNAME=myuser
ARTIFACTORY_PASSWORD=mypass

All of the above give me the same result error result (below).

When I provide the username and password directly inline in the pipfile it works fine, but when I use environment variables instead it fails with the following error:

Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.python.org/simple --extra-index-url http://${ARTIFACTORY_USERNAME}:${ARTIFACTORY_PASSWORD}@my-private-repo --trusted-host my-private-repo

                          ROUND 1                           
Current constraints:
  aiopubsub==1.0.0
  asynctest
  pytest==3.3.2

Finding the best candidates:
  found candidate aiopubsub==1.0.0 (constraint was ==1.0.0)
User for artifactory.firmglobal.com: 
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): artifactory.firmglobal.com
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pipenv/resolver.py", line 82, in <module>
    main()
  File "/usr/local/lib/python2.7/dist-packages/pipenv/resolver.py", line 71, in main
    clear=do_clear,
  File "/usr/local/lib/python2.7/dist-packages/pipenv/resolver.py", line 63, in resolve
    verbose=verbose,
  File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 425, in resolve_deps
    pre,
  File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 336, in actually_resolve_reps
    resolved_tree.update(resolver.resolve(max_rounds=PIPENV_MAX_ROUNDS))
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/resolver.py", line 193, in _resolve_one_round
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/resolver.py", line 193, in <setcomp>
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/resolver.py", line 257, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/repositories/pypi.py", line 116, in find_best_match
    all_candidates = self.find_all_candidates(ireq.name)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/repositories/pypi.py", line 101, in find_all_candidates
    candidates = self.finder.find_all_candidates(req_name)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/index.py", line 456, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/index.py", line 606, in _get_pages
    page = self._get_page(location)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/index.py", line 722, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/index.py", line 831, in get_page
    "Cache-Control": "max-age=600",
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 488, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 602, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/download.py", line 181, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line

/usr/local/lib/python2.7/dist-packages/pipenv/utils.py:1230: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/pipenv-ip4PTc-requirements'>
  warnings.warn(warn_message, ResourceWarning)

I assume it's just a confusing error message to tell me that the authentication fails, since I get the exact same error message if I put in an invalid username or password directly in the pipfile private source description.

Steps to replicate

pipfile which use source that require authentication. Use env variables for credentials:

[[source]]
url = "http://${ARTIFACTORY_USERNAME}:${ARTIFACTORY_PASSWORD}@private-repo"
verify_ssl = false
name = "my repo"

Run pipenv install where you provide credentials:
ARTIFACTORY_USERNAME=user ARTIFACTORY_PASSWORD=pass pipenv install

What am I doing wrong?

$ python -m pipenv.help output

Pipenv version: '11.9.0'

Pipenv location: '/usr/local/lib/python2.7/dist-packages/pipenv'

Python location: '/usr/bin/python'

Other Python installations in PATH:

2.7: /usr/bin/python2.7
2.7: /usr/bin/python2.7
3.5: /usr/bin/python3.5m
3.5: /usr/bin/python3.5
3.6: /usr/bin/python3.6m
3.6: /usr/bin/python3.6
2.7.12: /usr/bin/python
2.7.12: /usr/bin/python2
3.5.2: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.4.0-116-generic',
 'platform_system': 'Linux',
 'platform_version': '#140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018',
 'python_full_version': '2.7.12',
 'python_version': '2.7',
 'sys_platform': 'linux2'}

System environment variables:

LC_NUMERIC
MANDATORY_PATH
CONSCRIPT_OPTS
_LXSESSION_PID
XDG_GREETER_DATA_DIR
SPARK_HOME
PROJECT_HOME
LC_CTYPE
PYTHONDONTWRITEBYTECODE
XDG_CURRENT_DESKTOP
LC_PAPER
LOGNAME
XDG_SEAT
PATH
XDG_VTNR
QT_PLATFORM_PLUGIN
PYTHONUNBUFFERED
VIRTUALENVWRAPPER_SCRIPT
ZSH
DISPLAY
XDG_SESSION_DESKTOP
LANG
CONFIRMIT_CFG_DIR
TERM
SHELL
XDG_SESSION_PATH
XAUTHORITY
LANGUAGE
SHLVL
QT_QPA_PLATFORMTHEME
TERMINATOR_UUID
PIP_REQUIRE_VIRTUALENV
WINDOWID
_
SSH_AGENT_PID
HOME
LC_MONETARY
XDG_CONFIG_HOME
SAL_USE_VCLPLUGIN
CONSCRIPT_HOME
XDG_RUNTIME_DIR
WORKON_HOME
VIRTUALENVWRAPPER_PROJECT_FILENAME
LC_ADDRESS
SSH_AUTH_SOCK
VIRTUALENV_PYTHON
GDMSESSION
VIRTUALENVWRAPPER_WORKON_CD
CONFIRMIT_CFG_HOME
IBUS_DISABLE_SNOOPER
XDG_SEAT_PATH
PIP_PYTHON_PATH
XDG_SESSION_ID
DBUS_SESSION_BUS_ADDRESS
ORBIT_SOCKETDIR
VIRTUALENVWRAPPER_HOOK_DIR
LC_IDENTIFICATION
DESKTOP_SESSION
LSCOLORS
XDG_CONFIG_DIRS
DEFAULTS_PATH
XDG_SESSION_TYPE
OLDPWD
LS_COLORS
GDM_LANG
LC_TELEPHONE
LC_MEASUREMENT
PWD
COLORTERM
LC_NAME
XDG_MENU_PREFIX
LC_TIME
LESS
PAGER
XDG_DATA_DIRS
USER

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /home/andreas/.conscript/bin:/opt/spark-2.2.1/bin:/opt/flink/bin:/home/andreas/activator-dist-1.3.12/bin:/home/andreas/bin:/home/andreas/bin:/home/andreas/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
SHELL: /usr/bin/zsh
LANG: en_US.UTF-8
PWD: /path-to-project

Contents of Pipfile ('/path-to-project/Pipfile'):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "http://$USERNAME:$PASSWORD@my-private-repo"
verify_ssl = false
name = "artifactory"

[dev-packages]
pytest = "==3.3.2"
asynctest = "*"
aiopubsub = "==1.0.0"

[packages]
aio-pika = "==1.8.1"
deepdiff = "==3.3.0"
pika = "*"
async-timeout = "*"
async-generator = "*"
"e1839a8" = {path = ".", editable = true}

[requires]
python_version = "3.6"

The text was updated successfully, but these errors were encountered:

uranusjr · 2018-04-04T15:09:23Z

Dotenv are only loaded with run and shell, not install (and other commands). Yes, I know it’s weird, but that’s a design decision I don’t have the authority change ¯\_(ツ)_/¯ You’ll need to load your environs in other forms.

AndreasPresthammer · 2018-04-04T16:45:22Z

It's fine that the .env approach doesn't work, but shouldn't the other ones work?

JoeyBG · 2018-04-06T16:04:03Z

I am experiencing the same issue, it's downloading packages but failing to update/create Pipfile.lock

techalchemy · 2018-04-09T20:15:49Z

Yes if these are set outside of .env this should work just fine.

Tested this against current master and it is definitely working, so if it isn't for you try master

JoeyBG · 2018-04-09T21:15:32Z

Hi. It's not working with latest code, at least in my setup.

Here is my Pipfile

[[source]]
url = "https://${PYPI_USERNAME}:${PYPI_PASSWORD}@pypi.xpertsea.com/simple/"
verify_ssl = true
name = "pypi"

[packages]
dpath = "*"
scipy = "*"

[dev-packages]
mypy = "*"
pylint = "*"

[requires]
python_version = "3.6"

Here is what I get using pipenv lock :

Locking [dev-packages] dependencies…
candidates = self.finder.find_all_candidates(req_name)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/patched/notpip/index.py", line 456, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/patched/notpip/index.py", line 606, in _get_pages
    page = self._get_page(location)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/patched/notpip/index.py", line 723, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/patched/notpip/index.py", line 832, in get_page
    "Cache-Control": "max-age=600",
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 488, in get
    return self.request('GET', url, **kwargs)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 602, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/download.py", line 181, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line

Drilling down the code it appears that at line 1061 in core.py when venv_resolve_deps is called, it'll eventually use the Project object that has been explicitly told to not expand venvs when parsed, so the call to PyPI fails with invalid credentials and produce this weird error. It actually happens down the line in utils.py, around line 345 where the actual PyPI call is.

I'm just not sure where the best place is to actually fix this issue without causing another bug later down the row..

techalchemy · 2018-04-09T22:05:18Z

gotcha, so the issue occurs during locking because of project.py:

    @property
    def _lockfile(self):
        """Pipfile.lock divided by PyPI and external dependencies."""
        pfile = pipfile.load(self.pipfile_location, inject_env=False)
        lockfile = json.loads(pfile.lock())
        for section in ('default', 'develop'):
            lock_section = lockfile.get(section, {})
            for key in list(lock_section.keys()):
                norm_key = pep423_name(key)
                lockfile[section][norm_key] = lock_section.pop(key)
        return lockfile

techalchemy · 2018-04-09T22:08:34Z

I don't know if it's psosible to address this without also patching the piptools resolver debug logging functionality to prevent it from logging the interpolated index urls to the console...

eolo999 · 2018-04-12T09:54:41Z

I am also affected by this problem. Are there any news?

techalchemy · 2018-04-12T12:07:25Z

@eolo999 not yet unfortunately. We will update here when we get this sorted

- Fixes #1906 Signed-off-by: Dan Ryan <[email protected]>

Gr1N · 2018-04-22T17:04:56Z

@techalchemy when you plan to release new version of pipenv with this fix?

techalchemy · 2018-04-22T17:54:47Z

@Gr1N finishing up an automated suite that handles rebuilding vendored dependencies with licenses (due to some legal issues) => see #2035 -- I want to get this included because it adjusts for some SSL changes

Ideally this will be out today, at the very least I can put it up in prerelease for testing purposes once builds are passing

techalchemy · 2018-04-22T17:55:08Z

sorry for the holdup

techalchemy · 2018-04-23T06:51:00Z

pip install --upgrade --pre pipenv -- if things are working I'll move it out of prerelease tomorrow

Gr1N · 2018-04-23T07:52:31Z

@techalchemy version 11.10.1.dev1 works fine for me thanks! Also as I can see you need to updated docs:

Luckily - pipenv will hash your Pipfile before expanding environment variables (and, helpfully, will substitute the environment variables again when you install from the lock file - so no need to commit any secrets! Woo!)

It's not true anymore, as I see in Pipfile.lock env vars not hashed and it's great to my mind:

        ...
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            },
            {
                "name": "artifactory",
                "url": "${EXTRA_PYPI_INDEX}",
                "verify_ssl": true
            }
        ]
        ...

Gr1N · 2018-04-23T08:07:47Z

Found an issue #2037

jayvdb mentioned this issue Apr 7, 2018

Support Pipfile netlify/build-image#168

Merged

techalchemy closed this as completed Apr 9, 2018

techalchemy reopened this Apr 9, 2018

techalchemy added Type: Bug 🐛 This issue is a bug. Category: Dependency Resolution Issue relates to dependency resolution. Category: Private PyPIs 😎 Problem relates to private PyPI usage. labels Apr 9, 2018

techalchemy assigned techalchemy and jtratner and unassigned techalchemy Apr 9, 2018

AndreasPresthammer mentioned this issue Apr 13, 2018

install with '"e1839a8" = {path = ".", editable = true}' in Pipfile refuse to install other dependencies #1984

Closed

techalchemy added a commit that referenced this issue Apr 17, 2018

Add pipfile env var expansion and test

3722d8a

- Fixes #1906 Signed-off-by: Dan Ryan <[email protected]>

techalchemy mentioned this issue Apr 17, 2018

Add pipfile env var expansion and test #2015

Merged

techalchemy closed this as completed in #2015 Apr 19, 2018

coulonxyz mentioned this issue Jun 12, 2018

pipenv install with environment variable fails. #2351

Closed

ilons mentioned this issue Jun 20, 2018

Environment variables in Pipfile does not resolve #2389

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pipenv install with private source doesn't use provided environment variables #1906

pipenv install with private source doesn't use provided environment variables #1906

AndreasPresthammer commented Apr 4, 2018 •

edited

Loading

uranusjr commented Apr 4, 2018

AndreasPresthammer commented Apr 4, 2018

JoeyBG commented Apr 6, 2018

techalchemy commented Apr 9, 2018

JoeyBG commented Apr 9, 2018

techalchemy commented Apr 9, 2018

techalchemy commented Apr 9, 2018

eolo999 commented Apr 12, 2018

techalchemy commented Apr 12, 2018

Gr1N commented Apr 22, 2018

techalchemy commented Apr 22, 2018

techalchemy commented Apr 22, 2018

techalchemy commented Apr 23, 2018

Gr1N commented Apr 23, 2018

Gr1N commented Apr 23, 2018

pipenv install with private source doesn't use provided environment variables #1906

pipenv install with private source doesn't use provided environment variables #1906

Comments

AndreasPresthammer commented Apr 4, 2018 • edited Loading

Expected result

Steps to replicate

uranusjr commented Apr 4, 2018

AndreasPresthammer commented Apr 4, 2018

JoeyBG commented Apr 6, 2018

techalchemy commented Apr 9, 2018

JoeyBG commented Apr 9, 2018

techalchemy commented Apr 9, 2018

techalchemy commented Apr 9, 2018

eolo999 commented Apr 12, 2018

techalchemy commented Apr 12, 2018

Gr1N commented Apr 22, 2018

techalchemy commented Apr 22, 2018

techalchemy commented Apr 22, 2018

techalchemy commented Apr 23, 2018

Gr1N commented Apr 23, 2018

Gr1N commented Apr 23, 2018

AndreasPresthammer commented Apr 4, 2018 •

edited

Loading