Allow users to override default PyPI index URLs with PyPI mirror URLs #2281

JacobHenner · 2018-05-30T21:35:28Z

In response to #2075, allow users to override the default PyPI indices (pypi.org, pypi.python.org) with a mirror specified via CLI parameter or environment variable.

JacobHenner · 2018-05-30T21:37:07Z

WIP - Still need to write the test cases and update the docs, but figured I'd start the PR to encourage review.

uranusjr · 2018-05-31T02:23:28Z

pipenv/cli.py

+    nargs=1,
+    callback=validate_pypi_mirror,
+    help="Specify a PyPI mirror.",
+)


Is the option needed here? I’d be more inclined to put it only on install, lock, update, and sync. I don’t quite see the need of passing this for project initialisation.

It looks like this one is indeed for the install operation.

uranusjr · 2018-05-31T02:28:44Z

The implementation looks reasonable with a brief read (I didn’t actually run it, and yes tests are needed). I’m not sure if the flag is needed for bare pipenv calls.

JacobHenner · 2018-05-31T03:48:13Z

I've been working on adding tests for install, but I'm running into an issue testing the --pypi-mirror parameter. It looks like the test suite overrides the default sources to use a locally-run web server, but I've written a test to ensure setting --pypi-mirror doesn't modify the Pipfile's sources. I haven't been able to narrow down where this is occurring, or how to prevent it, however, so the test fails when it reads a source url "http://localhost...".

Also, right now this just covers install. I hope to test and address all functionality currently dependent on the two primary PyPI urls before proceeding.

techalchemy · 2018-05-31T05:36:11Z

Thanks for the PR! The test fixtures and the project.py itself are both configured to use an enviroment variable to configure the test source to always use a local pypi cache for testing. Your test just needs to verify that the real url, when you place it in the sources somewhere, gets overridden during the install process. So you can write a Pipfile with an extra source to actual pypi and use p.pipenv('install --pypi-mirror') and use the verbose flag and just verify that the actual url is not present in the output or something, since it should include source urls

JacobHenner · 2018-05-31T13:58:58Z

@techalchemy Thanks, I'll take a look at that.

JacobHenner · 2018-05-31T15:44:07Z

@techalchemy If I go that route, where would you suggest I store that Pipfile?

For the local pypi cache, I can't seem to see where it's set. During each test run, I see the server uses 127.0.0.1 with a random port, but I'm still looking (without much luck) for where this is set, or where PIPENV_TEST_INDEX is set by the test suite.

uranusjr · 2018-05-31T16:26:09Z

@JacobNenner PIPENV_TEST_INDEX is set in conftest.py (test configuration file used by Pytest), when a PipenvInstance is initiated. You can disable the PyPI mocking by passing pypi=False when you initiate the PipenvInstance.

JacobHenner · 2018-05-31T17:30:32Z

@pytest.mark.install
@flaky
def test_mirror_install(PipenvInstance, pypi):
    with PipenvInstance(pypi=False) as p:
        # This should sufficiently demonstrate the mirror functionality
        # since pypi.org is the default.
        c = p.pipenv('install requests --pypi-mirror https://pypi.python.org/simple')
        assert c.return_code == 0
        # Ensure the --pypi-mirror parameter hasn't altered the Pipfile or Pipfile.lock sources
        assert len(p.pipfile['source']) == 1
        assert len(p.lockfile["_meta"]["sources"]) == 1
        assert 'https://pypi.org/simple' == p.pipfile['source'][0]['url']
        assert 'https://pypi.org/simple' == p.lockfile['_meta']['sources'][0]['url']
        
        assert 'requests' in p.pipfile['packages']
        assert 'requests' in p.lockfile['default']
        assert 'chardet' in p.lockfile['default']
        assert 'idna' in p.lockfile['default']
        assert 'urllib3' in p.lockfile['default']
        assert 'certifi' in p.lockfile['default']

still returns

test_mirror_install failed; it passed 0 out of the required 1 times.
	<type 'exceptions.AssertionError'>
	assert 'https://pypi.org/simple' == 'http://127.0.0.1:44423/simple'
  - https://pypi.org/simple
  + http://127.0.0.1:44423/simple

techalchemy · 2018-06-03T21:26:28Z

Hey thanks for all your work on this front, so here's what you actually need to do to get your tests to work (almost there!)-- not sure you need to pass chdir=True here but it's possible:

@pytest.mark.install
@flaky
def test_mirror_install(PipenvInstance):
    with PipenvInstance(chdir=True) as p:
        with open(p.pipfile_path, 'w') as f:
            f.write("""
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
            """.strip())
        # This should sufficiently demonstrate the mirror functionality
        # since pypi.org is the default.
        c = p.pipenv('install requests --pypi-mirror https://pypi.python.org/simple')
        assert c.return_code == 0
        # Ensure the --pypi-mirror parameter hasn't altered the Pipfile or Pipfile.lock sources
        assert len(p.pipfile['source']) == 1
        assert len(p.lockfile["_meta"]["sources"]) == 1
        assert 'https://pypi.org/simple' == p.pipfile['source'][0]['url']
        assert 'https://pypi.org/simple' == p.lockfile['_meta']['sources'][0]['url']

        assert 'requests' in p.pipfile['packages']
        assert 'requests' in p.lockfile['default']
        assert 'chardet' in p.lockfile['default']
        assert 'idna' in p.lockfile['default']
        assert 'urllib3' in p.lockfile['default']
        assert 'certifi' in p.lockfile['default']

And FYI, I grabbed a copy of your branch and ran this test on it and it passes, so nice work! :)

JacobHenner · 2018-06-04T17:46:42Z

Thanks @techalchemy. I've gone ahead and added that.

I'm now going to work on the other functions which depend on connectivity to PyPI, such as updates.

JacobHenner · 2018-06-05T02:40:05Z

This is almost finished, I have two things left to address:

There's currently an issue with setting an environment variable on Windows running Python 2, according to appveyor.
I need to write additional integration tests for update, sync, lock, and uninstall. The challenge will be figuring out which mirror can be used for testing, and how to ensure that mirror is being used (and ensure that PyPI isn't being used silently in the background).

Hopefully I'll have time to work on this tomorrow. If not, I'll try to get to it by Wednesday.

uranusjr · 2018-06-05T03:48:05Z

Maybe you can use the actual PyPI, and override it with the local (mocked) PyPI as mirror? I guess you could peek into the logs or peek into it directly to see if it gets used as expected.

JacobHenner · 2018-06-05T03:50:53Z

Looks like all that's left are the additional tests. Thanks for the suggestions @uranusjr and @techalchemy, I'll investigate them soon (hopefully tomorrow).

techalchemy · 2018-06-05T05:08:10Z

That's roughly what i've been attempting to suggest on IRC, but it might not be clear :p

JacobHenner · 2018-06-05T14:32:39Z

Looks like that worked.

def test_mirror_install(PipenvInstance, pypi):
    with temp_environ(), PipenvInstance(chdir=True) as p:
        mirror_url = os.environ['PIPENV_TEST_INDEX']
        del os.environ['PIPENV_TEST_INDEX']
        # This should sufficiently demonstrate the mirror functionality
        # since pypi.org is the default when PIPENV_TEST_INDEX is unset.
        c = p.pipenv('install requests --pypi-mirror {0}'.format(mirror_url))
        assert c.return_code == 0
        # Ensure the --pypi-mirror parameter hasn't altered the Pipfile or Pipfile.lock sources
        assert len(p.pipfile['source']) == 1
        assert len(p.lockfile["_meta"]["sources"]) == 1
        assert 'https://pypi.org/simple' == p.pipfile['source'][0]['url']
        assert 'https://pypi.org/simple' == p.lockfile['_meta']['sources'][0]['url']

        assert 'requests' in p.pipfile['packages']
        assert 'requests' in p.lockfile['default']
        assert 'chardet' in p.lockfile['default']
        assert 'idna' in p.lockfile['default']
        assert 'urllib3' in p.lockfile['default']
        assert 'certifi' in p.lockfile['default']

I'll go ahead and use this pattern for the rest of my tests. So far, I intend to add tests for:

update
uninstall
lock
sync

This should cover all of the functions that now have the extra pypi_mirror parameter. Not sure if I'll get to this today. If not, I'll have a look tomorrow.

JacobHenner · 2018-06-06T03:15:10Z

I've written tests that should cover installs, uninstalls, locks, and syncs using the --pypi-mirror parameter. Please let me know if you think there are additional tests I should add.

JacobHenner · 2018-06-06T19:53:08Z

@uranusjr @techalchemy I performed additional manual tests in a restrictive environment, with no access to PyPI. All tested operations worked as expected. Looks like this one is ready for review, unless there are additional integration tests you'd like me to include.

uranusjr · 2018-06-07T01:20:38Z

pipenv/utils.py

@@ -277,6 +277,7 @@ class PipCommand(basecommand.Command):
    pypi = PyPIRepository(
        pip_options=pip_options, use_json=True, session=session
    )
+    print(pip_options)


stray debug?

Yep, missed in my pre-commit diff grep. Removed.

uranusjr · 2018-06-07T01:21:01Z

pipenv/utils.py

@@ -442,6 +445,7 @@ def resolve_deps(
            collected_hashes = []
            if any('python.org' in source['url'] or 'pypi.org' in source['url']
                   for source in sources):
+                #FIXME Add support for mirrors.


Is this still relevant?

Nope, this was a note-to-self that was missed in my pre-commit diff grep. Removed.

That being said, it looks like something might have been missed with venv_resolve_deps, I'll take a look now.

Fixed the missing bit, will push shortly.

JacobHenner · 2018-06-07T02:07:03Z

Removed some accidentally included debugging artifacts, added pypi_mirror support to venv_resolve_deps in get_vcs_deps. Should be good to continue review.

brettdh · 2018-06-07T16:43:03Z

One note: starting from scratch, I had expected the first pipenv install --pypi-mirror <package> to set url = <mirror url> in the Pipfile's [[source]] section, but it didn't. Either I have to manually correct this (even though I've already typed the URL once), or someone else coming along and running pipenv install would then unintentionally install the packages from PyPI.

Is there a more ergonomic way to support the use case of initializing a project to use a mirror for all the packages? Or configuring an existing project to use a mirror from now on? I'd expect that this is more common than selectively installing some packages from PyPI and some from a mirror.

uranusjr · 2018-06-07T17:08:01Z

I think the behaviour is intended. The flag is used to override the index during lock and install, not replace it in the declaration. The “right” way to do what you want is to write the mirror into pypirc, or produce your own Pipfile by hand. This really goes back to #2188 IMO—we can’t have an ~~ergonic~~ (typo, ergonomic) way to do this unless we have an actual init command.

brettdh · 2018-06-07T17:15:26Z

Ah! Yes, I absolutely agree - a hypothetical pipenv init would be the right place for this. Thanks.

brettdh · 2018-06-07T17:18:48Z

Side note: I'm not sure how to do this with .pypirc (does pipenv use settings from there?), but editing the Pipfile by hand is fine for now.

uranusjr · 2018-06-07T17:25:11Z

@brettdh Sorry, I meant pip.conf :p I believe (didn’t check) we do extract sources from it when initialising the project.

JacobHenner · 2018-06-07T20:24:29Z

Looks like there have been some conflicting changes. I'll rebase and iron out the conflicts later today.

Adds support for the --pypi-mirror command line parameter and the PIPENV_PYPI_MIRROR environment variable for most pipenv operations. This permits pipenv to function without pypi.org, which is necessary for users: 1. behind restrictive networks 2. facing strict artifact sourcing policies 3. experiencing poor performance connecting to pypi.org 4. who've configured a local cache for performance reasons When specified, the value of this parameter replaces all instances of pypi.org and pypi.python.org within pipenv operations without modifying or requring the modification of Pipfiles. - Resolves pypa#2075

JacobHenner · 2018-06-07T22:25:26Z

Rebased and (seemingly) ready to go.

techalchemy · 2018-06-08T04:29:04Z

pipenv/core.py

@@ -1131,7 +1138,7 @@ def do_lock(
            lockfile['default'][dep['name']]['markers'] = dep['markers']
    # Add refs for VCS installs.
    # TODO: be smarter about this.
-    _vcs_deps, vcs_lockfiles = get_vcs_deps(project, pip_freeze, which=which, verbose=verbose, clear=clear, pre=pre, allow_global=system, dev=False)
+    _vcs_deps, vcs_lockfiles = get_vcs_deps(project, pip_freeze, which=which, verbose=verbose, clear=clear, pre=pre, allow_global=system, dev=False, pypi_mirror=pypi_mirror)


FYI I did realize how ugly and horrible this is, there is a minor refactor PR over in #2315 that will change this a bit

techalchemy · 2018-06-08T04:47:29Z

Merging, thanks for your hard work 🍰

JacobHenner mentioned this pull request May 30, 2018

Please add feature that can set default source #2280

Closed

JacobHenner force-pushed the add_pypi_mirrors branch from 7475070 to eb55b19 Compare May 30, 2018 22:00

uranusjr reviewed May 31, 2018

View reviewed changes

JacobHenner force-pushed the add_pypi_mirrors branch 2 times, most recently from 7b63a5e to 299baa7 Compare June 4, 2018 17:21

JacobHenner force-pushed the add_pypi_mirrors branch from e92f856 to f3d00fb Compare June 5, 2018 02:31

JacobHenner force-pushed the add_pypi_mirrors branch from 606d6b0 to 787a466 Compare June 5, 2018 03:27

JacobHenner force-pushed the add_pypi_mirrors branch from 787a466 to 9154f88 Compare June 5, 2018 03:48

JacobHenner force-pushed the add_pypi_mirrors branch from 9154f88 to 3eabbbe Compare June 6, 2018 02:57

JacobHenner changed the title ~~WIP: #2075 Allow users to override default PyPI index URLs with PyPI mirror URLs~~ Allow users to override default PyPI index URLs with PyPI mirror URLs Jun 6, 2018

JacobHenner force-pushed the add_pypi_mirrors branch from 3eabbbe to 69b3e43 Compare June 6, 2018 03:29

This was referenced Jun 6, 2018

inherit pypi url from global pip.conf #1734

Closed

Repeatability: offline mirror #1731

Closed

uranusjr reviewed Jun 7, 2018

View reviewed changes

JacobHenner force-pushed the add_pypi_mirrors branch 2 times, most recently from ba6ad56 to 625d739 Compare June 7, 2018 02:05

JacobHenner force-pushed the add_pypi_mirrors branch from 625d739 to 18f8a69 Compare June 7, 2018 02:56

JacobHenner force-pushed the add_pypi_mirrors branch from 18f8a69 to af91eb6 Compare June 7, 2018 22:24

techalchemy reviewed Jun 8, 2018

View reviewed changes

techalchemy merged commit 0fa7bd8 into pypa:master Jun 8, 2018

JacobHenner mentioned this pull request Nov 26, 2019

Replacing the URL of a source (e.g. PyPI) at the global level python-poetry/poetry#1632

Open

2 tasks

kalebmckale mentioned this pull request Jan 26, 2023

Allow default name="pypi" source URL to be configured. #5574

Closed

Allow users to override default PyPI index URLs with PyPI mirror URLs #2281

Allow users to override default PyPI index URLs with PyPI mirror URLs #2281

Conversation

JacobHenner commented May 30, 2018 • edited Loading

JacobHenner commented May 30, 2018

uranusjr May 31, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uranusjr commented May 31, 2018

JacobHenner commented May 31, 2018

techalchemy commented May 31, 2018

JacobHenner commented May 31, 2018

JacobHenner commented May 31, 2018

uranusjr commented May 31, 2018

JacobHenner commented May 31, 2018

techalchemy commented Jun 3, 2018

JacobHenner commented Jun 4, 2018

JacobHenner commented Jun 5, 2018

uranusjr commented Jun 5, 2018

JacobHenner commented Jun 5, 2018

techalchemy commented Jun 5, 2018

JacobHenner commented Jun 5, 2018

JacobHenner commented Jun 6, 2018

JacobHenner commented Jun 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JacobHenner commented Jun 7, 2018

brettdh commented Jun 7, 2018

uranusjr commented Jun 7, 2018 • edited Loading

brettdh commented Jun 7, 2018

brettdh commented Jun 7, 2018

uranusjr commented Jun 7, 2018 • edited Loading

JacobHenner commented Jun 7, 2018 via email

JacobHenner commented Jun 7, 2018

Choose a reason for hiding this comment

techalchemy commented Jun 8, 2018

JacobHenner commented May 30, 2018 •

edited

Loading

uranusjr May 31, 2018 •

edited

Loading

uranusjr commented Jun 7, 2018 •

edited

Loading

uranusjr commented Jun 7, 2018 •

edited

Loading