New Python template for pip without pipenv #324
Comments
pipenv is really slow and sometimes it hangs for a really long time. Perhaps because I'm using a private PyPi instance that makes it worse, but that works fine with plain pip. This seems to be a common problem - see and the several other linked issues there: pypa/pipenv#3827 So far, this is just the result of grep/search/replace. Haven't tested this at all. Just want to start by getting some feedback if this is going to be something that everyone is OK with. Closes hashicorp#324
pipenv is really slow and sometimes it hangs for a really long time. Perhaps because I'm using a private PyPi instance that makes it worse, but that works fine with plain pip. This seems to be a common problem - see and the several other linked issues there: pypa/pipenv#3827 So far, this is just the result of grep/search/replace. Haven't tested this at all and there's more work to do. Just want to start by getting some feedback if this is going to be something that everyone is OK with. Closes hashicorp#324
pipenv is really slow and sometimes it hangs for a really long time. Perhaps because I'm using a private PyPi instance that makes it worse, but that works fine with plain pip. This seems to be a common problem - see and the several other linked issues there: pypa/pipenv#3827 So far, this is just the result of grep/search/replace. Haven't tested this at all and there's more work to do. Just want to start by getting some feedback if this is going to be something that everyone is OK with. Closes hashicorp#324
|
I have a rough draft PR going here - #327. |
|
My gut feeling right now is, that this isn't something which we should just do. I'm not a frequent Python user, so I'll have to familiarize myself with the implications this has.
Why is the private PyPi instance worse? |
|
I can't imagine why my private PyPi would make the situation worse - it's a CodeArtifact repo hosted by AWS, so it should be fast and reliable. But let's ignore that for now - at my company we have one other non-cdktf project where someone tried to use pipenv - that doesn't use our new private PyPi - and we've had the same sorts of problems with pipenv there. Install and "locking" takes infinitely long for some users at some times and for other users at other times it's only moderately slow. By taking infinitely long, I mean it's so frustratingly slow we might have to walk away from our computer for a while only to come back and control+c to give up. Then we might try the same command later in the day on the same computer and it will work. There's no debug output to even get a better idea why it hangs, so it's super annoying. The slowness seems to be related to pipenv re-downloading all dependencies for every operation just to check the hashes. So as a project grows larger, this problem becomes more common. You might never see it in simple examples, but for larger Python projects I'd consider pipenv not ready for production use. Personally, I don't see pipenv offering much over plain pip. It seems to mimic some of the features of other package managers from the JavaScript and Ruby communities, but I don't think Python had a problem in this area. It's just a solution looking for a problem and it's only made the situation worse and more confusing. A simple requirements.txt file is simple and good enough for most apps and traditional packaging via setup.py is rock solid. By the way, in case you're under the impression that pipenv is an official package manager for Python (the pipenv docs used to say something like that) -- it's not. pipenv is not common in the Python community and it's certainly not a standard. This blog post describes the situation very well: https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/ And here are more references showing how bad it is: pypa/pipenv#3827 https://stackoverflow.com/questions/57646310/is-pythons-pipenv-slow In closing, my recommendation is pyenv for managing multiple Python interpreter versions (e.g. 3.6.x, 3.7.x, etc) and the pyenv-virtualenv plugin (https://github.com/pyenv/pyenv-virtualenv) for managing application specific environments. Both are easy to install/use (e.g. homebrew), fast and reliable. pipenv has wasted so much of my time that I'm motivated to do the work to rip it out of cdktf, but I don't want to spend a bunch of time on the PR unless you're on board with the idea. |
|
I also ran into pypa/pipenv#1050 when trying to get cdktf python working on Windows. |
|
@cmclaughlin thanks for elaborating on this, very appreciated! I'll go through the links and collect more feedback on this.
So you think Pipenv makes it harder to use on Windows?
Still haven't gotten a precise version compatibility info about Jsii - just saw a comment today stating |
I don't think it's specific to Windows. Right now the python template says that |
|
@skorfmann I don't think I have seen issues with pipenv. My python projects have been fairly quick when initializing and synthesizing applications. I think we should gather more feedback, look at what @cmclaughlin has linked and other folks in the community. |
|
I spent some time looking into my specific/recent problem with pipenv... I've determined pipenv is not compatible with AWS Code Artifact... sorry I brushed that off as a possibility earlier. While most cdktf users won't experience this, it certainly frustrates me enough to not want to use pipenv and recommend against anyone else using - because it's not following what the rest of the Python community is doing. Add that to my previous bad experiences with it, heaps of other people that don't like it and the general feeling that it's slow and unnecessary. Here are more details: PyPi has a "legacy" API with a "/simple" endpoint. The PyPi reference implementation supports listing all packages via that endpoint, however as best as I can tell normal pip does not use it. Here you can see the endpoint is still supported. I'm under the impression that normal pip used to parse that HTML response to follow links to packages or perhaps it still uses the HTML response to search/list packages. But I'm pretty sure newer endpoints, such as the JSON API, are used for that now. I am fairly certain normal pip does not use the /simple endpoint to install packages. Instead it simply gets /simple/$PACKAGE_NAME. I scoured through the pip code to confirm this, but ultimately some simple verbose output shows that installing boto3 goes straight to /simple/boto3/ Now, on to AWS CodeArtifact - it implements the PyPi spec, but it does not support listing packages via the simple API - from https://docs.aws.amazon.com/codeartifact/latest/ug/python-compatibility.html: "CodeArtifact supports PyPI's Legacy APIs, except the simple API. CodeArtifact does not support PyPI's XML-RPC or JSON APIs." Clearly AWS left out support for listing the /simple endpoint intentionally - presumably because they knew pip no longer uses it. This is relevant because when you use a private PyPI with pipenv, you add it as a source to the Sadly, pipenv uses the /simple API to list packages. It took me a while to figure this out - here's an example where I waited over two hours with no errors. But after banging on it some more, the error is here in clear sight: At this point, I'd be happy to dig into the pipenv source to be absolutely certain I'm on the right track - that it's really trying to get /simple. And of course, as the AWS docs clearly state, the endpoint simply does not exist: All of this might seem more like a bug report for pipenv rather than cdktf, but I really don't care about pipenv. I'm loving cdktf so far and would rather spend my time ripping pipenv out of cdktf than filing a bug report against pipenv - there are plenty examples of issues on that repo going no where. PS While I didn't show an example of normal pip installing from Code Artifact, that does in fact work - and as I previusly mentioned I think normal pip and a basic requirements.txt in the root of any cdktf stack is simple and sufficient. Combined with a recommendation of pyenv and virtualenv, I think the community will have less problems in the long term. |
|
Cool, thanks for clarifying this.
I think a good first step would be adding an additional Python template to the |
|
Sounds good! I'll work on that. |
cmclaughlin
changed the title
|
This was addressed by #327 |
|
I'm going to lock this issue because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
Community Note
Description
pipenv is really slow and sometimes it hangs for a really long time.
Any objection to removing it? I think leaving the Python env setup to the user is fine - and just recommending Pyenv + virtualenv. Perhaps writing out a simple requirements.txt file instead of a Pipfile in new templates and docs.
Example where it hangs for me - perhaps because I'm using a private PyPi instance that makes it worse, but that works fine with plain pip.
References
pypa/pipenv#3827
The text was updated successfully, but these errors were encountered: