Release v2023.6.26: error: invalid command 'egg_info'

[cclauss]

Upgrading to pipenv v2023.6.26 generates an error: invalid command 'egg_info' in the CircleCI builds of our Django project.

pipenv works as expected on local machines but fails on CircleCI until we pip install --upgrade "pipenv<2023.6.26

Expected result

pipenv builds without errors.

Actual result

Successfully installed pipenv-2023.6.26 setuptools-68.0.0
pipenv, version 2023.6.26
Creating a virtualenv for this project...
Pipfile: /home/circleci/circleci-workdir/Pipfile
Using /home/circleci/.pyenv/versions/3.9.17/bin/python3.9 (3.9.17) to create virtualenv...
created virtual environment CPython3.9.17.final.0-64 in 162ms
  creator CPython3Posix(dest=/home/circleci/circleci-workdir/.venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/circleci/.local/share/virtualenv)
    added seed packages: pip==23.1.2, setuptools==67.8.0, wheel==0.40.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Successfully created virtual environment!

Virtualenv location: /home/circleci/circleci-workdir/.venv
Installing dependencies from Pipfile.lock (ef2d1f)...
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'egg_info'

Exited with code exit status 1

When possible, provide the verbose output (--verbose), especially for locking and dependencies resolving issues.

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

---

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

If you're on macOS, run the following:

$ pipenv --support | pbcopy

If you're on Windows, run the following:

> pipenv --support | clip

If you're on Linux, run the following:

$ pipenv --support | xclip

[oz123]

@cclauss can you please show you dependencies ?

Do you have somewhere in the tree distribute. The seems to be related to one of the packages.
(I guess earlier versions of pipenv were more relaxed about the underlying error).

[matteius]

This was certainly caused by (unintentionally) the large refactor to requirementslib to migrate away from attrs to pydantic. It seems that you have some requirement that is going down the path of getting the metadata by calling python setup.py egg_info -- but it seems something either with this package or otherwise doesn't allow for this command. It would be helpful to figure out what path for getting the metadata this package should take, and more information about how the package setup is configured that its failing on.

[juanitorduz]

I have a similar issue when locking this Pipfile. Any tips on how to help you debug the error?

[matteius]

@juanitorduz that Pipfile will be gold for trying to figure it out -- know that this is a top priority bug for pipenv, but it will require some tinkering with requirementslib to get it nailed down. I'll try to find some time to triage further later this week, but if someone else wants to dig deeper, I think the strategy is:
1.) Narrow down which package type is causing this issue, is the vcs isntall?
2.) Then I would try comparing in an editable pipenv install from main the two release points to determine how the metadata is determined in the prior version, because clearly in the new version its trying to build the egg_info, we may need a different code path to take precedence in requirementslib for this case.

The easiest way to test a potential fix would be to modify requirementslib within pipenv and open a PR -- if all the tests in pipenv pass, there is a high probability the tests in requirementslib will pass also when the change is ported over the the main library there.

[cclauss]

Pipfile:
{git = "https://github.com/shtalinberg/django-el-pagination.git"}

[cclauss]

@juanitorduz Try commenting out your line 90 and running again.

sts-jax = {git = "https://github.com/probml/sts-jax.git", editable = true, ref = "main"}

[matteius]

I was able to reproduce it briefly with an existing old virtualenv and just the Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "*"
sts-jax = {git = "https://github.com/probml/sts-jax.git", editable = true, ref = "main"}

[dev-packages]

[requires]
python_version = "3.9"

For some reason when I had pipenv create a new virtualenv, it did not get the egg_info issue. I then wasn't able to lock for a different reason which is equally odd, despite there being a regular release 2.12, I got:

CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement tensorflow>=2.11 (from versions: 2.13.0rc0, 2.13.0rc1, 2.13.0rc2)

So I noted those that it did find are pre releases, and I ran again: pipenv lock -v --pre and it locked successfully and installed successfully.

Noting also that in the issue report from @cclauss that it appears to be creating a fresh virtualenv but is getting the no egg_info command -- from what I can tell, python setup.py egg_info should be a valid command for the dependencies that are getting the error, they have a setup.py and use setuptools.

[matteius]

I think I just figured out what is happening, and its not the simplest fix but let me explain. requirementslib basically is using the system python and not the virtualenv python -- I have a fix, but the problem is, requirementslib starts using pipenv mechanics to figure out the correct python to use. Your system python has a really old setuptools is my guess which doesn't have the setup_egg command. Please have a try of this branch to see what I mean, for me it fixes the issue I was able to reproduce in my Ubuntu VM: #5759

The problem is, it opens the can of worms of what the heck is requirementslib, and why doesn't it just become part of pipenv. Or maybe it pushes us to have a better interface to pass in the right python to use within requirementslib, but yeah -- this is a great discovery of an issue and kind of a bummer.

[shinkawk]

Hi @matteius

Your system python has a really old setuptools is my guess which doesn't have the setup_egg command.

Just tested 2023.6.26 with pipenv lock command againt different Python Offical docker images.
Looks like the issue happens on python:3.9 or lower. 3.10 and 3.11 has no issue installing the package.

Hope this helps!

[matteius]

@shinkawk I am guessing if you do a regular pip install setuptools --upgrade ahead of the test that python:3.9 and lower would actually succeed, but it highlights an implementation flaw in pipenv we want to address.

[intotecho]

A client with limited python experience spent two days trying to understand why their builds were failing.
Is there a plan to quickly rollback this release or push a new fix urgently to stop these failures?

[matteius]

@intotecho The workaround should be to adjust the CI to pip install setuptools --upgrade ahehad of using pipenv, or to pin pipenv to the prior version. I do not plan to yank the release over this edge case; as I pointed out in my prior comments, it will take some time to properly patch requirementslib to use the right python executable, which I don't have time to do immediately.

[matteius]

Ok -- I thought of a potential workaround today and just tried it out; actually I tried it out earlier and it failed, started on the full on fix (which brought to light further how there is no parent model in requirementslib to track things and so adding python to every single class was proving time consuming) -- I stumbled across a line that was overriding the environment variable again to be sys.executable, so I went with my patch idea from earlier and got it working locally. Please try out this branch and let me know -- if its good, I can try to cut a release tomorrow: #5760

[matteius]

There is an issue with the fix and the windows CI -- I am looking into it, probably a path specification difference.

[matteius]

I plan to cut a release over the weekend after @oz123 has a chance to catch up with the latest merges.