PyPy hard-codes the dlls and sos installed into a virtualenv

[mattip]

What's the problem this feature will solve?
In the pypy-specific create/via_global_ref/builtin/pypy/common sources() function the super class calls cls._add_shared_libs(interpreter), which is implemented in each of [PyPy2, PyPy3] times [Windows, Posix]. In each of these 4 implementations, the class method explicitly lists the files it expects to find.

This is too restrictive and leads to churn when PyPy wants to change something.

Describe the solution you'd like
CPython3Windows uses a more flexible cls.include_dll_and_pyd which uses Path.iterdir to find all the relevant files. I think on CPython, Posix symlink is always used, so the problem is avoided?

For an upcoming py3.9 I am toying with changing the dll name from libpypy3-c.dll to libpypy3.9-c.dll. Testing creating a virtualenv is failing. PRs #2218, #2141, #1103 would have been avoided if specific dll/so naming was avoided.

Alternative Solutions

I could just continue making PRs to add to the hard-coded list

Additional context

Thanks! If this seems like a desired direction I can open a PR

[gaborbernat]

The idea with not blank accepting just all files found is to fail upfront when the required files are not found. E.g. virtualenv should not create a virtual environment if crucial files are missing. The cost of this is that whenever we change virtualenv implementation we need to alter our list of required files accordingly. If we'd go blank accepting all files and assume it would work users would get more cryptic error when they have a broken pypy that's missing some crucial files. So, don't know what to say. I'm open to either approaches, but the idea was that generally, virtualenv implementations change rarely (perhaps not with pypy now, but in general) so the churn shouldn't be that hard. What do you think?

[mattip]

Hmm. I would like to make PyPy as much like CPython as possible. CPython3Posix has no sources() method at all, and CPython3Windows has a shim launcher that PyPy does not have, but falls back to a "copy everything" wildcard if that shim is not found.

We now test that pip install virtualenv produces a working environment on our CI (a run with windows + pypy3.9, note how the steps after "Create virtualenv" fail which is how I knew something was off) so I am less worried about unknown incompatibilities like in the past.

[gaborbernat]

Fair enough. Note though on Linux CPython has no dll/so to copy, so doesn't need sources. That being said I leave the question up to your judgment; put in a PR in whichever direction you lean and I'll merge after review. (when I was speaking of broken environments I meant more like if the user has deleted files/etc).

[mattip]

Thanks for the feedback. After all I can make a PR but frustrated users end up here first, and you have to deal with the fallout

[gaborbernat]

That's fine as long as you don't mind me pinging you here in case is something out of my control 😆

[gaborbernat]

Guess this is no longer the case as no one reported it.