Windows embeddable throws FileNotFoundError

[zerocewl]

Issue

Using the latest (20.15.0) version of virtualenv together with python 3.9 Windows embeddable package throws a FileNotFoundError if calling any module code e.g. python.exe -m virtualenv testEnv.

While following the issue Windows embedable support #1774 and the depending MR Windows embedable support i though it should now be possible to create and handle any venv with a windows embeddable python, but after proceeding the mentioned steps i was still not able to use the virtualenv module. Am i missing any step?

I unzipped the embedded package: https://www.python.org/ftp/python/3.9.13/python-3.9.13-embed-amd64.zip
Then I run the get-pip.py: https://bootstrap.pypa.io/get-pip.py
Then I uncomment in python39._pth the import statement.
Then I add some path to PATH env variable. (add path to python.exe and pip.exe)
Then I run pip install virtualenv.
Then I tried to call python.exe -m virtualenv testEnv

Note: the unzipped python folder does not contain any DLLs subfolder.

(Tested with latest python 3.9 and 3.10.)

Environment

Provide at least:

• OS:

C:\Users\tester\Downloads\python-3.9.13-embed-amd64>systeminfo

Host Name:                 WIN10
OS Name:                   Microsoft Windows 10 Home
OS Version:                10.0.19043 N/A Build 19043
OS Manufacturer:           Microsoft Corporation
...

• pip list of the host python where virtualenv is installed:

C:\Users\tester\Downloads\python-3.9.13-embed-amd64>python -m pip list
Package      Version
------------ -------
distlib      0.3.4
filelock     3.7.1
pip          22.1.2
platformdirs 2.5.2
setuptools   62.6.0
six          1.16.0
virtualenv   20.15.0
wheel        0.37.1

Output of the virtual environment creation

Make sure to run the creation with -vvv --with-traceback:

C:\Users\tester\Downloads\python-3.9.13-embed-amd64>python.exe -m virtualenv testENV --vvv --with-traceback
Traceback (most recent call last):
  File "runpy.py", line 197, in _run_module_as_main
  File "runpy.py", line 87, in _run_code
  File "C:\Users\tester\Downloads\python-3.9.13-embed-amd64\lib\site-packages\virtualenv\__main__.py", line 80, in <module>
    run_with_catch()  # pragma: no cov
  File "C:\Users\tester\Downloads\python-3.9.13-embed-amd64\lib\site-packages\virtualenv\__main__.py", line 65, in run_with_catch
    run(args, options, env)
  File "C:\Users\tester\Downloads\python-3.9.13-embed-amd64\lib\site-packages\virtualenv\__main__.py", line 18, in run
    session = cli_run(args, options, env)
  File "C:\Users\tester\Downloads\python-3.9.13-embed-amd64\lib\site-packages\virtualenv\run\__init__.py", line 30, in cli_run
    of_session = session_via_cli(args, options, setup_logging, env)
  File "C:\Users\tester\Downloads\python-3.9.13-embed-amd64\lib\site-packages\virtualenv\run\__init__.py", line 48, in session_via_cli
    parser, elements = build_parser(args, options, setup_logging, env)
  File "C:\Users\tester\Downloads\python-3.9.13-embed-amd64\lib\site-packages\virtualenv\run\__init__.py", line 74, in build_parser
    CreatorSelector(interpreter, parser),
  File "C:\Users\tester\Downloads\python-3.9.13-embed-amd64\lib\site-packages\virtualenv\run\plugin\creators.py", line 15, in __init__
    creators, self.key_to_meta, self.describe, self.builtin_key = self.for_interpreter(interpreter)
  File "C:\Users\tester\Downloads\python-3.9.13-embed-amd64\lib\site-packages\virtualenv\run\plugin\creators.py", line 25, in for_interpreter
    meta = creator_class.can_create(interpreter)
  File "C:\Users\tester\Downloads\python-3.9.13-embed-amd64\lib\site-packages\virtualenv\create\via_global_ref\builtin\via_global_self_do.py", line 34, in can_create
    cls._sources_can_be_applied(interpreter, meta)
  File "C:\Users\tester\Downloads\python-3.9.13-embed-amd64\lib\site-packages\virtualenv\create\via_global_ref\builtin\via_global_self_do.py", line 39, in _sources_can_be_applied
    for src in cls.sources(interpreter):
  File "C:\Users\tester\Downloads\python-3.9.13-embed-amd64\lib\site-packages\virtualenv\create\via_global_ref\builtin\cpython\cpython3.py", line 67, in sources
    for ref in refs:
  File "C:\Users\tester\Downloads\python-3.9.13-embed-amd64\lib\site-packages\virtualenv\create\via_global_ref\builtin\cpython\cpython3.py", line 98, in dll_and_pyd
    for file in folder.iterdir():
  File "pathlib.py", line 1160, in iterdir
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\tester\\Downloads\\python-3.9.13-embed-amd64\\DLLs'

[reksar]

Creating an empty DLLs folder should fix the issue.

[zerocewl]

@reksar
Ok, thx for the workaround.

Venv creation and pip install works now for python 3.10 and 3.9.

But if i try to use my custom pkg inside the activated py 3.9 env, i got an AttributeError:

  File "C:\Users\tester\Downloads\testEnv1\lib\site-packages\dateparser\__init__.py", line 3, in <module>
    from .date import DateDataParser
  File "C:\Users\tester\Downloads\testEnv1\lib\site-packages\dateparser\date.py", line 9, in <module>
    from dateparser.date_parser import date_parser
  File "C:\Users\tester\Downloads\testEnv1\lib\site-packages\dateparser\date_parser.py", line 5, in <module>
    from .conf import apply_settings
  File "C:\Users\tester\Downloads\testEnv1\lib\site-packages\dateparser\conf.py", line 6, in <module>
    from .parser import date_order_chart
  File "C:\Users\tester\Downloads\testEnv1\lib\site-packages\dateparser\parser.py", line 13, in <module>
    from dateparser.utils.strptime import strptime
  File "C:\Users\tester\Downloads\testEnv1\lib\site-packages\dateparser\utils\strptime.py", line 57, in <module>
    __strptime = patch_strptime()
  File "C:\Users\tester\Downloads\testEnv1\lib\site-packages\dateparser\utils\strptime.py", line 28, in patch_strptime
    _strptime_spec.loader.exec_module(_strptime)
AttributeError: 'zipimporter' object has no attribute 'exec_module'

Note: The python 3.10 environment works as expected.

Maybe the delivered dlls only work with python 3.10, the tests from the MR seem only to test this version?

[reksar]

I can't to find the difference between 3.9 and 3.10 behaviors. Note that in the embeddable Python std lib is zipped and you have problem with zipimporter during importing std datetime module from python39.zip. I guess. So the issue is predictable, but why only for 3.9? Maybe 3.10 tries to import from elsewhere? You can test Python's sys.path by adding the <local venv dir>\Scripts\python39(310)._pth file. Take the python*._pth file from Python host dir as example.

[reksar]

Ha! Deprecated since Python 3.10. Use exec_module() instead.

It seems you need to add a hack for old zipimporter if you want to support old versions of embeddable Python in your project.

[reksar]

All I can help is to automatically create the DLLs dir (if it doesn't exist) during the virtualenv setup 😄 Will need to be added this feature soon.

[zerocewl]

Ha! Deprecated since Python 3.10. Use exec_module() instead.

It seems you need to add a hack for old zipimporter if you want to support old versions of embeddable Python in your project.

Maybe this can be added to the docs? 🤔

[reksar]

Hm, where exactly? Looks like everything is in place.

The embeddable package

The standard library is included as pre-compiled and optimized .pyc files in a ZIP

zipimport.load_module

Deprecated since version 3.10: Use exec_module() instead.

[zerocewl]

Sry i thought virtualenv will only work with embedded python >= 3.10, but i found the depending issue in the dateparser repo.

Before python3.10 the exec_module() was missing in zipimport.zipimporter.

scrapinghub/dateparser#849

https://docs.python.org/3.10/library/zipimport.html#zipimport.zipimporter.exec_module

[reksar]

Yes, this is not a virtualenv problem.

But initially you got the error because the DLLs dir was missing. I think, the virtualenv should take care of it during setup. 👍

Or just ignore it absence when creating the virtual environment🤔

[zerocewl]

Maybe ignore only if embedded python is used? In the other case the error would you point to the right direction.

[pfmoore]

I feel I should reiterate my point from #1774 (comment) - the embedded Python distribution is not intended for this sort of use, and I don't think virtualenv should add support for being used with it.

It's worth pointing out that the embedded distribution doesn't include the stdlib venv module, indicating fairly clearly that it's not intended as a base for creating virtual environments from.

If you want a portable Python environment that can be used like this, maybe what you actually want is the nuget package?

[reksar]

If you know what you do and it just works - why not?

Most use-cases is simple and portable Python environment. I think closest alternative is PyPy (if you don't pay attention that this is not CPython), but nuget Python package can be too. But both of these alternatives have stdlib venv module, while embedded Python does not. And only virtualenv can help in this case.

Using pip to manage dependencies as for a regular Python installation is not supported with this distribution

but we make pip work

It's worth pointing out that the embedded distribution doesn't include the stdlib venv module

and we install virtualenv through pip

Are we insane? Yes! Does it hurt someone?)

[pfmoore]

If you know what you do and it just works - why not?

Sure 🙂 But if it breaks, you're on your own. Of course it's down to @gaborbernat whether he wants to support this. If it was my choice, though, I wouldn't.

[reksar]

@pfmoore Call it a non-standard use case if you like)

All we do is:

• copy one extra zip file if it exists
• do not crash with an error if the DLLs dir does not exists

[gaborbernat]

The PR has been now merged.