{Packaging} Bump PyWin32 to 305

[jiasli]

Change

Bump PyWin32 (to >= 303) so that Azure CLI no longer loads pywintypes310.dll or pythoncom310.dll from the current working directory (CWD), but always from C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\pywin32_system32\.

Context

LoadLibrary and LoadLibraryEx Win32 API

According to https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order#standard-search-order-for-desktop-applications, even if SafeDllSearchMode is enabled and it places the user's current directory later in the search order, the current directory is still searched by LoadLibrary and LoadLibraryEx. LoadLibraryEx supports additional flags such as LOAD_LIBRARY_SEARCH_DEFAULT_DIRS and LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR. When these flags are specified, directories in the standard search path are not searched, so the current directory is no longer searched.

Python 3.8's DLL loading mechanism change

https://docs.python.org/3/whatsnew/3.8.html#changes-in-the-python-api

DLL dependencies for extension modules and DLLs loaded with ctypes on Windows are now resolved more securely. Only the system paths, the directory containing the DLL or PYD file, and directories added with add_dll_directory() are searched for load-time dependencies. Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() and if it exists, use it to add your DLLs directory while loading your library. Note that Windows 7 users will need to ensure that Windows Update KB2533623 has been installed (this is also verified by the installer). (Contributed by Steve Dower in bpo-36085.)

bpo-36085 is migrated to python/cpython#80266.

In summary, when loading DLLs:

• in Python 3.7, LoadLibraryExW is called with LOAD_WITH_ALTERED_SEARCH_PATH.
• in Python 3.8, LoadLibraryExW is called with LOAD_LIBRARY_SEARCH_DEFAULT_DIRS and LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR, more securely (bpo-36085: Enable better DLL resolution on Windows python/cpython#12302)

PyWin32's DLL loading mechanism change

Since version 303, PyWin32's _win32sysloader switched from calling LoadLibrary to LoadLibraryEx with the same LoadLibraryEx flags as Python itself mhammond/pywin32#1794.

Therefore, PyWin32's change causes the current directory no longer to be searched, but its change log doesn't mention this:

https://github.com/mhammond/pywin32/blob/d73b0200eba81fa551b90d15cf5da097f4197f8f/CHANGES.txt#L92-L93

Tweaks to how DLLs are loaded and our installation found, which should improve virtualenv support and version mismatch issues (#1787, #1794)

Testing guide

The test can be done with system Python, without Azure CLI's embedded Python.

With PyWin32 302, DLL can be loaded from the current directory:

> cd D:\temp
> pip install pywin32==302
> mv $env:LOCALAPPDATA\Programs\Python\Python310\Lib\site-packages\pywin32_system32\pythoncom310.dll D:\temp
> python -c "import win32com"

With PyWin32 305, DLL cannot be loaded from the current directory and results in an ImportError:

> cd D:\temp
> pip install pywin32==305
> mv $env:LOCALAPPDATA\Programs\Python\Python310\Lib\site-packages\pywin32_system32\pythoncom310.dll D:\temp
> python -c "import win32com"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\jiasli\AppData\Local\Programs\Python\Python310\lib\site-packages\win32com\__init__.py", line 6, in <module>
    import pythoncom
  File "C:\Users\jiasli\AppData\Local\Programs\Python\Python310\lib\site-packages\pythoncom.py", line 4, in <module>
    pywintypes.__import_pywin32_system_module__("pythoncom", globals())
  File "C:\Users\jiasli\AppData\Local\Programs\Python\Python310\lib\site-packages\win32\lib\pywintypes.py", line 105, in __import_pywin32_system_module__
    raise ImportError("No system module '%s' (%s)" % (modname, filename))
ImportError: No system module 'pythoncom' (pythoncom310.dll)

[yonzhan]

Bump PyWin32 to fix vulnerability issue

[jiasli]

Why/How pythoncom310.dll is loaded

I put these lines at site-packages\win32\lib\pywintypes.py:L30:

print(filename)
import traceback
traceback.print_stack()

Result:

> az --version
pywintypes310.dll
  File "runpy.py", line 196, in _run_module_as_main
  File "runpy.py", line 86, in _run_code
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/__main__.py", line 14, in <module>
  File "<frozen importlib._bootstrap>", line 1078, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/telemetry.py", line 19, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/telemetry/__init__.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\portalocker/__init__.py", line 4, in <module>
  File "<frozen importlib._bootstrap>", line 1078, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\portalocker/portalocker.py", line 12, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\win32\lib\pywintypes.py", line 120, in <module>
    __import_pywin32_system_module__("pywintypes", globals())
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\win32\lib\pywintypes.py", line 32, in __import_pywin32_system_module__
    traceback.print_stack()
pythoncom310.dll
  File "runpy.py", line 196, in _run_module_as_main
  File "runpy.py", line 86, in _run_code
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/__main__.py", line 39, in <module>
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\__init__.py", line 895, in get_default_cli
    from azure.cli.core.azlogging import AzCliLogging
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/azlogging.py", line 30, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 25, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/extension/__init__.py", line 11, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 945, in _find_spec
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\_distutils_hack/__init__.py", line 97, in find_spec
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\_distutils_hack/__init__.py", line 108, in spec_for_distutils
  File "importlib\__init__.py", line 126, in import_module
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\setuptools/__init__.py", line 16, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\setuptools/version.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\pkg_resources/__init__.py", line 80, in <module>
  File "<frozen importlib._bootstrap>", line 1078, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\pkg_resources/extern/__init__.py", line 52, in create_module
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\pkg_resources/extern/__init__.py", line 37, in load_module
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\pkg_resources/_vendor/appdirs.py", line 560, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\win32com/__init__.py", line 6, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\pythoncom.py", line 4, in <module>
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\win32\lib\pywintypes.py", line 32, in __import_pywin32_system_module__
    traceback.print_stack()

The call stack indicates pythoncom310.dll was loaded by site-packages\pkg_resources/_vendor/appdirs.py:L560:

if system == "win32":
    try:
        import win32com.shell
        _get_win_folder = _get_win_folder_with_pywin32
    except ImportError:
        try:
            from ctypes import windll
            _get_win_folder = _get_win_folder_with_ctypes
        except ImportError:
            try:
                import com.sun.jna
                _get_win_folder = _get_win_folder_with_jna
            except ImportError:
                _get_win_folder = _get_win_folder_from_registry

That's explains why win32com is imported.

The dev environment doesn't have this issue because the dev environment pins setuptools to 52.0.0:

azure-cli/requirements.txt

Line 2 in 99c5fb4

setuptools==52.0.0

which doesn't injects _distutils_hack. But the Windows MSI bundles latest setuptools 65.6.3 and results in the following call chain:

1. setuptools injects _distutils_hack
2. _distutils_hack calls setuptools
3. setuptools calls pkg_resources ([BUG] Importing setuptools is slow and takes nearly half a second pypa/setuptools#3441)
4. pkg_resources imports win32com