USVFS Issues on Windows 11 24H2

[wxMichael]

The problem:

On Windows 11 24H2, checks for files existing in Python apps launched via MO2 fail for any file that is in the VFS but not overriding a file actually present in Data/.
None of the following work:

• os.path: exists() / isfile() / isdir()
• pathlib.Path: exists() / is_file() / is_dir() / glob()

e.g. path.is_dir() will return False while path.iterdir() returns its contents.

To Reproduce:

Steps to reproduce the behavior:

1. Create a bat file to launch repro.py: python repro.py
2. Create a file repro.py with the below code.
3. Change the file path to a file that is present in the VFS but not present in the actual Data folder.
4. Launch the bat via MO2.

import sys
from pathlib import Path

if sys.getwindowsversion().build != 26100:
	print("This repro requires Windows 11 24H2")
else:
	# Provide a file or folder present only in the VFS
	file = Path(R"C:\Program Files (x86)\Steam\steamapps\common\Fallout 4\Data\F4SE\Plugins\Buffout4.dll")
	print(f"""path: {file}
exists: {file.exists()}
is_file: {file.is_file()}
is_dir: {file.is_dir()}
iterdir: {next(f for f in file.parent.iterdir() if f.name == file.name)}""")
input()

Environment:

• Mod Organizer Version that exhibits the issue: 2.5.2
• Last Mod Organizer Version that did not exhibit the issue (if applicable): N/A
• Desktop OS/version used to run Mod Organizer: Windows 11 Home 24H2 26100.2605

Details:

Output showing the file existing but checks failing:

path: C:\Program Files (x86)\Steam\steamapps\common\Fallout 4\Data\F4SE\Plugins\Buffout4.dll
exists: False
is_file: False
is_dir: False
iterdir: C:\Program Files (x86)\Steam\steamapps\common\Fallout 4\Data\F4SE\Plugins\Buffout4.dll

Link to Mod Organizer logs:

USVFS: https://gist.github.com/wxMichael/9e8b04c3865495c2108863fec2f7f6cd

MO Interface: https://gist.github.com/wxMichael/8a40ac3e0b788515c86ee744644c4416

[wxMichael]

Related issue/commits from Wrye Bash:

• AttributeError: 'NoneType' object has no attribute 'description' wrye-bash/wrye-bash#689
• wrye-bash/wrye-bash@d3f11ab
• wrye-bash/wrye-bash@a8339b8

[Cutleast]

This may be related to the starting issues of Python executables built with cx_Freeze or PyInstaller, that I encountered, recently:

• Standalone executable issues Cutleast/Dynamic-Interface-Patcher#7
• Can't start via MO2: Failed to start embedded python interpreter! Cutleast/Dynamic-Interface-Patcher#5

The AV part was fixed in the meantime, but the issues with launching via MO2 persist.

[Holt59]

The above code works fine with 24H2 and Python 3.11. Are you sure this is related to Windows 24H2 and not Python 3.12?

[Holt59]

Unless someone reproduces the issue with Python 3.11 on Windows 24H2, I would say the issue comes from the use of GetFileInformationByName in the os.stat implementation starting from Python 3.12.

Hooking GetFileInformationByName seems to fix the issue although I am not sure if this is would be the lowest-level function to hook for this.

[AnyOldName3]

Wine doesn't seem to have an implementation of GetFileInformationByName (based on a search of their GitHub mirror), so we can't base our hook on where they decide to stop pretending to be Windows and start calling Unix functions.

[Holt59]

Wine doesn't seem to have an implementation of GetFileInformationByName (based on a search of their GitHub mirror), so we can't base our hook on where they decide to stop pretending to be Windows and start calling Unix functions.

I went down to NtQueryInformationByName which seems to also fix the issue (without hooking GetFileInformationByName).

As far as I can tell, Wine does that even deeper by hooking ObXXX calls, like ObOpenObjectByName or ObQueryNameString. Since we have never dealt with this kind of API Calls, I don't want to go down that rabbit hole for now.

[Holt59]

@wxMichael @Cutleast You can try this version of USVFS if you want - I am not 100% sure I did not modify the API between MO2 and USVFS, so these may not work at all but let me know if you try them.

usvfs_0.5.7.0.zip

[Cutleast]

Can confirm that it fixes the issues with the Pyinstaller build of DIP, didn't test it with cx_Freeze, though.

And since DIP makes use of pathlib.Path.is_file() and pathlib.Path.is_dir(), those seem to be fine, too.

[wxMichael]

The above code works fine with 24H2 and Python 3.11. Are you sure this is related to Windows 24H2 and not Python 3.12?

It's possible it's also specific to Python 3.12+.
My project has been using 3.12 for a while without issue until I updated to 24H2 to test my suspicion that it's what was causing my users trouble.

[Cutleast]

Update: While fixing the issues with the cx_Freeze build, it broke the Nuitka build. It just throws this error message:

and then crashes immediately.

[wxMichael]

@Holt59 That version of USVFS does work with the repro code on my PC, both when run from source and when compiled with PyInstaller.

[Holt59]

Update: While fixing the issues with the cx_Freeze build, it broke the Nuitka build. It just throws this error message:
and then crashes immediately.

Can you provide a reproducible example? I tried building DIP but it fails to run even outside of MO2 so I cannot really check anything.

[Cutleast]

Here you go, a minimal reproducable example (I just adapted OP's repro to test an import from PySide6). The command to build the executable is in the first line.

# nuitka --msvc="latest" --standalone --windows-console-mode=force --enable-plugin=pyside6 --remove-output --nofollow-import-to=tkinter "usvfs_test.py"

import sys
from pathlib import Path

from PySide6.QtWidgets import QApplication

if sys.getwindowsversion().build != 26100:
    print("This repro requires Windows 11 24H2")
else:
    # Provide a file or folder present only in the VFS
    file = Path(R".\DIP\DIP.exe")
    print(f"""path: {file}
exists: {file.exists()}
is_file: {file.is_file()}
is_dir: {file.is_dir()}
iterdir: {next(f for f in file.parent.iterdir() if f.name == file.name)}""")
input()

When launched via MO2, the same error and crash occur:

PySide6/__init__.py: Unable to import Shiboken from C:\Games\Skyrim Special Edition\data\usvfs_test.dist, C:\Games\SKYRIM~1\data\usvfs_test.dist
Traceback (most recent call last):
  File "C:\Games\SKYRIM~1\data\usvfs_test.dist\usvfs_test.py", line 6, in <module>
  File "C:\Games\SKYRIM~1\data\usvfs_test.dist\PySide6\__init__.py", line 123, in <module PySide6>
  File "C:\Games\SKYRIM~1\data\usvfs_test.dist\PySide6\__init__.py", line 62, in _setupQtDirectories
  File "C:\Games\SKYRIM~1\data\usvfs_test.dist\shiboken6\__init__.py", line 27, in <module shiboken6>
ImportError: LoadLibraryExW 'C:\Games\SKYRIM~1\data\usvfs_test.dist\shiboken6\Shiboken.pyd' failed: The specified module could not be found.

[Holt59]

Here you go, a minimal reproducable example (I just adapted OP's repro to test an import from PySide6). The command to build the executable is in the first line.

I cannot reproduce... I get

[WinError 3] The system cannot find the path specified: 'DIP'

...which is 100% expected since I do not have such folder, but that also means that the PySide6 import went well.

Does C:\Games\SKYRIM~1\data\usvfs_test.dist\shiboken6\Shiboken.pyd actually exists on your side? Does it work outside of MO2?

[Cutleast]

It works outside of MO2 and Explorer++ says, it exists within the VFS:

[Cutleast]

Here's some more information about the environment:
Python: 3.12.4
Nuitka: 2.5.6
PySide6: 6.8.0.2
shiboken6: 6.8.0.2
MO2: 2.5.2
UVFS: the files you provided, some hours ago
Windows: 26100.2605

UVFS Log: https://gist.github.com/Cutleast/451bb5d0e9119bb621bdf0c4904d167e
MO2 Interface Log: https://gist.github.com/Cutleast/eed0b4895174a007d08c3e303d85a156

[Holt59]

Here's some more information about the environment: Python: 3.12.4 Nuitka: 2.5.6 PySide6: 6.8.0.2 shiboken6: 6.8.0.2 MO2: 2.5.2 UVFS: the files you provided, some hours ago Windows: 26100.2605

UVFS Log: https://gist.github.com/Cutleast/451bb5d0e9119bb621bdf0c4904d167e MO2 Interface Log: https://gist.github.com/Cutleast/eed0b4895174a007d08c3e303d85a156

Okay I was simply launching the executable through MO2 but if I understand correctly you are actually launching a bat file that launches the executable that is in the VFS? What is the content of the bat file?

[Cutleast]

I just created it to better capture the error message since it immediately closes:

usvfs_test.exe
pause

Starting the exe directly doesn't change the outcome.

[Holt59]

I just created it to better capture the error message since it immediately closes:

usvfs_test.exe
pause

Starting the exe directly doesn't change the outcome.

I cannot reproduce, with or without .bat, I tried with relative/absolute paths everywhere, etc., but to no avail...

Can you try with a MO2 instance with only a mod containing the build executable (+ dependencies) and launch it once (with debug logs) to get clean USVFS logs?

[Cutleast]

I now created a new portable instance, fresh from MO2 v2.5.2 (+ your USVFS files) and set the log level to "Debug":
USVFS Log: https://gist.github.com/Cutleast/6ba546e15586872c108d35eeab984d2e
MO2 Interface Log: https://gist.github.com/Cutleast/e4340f85db8fbe2baf7bcc37052b30db