Fix FileSystem dock won't show any file folders while loading large projects (v2)

[Hilderin]

Fixes #92335, Fixes #47053 and Fixes #85082
This is a new PR for the reverted PR #92650.

This PR improves:

• FileSystem Dock not showing progress bar on first file scanning
• Better progression (%) for scanning large projects with a lot of folders and files
• Performance optimization on scanning project files for large projects
• Add progression popup for _update_scan_actions step (instead of Editor freeze)
• Add progression popup for _update_script_classes step (instead of Editor freeze)
• Add progression popup for loading editor layout and reopening scenes (instead of Editor freeze)

I tested these modifications with a project having all the files from https://github.com/PMDCollab/SpriteCollab/tree/master/sprite (more than 125000 files and 385Mb).

Before:
FileSystem was empty without any feedback for a couple of minutes and after that the editor was just frozen:

After modifications:

Performance optimization on project scanning before asset importation
For a 30000 files project, it took around 2m50s on startup to scan the project.
After a couple of optimizations, it now takes 1m30s for the same project.

Before:

After:

Causes:

• The FileAccess:exists was calling the open method which has a couple of overhead over file_exists.
• The _scan_new_dir was double sorting the folders.
• The reimport_files was recalculating the import_groupe_file even if _update_scan_actions did that just before.

[Hilderin]

All the correction should be done. Thanks for the review.

[Hilderin]

I think it's important to note that some issues that this PR fixes have also been addressed by #92303, but in a different way. I suggest merging #92303 before this one, and then I will rebase and adjust this PR.

[Hilderin]

I rebased from master and reworked this PR.

With the modifications from #92303, I was able to simply this PR a lot. It's now almost only adding EditorProgress at different places during first startup and importation process.

I also kept some optimizations like the FileAccess::exists, ResourceFormatImporter::get_resource_import_info and ResourceLoader::_path_remap.

[Calinou]

Tested locally (rebased on top of master 97b8ad1) on various official demos, it works as expected. The --import command line argument still works correctly too.

Code looks good to me.

[Hilderin]

Rebased to fix conflict with master.

[Saul2022]

Testing on https://github.com/PMDCollab/SpriteCollab/tree/master now without the pr that heavily improves initial anlysis before scanning files dialog, because of curiosity to see what's going on.

Initial transition from godot logo to editor scan( over 15 minutes( luckily with the load project pr is down to 1:30 or 3 minutes, so i thank that.).

Then on godot scan, update action and preparing for import dialog 11:30 minutes. Though on preparing for import it gets stuck at 99% which seems to still freeze with no additional progress bar, and atm it still going now with over 12 minutes and still going

Aside from this something i thought could be done in the godot dialog, which i would guess it might freeze too after the circle wheel dissapears , is make it have a progress bar that says what it doing, because it pretty weird how the editor gets staring without seeing anything going on, which might make people think it got frozen..

[Hilderin]

Wow, thank you to have tested it, I found why it's stuck before the reimportation. All the resources must pass in EditorNode::_resources_reimporting and EditorNode::_resources_reimported and for each one ResourceLoader::get_resource_type(res_path) is called forcing to open the asset file a couple of times each. I did another modification to optimize this process by using the resource type already calculated in EditorFileSystem and I added another progress step while doing in.

Time on reimporting and reimported (each):
Before: 90sec
After: 8sec

[Hilderin]

Aside from this something i thought could be done in the godot dialog, which i would guess it might freeze too after the circle wheel dissapears , is make it have a progress bar that says what it doing, because it pretty weird how the editor gets staring without seeing anything going on, which might make people think it got frozen.

Are you speaking about the Godot splash screen at startup? If I understand correctly you suggest to add a progress bar here?

[Saul2022]

If I understand correctly you suggest to add a progress bar here?

Yea, as there's no visual indication to the user , of whats going on , like staying 1:30 minutes staring at the godot logo, without knowing what's going on , might make people think it got frozen.

Time on reimporting and reimported (each): Before: 90sec After: 8sec

Edit : Honestly that's so fascinating as a speedup, awesome job on this great prs, though in general what would be the general speed improvement on all the asset reimport/reimported after the preparing import dialog is done ( like what you tested with the sprites of your linked project)?

Atp what is just left to do would be fix the cache thing as Juan suggested, and maybe cache the scenes that were opened before closing., as they only seem to be cached upon switching scenes, so when opening they load way faster.

Edit 2: testing calinuo test project of obj files from the normal raycaster pr which contains 2391 files without godot folder
https://github.com/user-attachments/files/16075644/test_base_mesh.zip

Times recorded since opening project to scene load
4.4 rc debug latest checks 3:19:33 minutes

4.4 rc debug #95678 4:4:14 minutes

4.3 stable official 4:05:85

Prob with project load pr && release - optimized times would decrease 3 times or more based on numbers.

[Hilderin]

Thanks again for all your benchmarks, very appreciated.

Yea, as there's no visual indication to the user , of whats going on , like staying 1:30 minutes staring at the godot logo, without knowing what's going on , might make people think it got frozen.

It's challenging technically to add a progress bar on the logo splash screen, but, I added another progress popup while executing the first scan. That way, the splash screen will disappear sooner when the first scan starts and as a bonus, I added steps in the progress bar for the creation of autoload scripts and plugins which could take a while if you have a lot of autoloads or plugins.

though in general what would be the general speed improvement on all the asset reimport/reimported after the preparing import dialog is done ( like what you tested with the sprites of your linked project)?

This optimization should affect all asset types. When reimporting a small number of assets, the difference will be minimal, but in large quantity, removing the need to read the asset should improve performance in the same magnitude for any kind of asset. Still better, I just pushed another little modification and I was able to cut the processing time in EditorNode::_resources_reimported just by caching the resource types calculated in EditorNode::_resources_reimporting.

Atp what is just left to do would be fix the cache thing as Juan suggested, and maybe cache the scenes that were opened before closing., as they only seem to be cached upon switching scenes, so when opening they load way faster.

I'm working on it.

[Calinou]

It's challenging technically to add a progress bar on the logo splash screen

We could display a loading percentage in the window title, but it'll be missed by those having the editor in fullscreen while the editor is loading (which is unlikely).

[Hilderin]

We could display a loading percentage in the window title, but it'll be missed by those having the editor in fullscreen while the editor is loading (which is unlikely).

That could be a brilliant idea!! Right now, I added a EditorProgress in _first_scan_filesystem which should do the trick. Also, on Android the window title is not displayed :(

[Saul2022]

. Also, on Android the window title is not displayed :(

That is weird , if you mean your project like how it appears at the bottom left on windows, ut normal for it to not appear on android.

Else if you mean godot engine logo it there

Either way this changes are really appreciated , so thank you for this amazing job.

[Hilderin]

I think Calinou was referring to the window title bar that we have on desktop:

[Saul2022]

Final update on it 17: mins for initial scan , 6 minutes for scan, 7 for action scans, then 4 on preimport preratation , 6 for exectuting reimport precess and 2 hours for reimport to get to 50% using sprite collab whole project.

Some notes( though this is for a future pr, as this one is pretty well as it is, and is not a regression of 4.3) :
I think this is now pretty much max speed this can get in first startup aside from initial scan before file scan, which is addresed on the other pr, though maybe for some stuff like action scan or reimport could use some improvements like as the pngs reimport is instant, like i think what is limiting here is that it going over 1 by 1 like you reimported this image now? Ok then go for the next one ,add up to that , that this files are just some kb ( not even getting to 1 mbs )..and the burden here being that there are tons and tons of them and while the import of each is instant, having so many and just importing one at a time instead of say 10 and then pass over to other ten and go on until import is over. It might seem pretty confusing like this so i might create a issue later, though this might pretty much affect models too., or i might get too crazy and is just that multi threading bug or lack of it for import.

[akien-mga]

Thanks!