fix: better way to do LSP file overrides

[asterite]

Description

Problem

Resolves #8670

Summary

When you open a Noir file in LSP, and whenever you edit it, we cache that file's path and source code. Then when building a FileManager we pass these files as "overrides" so their contents are read from this cache instead of the filesystem. We do this so that even if you don't save a file, code that references that unsaved content works fine. This is how Rust Analyzer seem to work too (I'm not sure how they implemented it, though, but the exhibited behavior is that).

For overrides, what we used to do before this PR is traversing all files in the filesystem for the relevant packages and adding them to the FileManager, except that if there's an override for a path we'd use that override's source code. That works fine except when a file is deleted from the filesystem. In that case the override won't be added because we never find that path in the filesystem to be able to override it!

Initially I thought we could add the overrides at the end, at least those that we didn't find in the filesystem. The problem with that is that the override's FileID will end up being different than the one before the file was deleted, so things like "Go to definition", etc., which rely on FileIDs won't work well.

An alternative approach, which is what this PR does, is to first compute all the filenames that need to go into the FileManager. Those come from the filesystem, but also from the overrides. Then, to have a consistent FileID order, we can sort all of these filenames, and just then add them to the FileManager (considering overrides here too).

That said, this is only done if there are overrides specified, otherwise in a regular compilation the order will change on where the files are stored (their parent absolute path). When there are no overrides the order will be that given by the packages traversal and the filesystem directory entries.

Here's how it works in Rust when you delete a file but continue editing it. Notice how we can still jump to that file from another file, and even if you add a new definition to the deleted file, it will be suggested in other files:

And now it works the same way in Noir LSP:

Additional Context

Documentation

Check one:

• No documentation needed.
• Documentation included in this PR.
• [For Experimental Features] Documentation to be submitted in a separate PR.

PR Checklist

• I have tested the changes locally.
• I have formatted the changes with Prettier and/or cargo fmt on default settings.

[github-actions]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Test Suite Duration'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.

Benchmark suite	Current: 940d829	Previous: 9b73209	Ratio
test_report_noir-lang_noir_bigcurve_	284 s	230 s	1.23

This comment was automatically generated by workflow using github-action-benchmark.

CC: @TomAFrench

[asterite]

Hmm... the order of the files will change depending on the absolute filename they have.