Use the faster implmentation of the 2020 syntaxtical classifications

[orta]

Fixes #38435

This builds on the 2020 semantic classifier by moving it to use a visitor pattern, as seen in aeschli/typescript-vscode-sh-plugin#10 which means it only looks at nodes which are relevant, based on the implementation in aeschli/typescript-vscode-sh-plugin#10

This makes minor improvements to some of the results (correctly adding local, converting module to be classified as a namespace in the protocol)

[orta]

Reviewed:
1: Convert to use the ts API for visitNode etc instead of re-creating
2: Convert tokenModifiers to act more like the other flags enums in the codebase

[andrewbranch]

@alexdima @orta is there any context on this I should be aware of before diving into a review? Is this ready to go as far as you know, or is there a reason we never reviewed it at the time? 🙈

[orta]

My guess is more that it's just an overwhelmingly big PR, and while there's a reasonable amount of tests which are not breaking it's probably a slog to be certain

[alexdima]

@andrewbranch I did this as an exploration some time ago in order to investigate if TS could compute semantic tokens faster. Currently, the TS extension in VS Code defines a document range semantic tokens provider, which gets invoked for the range of the viewport together with a viewport above and one below. But if the user scrolls quickly to a different region, then, depending on theme settings and individual sensitivity, some flashing can be observed. The flashing can be eliminated by using a document semantic tokens provider, but that means computing semantic tokens for the entire document.

The exploration implements a custom AST walker which maintains custom state useful for semantic tokens for speedier access (like cached symbols, cached locals). In the end, I could measure 10x improvements for computing semantic tokens for checker.ts with this approach.

However, I realise that maintaining a separate visitor just for semantic tokens is probably not something anyone would be keen to do, plus I don't think I came across a lot of negative user feedback about the viewport semantic tokens provider. So, personally, I don't mind if you don't want to take this.

[andrewbranch]

That context is very helpful, thanks! So this change essentially just ignores the span parameter identifying the viewport + buffer range, right? In other words, as a drop-in replacement, it might actually be slower on large files since it’s tokenizing the whole thing instead of a small span?

[andrewbranch]

I think additional updates to the semantic classifier are going to need to be driven by concrete goals and design discussions that take into account what’s changed since this PR was opened.

[alexdima]

So this change essentially just ignores the span parameter identifying the viewport + buffer range, right? In other words, as a drop-in replacement, it might actually be slower on large files since it’s tokenizing the whole thing instead of a small span?

Yes, this implementation is not meant to be a drop-in replacement for the document range semantic tokens provider. This is an implementation suitable for a document semantic tokens provider.