Handle dynamic registration of semantic tokens capability

[tatsudo]

In C# files nothing gets semantic highlighting — class fields and other identifiers are colored by tree-sitter only, so a private field looks the same as a plain local variable.

The reason: Roslyn (the C# language server) doesn't declare semanticTokensProvider statically in its initialize response. It registers it dynamically (client/registerCapability), and only when the client advertises textDocument.semanticTokens.dynamicRegistration = true. Zed advertised false and didn't handle such a registration, so Roslyn never offered semantic tokens at all. (rust-analyzer/gopls are unaffected — they declare the capability statically.)

This is the first of two PRs. This one makes Roslyn actually send semantic tokens. The companion PR (#60027) maps Roslyn's C#-specific token types to theme styles — without it the tokens arrive but most are dropped, since their types aren't in Zed's default rules.

Solution

• Advertise textDocument.semanticTokens.dynamicRegistration = true.
• Handle the textDocument/semanticTokens registration and unregistration so the capability is stored, following the existing arms for documentLink, diagnostics, etc.

Testing

• Added a test that dynamically registers and unregisters textDocument/semanticTokens and checks the stored capability appears and is cleared.
• Verified manually against Roslyn on a C# project: Zed now sends textDocument/semanticTokens/full and gets tokens back; before this change there was no semantic-token traffic at all.

Self-Review Checklist:

• I've reviewed my own diff for quality, security, and reliability
• Unsafe blocks (if any) have justifying comments
• Tests cover the new/changed behavior
• Performance impact has been considered and is acceptable

---

Release Notes:

• Support dynamic registration of the textDocument/semanticTokens capability.

[SomeoneToIgnore]

Thank you for the fix.

I suspect we need to refresh semantic tokens after the registration, similar to #60009 ?
If so, would be great to have a test and a refresh code; otherwise would be interesting to know what's different between two cases.

[tatsudo]

Good point, that should be refreshed too, I missed it. I've added the refresh after registration (the case is the same as #60009) plus a test that re-queries the open document.

[SomeoneToIgnore]

1. Both tests seem to pass if I omit prod changes from the latest commit — this is wrong.
   Either my proposal is incorrect and we already refresh the semantic tokens on capability changes (curious to know, why there is such a difference?) or the test is wrong

2. If I apply the test changes only, one of them hangs instead of panicking

First is crucial to fix properly, second is optional but good to have fixed nonetheless.

[tatsudo]

You're right. I missed that the editor requests semantic tokens with a ~50ms debounce, and the test never advanced the timer — so the refresh scheduled when the document opens stayed pending and fired after the registration. That's why a request happened even without the prod change (and why next().await hung instead of failing when it didn't).

I added the clock advance to drain that initial refresh first, and switched to asserting via a request counter. Now the test fails fast (no hang) without the registration refresh and passes with it — verified both ways.

Regarding your question the refresh is needed: with the registration refresh removed, the corrected test fails because the already-open document is never re-queried. So we don't already refresh on capability registration. So it looks like the same fix as in that PR #60009.

[SomeoneToIgnore]

Thanks!