[Sonic] Fix Semantic tokens - #83944
Merged
chsienki merged 3 commits intoJun 1, 2026
Merged
Conversation
Adds an L1 primitive for fetching the decl-half source generated document of
a Razor document. The Sonic 3.2 source generator now emits two C# documents
per component (impl + decl); IDE features that previously read the single
generated document need access to both halves to be split-aware.
This commit makes the decl-half accessor a first-class member of
IDocumentSnapshot so cohost features can use it directly without downcasting
to RemoteDocumentSnapshot.
The L1 primitive chain:
- GeneratorRunResult.TryGetDeclSourceGeneratedDocumentForRazorFilePathAsync:
given a Razor file path, computes the decl hint name (impl hint name with
the .g.cs suffix replaced by .decl.g.cs) and fetches the corresponding
source generated document, or null if the generator did not emit a decl
document for that file.
- RemoteProjectSnapshot.TryGetDeclGeneratedDocumentAsync: thin wrapper
around GeneratorRunResult, matching the existing
GetRequiredGeneratedDocumentAsync pattern.
- RemoteDocumentSnapshot.TryGetDeclGeneratedDocumentAsync: caches the
(possibly null) result. Uses a separate _declGeneratedDocumentInitialized
bool because the cached value being null is meaningful (no decl half
exists) and shouldn't trigger a re-query.
- IDocumentSnapshot.TryGetDeclGeneratedDocumentAsync: new interface member.
Cannot use a default interface implementation because the project
multi-targets netstandard2.0 / net472 which don't support DIMs.
- TestDocumentSnapshot returns null, matching the no-source-generator
contract noted in the interface XML doc.
Additive change: no existing callers are affected, no behavior changes for
.cshtml files or non-component documents.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adopts the 'Type A' pattern (read-only aggregation across both halves) for the semantic tokens cohost feature. Results from both halves land in the same razor-coordinate sink; the existing sort and overlap-rejection in ConvertSemanticRangesToSemanticTokensData handles any overlap between impl and decl ranges. ICSharpSemanticTokensProvider: - Add GetDeclCSharpSemanticTokensResponseAsync. Returns null when no decl half exists, which the caller treats as 'skip the decl pass'. RemoteCSharpSemanticTokensProvider: - Implement the decl method using the new IDocumentSnapshot.TryGetDeclGeneratedDocumentAsync L1 primitive (added in the previous commit), then forward to SemanticTokensRange.GetSemanticTokensAsync against the decl Roslyn document. No downcast to RemoteDocumentSnapshot required. AbstractRazorSemanticTokensInfoService: - Extract the per-document body of AddCSharpSemanticRangesAsync into a new private helper AddCSharpSemanticRangesForDocumentAsync that takes the RazorCSharpDocument and an isDeclHalf flag. The virtual entry point now calls the helper once for the impl half, then again for the decl half if codeDocument.GetDeclCSharpDocument() returns non-null. - Overload TryGetSortedCSharpRanges to accept a RazorCSharpDocument parameter; keep the original (codeDocument, range) signature as a convenience overload that uses the impl half (preserves existing callers and tests). - Tweak ranges.SetCapacityIfLarger to use ranges.Count + ... so the impl-pass growth isn't blown away when sizing for the decl pass. Tests: - Remove all 7 PROTOTYPE-skip annotations on CohostSemanticTokensRangeEndpointTest. With combinatorial Theory data this enables 76 test cases (was 48). Verification: - VS unit tests (net472): 76/76 pass. - VS Code unit tests (net10.0): 76/76 pass. This is the first feature migration in the Sonic 3.3 series; it follows the canary 2 pattern from sonic/3_3_cohosting_canaries (cleaned up to use the new interface method, no PROTOTYPE workarounds). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
davidwengier
approved these changes
May 31, 2026
davidwengier
left a comment
davidwengier
left a comment
Member
There was a problem hiding this comment.
I think there is a nicer refactoring of the semantic tokens service (potentially even, if it helps, going as far as to remove the ICSharpSemanticTokensProvider indirection entirely since it's no longer necessary), but given this is a feature branch, I'm approving as that can be done in a follow up.
| .TryGetDeclGeneratedDocumentAsync(cancellationToken) | ||
| .ConfigureAwait(false); | ||
|
|
||
| if (declGeneratedDocument is null) |
Member
There was a problem hiding this comment.
Annoying to duplicate all this code. Can we either extract everything below this into a separate method than the other GetCSharpSemanticTokensResponseAsync method calls? Or can the caller pass the RazorCSharpDocument in to that method, and somehow go from that instance to the generated document in a more general fashion?
| _logger.LogDebug($"Requesting C# semantic tokens for host version {documentContext.Snapshot.Version}, correlation ID {correlationId}, decl half: {isDeclHalf}, and the server thinks there are {generatedDocument.Text.Lines.Count} lines of C#"); | ||
|
|
||
| var csharpResponse = await _csharpSemanticTokensProvider.GetCSharpSemanticTokensResponseAsync(documentContext, csharpRanges, correlationId, cancellationToken).ConfigureAwait(false); | ||
| var csharpResponse = isDeclHalf |
Member
There was a problem hiding this comment.
Somewhat as an alternative to my previous comments, consider refactoring this method a bit, so that the code up to this point is called from the above method directly, which already knows whether it's doing the decl or impl. Not sure how nice or ugly that would end up looking though, so feel free to ignore this is if it ends up horrible.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
chsienki
marked this pull request as ready for review
June 1, 2026 04:24
chsienki
enabled auto-merge (squash)
June 1, 2026 04:24
Contributor
There was a problem hiding this comment.
Pull request overview
Updates Razor semantic token generation to account for the decl/impl source-generator split by performing an additional semantic token pass over the decl-half generated C# document when available, improving token coverage for component code bodies.
Changes:
- Add a decl-half semantic token request path and process results alongside the existing impl-half tokens.
- Extend
IDocumentSnapshot(and remote/test implementations) to optionally provide the decl-halfSourceGeneratedDocument. - Re-enable previously skipped cohosting semantic token range tests now that decl/impl split is handled.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/Razor/src/Razor/test/Microsoft.CodeAnalysis.Razor.CohostingShared.UnitTests/Endpoints/CohostSemanticTokensRangeEndpointTest.cs | Unskips semantic token tests that depended on decl/impl split awareness. |
| src/Razor/src/Razor/test/Microsoft.AspNetCore.Razor.Test.Common.Tooling/ProjectSystem/TestDocumentSnapshot.cs | Implements the new IDocumentSnapshot.TryGetDeclGeneratedDocumentAsync API for test doubles (returns null). |
| src/Razor/src/Razor/src/Microsoft.CodeAnalysis.Remote.Razor/SemanticTokens/RemoteCSharpSemanticTokensProvider.cs | Adds decl-half semantic token support and centralizes the token request logic. |
| src/Razor/src/Razor/src/Microsoft.CodeAnalysis.Remote.Razor/ProjectSystem/RemoteProjectSnapshot.cs | Adds decl-half generated document retrieval via source generator run results. |
| src/Razor/src/Razor/src/Microsoft.CodeAnalysis.Remote.Razor/ProjectSystem/RemoteDocumentSnapshot.cs | Caches decl-half SourceGeneratedDocument? retrieval on the remote snapshot. |
| src/Razor/src/Razor/src/Microsoft.CodeAnalysis.Remote.Razor/ProjectSystem/GeneratorRunResult.cs | Adds helper to locate the decl-half source-generated document from the impl hint name. |
| src/Razor/src/Razor/src/Microsoft.CodeAnalysis.Razor.Workspaces/SemanticTokens/ICSharpSemanticTokensProvider.cs | Extends the provider contract with a decl-half semantic tokens method. |
| src/Razor/src/Razor/src/Microsoft.CodeAnalysis.Razor.Workspaces/SemanticTokens/AbstractRazorSemanticTokensInfoService.cs | Performs two-pass C# token collection (impl + optional decl) and merges mapped results. |
| src/Razor/src/Razor/src/Microsoft.CodeAnalysis.Razor.Workspaces/ProjectSystem/IDocumentSnapshot.cs | Adds optional decl-half SourceGeneratedDocument retrieval API. |
| @@ -1,4 +1,4 @@ | |||
| // Licensed to the .NET Foundation under one or more agreements. | |||
| // Licensed to the .NET Foundation under one or more agreements. | |||
davidwengier
approved these changes
Jun 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Microsoft Reviewers: Open in CodeFlow