[ty] First cut at semantic token provider

[UnboundVariable]

This PR implements a basic semantic token provider for ty's language server. This allows for more accurate semantic highlighting / coloring within editors that support this LSP functionality.

Here are screen shots that show how code appears in VS Code using the "rainbow" theme both before and after this change.

The token types and modifier tags in this implementation largely mirror those used in Microsoft's default language server for Python.

The implementation supports two LSP interfaces. The first provides semantic tokens for an entire document, and the second returns semantic tokens for a requested range within a document.

The PR includes unit tests. It also includes comments that document known limitations and areas for future improvements.

[github-actions]

mypy_primer results

No ecosystem changes detected ✅
No memory usage changes detected ✅

[codspeed-hq]

CodSpeed WallTime Performance Report

Merging #19108 will not alter performance

_{Comparing UnboundVariable:semantic_tokens (3be32e1) with main (4dd2c03)}

Summary

✅ 7 untouched benchmarks

[carljm]

Thanks for the PR! I'd prefer for Micha or Dhruv (who know the LSP and our plans in that area better) to review this; just one initial comment.

[dhruvmanila]

This is a great start, thank you for doing this!

I've been reviewing this for an hour and I need to take a lunch break but here are my initial comments. I'll look at the remaining parts after the lunch break

[dhruvmanila]

Ok, my review is finally done, this is an awesome PR!

The test cases are extremely thorough, thank you for writing them!

I think the only required change would be to make sure that we convert the ty location values back to the LSP values in the common function for the semantic tokens handler. And, around the visitor implementation where some of the calls that might lead to duplicate tokens could be removed.

A lot of my review comments could be considered as suggestions so feel free to either apply them or discard them. It's totally fine if those are done as follow-up. I can also own them as follow-up to this PR if you prefer.

[UnboundVariable]

@dhruvmanila, thanks for the thorough and insightful code review comments! I think I've addressed them all.

[dhruvmanila]

This is great! Thank you for taking this on and addressing all the review comments!

[MichaReiser]

This can be done in a sepreate PR, but it would be great to add semantic highlighting to the playground too. It should be "as easy as" implementing the necessary provider and registering them here:

ruff/playground/ty/src/Editor/Editor.tsx

Lines 143 to 180 in a663796

	class PlaygroundServer
	implements
	languages.TypeDefinitionProvider,
	editor.ICodeEditorOpener,
	languages.HoverProvider,
	languages.InlayHintsProvider,
	languages.DocumentFormattingEditProvider,
	languages.CompletionItemProvider
	{
	private typeDefinitionProviderDisposable: IDisposable;
	private editorOpenerDisposable: IDisposable;
	private hoverDisposable: IDisposable;
	private inlayHintsDisposable: IDisposable;
	private formatDisposable: IDisposable;
	private completionDisposable: IDisposable;
	constructor(
	private monaco: Monaco,
	private props: PlaygroundServerProps,
	) {
	this.typeDefinitionProviderDisposable =
	monaco.languages.registerTypeDefinitionProvider("python", this);
	this.hoverDisposable = monaco.languages.registerHoverProvider(
	"python",
	this,
	);
	this.inlayHintsDisposable = monaco.languages.registerInlayHintsProvider(
	"python",
	this,
	);
	this.completionDisposable = monaco.languages.registerCompletionItemProvider(
	"python",
	this,
	);
	this.editorOpenerDisposable = monaco.editor.registerEditorOpener(this);
	this.formatDisposable =
	monaco.languages.registerDocumentFormattingEditProvider("python", this);
	}

[MichaReiser]

I'll leave it up to you if this PR should close astral-sh/ty#260

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.