[ty] First cut at semantic token provider (#19108)

This PR implements a basic semantic token provider for ty's language
server. This allows for more accurate semantic highlighting / coloring
within editors that support this LSP functionality.

Here are screen shots that show how code appears in VS Code using the
"rainbow" theme both before and after this change.


![461737617-15630625-d4a9-4ec5-9886-77b00eb7a41a](https://github.com/user-attachments/assets/f963b55b-3195-41d1-ba38-ac2e7508d5f5)


![461737624-d6dcf5f0-7b9b-47de-a410-e202c63e2058](https://github.com/user-attachments/assets/111ca2c5-bb4f-4c8a-a0b5-6c1b2b6f246b)

The token types and modifier tags in this implementation largely mirror
those used in Microsoft's default language server for Python.

The implementation supports two LSP interfaces. The first provides
semantic tokens for an entire document, and the second returns semantic
tokens for a requested range within a document.

The PR includes unit tests. It also includes comments that document
known limitations and areas for future improvements.

---------

Co-authored-by: UnboundVariable <[email protected]>

Author: UnboundVariable

Cargo.lock

@@ -11,6 +11,7 @@ repository = { workspace = true }
 license = { workspace = true }
 
 [dependencies]
+bitflags = { workspace = true }
 ruff_db = { workspace = true }
 ruff_python_ast = { workspace = true }
 ruff_python_parser = { workspace = true }

crates/ty_ide/Cargo.toml

@@ -5,13 +5,17 @@ mod goto;
 mod hover;
 mod inlay_hints;
 mod markup;
+mod semantic_tokens;
 
 pub use completion::completion;
 pub use db::Db;
 pub use goto::goto_type_definition;
 pub use hover::hover;
 pub use inlay_hints::inlay_hints;
 pub use markup::MarkupKind;
+pub use semantic_tokens::{
+    SemanticToken, SemanticTokenModifier, SemanticTokenType, SemanticTokens, semantic_tokens,
+};
 
 use ruff_db::files::{File, FileRange};
 use ruff_text_size::{Ranged, TextRange};