Add telemtry for how long it takes to parse files with tree-sitter #213565

alexr00 · 2024-05-27T14:26:52Z

Part of #210475

…aker

src/vs/editor/browser/services/treeSitter/treeSitterParserService.ts

hediet · 2024-05-27T15:40:49Z

CI failure

…serService

hediet

Nice work!
There are still some race conditions that I think should be fixed before merging this though.

src/vs/editor/browser/services/treeSitter/treeSitterParserService.ts

alexr00

@hediet, thanks for the review! I tried to capture all of your feedback.

src/vs/editor/browser/services/treeSitter/treeSitterParserService.ts

hediet · 2024-07-18T10:13:34Z

src/vs/editor/browser/services/treeSitter/treeSitterParserService.ts

 	}

 	override dispose() {
 		super.dispose();
-		this._treeSitterTree?.dispose();
-		this._unregisterModelListeners();
+		this._languageSessionDisposables.dispose();


This is not needed, as the this._register call takes care of that :)
The nice thing about _register is that you can see at the creation location in the source that it is eventually going to be disposed.

🤦‍♀️

src/vs/editor/browser/services/treeSitter/treeSitterParserService.ts

hediet · 2024-07-23T12:51:47Z

src/vs/editor/browser/services/treeSitter/treeSitterParserService.ts

+			}
+			return this._onDidChangeContent(model, e);
+		}).catch((e) => {
+			this._logService.error('Error parsing tree-sitter tree', e);


I think onUnexpectedError might also be good here, because then you'd see it in telemetry.

hediet · 2024-07-23T12:57:52Z

src/vs/editor/browser/services/treeSitter/treeSitterParserService.ts

+			try {
+				tree = this.parser.parse((index: number, position?: Parser.Point) => this._parseCallback(model, index), this.tree);
+			} catch (e) {
+				// parsing can fail when the timeout is reached, will resume upon next loop


Can you check the error type and re-throw the error for non-expected errors?

…icrosoft#213565) * Make space for tree sitter * Add the tree sitter wasm file * Very naive tree-sitter syntax highlighting for html, with a layer breaker * Update tree when content changes * WIP for making abstract tokens class * Handle theme changes * Replace entire text model value with parse callback * Perf improvements * Add tree-sitter-typescript * Add typescript + better initial parsing * Refactor into tree parsing service and fix flaw in parse callback * Remove things that aren't the parser service * Add yielding * Remove changes that aren't required for PR * Remove more file changes * Reduce yield to 50 ms * Fix incremental parsing * Try update node-abi * Revert "Try update node-abi" This reverts commit df28801. * Update text buffer chunk api * fix build * Remove tree-sitter dependency * Adopt new, as yet unpublished, `@vscode/tree-sitter-wasm` package * Use published `@vscode/tree-sitter-wasm` package * Break `TreeSitterTree` and `TreeSitterParserService` into better pieces and: - document the order of editor changes - use service injection where `TextModel` is constructed * Fix tests * Remove unneeded import * Fix missing tree-sitter-wasm in web and remote * Make package.jsons match * Add @vscode/tree-sitter-wasm to web loader config * Try using importAMDNodeModule * PR feedback * Add race condition test for changing language while loading language * Use same timeout * Queue content changes * Remove override dispose * Move queue into TreeSitterTree --------- Co-authored-by: Peng Lyu <[email protected]>

alexr00 self-assigned this May 27, 2024

alexr00 enabled auto-merge (squash) May 27, 2024 14:26

alexr00 requested a review from hediet May 27, 2024 14:27

VSCodeTriageBot added this to the May 2024 milestone May 27, 2024

alexr00 added 15 commits May 27, 2024 16:43

Make space for tree sitter

e1ea829

Add the tree sitter wasm file

150354b

Very naive tree-sitter syntax highlighting for html, with a layer bre…

1ee4fe2

…aker

Update tree when content changes

3614235

WIP for making abstract tokens class

2703b6b

Handle theme changes

edde184

Replace entire text model value with parse callback

5c20e64

Perf improvements

4cd3e08

Add tree-sitter-typescript

fef5067

Add typescript + better initial parsing

5c65dfa

Refactor into tree parsing service and fix flaw in parse callback

c372e40

Remove things that aren't the parser service

d411bfc

Add yielding

7c0e5d5

Remove changes that aren't required for PR

e32a24d

Remove more file changes

7f0678a

alexr00 force-pushed the alexr00/treeSitterParserService branch from 42c4efa to 7f0678a Compare May 27, 2024 14:48

alexr00 commented May 27, 2024

View reviewed changes

src/vs/editor/browser/services/treeSitter/treeSitterParserService.ts Outdated Show resolved Hide resolved

alexr00 modified the milestones: May 2024, June 2024 May 28, 2024

alexr00 mentioned this pull request May 28, 2024

Explore using tree sitter for syntax highlighting #210475

Open

rebornix mentioned this pull request Jun 13, 2024

Update treesitter parsing with text buffer chunk api #215471

Merged

alexr00 added 4 commits June 18, 2024 12:41

Reduce yield to 50 ms

eb1bba2

Fix incremental parsing

d56de42

Merge branch 'main' into alexr00/treeSitterParserService

194527c

Try update node-abi

df28801

alexr00 added 8 commits July 15, 2024 10:31

Merge remote-tracking branch 'origin/main' into alexr00/treeSitterPar…

3a9a103

…serService

Fix tests

7148842

Remove unneeded import

2c2e321

Fix missing tree-sitter-wasm in web and remote

028c770

Make package.jsons match

539a273

Add @vscode/tree-sitter-wasm to web loader config

e6c2529

Merge branch 'main' into alexr00/treeSitterParserService

4f98572

Try using importAMDNodeModule

04232a4

hediet reviewed Jul 17, 2024

View reviewed changes

PR feedback

0ca55da

alexr00 commented Jul 18, 2024

View reviewed changes

alexr00 added 2 commits July 18, 2024 12:08

Add race condition test for changing language while loading language

6435924

Use same timeout

7e82f06

alexr00 requested a review from hediet July 18, 2024 10:09

hediet reviewed Jul 18, 2024

View reviewed changes

src/vs/editor/browser/services/treeSitter/treeSitterParserService.ts Outdated Show resolved Hide resolved

Queue content changes

13a6a3b

alexr00 requested a review from hediet July 18, 2024 13:17

alexr00 added 2 commits July 19, 2024 17:04

Remove override dispose

d08bbb1

Move queue into TreeSitterTree

26edd5c

hediet reviewed Jul 23, 2024

View reviewed changes

hediet approved these changes Jul 23, 2024

View reviewed changes

alexr00 merged commit 6c15586 into main Jul 23, 2024
6 checks passed

alexr00 deleted the alexr00/treeSitterParserService branch July 23, 2024 12:59

alexr00 mentioned this pull request Jul 23, 2024

Revert "Add telemtry for how long it takes to parse files with tree-sitter" #223317

Merged

alexr00 restored the alexr00/treeSitterParserService branch July 24, 2024 08:14

alexr00 deleted the alexr00/treeSitterParserService branch July 29, 2024 09:31

vs-code-engineering bot locked and limited conversation to collaborators Sep 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add telemtry for how long it takes to parse files with tree-sitter #213565

Add telemtry for how long it takes to parse files with tree-sitter #213565

alexr00 commented May 27, 2024

hediet commented May 27, 2024

hediet left a comment

alexr00 left a comment

hediet Jul 18, 2024

alexr00 Jul 18, 2024

hediet Jul 23, 2024

hediet Jul 23, 2024

Add telemtry for how long it takes to parse files with tree-sitter #213565

Add telemtry for how long it takes to parse files with tree-sitter #213565

Conversation

alexr00 commented May 27, 2024

hediet commented May 27, 2024

hediet left a comment

Choose a reason for hiding this comment

alexr00 left a comment

Choose a reason for hiding this comment

hediet Jul 18, 2024

Choose a reason for hiding this comment

alexr00 Jul 18, 2024

Choose a reason for hiding this comment

hediet Jul 23, 2024

Choose a reason for hiding this comment

hediet Jul 23, 2024

Choose a reason for hiding this comment