Disabling tree-sitter on big files

[kirawi]

Opening large files (e.g. >100mb) that have an associated syntax highlighting grammar leads to high memory usage and the file taking a while to load: tree-sitter/tree-sitter#222

Ideally opening such a file would prompt the user with disabling syntax highlighting for the file.

[archseer]

Refactor tree-sitter-highlight to work like the atom one, recomputing partial tree updates.

This is already done, the base layer is incremental, and injections are recomputed.

I personally don't think we should bother highlighting above a certain very large file size.

[kirawi]

I agree, but some people (like me) like syntax highlighting on big files even if it's not actually that useful. So yeah, I agree with you which is why I want this to be a plugin in the future. I might do it myself if I'm not too busy. For the prompt, I'm not sure how dependent it would be on tui since we're switching to termwiz.

[kirawi]

I'll take this issue on (prompt to disable syntax highlighting on big files).
Unassigning myself.

[sudormrfbin]

Wouldn't we have to use syntect anyway to provide highlights for languages without treesitter at some point in the future ? Installing a plugin for syntax highlighting of some filetypes seems annoying.

[kirawi]

Maybe, syntect seems fairly light and there are a lot of common dependencies between it and Helix. However it might be a problem because it's Regex-based, and we're already trying to move away from the standard Regex library.

[Aloso]

@kirawi syntect uses the oniguruma regex engine by default, though it can be configured to use fancy-regex instead. If you mean regex with the "standard Regex library", why are you trying to move away from it? cargo tree shows that regex is used by 5 different crates in the dependency graph.

[archseer]

I don't have any plans to drop the regex library, I also don't plan on supporting syntect. The scope of this issue should purely be on disabling highlighting on 200MB+ files.

[pppKin]

Hi, any update on this one? I opened a 21k+ lines lua file and it stack overflowed tree-sitter and crashing helix. While I try to gather more info on that particular stack overflow and report it to upstream, it would be nice to be able to disable tree-sitter on large file right within helix.

[tgharib]

This is a deal-breaker for me as well. As soon as I open a 820k+ LoC C file for registers, helix crashes. Change the file extension to txt and it instantly loads.

[pppKin]

Apart from the disable ts for files larger than xx mb thing , I think we can also implement a open as plain text command in file picker.

edit: not just file picker, search results, references, etc

[msdrigg]

I am seeing this as well in a long .json file. Thankfully I found this issue and changing the lang to .txt fixed it.

This should be the default for large files.

[archseer]

I still think there should be an upper bound but we can set it to something really high (200~500MB?)

[pascalkuthe]

I still think there should be an upper bound but we can set it to something really high (200~500MB?)

Are you just concerned about the 500ms delay when first opening such a large file (the time until the parser times out) or what usecase did you have in mind?

Atlwadt for 200MB headers in the kernel I have seen TS perform pretty decently (and I think I it can be optimized further so.it endsup pretty usable) so the limit should be pretty large (like 500MB) IMO

[archseer]

I think past a certain size size limit there's just no point in wasting CPU cycles to even attempt highlighting, e.g. 500mb+

[pascalkuthe]

Makes sense adding a limit should be really straightforward now. Just need to return an error from the parser function if the filesize is larger than 500MB.

[iocron]

I am not even able to open huge files, because it crashes on the file picker code-preview beforehand.

[MasterAwesome]

Is there a way to support partial syntax highlighting not based on TS? Possibly using something like syntect or similar?

Alternatively, I also think the ability to force treesitter parsing to run through a command like :ts-force-parse might be useful since we have default timeouts of 500ms and possibly file size based limits too in the future.

[kirawi]

We don't want to support any highlighter except tree-sitter. It's possible that we may support LSP highlighting in the future, but regex highlighting would have to be implemented as a plugin.

[leath-dub]

Can you not just parse chunks of the file, treesitter was made to be error tolerant so especially if you make the chunk size large enough it is very likely an error would be recovered far away from the users view.

Edit: I suppose this is quite a refactor though as a lot of other features rely on the whole tree