theme keys static and documented

[poliorcetics]

This PR adds a ThemeKey type that contains as much static scopes as possible for Tree Sitter
highlight queries.

It also adds a new generated file for the book which fully documents the highlight scopes and a
new cargo xtask subcommand to ensure all of the default highlight queries are in the static
scopes and will not fall in ThemeKey::OtherQuery.

Closes #3649

[poliorcetics]

@the-mikedavis I didn't add the theme checking but having a single source and always up to date source of truth for theme keys should make it much easier to know if a theme is up to date for maintainers, even with a manual workflow for now.

I added all the rest I think, and the resulting documentation is also more searchable IMO since I can now look for keyword.<something> whereas that was not possible before. because it used a manual tree structure.

[the-mikedavis]

I preferred seeing keys as "scope.scope.scope" rather than "ScopeScopeScope" - the fallback behavior was more intuitive. Is there a different way of approaching this, maybe by using a macro when getting a key from a theme?

[poliorcetics]

I preferred seeing keys as "scope.scope.scope" rather than "ScopeScopeScope" - the fallback behavior was more intuitive. Is there a different way of approaching this, maybe by using a macro when getting a key from a theme?

I could do Ui_Menu instead of UiMenu, that would make it much easier to read with a separator. It's not as Rust-y but it's a generated enum with dozens of variants so 🤷

[kirawi]

We could make use of namespacing.

[poliorcetics]

We could make use of namespacing.

I can try generating something like ui::Menu, and Ui constants yes, with the enum as backing type, good idea. I'll see what I can do

[poliorcetics]

We could make use of namespacing.

Well it's very hard in macro_rules and I'm not willing to introduce such a proc macro for just this, the complexity and compile time would suffer a lot. Nested elements in macros are always hard, both with types and modules. We could maintain a manual mapping of modules to the keys used in helix, or I can make another macro_rules.

@the-mikedavis, what do you think about Ui_Menu ? From an implementation POV, it's very easy, and the separator is there. I can make everything lowercase too, like ui_menu and wrap the whole thing in something like this:

pub type tk = ThemeKeys;

That way the usage point becomes tk::ui_menu, and the import is simply use helix_view::theme::tk;

[poliorcetics]

I moved the names to Ui_Linenr_Selected format, to make it easier to read and rebased on most recent master

[poliorcetics]

I don't think CI failures are related to my changes

[poliorcetics]

What do I have to do for this to move forward ? IMO it would greatly improve documentation for theme keys, and avoid allocating strings in several places

[archseer]

avoid allocating strings in several places

Are these allocations that costly? From what I can tell, we use static strings everywhere apart from parsing the theme file.

---

This might make sense for ui. but I'm not a huge fan of hardcoding the syntax scopes (even with the OtherScope escape hatch), it's a huge macro that's essentially a string interner just so we can use the syntax scopes in a couple of spots in the UI via get/try_get. When rendering we use raw indexes anyway.

The query validation could then be done externally: a file could essentially contain a list very similar to the macro input. We'd then use that to generate the documentation tree. A lint script could then cut the first column of known values, rg @\w+ (dropping any that start with @_) and cross reference against that list.

[poliorcetics]

The documentation benefits to know what is used in themes and what is not are nice and not to be dismissed.

For example, this PR surfaces several duplications between sshconfig and the rest, e.g. constant.builtin.boolean vs boolean (only used in sshconfig). I have also caught people unnecessarily introducing new scopes when existing ones were already used elsewhere.

Theming is very important to users, having a list of all keys, documented (which they would not be with a parser from the runtime queries), allows trimming down that list on our side (making theme definition easier) and correctly define new themes on the user's side.

Ideally we would even document which queries are used in which language, but that would probably make the table too big for now so I left it for later, once I have a better idea for presentation.

[poliorcetics]

@archseer @the-mikedavis I completely changed the way this is done.

1. The big macro is now only compiled when compiling the xtask crate, which means regular helix compilation won't pay the cost
2. I removed the big enum, leaving the &str in the code as currently in master.
3. I kept the check and documentation to ensure languages don't introduce similar-but-not-the-same keys in their highlight queries, if you look at the first commit you'll see that master has several such instances.

[poliorcetics]

Updated to latest master, this PR is no fun to maintain 😓

[archseer]

Hmm why were these closed? We were going to merge this in the next cycle