language: Automatically detect language for new buffers

[maxstevens-nl]

Closes #4868

This is a draft for the automatic detection of the language of new buffers using Magika, an open-source AI model from Google for detecting file types.

The file-detection part of this PR works, but some unknowns need to be solved before it's ready to merge:

• The magika crate is built on top of ort, which uses a pulled in ONNX binary to run the model. This really bloats the final build, adding about 25mb (+ they don't provide a prebuilt binary for the x86_64-apple-darwin target). There are roughly three options to tackle this:
  • Fork the magika crate and port it to candle-onnx
  • Implement candle-onnx bindings to the Magika model in a local crate, removing the Magika dependency entirely
  • Accept the bundle size increase
• Ideally we want to run this detection only once per buffer, only if its language is yet undetermined. However, there is currently no 'undetermined' language for a buffer, since every new buffer has plaintext set as its language. As a workaround this PR currently runs the classification on every edit of a plaintext buffer, but this is not ideal.
• Creating the Magika session takes ~20ms. It seems fine to do that for every classification, instead of putting it in a global? Especially since classification of a file is a relatively uncommon operation.
• The first classification takes around 500ms on an M1 macbook. Every subsequent classification is <50ms. This difference seems to stem from the following added line in editor.rs, which is sometimes instant, and sometimes takes more than 400ms:

            if let Ok(detected_language) = detected_language.await {

This behavior is also observed in an implementation with a global indicating it's not related to session creation. Pointers on how to get the first classification down to <50ms would be appreciated.

• In this comment the tokio dependency of Magika is mentioned as an issue. Is that still relevant in this implementation? A quick glance reveals that Tokio is already in the dependency-tree of the editor crate, and it's not being used for the detection.

Release Notes:

• Added automatic language detection for new files

[zed-industries-bot]

	Messages
📖	This PR includes links to the following GitHub Issues: #4868 If this PR aims to close an issue, please include a Closes #ISSUE line at the top of the PR body.

Generated by 🚫 dangerJS against 4aa685b

[maxstevens-nl]

@dvdsk the PR I was telling you about. I might look into porting the Magika crate, and any insight regarding the time diff would also be helpful

[SomeoneToIgnore]

Thank you for the detailed report.

Binary size is somewhat tolerable as no good pass on reducing its size was done we sure can mitigate that one way or another.

But reading

The first classification takes around 500ms on an M1 macbook

and wondering, if this all is worth the effort?

which is sometimes instant, and sometimes takes more than 400ms:

Just to confirm, are we talking about the first classification only?
Does it happen for every new language/type of file or just the very first time on any random language?

If the latter, we could run any random file during the init phase and keep the session running?
If the former, I am not sure we want to move on with this approach.

---

Overall, even 50ms is somewhat an interesting difference: I imagine we can run all Zed's languages' tree-sitter queries on the document range and score the matches within the same timeframe.

Then, we'll get a way to detect Zed's language with Zed's infra with no extra dependencies and shenanigans?

[maxstevens-nl]

@SomeoneToIgnore

Just to confirm, are we talking about the first classification only? Does it happen for every new language/type of file or just the very first time on any random language?

I figured out the bottleneck, it's this line:

let language = language_registry.language_for_name(language_name).await;

that can take up to 800ms to execute. I'm not familiar with the language_registry, but I assume it's waiting to be initialized or on a lock to be released? If we can speed this call up, we can probably get the classification down to 5-20ms. Any suggestions or advice to speed up the language_for_name call?

Overall, even 50ms is somewhat an interesting difference: I imagine we can run all Zed's languages' tree-sitter queries on the document range and score the matches within the same timeframe.

Then, we'll get a way to detect Zed's language with Zed's infra with no extra dependencies and shenanigans?

Interesting idea. That's a choice that you'll have to make. Magika is probably better with partial files (doesn't care about syntax errors), while the tree-sitter approach minimizes binary size at the expense of accuracy (although I haven't tested that).

[SomeoneToIgnore]

Thank you for checking, alas, no good pointers on how to optimize the language loading.
One related note: the tree-sitter approach will have to do something similar anyway, so might be that we gain no latency benefits with this approach, and one way or another we have to find a way to load languages faster.

[SomeoneToIgnore]

Sounds like we need to deal with a few more latency things first, so I'll close this for now so it does not hang in the queue for eternity.