Add Injected Language Support

[thecoolwinter]

Description

This PR adds support for injected languages using tree-sitter as described in #16 and in the tree-sitter documentation. Languages can contain injected languages which are described in a language's injections.scm file. Some examples of injected languages are:

• HTML contains CSS and Javascript in style and script tags
• Javascript contains Regex literals
• PHP contains HTML between the <php tags
• C++ can contain rawstring literals of arbitrary languages

Details

This PR is a rework of the TreeSitterClient class. Specifically it:

• Adds a layers array and primaryLayer property. layers contains all language layers in the document, and primaryLayer is the ID of the document's primary language.
• Each layer is a LanguageLayer object. These objects represent an injected language-range(s) combination. Each layer can have one or more range associated with it depending on if it should be parsed as one document or multiple.
• When editing:
  • Each layer's ranges are updated using a similar algorithm to tree-sitter, and edits are applied to make use of the incremental parsing tree-sitter gives.
  • Each layer is checked for any injections, inserting any new injected layers and keeping track of any 'untouched' layers to remove.
  • Any layers that were not touched are removed.

The highlight query algorithm is largely the same, but keeps track of any ranges not used by any injected layers, and only queries the primary layer for those ranges so as not to override any injected highlights.

Related Issues

• Closes ✨ Injected Languages #16

Screenshots

Before, other languages were detected but parsed and highlighted as normal text.

With Injected languages, in this case CSS and JS embedded in HTML and a second layer of Regex embedded in JS embedded in HTML:

[lukepistrol]

Is this still WIP?

[thecoolwinter]

Is this still WIP?

Yeah it is. There are still some glaring issues right now that I thought would be trivial to fix but are ending up being trickier than expected.

[lukepistrol]

Just a couple of code style considerations about function parameter indentation. I provided fixes for most of them.

Refs: Code Style

[lukepistrol]

LGTM 👍

[austincondiff]

This is really great work @thecoolwinter! 👏

[lukepistrol]

@thecoolwinter Are there any breaking changes regarding the public API? Need to know when releasing a new version.

[thecoolwinter]

@thecoolwinter Are there any breaking changes regarding the public API? Need to know when releasing a new version.

Yes, there's an additional method on HighlightProvider which external highlight providers will have to implement when updating.