Python highlighting could be improved to cover new type alias syntax (PEP 695)

[kas2020-commits]

As per PEP 695, starting in version 3.12 Python has new syntax for defining type parameters and type aliases.

To be clear, the tree sitter parser works just fine, it's the highlighting that could be improved.

You can now define type aliases in Python like so:

type C = ...

where type should be defined as a @keyword in /runtime/queries/python/highlights.scm same as class in class definitions. The Tree-sitter sub-tree for the provided code is this:

(type_alias_statement
  (type
    (identifier))
  (type
    (elipsis)))

An important caveat is that the literal type is not always a keyword - only in this context is it a keyword. By default, type is the symbol name for a built-in callable class definition, but it can also be considered a type hint.

Basically, these are all the major contexts of how type should be identified for highlighting purposes:

# `type` should be highlighted as a @keyword
type C = ...

# `type` Should be highlighted as a @type
MyInt: type[int] = int

# `type` should be highlighted as an @identifier
_ = type

# `type` should be highlighted as a @function.builtin 
_ = type(None)

[7ombie]

Excuse my ignorance, but I was confused about the last two cases:

_ = type
_ = type(None)

As I understand it, in both cases, type is an identifier that refers to a builtin function. The second case simply uses the identifier in an expression (the parens are essentially just a suffix operator). Why do we introduce a distinction when there's no meaningful difference?

Edit: Just noticed that GitHub highlights them differently too (so it must assign different token types as well).

[kas2020-commits]

@7ombie While all functions are variables it's not the case that all variables are functions, so when used as a variable you can't/shouldn't highlight it like a function, because you would be inconsistently highlighting variables differently depending on their type. [I think] this is the case for all programming languages that have "first-class functions" or higher order functions.

[7ombie]

@kas2020-commits - I see. I'm sure you're right (about how it works), but that makes no sense to me.

I thought it would use the type of the variable, so each variable would be consistently highlighted throughout, indicating what it is. Instead, we're highlighting the same variable differently, based on whether it occurs in a terminal expression (just the name) or a compound expression (like an invocation), which only indicates what I'm doing with it at that time, which isn't informative (I can already see what I'm doing). It's just noise.

[arthur-st]

Instead, we're highlighting the same variable differently, based on whether it occurs in a terminal expression (just the name) or a compound expression (like an invocation), which only indicates what I'm doing with it at that time, which isn't informative (I can already see what I'm doing). It's just noise.

@7ombie It's not quite the same:

• in an expression like x = type(None), type() is a built-in function (technically a class) identifying the type of the object;
• in an expression like x = type, type is a metaclass (i.e., its own type, output of type(type)), representing an implementation edge case.

The example above, therefore, produces the following:

Python 3.12.2 (tags/v3.12.2:6abddd9, Feb  6 2024, 21:26:36) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> type
<class 'type'>
>>> type(type)
<class 'type'>
>>> type(None)
<class 'NoneType'>
>>> type(type(type))
<class 'type'>
>>> type(type(None))
<class 'type'>