Code using python 3.12 syntax does not render with correct syntax highlighting

[matt-gardner]

Select Topic Area

Bug

Body

(It's possible this belongs in General instead of Code Search and Navigation, but this looked most appropriate.)

If you have code that uses python 3.12 syntax, such as the following:

class Foo[Bar]:
    x: Bar

GitHub does not render the code with correct syntax highlighting. The new type parameter syntax messes with the code rendering, both during viewing normal code, and during PR review.

The code above doesn't show the error, but if you have a complex enough file with 3.12 syntax sprinkled throughout, pretty quickly there are syntax highlighting errors.

---

Editing to add the failing example below, so the problem is more obvious from the original post:

class Foo[Input](Bar[Input, Baz], ABC):
    """This :py:class:`Foo` outputs :py:class:`Baz`es."""

    def __init__(self, bat: Bat[str], name: str | None = None) -> None:
        Bar.__init__(self, name).  # need to call Bar.__init__ instead of super().__init__ for unimportant reasons
        self.bat = bat

    def with_mapped_input[MappedInput](
        self,
        input_mapping_fn: Callable[[MappedInput], Input] | Callable[[MappedInput], Awaitable[Input]],
    ) -> "Foo[MappedInput]":
        return _MappedFoo(self, input_mapping_fn)

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[fxrdhan]

This appears to be a valid bug report related to GitHub's syntax highlighting for the new Python 3.12 type parameter syntax (PEP 695). You're describing how code using features like:

class Stack[T]:
    def push(self, item: T) -> None: ...
    def pop(self) -> T: ...

def max_value[T: Comparable](args: list[T]) -> T:
    ...

isn't being highlighted correctly in GitHub's interface.

This makes sense as a bug since:

• Python 3.12 introduced new syntax for generic type parameters directly in class and function definitions
• GitHub's syntax highlighter (likely using Linguist) needs to be updated to recognize these new syntax patterns
• Without proper recognition, the highlighter may interpret the square brackets incorrectly, leading to cascading highlighting errors

To help the GitHub team investigate, it would be helpful if you could:

• Share a minimal complete example that reliably reproduces the highlighting issues
• Specify where exactly the highlighting breaks down (e.g. does it affect the entire file or just specific constructs?)
• Note if this occurs in all GitHub contexts (raw files, PR diffs, code snippets in issues/comments)

Would you be able to provide any of these additional details to help track down the root cause?

[matt-gardner]

I found a case from our private repo that I could anonymize and paste in here, demonstrating the error. As far as I can tell, this error happens everywhere that syntax is highlighted, in snippets, in PR reviews, in viewing files, in comments, etc. I imagine the error I'm pasting below is just one of many cases where syntax highlighting fails, it was just the first case that I came across when trying to find something I could put in here.

class Foo[Input](Bar[Input, Baz], ABC):
    """This :py:class:`Foo` outputs :py:class:`Baz`es."""

    def __init__(self, bat: Bat[str], name: str | None = None) -> None:
        Bar.__init__(self, name).  # need to call Bar.__init__ instead of super().__init__ for unimportant reasons
        self.bat = bat

    def with_mapped_input[MappedInput](
        self,
        input_mapping_fn: Callable[[MappedInput], Input] | Callable[[MappedInput], Awaitable[Input]],
    ) -> "Foo[MappedInput]":
        return _MappedFoo(self, input_mapping_fn)

[jgiacobbi]

Another example:

import asyncio
from multiprocessing import Event, Process, log_to_stderr
from typing import Awaitable, Callable

from .payload_queue import Payload, PayloadQueue


class Consumer[T](Process):
    """A consumer process that reads data from a queue and processes it."""
    def __init__(
        self,
        sink: Callable[[T], Awaitable[None] | None],
        work_queue: PayloadQueue[T],
        *args,
        **kwargs,
    ):
        self.sink = sink
        self.work_queue = work_queue
        self.done = Event()
        super().__init__(*args, **kwargs)

I thought it was the 'generic class with inheritance' doing it because regular generic classes seem fine but I failed to create a smaller example.

[matt-gardner]

Reposting the above snippet using python after the triple backticks to demonstrate the highlighting failure inline:

import asyncio
from multiprocessing import Event, Process, log_to_stderr
from typing import Awaitable, Callable

from .payload_queue import Payload, PayloadQueue


class Consumer[T](Process):
    """A consumer process that reads data from a queue and processes it."""
    def __init__(
        self,
        sink: Callable[[T], Awaitable[None] | None],
        work_queue: PayloadQueue[T],
        *args,
        **kwargs,
    ):
        self.sink = sink
        self.work_queue = work_queue
        self.done = Event()
        super().__init__(*args, **kwargs)

[jgiacobbi]

I noticed that my example is highlighting fine in matt's repost so I went back and added more of the class to it until it broke in my private repo

import asyncio
from multiprocessing import Event, Process, log_to_stderr
from typing import Awaitable, Callable

from .payload_queue import Payload, PayloadQueue

class Consumer[T](Process):
    """A consumer process that reads data from a queue and processes it."""

    def __init__(
        self,
        sink: Callable[[T], Awaitable[None] | None],
        work_queue: PayloadQueue[T],
        *args,
        **kwargs,
    ):
        self.sink = sink
        self.work_queue = work_queue
        self.done = Event()
        super().__init__(*args, **kwargs)

    def run(self):
        self.logger = log_to_stderr()
        try:
            if asyncio.iscoroutinefunction(self.sink):
                asyncio.run(self.__arun())
            else:
                self.__run()
        finally:
            self.logger.info(f"exiting")
            self.done.set()

edit: added imports back in to eliminate that variable, made no difference

[matt-gardner]

The docstring is still wrong on my posting of it. Interesting that adding another method after __init__ causes a failure in the rendering of __init__ itself...

[look]

Thank you for the report, folks. I filed an issue to investigate if updating our Python highlighter will fix the issue.