autodoc: Parser agnostic content build helper

[LecrisUT]

This is more of a show-and-tell to consider if something similar can be adopted by sphinx. The idea here is to have some helpers to construct and manipulate the SphinxDirective.content and equivalents. The high level interface that would be used is something like this

High level interface

class MyAutodocDirective(SphinxDirective):
    has_content = False

    def run(self) -> list[Node]:
        """
        equivalent to:

        .. code-block rst

            .. mydom:obj /path/to/obj

                Some random hard-coded content:
                * some list
                  with multi-line
                * and the next item

        or if it is run in myst-parser

        .. code-block markdown

            ```{mydom:obj} /path/to/obj

            Some random hard-coded content:
            * some list
              with multi-line
            * and the next item
            ```
        """
        # High-level interface to construct the content in the docstring
        with self.content.directive("mydom:obj", "/path/to/obj"):
            self.content.append("Some random hard-coded content:", source="")
            with self.content.list():
                self.content.append("some list", source="")
                self.content.append("with multi-line", source="")
                self.content.new_item()
                self.content.append("and the next item", source="")

I have implemented this in my project and I am hoping that some similar interface can be added here in upstream.

Implementation mock-up

RST_DIRECTIVE_INDENT = 4

class Content(StringList):
    """
    Wrapper around ``StringList`` with helper functions for formatting rst contents.
    """

    #: Current rst content indent of the content
    indent: int

    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.indent = 0

    def _indent_str(self, orig: str) -> str:
        """
        Prepend all the necessary indent and possibly the list symbol.
        """
        if not orig.strip():
            # No need to indent if the line is empty
            return ""
        # Indented line
        return " " * self.indent + orig

    @overload
    def _indent_other(self, other: StringList) -> StringList: ...

    @overload
    def _indent_other(self, other: list[str]) -> list[str]: ...

    @overload
    def _indent_other(self, other: str) -> str: ...

    def _indent_other(self, other):
        """
        Main helper function to indent any operand that is using
        """
        if isinstance(other, StringList):
            other.data = [self._indent_str(line) for line in other.data]
            return other
        if isinstance(other, list):
            return [self._indent_str(line) for line in other]
        if isinstance(other, str):
            return self._indent_str(other)
        raise NotImplementedError(f"Trying to indent an unknown input type: {type(other)}")

    # We need to override any methods used to insert items
    # Just kept append as example here
    def append(self, item, source=None, offset=0):
        item = self._indent_other(item)
        super().append(item, source, offset)

    # Context helpers
    @contextmanager
    def directive(
        self,
        name: str,
        *directive_args: str,
        **directive_kwargs: Optional[str],
    ) -> "Generator[Self]":
        """
        Add the directive header and start appending its content.

        This handles the rst indentation of the directive content introduced.

        :param name: directive name
        :param directive_args: directive's parameters
        :param directive_kwargs: other directive arguments
        """
        # Add the directive header
        self.append(f".. {name}:: {' '.join(directive_args)}".rstrip(), source="")
        self.indent += RST_DIRECTIVE_INDENT
        for key, value in directive_kwargs.items():
            self.append(f":{key}: {value or ''}".rstrip(), source="")
        self.append("", source="")
        # Start adding other contents
        yield self
        # exit the directive
        self.append("", source="")
        self.indent -= RST_DIRECTIVE_INDENT
        assert self.indent >= 0

Things that need to be considered:

• How to detect in which nested parser are we currently in, e.g. we could be in a markdown file but under a rst-eval directive:

  ```{eval-rst}
  .. mydom:autoobj /path/to/obj
  
      The content here needs to be injected as an rst content
  ```

  Some collaboration with myst-parser team would be nice since I haven't figured out a way to extract the current parser from that side either.
• Any implementation would be fine, not necessarily subclassing StringList, but it should at least be under a class so that the specific implementation for either rst-type or markdown-type could be overwritten depending on the state object
• How to make room for detecting what the additional content is written in? It could be a simple directive or conf-val level switch to say the docstring contents are rst or markdown based and then it could decide if it will wrap the content under a eval-rst directive or the reverse equivalent (which doesn't seem to be available in myst-parser?) depending on the state object

[taminomara]

In my autodoc implementations, I've ended up creating instances of directives and running them directly, instead of creating intermediate RST string and parsing it. Maybe this would be a better approach for python autodoc as well?

[taminomara]

This would be a good thing for tools like autosummary/apidoc, though!