autodoc: Usage/Clashes with non-python documenters

[LecrisUT]

Is your feature request related to a problem? Please describe.
sphinx.ext.autodoc is used for documenting only python objects, and that is perfectly fine. With the recent refactoring, it is a bit clearer that there is no special sauce in there, one can replicate it for their own domain just by populating the contents appropriately.

The suggestion here for "supporting" non-python documenters is more about providing some documentation and guides for how people can replicate the workflow of sphinx-apidoc -> auto* directives -> domain-specific directives.

• Have a tutorial for such a workflow. We can extend the recipe example and have it parse yaml files as the "auto" part there.
• Have the sphinx.ext.autodoc directives be owned by the python domain so that it would not clash with another domain that could similarly have an automodule (e.g. CMake also has a module entity that could clash there)
• Provide some useful helpers so that users avoid repeating themselves when constructing the content. I am primarily thinking of the context of autodoc: Parser agnostic content build helper #14051 and having a simple way of supporting more arbitrary text formats (both rst and markdown) that the content could have been written in.

Additional context

• Split from Restore the legacy implementation of autodoc for compatibility #14091 (comment)

[AA-Turner]

See also https://github.com/jnikula/hawkmoth (cc @jnikula)

[jnikula]

See also https://github.com/jnikula/hawkmoth (cc @jnikula)

Thanks for the ping.

I'll note that I've deliberately chosen not to try to plug any of this into sphinx.ext.autodoc Documenter classes. Hawkmoth just converts the documentation comments into C and C++ domain directives (like .. c:function::) and inserts them into the document. So the "Sphinx Autodoc for C" tagline is not entirely accurate.

There are obviously upsides and downsides to this. For example, we get none of the built-in features of the Documenters, and we have to replicate the applicable ones ourselves. Things like :members: option, or additional events. But then I can't even imagine how to sensibly adapt something like autodoc-process-docstring with its obj parameter to C/C++.

The huge upside is that we don't have to try to understand the inner workings of the Documenters, or to try to adapt them for C/C++, or to keep up with or adapt to Sphinx upstream changes across versions. It simply always felt easier to dodge that altogether. The Hawkmoth core functionality is just 1300 lines of code, of which 300 is for the Sphinx extension.

Hawkmoth does insert its own directives into the C and C++ domains, e.g. app.add_directive_to_domain('c', 'autofunction', CAutoFunctionDirective), and in hindsight I wonder if that was the right thing to do, and what would go haywire if Sphinx upstream added them itself. Fingers crossed that never happens...

[LecrisUT]

The huge upside is that we don't have to try to understand the inner workings of the Documenters, or to try to adapt them for C/C++, or to keep up with or adapt to Sphinx upstream changes across versions. It simply always felt easier to dodge that altogether.

FYI, the Documenters were refactored in this cycle. Specifically they were dropped 🙂. But in principle the idea is the same, the sole goal of auto* directives is to just create the content with the appropriate domain directives inside, and the recent refactor made it a bit clearer about that.

I don't think we should revive something like Documenter and try to make it domain independent. But if we can define what should be in such a directive, like events, dependency notes, helpers to write the content, content invalidators? then it would be quite a lot easier to write these.

[AA-Turner]

From a maintenence standpoint, I'd much prefer if third-party implementations don't rely on sphinx.ext.autodoc (similar to Hawkmoth). As @LecrisUT notes, there's nothing particularly unique to how autodoc inserts the parsed content into the document, so third-party projects can just do similar things.

As such, I'd be very hesitant to define anything at the centre.

A

[LecrisUT]

Maybe not definition wise, but just documentation and example. The recipe domain has been a very good starting point to understand some of the underlying concepts, and having something similar around this part would be really beneficial to understand more about the content invalidation like adding/deleting members of a domain and how that plays with a Domain.data.

[jnikula]

FYI, the Documenters were refactored in this cycle. Specifically they were dropped 🙂.

Oh, wow, I feel like I just dodged a bullet there. 😅