Keep the centralized module documentation at docs.saltproject.io

[max-arnold]

This was proposed at the Salt Extensions Working Group meeting: https://youtu.be/m7_ZVsxzioc?feature=shared&t=2905

The idea is to keep the existing module documentation in place at https://docs.saltproject.io even after some modules will be extracted into community-maintained repositories. It doesn't add a lot of burden on the Salt Core Team, but could solve multiple problems that are caused by the code split. The benefits are:

• A familiar user experience
• Extension discovery (https://extensions.saltproject.io/ hasn't been updated since 2021)
• Documentation search (and also the existing search engine rankings)
• The existing offline docset for Dash/Zeal/Velocity will keep working https://github.com/Kapeli/feeds/blob/master/SaltStack.xml
• Common documentation standards
• Avoid module name collisions (due to common documentation URL namespace)
• Easy to find a corresponding issue tracker for any particular module

A module page header can have additional fields/badges:

• A clear module status indication: core/extended core/community
• A link to the repo and issue tracker
• Latest version and Salt compatibility status
• Test coverage status, number of issues, etc

Some problems with this approach:

• Security considerations
• Different versioning cycles for Salt itself and external modules

Any new extensions should have an opt-in mechanism to publish their docs on docs.saltproject.io.

P.S. Now is the best time to tackle this idea, because with time the module documentation structure will diverge and it will be harder to unify it again.

[ITJamie]

What does this mean for future new extensions though.

Eg If new extensions cant add to the salt docs It will mean new extensions wont be as discoverable as existing pulled out code

id also argue that salt core having a massive manpage isnt great #64255

[max-arnold]

Thanks for moving the discussion here from Slack. Yes, I believe that any new extensions should have an opt-in mechanism to publish their docs on docs.saltproject.io. I added this paragraph to the proposal.

[max-arnold]

Did a bit of research on how the Dash/Zeal offline docsets get built:

• The SaltStack docset is maintained by Kapeli itself (the company behind Dash). I was unable to find a GH action or something similar that is public
• The resulting docset is published here: https://github.com/Kapeli/feeds/blob/master/SaltStack.xml Inside of the archive are Salt docs in HTML format and SQL index (I believe it uses sqlite)
• Community-maintained docsets are kept in a separate repo https://github.com/Kapeli/Dash-User-Contributions#contribute-a-new-docset
• The docset generation process and tools are described at https://kapeli.com/docsets
• One of the official docset generator tools is called doc2dash and it can ingest Sphinx-generated html https://pypi.org/project/doc2dash/ https://doc2dash.hynek.me/en/latest/how-to/ https://doc2dash.hynek.me/en/latest/extending/

There are three rough ideas of how we can help keep this docset working (ordered from most optimal to less optimal, from a docset maintainer point of view):

1. Maintain a single core repository that also includes (or pulls as submodules) all extension docs and builds a single html doc tree. Basically the way things currently are.
2. Maintain a single core repository that also includes short stubs for all extensions. The extension stubs will be searchable, but their full docs aren't.
3. Maintain a machine-readable extension index (with metadata, possibly fetched from PyPI), in order to simplify the integrated docset generation process for Kapeli

[max-arnold]

I think that a machine-readable extension index can be beneficial for many purposes:

• A static site generator can use it
• Salt can have a CLI tool that searches it (pip search doesn't work). salt-pip search? spm?
• An offline docset generator can use it

The index can be organized as a directory with yaml/json files on Github (authors have to submit a PR to get into it). A GH action can periodically refresh these files by pulling fresh metadata from the linked PyPI packages and GH repos. Both extended core and community extensions should be listed here (the ones maintained by the core team can have a special mark).

[max-arnold]

Created the extension metadata repo: https://github.com/max-arnold/salt-extensions-metadata