Systematize builtins.fetchTree docs

[Ericson2314]

Motivation

Render the docs nicely.

Context

I would like to use a markdown AST for this, but to avoid new deps (lowdown's AST doesn't suffice) I am just doing crude string manipulations for now.

Priorities and Process

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[roberth]

This seems more productive than what I've been up to with a self-documenting codec abstraction, which may be too ambitious for something in C++.

[Ericson2314]

OK @roberth @fricklerhandwerk @infinisil the infra here is now set up! Not sure whether we want to

• Just merge it
• Hide not-yet-documented fields
• Go through and document everything

[fricklerhandwerk]

I vote to merge as is, and agree it's a useful step forward.

But I sigh loudly as well because it's adding notable tech debt. Now we have 5 (!) mechanisms for producing documentation:

1. plain markdown via mdBook
2. globals.hh, the Config object, and --dump-cli
3. pre-processing in Makefiles
4. processing in Nix language
5. this

Especially the additional document composition code path makes it substantially more messy.

The ideal end state for me would be documentation living exclusively in C++ headers (with plain markdown files for optional free-form blurbs) and processing structured output – templating and such – happening exclusively in the Nix language, decoupled from building the all of Nix. Currently it's hacks on top of hacks and it takes forever to compile.

[Ericson2314]

I agree there are too many. I would like to replace the stuff we do with the interpreter inside Nix with this method, because then it will work with the store-only Nix.

[fricklerhandwerk]

replace the stuff we do with the interpreter inside Nix with this method

That would make it a lot harder to contribute to for most people, and I don't see why docs can't be built by a full Nix as a separate step, but let's discuss it somewhere else.

[thufschmitt]

Discussed during the maintainers meeting today. Deferred for the time being

Details

• Adds an extra dep (cmark)
  • Having another markdown renderer in the closure is meh
  • @Ericson2314: Required because manipulating the lowdown AST doesn't work
  • @thufschmitt Embedding the Markdown directly probably works and is 10% of the work, although it's a bit less nice in theory
    • We're already doing it in a bunch of places
• @Ericson2314: Would be happy to switch to cmark everywhere, but doesn't support outputing to the terminal
  • Commented on Add support for ANSI output format commonmark/cmark#362 (comment) to offer implementing it

Decision: Defer until commonmark/cmark#362 (comment) gets an answer. The the plan is to

• If the answer is positive, @Ericson2314 will implement the cmark terminal renderer
• Otherwise just concatenate markdown snippets

[tribals]

Our use case includes having a self-contained nix executable directly output some pretty-pretty documentation on the terminal without relying on other programs.

Personally I want to state that such style of documentation for CLI tools is worst-case - it is neither man nor help, you're forcing user to pipe output to pager and leaving out man convetions (no one interested in following them because "help files" is written in Markdown, and typical markdown user is rarely even avare of existence of manual pages, and their conventions).

Not to mention that you're forcing user to execute your command in order to get help (rather than "viewing static files" with man).

(From here: commonmark/cmark#362).

[Ericson2314]

OK. With the aid of #14545 (which we should land first), I hand-diffed the :doc builtins.fetchTree output to make sure that no doc changes in the past 2 years were lost in the rebase. This should be good to go!

[dpulls]

🎉 All dependencies have been resolved !

[Ericson2314]

OK @edolstra and @xokdvium it just concats strings now. I will do the rest separately as requested.

[edolstra]

Maybe we can drop stripIndentation() here and have the caller do that?

[Ericson2314]

I would rather not do that because we have both the doc builder caller, and the main.cc caller that dumps the machine-readable JSON info.