[CI, performance] .. code-block:: :test: snippets should combine in a single compilation & run

[timotheecour]

This commit 5fc78d8 addressed #9300 by making N runnableExamples run N x faster, which resulted in a major speedup, in particular during CI or local testing.

We should apply the same trick for .. code-block:: :test: snippets, which are still being run independently.

Example

nim rst2html $nim/doc/manual.rst
nim doc src/std_vector.nim

Current Output

nim rst2html $nim/doc/manual.rst
~ takes 40 seconds

git clone https://github.com/kaushalmodi/std_vector && cd std_vector # as of ffa9155f24e01b65a267fcd7d6d382a1f7d0a527
nim doc src/std_vector.nim
~ takes 20 seconds

Expected Output

takes ~2 seconds

Possible Solution

• apply the same idea as 5fc78d8 for test snippets (but look at latest code; eg also needs followups like make runnableExamples use a private nimcache so that they can be tested in parallel #8281)
• likewise for .. code-block:: blocks that are not test:, we should be able to combine them in a single file that gets compiled (but not run)

Additional Information

• devel a39123c

[kaushalmodi]

I agree with this observation. After this commit in my std_vector repo, where I convert all :test: code blocks to runnableExamples, time to generate the docs went down from about 30 seconds to about 7 seconds.

[timotheecour]

also, I really want to know what are valid use cases left for .. code-block:: test: when we now have runnableExamples ?
in particular, runnableExamples now also works at module scope even if you have multiple interleaved doc comments + runnableExamples. Should we simply convert all .. code-block:: test: to runableExamples or am I missing something?

note: despite common misconception, runnableExamples can be used without a body, simply by adding = (a gotcha which we could relax in compiler code)

[Araq]

How can you use runnableExamples inside an .rst file?

[timotheecour]

original post should apply to both nim and rst

also, I really want to know what are valid use cases left for

that comment should only apply to nim files, not rst, (eg like before kaushalmodi/std_vector@b1502d5) or even .. code-block blocks without test (which, if only meant to be compiled and not run, can still use when false or if false or maybe be handled via runnableExamples(doccmd = "compileOnly"))

[maxgrenderjones]

also, I really want to know what are valid use cases left for .. code-block:: test: when we now have runnableExamples ?

This might not be quite the answer you were looking for, but I ran away from the runableExamples implementation of 1.2.0 when I discovered that it reformatted my code, with no obvious way to stop it (it also hid comments that I put there to force linebreaks to be where I wanted them).

I'm now mostly happy with a switch to .. code-block (which also work in my README.rst, only it would appear that bitbucket's .rst formatter doesn't like blocks with :test: in them and they end up hidden 😢

[timotheecour]

when I discovered that it reformatted my code

• that's tracked here runnableExamples should preserve source code doc comments, strings, and (maybe) formatting #8871

the other blockers are: #9227 and #14174; hopefully all can be fixed soon

[maxgrenderjones]

Thanks! On reflection, I think I buried the lead. In the README of my project, I’d like to provide some guidance on how to use it, and I’d like those examples to be known good via testing. I think the best bet for getting these to display right is ..code-block but there are problems with the syntax (see #14352)

[timotheecour]

well i just fixed #14174 so we're down to 2 more blockers;

/cc @Araq
The more I think about it the more I believe we can just completely get rid of rst files altogether and write all the docs in pure nim code; it's not as crazy as it seems, and in fact it just works:

mv doc/manual.rst doc/manual.nim
enclose all content inside a single ##[ ... ]##
nim doc doc/manual.nim

that's it, it works; the generated html looks just as good.

• no need for a separate rst2html command
• we can use arbitrary nim code & macros inside the manual.nim if there's a need to auto-generate some doc strings (eg date, urls, href, index logic etc); no need to rely on sub-par RST specific include syntax (that doesn't even work reliably, see code-block:: nim :test: syntax causes rst2html.py to choke #14352 (comment)) or any of that (that, IMO, is a big plus and I have concrete use cases in mind)
• writing tables become as easy as writing a reusable macro that converts nim AST to RST table so users can use nim instead of RST to write tables in docs eg:

table: # will generate RST table
  0: "country", "population"
  1: "france", "70M"

• we can embrace the fact that nim's RST parser handles markdown too
• we can replace all the ugly rst test snippets with runnableExamples that syntax highlight correctly
• this issue and other RST issues could just be closed as wontfix
• the only counter argument I can think of (that the un-rendered rst docs would render better on github than a big comment file) isn't even valid: our biggest rst file, https://github.com/nim-lang/Nim/blob/devel/doc/manual.rst doesn't render correctly on github (it should look like https://github.com/nim-lang/Nim/blob/devel/doc/contributing.rst)

[Araq]

No... I mean, no. It's a brilliant hack but I don't want to have my spec inside a .nim file.