nim doc .. compiler crash (regression 0.19.6 => 1.0)

[kaushalmodi]

Today, while running nim doc .. on a nimterop generated file, I saw this crash.

Example

proc foo*(): int =
  ## ```
  ##    *******************************************************************************
  ##      Foo
  ##    *******************************************************************************
  ## ```
  return

Current Output

fatal.nim(49)            sysFatal
Error: unhandled exception: rst.nim(1197, 11) `p.tok[p.idx].kind == tkIndent`  [AssertionDefect]

Expected Output

No crash

Workaround

I can prevent the crash if I alter the nimterop-generated files, by removing the *******.. lines:

proc foo*(): int =
  ## ```
  ##      Foo
  ## ```
  return

Additional Information

• I was generating documentation of some modules that use Nim type definitions generated by nimterop
• This issue is blocking the documentation of my project
• This issue can be seen on both nim 1.2.0 (haven't tried older nim versions) and nim devel (1fc40db)

$ nim -v
Nim Compiler Version 1.2.0 [Linux: amd64]
Compiled at 2020-04-03
Copyright (c) 2006-2020 by Andreas Rumpf

active boot switches: -d:release

FYI @genotrance @jyapayne .. above dummy code is a minimal example from a real project where I used nimterop v0.5.7.

[timotheecour]

fails in 1.0.0 with same error, succeeds in 19.6; but the triple backticks are rendered weirdly:

[timotheecour]

So the regression happened between 19.6 and 1.0.

Also, the problem disappears with #14439 but for the following reason: it ignores the doc comment because it appears after discard (but the underlying issue remains if you remove discard or move it below doc comment)

The question is whether doc comments after proc body should be docgen'd:

proc fn*() = discard
  ## foo
proc fn2*(): int = 12
  ## foo

I actually think it indeed should but that it wasn't supposed to work, even though apparently it does

[kaushalmodi]

Also, the problem disappears with #14439 but for the following reason: it ignores the doc comment because it appears after discard (but the underlying issue remains if you remove discard or move it below doc comment)

Sorry, I just fixed the minimal example to reflect a valid doc string case. The crash can still be reproduced on nim 1.2.0, nim devel and also with #14439.

The question is whether doc comments after proc body should be docgen'd:

Nah, I wouldn't worry about that.. that's a different rabbit hole :)

[timotheecour]

IMO nim doc should not error for unsupported or invalid doc comments, but instead use best effort and output a warning (with sufficient debugging context) when it can't handle a doc comment, and then move on to next doc comment. There could be many more such instances.
Unlike for the compiler, the recovery state after an error is sane because each doc comment should be independent.

Best effort is TBD but simplest is to stop on 1st invalid character, replace that doc comment with a string (eg: nimdoc:unrecognized:foo.nim:12:24) and continue with other doc comments