Documentation System

[MikeInnes]

A metadata system for global objects + pretty markdown parsing and display (see the readme). For example:

@doc """
... flowing rivers of technical poetry ...
""" ->
function foo() ...

Documenting multiple methods of a function will concatenate the doc strings together, in the order the methods were originally defined, and this is overloadable.

By default @doc treats unadorned strings as equivalent to md"" (i.e. markdown syntax) but you can actually put anything before the arrow to associate it with the function.

Docs can be queried with e.g.

doc(Docs.doc)
@doc Docs.doc
@doc @doc # For macros
@doc text""

(and you can try these out on this branch – the repl help key ? will work too)

General Todos:

• Prettier syntax ("..." -> @doc doc"..." ->)
• Add Docs.@init call to module init code
• REPL ? integration
• Make sure exports are present and correct
• Move away from the current help system and deprecate it
• Fix the segfault when Text docs are uncommented (!?)

[ViralBShah]

It seems to me that not all the TODOs need to be done before we merge this. It would be great if we can have something that exists along with the existing help for a bit, while we transition and fix everything and deprecate the existing system.

[MikeInnes]

I agree completely – the TODOs are more general things that would be nice to have eventually, but sure, this is basically usable as is.

[mauro3]

I found that it was quite easy to just parse the old helpdb.jl file, put its contents into the new help storage and have the new help-system displaying 99% of what there was before:
https://github.com/mauro3/julia/blob/metadata/base/help.jl#L89

[ViralBShah]

I suggest merging this and then taking it from there.

[MikeInnes]

+1

[johnmyleswhite]

Merging something (or the threat of merging something) does seem like the best way to make progress.

[StefanKarpinski]

I'm a little concerned about packages starting to use this in a way that doesn't degrade gracefully, but I can get over that fear for the sake of making forward progress. Go for it.

[ViralBShah]

This is exactly what we want to sort out in the middle of the dev cycle. @one-more-minute Please merge when you are ready.

We may want to start using in a few packages and then improve from there.

[ViralBShah]

Perhaps squash some of the commits too?

[timholy]

I would say squashing isn't necessary for merging this. (If there were intermediate breakages I might think differently, but that's a lot of green checkmarks above.)

[MikeInnes]

Awesome, can't wait to see what you come up with

[StefanKarpinski]

This is very exciting. Thank you all for pushing this through.

[milktrader]

I guess now is a good time to bring up the matter of standardizing headers. For comparison, R has the following ...

Description 
...
Usage
...
Arguments
...
Details
...
References
...
See Also
...

[stevengj]

@milktrader, see #8966

[MikeInnes]

@StefanKarpinski Glad to help!

[MikeInnes]

I added some really basic documentation examples in basedocs.jl – hope that's OK. It's not meant to be final in any way so much as to get something going, and give people something to try out in the repl (e.g. you can do ?fft, @time, ?r"" etc. now) to get a feel for it.

[mauro3]

I'm using it over in https://github.com/mauro3/Traits.jl, works a treat!

[MikeInnes]

That's great to see 👍

[timholy]

One of the really sweet things @MichaelHatherly added to Lexicon was the ability to dispatch on help. I confess that due to other commitments I haven't followed this effort at all; is that functionality supported?

[MikeInnes]

I actually just added this (3c2e2c4):

julia> @doc "foo1" foo() = 1
foo (generic function with 1 method)

julia> @doc "foo2" foo(x) = 2
foo (generic function with 2 methods)

help?> foo()
  foo1

help?> foo(2)
  foo2

[MikeInnes]

Ok, I see there's also the idea of having a searching version of @which, which is pretty neat. How well does it work? I'd definitely be interested in integrating something like that.

[timholy]

I haven't tested it extensively (I'm only using Docile/Lexicon in a subset of my private code tree, basically the stuff that was developed after these packages became available). But everything I've tried so far suggests it's the cat's meow.

Specifically, I was wondering about this for ? or @doc, not for @which (although implementing for @which would be awesome too).

[MikeInnes]

I only mention @which because it seems like a good idea to implement a general @whiches (or whatever) which returns the array of methods which fuzzy-match – then adding support to ? in the repl should be pretty easy.

That's only an implementation detail, though – I'm definitely on board with fuzzy searching for docs in general. @MichaelHatherly So you have this to working in Docile/Lexicon already?

[MichaelHatherly]

The query(::Function, (DataType...)) method in Lexicon does left-to-right partial matching, since I found doing an unordered ⊆ match on the type signatures typically gave more results than was helpful. The @query macro does exact method matching in the same way as @which.

Having a more intelligent fuzzy-matcher could be quite nice. Not sure what it would look like though.

[ViralBShah]

Pushing forward on completing the transition - it seems that the next step is to transition help to the new documentation system.

Does this mean that our standard library documentation should be translated to Markdown first to leverage the new system best? This may not work best with readthedocs - does it have to use rst?

[MikeInnes]

I think readthedocs requires RST, yes. That said, we already have our own docs template and website, so I'm not sure readthedocs really buys us that much – @MichaelHatherly has done a lot of work on generating great documentation pages and might be interested in helping to move away from it entirely.

OTOH, it wouldn't be hard to generate RST manual pages from the inline docs using Markdown.jl, so that's another option.

The important thing is to have some way to share content between Base docs and the manual – otherwise there'll be a lot of repeated work.

[MichaelHatherly]

There's also http://www.mkdocs.org/, which @tshort has been using recently (https://tshort.github.io/Sims.jl/) in conjunction with Docile.

[ViralBShah]

I was thinking the same about readthedocs. The manual and Base docs should just be equally accessible, but they don't share much of anything. They can be done in different ways without impacting anything, I feel.

mkdocs looks really cool!

[tshort]

Embedded Docile/Base 0.4 docs can be used with Mkdocs and published on either readthedocs or GitHub Pages. It's really easy.

Github Pages works great with Mkdocs. I haven't tried it, but one advantage of readthedocs is it provides online searching.