The Doc Knight Rises

[MikeInnes]

This is a first (WIP) pass at converting as many of the docstrings as possible to Markdown. There's also an RST render backend for the Markdown parser so that we can keep the RST docs as they are while having docstrings in Markdown.

Once I've sorted out some code-block related issues I'll push the manual regeneration step so that people can check for information loss.

[tkelman]

Could this wrap long lines in the output? Generally a lot nicer to edit that way.

[MikeInnes]

Yeah that would be better. Will look for a pandoc option to do that.

[shashi]

Oops sorry about the assigning myself and then unassigning, stupid vimium tricked me.

Also, @one-more-minute is the hero we deserve need!

[hayd]

There's still some (256) ````rst` in the docstrings. Is the objective to convert these too in this PR?

This is great work!!

[MikeInnes]

@shashi Or at least, the one you need right now ;)

Thanks @hayd. Not all docstrings can be converted because they contain cross references (:func: etc.) or .. doctest:: directives which can't be converted by pandoc. Once cross refs are working in Markdown we can figure out a way to do the last conversions.

[tkelman]

Do we have doctest capabilities in the base docsystem, or standard markdown syntax for them? I see Lexicon has something for that? I think it would be fine to have the doctest-running eventually dependent on a Julia package as a replacement for Sphinx and the JuliaDoc shim.

[StefanKarpinski]

Is the huge number of deletions in helpdb.jl expected?

[tkelman]

I think a lot of that is line count differences due to markdown not breaking long lines where RST was. You can do git diff 04dc8061ad10ed36f33659537877d0c18fe20847~ 04dc8061ad10ed36f33659537877d0c18fe20847 to look at this locally.

[StefanKarpinski]

@one-more-minute, what's going on this?

[ViralBShah]

@one-more-minute This is one of the things that is preventing the 0.4 release. Would it be useful for some of us to get on a hangout and figure out how to get things done?

[MikeInnes]

This is essentially already there, I just need to regenerate the manual from it and make sure nothing's going wrong in the round trip. The main remaining issue is the text-wrapping thing – I played around with pandoc to get that working but no permutation of the --columns=80 command seemed to work. It may be to do with working over stdout and I'll see if I can work around that, otherwise if anyone has any ideas let me know.

[MikeInnes]

I polished this up and regenerated the manual from the new md docstrings. I also fixed the code block syntax issues, and added a warning for autogenerated docs, which shouldn't show up in the rendered manual. Once this is in I'll put up examples for how to move methods to the right place.

[MichaelHatherly]

@one-more-minute this diff is what I get when running genstdlib.jl and after make. Should there be that much of a change between builds? It looks kind of as though there's some different line endings in places.

[MikeInnes]

The whitespace thing is annoying. I'll run the conversion on a unix soon and see if that helps.

I'll also avoid auto-converting things that pandoc isn't recognising correctly; thanks for looking over the diff.

[MichaelHatherly]

I'll run the conversion on a unix soon and see if that helps.

Perhaps having genstdlib.jl run just once per day separately or something like that would avoid that kind of problem. Then contributors wouldn't have to worry about running it themselves. That would mean the docs wouldn't be completely up-to-date all the time though.

[MikeInnes]

Another day, another rebase. How is this looking now?

[ViralBShah]

Why are all the builds failing?

[MikeInnes]

Pandoc seems to introduce some whitespace issues – fixed.

[ViralBShah]

@tkelman What next here? Should we rebase and get it ready?

[StefanKarpinski]

Let's merge this. The remaining issues are not going to get fixed until this gets merged.

[KristofferC]

Reflecting my own view here but I am not sure what the remaining issues are and how I can help. Can I start making PRs moving documentation to method definitions now? Tell me what to do hehe.

[MikeInnes]

Rebased.

@KristofferC Working on that now ;)

[MichaelHatherly]

@one-more-minute, thanks for all the effort you've put into this one!

[MikeInnes]

@MichaelHatherly Very close now!

[timholy]

Wow, this will be nice!

[tkelman]

ifft (unless inline latex in markdown works - does it?) and @sprintf look like they need to be fixed up manually, and some mismatched double-backticks confused things in readdlm.

doc/convertmd.jl was intended to be temporary and probably won't be needed again, right?

[ViralBShah]

Hurray!

[ihnorton]

What's the deal with the thousands of lines of code-block warnings? They make CI output nearly unreadable.