doxygen + markdown to document C/C++ code

[ScottPJones]

This has already been done in JuliaLang/utf8proc, and can be of great help in understanding the different C/C++ modules used to build the core of Julia.
This has already been discussed in #11107, with the suggestion to open a new issue.

[tkelman]

I think there's a pretty strong need to get better about documenting the internals.

Now, to argue the technical merits of the specific approach we use, if we want to do this - it would be quite helpful (but maybe wishful thinking?) if the same system could be used for the C/C++ parts of the runtime as for the Julia parts of the standard library (@doc has some modularity issues for documenting base as you've noted elsewhere).

[StefanKarpinski]

@ScottPJones, it's good practice to keep titles short since they are clipped in various interfaces, including GitHub's issue listings and email subjects. Additional explanation can go into the issue body.

[ScottPJones]

@StefanKarpinski Thanks again for the helpful advice!

[ScottPJones]

@tkelman I've used doxygen for C/C++ code for about 10 years, and found it very useful. Where I'm currently consulting, we've also standardized on doxygen, and are using it for C, C++, Java, JS, Python, and Julia code (currently, in Julia it's just for human eyes, I think there's work to do to on a filter for doxygen to understand Julia, just like Python used to need...). So I think it definitely could be used for all of the code in Julia, and packages written in Julia... (including Base if the issue of making @doc just save the information for later Markdown processing is solved... for now I just wrapped #= =# around the documentation... There is also even a filter for MatLab ;-)

[StefanKarpinski]

Seems like once the doc string format settles down we could teach doxygen to parse it and then use doxygen for everything.

[garrison]

+1. Having better-documented internals will have countless benefits. In particular, I expect that documenting the ownership of various pointers/handles will improve (or at least help us better assess) memory leaks going forward.

[tkelman]

This came up again in #11573.
I'm very meh on doxygen specifically. If we go that way, it looks like there is a choice between annotating with @ markers and annotating with \ markers (or something else?), and I think the latter makes much more sense for Julia code since otherwise there's too much potential for confusion with Julia macros.

[ScottPJones]

As I said in #11573, we'd probably want to pick another character besides @, that could be used inside doc strings... that could simply be replaced by @ before passing on to doxygen processing.
@tkelman Have you actually used doxygen? It helps a lot when dealing with very large projects, with lots of dependencies, and it handles a lot of different languages...

[tkelman]

As a user of libraries that themselves use doxygen, yes. http://www.coin-or.org/Ipopt/doxygen/namespaceIpopt.html

It's an old-fashioned ugly giant disorganized dump of API listings and class diagrams that aren't particularly useful, from what I've seen. Maybe there are more modern better-looking themes for it, or better ways of structuring its output, but I've never really seen it done in a way that I found useful for learning how to use or how to hack on a library.

[ScottPJones]

I use it for C really, so I don't deal with the class diagrams.
To me, it is very useful because you can very quickly find the information on things like the enumerations or defines used with a library, see how a struct is defined, find internal documentation quickly, see what calls what...
It may not be perfect, but I think it is way better than not having any documentation of the internals...

[IainNZ]

Well I mean, you don't need doxygen to write comments. I've written plenty of code, in plenty of settings, in plenty of languages, and doxygen or doxygen-like things weren't the magic, its was the comments themselves.
I'm in the @tkelman camp (probably due to using the same software!) of finding doxygen-generated external docs usually a pile of not very useful HTML.
I feel like what we really need is a consistent style for documenting internals, that might make it easier to jump around. Surely if you are digging around in the C/C++ code of Julia, you just need comments inline, and there isn't much utility to the external generated docs? I've only been spelunking down there once though, so maybe there is some utility to the doxygen html lump.

[garrison]

I've personally rarely found the actual output of Doxygen to be very useful, but I have found that having Doxygen-styled comments in a code base often leads to an improvement in the overall documentation level of the code. I think we could get most of the benefits by simply having a consistent style for documenting the behavior of each function, as well as (in places) the members of data structures. But the Doxygen syntax is nice in that it is fairly standardized and also allows cross references. Perhaps we can even hope that there will one day be better tools that use the syntax and create more useful output.

[garrison]

Another thing to consider is that LLVM uses Doxygen. I'd be curious to know whether the Julia developers diving deeply into LLVM have found this form of documentation useful. Also, it's an ongoing goal to get more influence in the LLVM community; I'd wonder if using Doxygen to document the C/C++ code might make Julia development more approachable to developers who already work with LLVM.

[yuyichao]

I'd be curious to know whether the Julia developers diving deeply into LLVM have found this form of documentation useful.

I wouldn't call myself "diving deeply into LLVM" but I found the LLVM doxygen doc (for the C++ API at least) hard to read / find out how to do the thing I want. I actually find julia source code a better way to learn LLVM than the doxygen doc.

[jiahao]

Last year, I put together a hacky doxygen setup with a simple Julia to C filter so that Doxygen can learn to read simple method signatures. Personally I think that doxygen is much better as a software archaeology tool than as a documentation tool, for teasing out code interdependencies.

If anyone is interested, here is the Doxyfile

https://github.com/jiahao/julia/blob/cjh/doxygen/contrib/Doxyfile

and the filter is in this directory

https://github.com/jiahao/julia/tree/cjh/doxygen/contrib/jldoxy

The filter doesn't understand polymorphism (trying to map Julia constructs onto equivalent C++ expression templates was just too painful), but the results are surprisingly useful nonetheless.

[Keno]

The generated doxygen docs are terrible for actually understanding what's going on. The only useful thing they do is that they're googlable and have links to the source files.

[yuyichao]

The generated doxygen docs are terrible for actually understanding what's going on. The only useful thing they do is that they're googlable and have links to the source files.

That's exactly what I used them for.... I think I've seen some good doxygen docs but the one for LLVM is definitely NOT one of them....