doc pipeline not working with sphinx 2.1.2

[raphaeldussin]

not a pressing issue, but I'd like to put it out there. Current extensions to doxygen are not working with more recent sphinx versions so I couldn't update the rtd yml file. Currently works with sphinx 1.5.

[jr3cermak]

I am taking a stab at this. It would be helpful to have a working doc pipeline, if possible. So far, my efforts to get a doc pipeline working with sphinx 1.5 has not worked out. If you have a working pipeline, it would be useful to have the version numbers to most of the relevant packages/modules: python, lxml, doxygen, sphinx, latex, bibtex and perl. Angus's last update to supporting modules was back in 2017. I suppose I could try a OS(vm) from 2017 to see if I can get appropriate versions of stuff to line up?

My initial goal is to get the doc pipeline working in Ubuntu 18.04.4 LTS and Ubuntu 20.04 LTS.

[jr3cermak]

Just ran across something in dev. Does #979 replace this?

[marshallward]

Hi Rob, I took a deep dive into this sometime last year, I will try to remember what I learned to help you out. Hopefully @angus-g will correct any details that I get wrong.

The main problem is that we rely on two sphinx extensions which are no longer compatible with our pipeline:

• sphinxcontrib.autodoc_doxygen: This converts the Doxygen XML into readable reST docs (stored as temporary files, I think) which Sphinx can understand. It is somewhat language-agnostic though, and requires the other extension below.
  https://github.com/rmcgibbo/sphinxcontrib-autodoc_doxygen

• sphinxfortran.fortran_domain: This adds additional structures which allow for the parsing of Fortran source.
  https://github.com/VACUMM/sphinx-fortran

Angus forked these two projects some time ago and made significant changes to them in order to get the MOM6 source to look how it does today. Since this work was completed, Sphinx has changed quite a bit and is no longer compatible with Angus' forks. And while the extensions have mostly stayed up-to-date with the latest Sphinx, it's no longer possible to simply merge Angus' changes into these updated extensions.

I was able to do a very crude merge of his changes into the updated versions, but only got far enough to get a partial documentation. I don't think to finish this would be an impossible job, but it will be time consuming.

I think there are three paths forward:

• Keep using an outdated Sphinx (a bad idea, but the least amount of work)
• Update these proejcts and integrate the changes
• Switch to a new documentation tool (e.g. WIP: using doxyrest for documentation #979).

Raf looked into Breathe and Doxyrest, and I looked a bit into FORD, but all had their issues and limitations. Doxyrest in particular would take a significant amount of work, since it currently cannot parse Fortran.

Stepping back a bit, Doxygen is actually somewhat limited in its ability to parse Fortran. I can't remember the details, but I do recall it makes a fair number of mistakes, misinterprets many terms, and even completely ignores some structs. (I know that's a big claim without evidence, so just take it as anecdotal for now, but I'll try to back it up later).

As a wildcard solution, I have been trying to update my flint Fortran parser (https://github.com/marshallward/flint/) to produce reST files which could be fed to Sphinx. This would fully parse the Fortran and would not rely on Doxygen. I have made a lot of progress, but this will always be a moonshot and still needs a lot of work.

If I were being asked to solve this in the quickest way, I'd probably try to update those two extensions and make them work in Sphinx 2. It would not be trivial but it ought to be doable.

[jr3cermak]

Thank you for this information. I found a combination that is operational for html and reproduces what is out on RTD sites presently. I can attempt to move forward at this point.

I like your suggestion to update the current extensions (if possible) as an interim step to get us through the current documentation task. We can circle back on Breathe/Doxyrest and/or flint after.

For those in need of using the old pipeline, this is what worked for me. I grabbed a server version of Ubuntu 16.04.6 LTS. Ran apt-get upgrade.

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.6 LTS
Release:	16.04
Codename:	xenial
cermak@ubuntu:~/MOM6/docs$ python3 --version
Python 3.5.2
$ doxygen --version
1.8.11
$ sphinx-build --version
Sphinx (sphinx-build) 1.5

$ sudo apt-get install python3-pip
$ sudo apt-get install doxygen
$ sudo apt-get install texlive-binaries texlive-latex-base texlive-latex-recommended texlive-latex-extra
$ git clone <MOM6 doc repo>
$ cd MOM6/docs
$ pip3 install -r requirements.txt
$ make html

$ pip3 list
alabaster (0.7.12)
Babel (2.8.0)
certifi (2020.6.20)
chardet (3.0.4)
command-not-found (0.3)
docutils (0.16)
idna (2.10)
imagesize (1.2.0)
Jinja2 (2.11.2)
language-selector (0.1)
lxml (4.5.2)
MarkupSafe (1.1.1)
mock (4.0.2)
pip (8.1.1)
pycurl (7.43.0)
Pygments (2.6.1)
pygobject (3.20.0)
python-apt (1.1.0b1+ubuntu0.16.4.9)
python-debian (0.1.27)
python-systemd (231)
pytz (2020.1)
requests (2.24.0)
setuptools (20.7.0)
six (1.15.0)
snowballstemmer (2.0.0)
Sphinx (1.5)
sphinx-fortran (1.0.1)
sphinxcontrib-autodoc-doxygen (0.3.3.dev49)
ssh-import-id (5.5)
ufw (0.35)
unattended-upgrades (0.1)
urllib3 (1.25.10)
wheel (0.29.0)

The trick is to go WAY back to an ancient doxygen. You have to go back to doxygen version 1.8.11. We went as far back as 1.8.12. With 1.8.11, I can use a modern OS with sphinx 1.5.

[jr3cermak]

@raphaeldussin: I took a look at doxyrest via https://github.com/raphaeldussin/MOM6/tree/doxyrest_doc and was only able to get a very partial html and pdf rendering. See: https://recon.lccllc.info/MOM6/doxyrest/_build

@marshallward: I worked a bit with flint. I can get it to parse all the MOM6 code. Would the next step to build a sphinx extension around flint and the parsed structures? We would still need doxygen to pull forward all the references and latex/bib/citation materials? (see below)

I have re-enabled doxygen output of html and pdf. It is not the RTD style, but at least references and links to citations work as intended. It is not the RTD style, but it gives us something to look at.
An example can be found https://recon.lccllc.info/MOM6/esmg/_build/doxygen_html.

For both the examples above, I have renamed the *.log files to *_log.txt files so they can be viewed through the web browser instead of triggering a download. e.g. latex log: https://recon.lccllc.info/MOM6/esmg/_build/doxygen_latex/latex_log.txt

The disturbing issue is there are too many errors (>100) to produce a PDF.

(That makes 100 errors; please try again.)
!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on refman.log.
make: *** [Makefile:8: refman.pdf] Error 1

It will take some time to discover how to fix this. I may need to start with a smaller set of code/material to get started and see where the errors are being generated. (A lot of the formula errors seem to stem from trying to support \eqref).

My proposal at this point is complete reboot with the latest sphinx and doxygen and bring other tools into the pipleline as they become working. In the end, we should have two or more versions of html and pdf in at least doxygen and eventually RTD styles.

If you find some workable pieces, @raphaeldussin or @marshallward, please let me know. I have rediscovered the usefulness of python virtual environments so I can easily switch between all the various pipelines.

---

Back to flint...

The flint tokenized data structures seem to be very simple. [ [line1tok1, line1tok2], [line2], ...]

(Pdb) p proj.files[2].tokenize()
/src/user/dumbbell_surface_forcing.F90 (/user/dumbbell_surface_forcing.F90)
[['!> Surface forcing for the dumbbell test case'], ['module', 'dumbbell_surface_forcing'], ['use', 'mom_diag_mediator', ',', 'only', ':', 'post_data', ',', 'query_averaging_enabled'], 

I know the fortran compiler is case insensitive. The flint tokenization seems to push everything to lowercase. Should case be preserved for documentation? I would find it odd that the documentation is in one case and the actual code is in another. (I think doxygen and/or sphinx do this as well...)

The one issue that flint could fix is doxygen's need for subroutine arguments to appear on separate lines.

Doxygen complains about:

 !subroutine subStats(HI, array, mesg, sym_stats)
  subroutine subStats(HI, array, sym_stats, aMean, aMin, aMax)
    type(hor_index_type), intent(in) ::  HI     !< A horizontal index type
    real, dimension(HI%isd:,HI%JsdB:,:), intent(in) :: array !< The array to be checksummed
    logical,          intent(in) :: sym_stats !< If true, evaluate the statistics on the
                                              !! full symmetric computational domain.
    real, intent(out) :: aMean, aMin, aMax    !< Mean/min/max of array over domain

aMean, aMin, aMax need to be on separate lines to be properly documented.

There are not too many of these that could be unwound. Should this be done for now?

change:

    real, intent(out) :: aMean, aMin, aMax    !< Mean/min/max of array over domain

to:

    real, intent(out) :: aMean   !< Mean of array over domain
    real, intent(out) :: aMin    !< Minimum of array over domain
    real, intent(out) :: aMax    !< Maximum of array over domain

[marshallward]

I'll try to pull together my modified Sphinx plugins and see how far I got with them. I got past the errors, but the final result was incomplete. (Hopefully they are sitting around somewhere).

The flint output that you're looking at is the final tokenized source code, but there are intermediate stages where things like whitespace and case are handled. For example, this is where the docstrings are stored before they are stripped. This would be the place to record the original, case-sensitive input. This is not yet done but I believe it can be added. Let's move this discussion to the flint repo, though. (I've opened this issue: marshallward/flint#5)

On your third point, I believe Doxygen has a syntax for bundling multiple variables under a single docstring (!<@ ... !>@). But I would also be supportive (perhaps even enthusiastically) of unwinding these to one variable per line.

[jr3cermak]

Making slow progress on the reboot using sphinx-3.2.1. Working on Agnus's first module https://github.com/rmcgibbo/sphinxcontrib-autodoc_doxygen to process the _*.dox pages first and then move onto the fortran files. The sphinx-fortran adds the fortran domain which might be a good time add an option to switch between Angus and flint processing.

Current rendering:

• doxygen: html pdf sphinx: html pdf (https://recon.lccllc.info/MOM6/) and various logs
• RTD (https://mom6devesmgnew.readthedocs.io/en/latest/index.html)

Discoveries:

• xml generation with doxygen 1.8.13 is broken for our purposes; have a method to make RTD use 1.8.19
• embedding images requires \image html and \image latex (duplicate lines at the moment)
• references and citations work with doxygen
  • this includes a limited hack for \footnote material that can be improved
  • \eqref does work but seems to be limited to just the currently viewed/rendered page
  • rework how citations are placed in *.rst files; use :cite:`ref`
• references and citations are unprocessed by sphinxcontrib-autodoc_doxygen
• MathJax: formula autonumbering/references confined to single pages in html
• readthedocs uses an older doxygen
• sphinxcontrib.autodoc_doxygen automatically activates sphinx extensions:
  • sphinx.ext.autodoc
  • sphinx.ext.autosummary

Tasks:

• Look at doxygen 1.8.19/sphinx 1.5 + Angus modules
• Render clean html via doxygen only (1.8.19); \ref, \cite, \eqref, \footnote and call/caller graphs do work
• Render clean pdf via doxygen only (1.8.19); Somewhat possible just processing _*.dox files
• Restart with a clean slate sphinx 3.2.1;doxygen-1.8.19
• Create two different doxygen.conf files to support RTD (doxygen_rtd.conf) and non-RTD (doxygen.conf)
• Install sphinxcontrib.bibtex to support citations
• [defer] Option: doxyrest/sphinx (https://github.com/raphaeldussin/MOM6/tree/doxyrest_doc)
• Option: doxygen/flint/sphinx (https://github.com/marshallward/flint/)
• Option: doxygen/sphinx;rework Angus module from (https://github.com/rmcgibbo/sphinxcontrib-autodoc_doxygen) FORK: (https://github.com/jr3cermak/sphinxcontrib-autodoc_doxygen)
• Option: doxygen/sphinx;rework Angus module from updates from (https://github.com/VACUMM/sphinx-fortran)
• Reapply RTD theme and refresh readthedocs website (https://mom6devesmgnew.readthedocs.io/en/latest/index.html)
• Feature: MathJax formula references (\eqref) linking across pages
• Bug: \footnote{} support (works in doxygen html/pdf but not sphinx/html)
• Bug: orderedlist and listitems are broken + Feature: nesting of items should work
• Bug: fix for duplicate More... labels across pages
• Bug: render embedded html in titles passed through by doxygen/xml
• Created a line in the sand so development can proceed without tripping up documentation effort; see README.md
• Bug: handle fortran functions e.g. function EOS_domain(HI, halo) result(EOSdom) and others
• Bug: sphinx-pdf docstrings do not wrap; works in doxygen-pdf
• Feature: Case sensitive Fortran tokens brought through to html/pdf
• Feature: sphinx support of call/caller graphs (works in doxygen html/pdf; textual references to/from work in sphinx-html/pdf)

[jr3cermak]

Some progress porting pieces of sphinxcontrib-autodoc_doxygen to sphinx-3.2.1.

Citations still break on RTD. They work locally. Very strange.

Local: https://recon.lccllc.info/MOM6/esmgNew/_build/html/index.html
RTD: https://mom6devesmgnew.readthedocs.io/en/latest/index.html

[jr3cermak]

I have managed to pull together code that generates html and pdf using sphinx-3.2.1 and doxygen-1.8.19. RTD's default doxygen is 1.8.13 which produces incorrect XML for our use. It turns out you can feed RTD a binary via git and force it to run a 1.8.19 version. The XML is still not perfect in 1.8.19. There is still some work to do, but the major hurdles I think are out of the way.

If you like to test this before we push it upstream further:

• esmg-docs branch; upstream dev esmg-docs
• RTD html rendering: https://mom6-devesmg.readthedocs.io/en/latest/
• RTD pdf rendering: https://mom6-devesmg.readthedocs.io/_/downloads/en/latest/pdf/

If things look good, we can push this upstream to update the RTD for the ESMG and the official mom6.readthedocs.io.

[jr3cermak]

@marshallward : I just got bitten by libpoppler in a CI run. Did you discover this was a transitory bug? https://github.com/NOAA-GFDL/MOM6/pull/1264#issuecomment-737277996

This is what I just saw with a commit attempting to re-sync my development branches without the errant embedded doxygen binary. It looks like an internal fetching issue and not necessarily a package issue. If it manages to get the package properly from a mirror, it should work. We will likely continue to see errors until the mirrors get ironed out.

Get:48 http://azure.archive.ubuntu.com/ubuntu bionic/universe amd64 texlive-bibtex-extra all 2017.20180305-2 [56.0 MB]
Fetched 108 MB in 2s (61.1 MB/s)
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/p/poppler/libpoppler73_0.62.0-2ubuntu2.10_amd64.deb  404  Not Found [IP: 52.147.219.192 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Error: Process completed with exit code 100.

Things may break again when they upgrade the workflows to 20.04.

Ubuntu-latest workflows will use Ubuntu-20.04 soon. For more details, see https://github.com/actions/virtual-environments/issues/1816

[marshallward]

AFAIK @adcroft fixed it by adding a apt-get update, which seemed to resolve the problem for now. I reckon we just monitor and see how it holds up.