Sphinx 4.0 Support (including Mathjax 3)

[chrisjsewell]

Sphinx 4.0 is now out 🎉 and I'm sure there are some that are anxious to upgrade their documentation to use it.

We also certainly want to allow this.
However, there is currently one known blocker for trivially changing the dependency pinning in setup.cfg: sphinx-doc/sphinx#7425 / sphinx-doc/sphinx#9094 (Mathjax 3 support) and its interplay with the "dollarmath"/"amsmath" extension.

In https://myst-parser.readthedocs.io/en/latest/using/syntax.html#mathjax-and-math-parsing, I discuss why we want to override default mathjax configuration;
in short because we do not want Mathjax to act as a parser (searching for math, since the Markdown parser already does this), only a renderer (rendering already identified math).

For example, if you set myst_update_mathjax = False, and did not have "dollarmath" nor "amsmath" in myst_enable_extensions, then all of these would still be rendered as math (only with HTML builders but not with other builders like LaTeX)

```{math}
a = 1
```

$$b=2$$

\begin{equation}c=3\end{equation}

or, another example, if you did have "dollarmath" enabled, but myst_update_mathjax = False, then escaped dollar signs would still be rendered as math (again only for HTML builders):

\$\$b=2$$

The way we stop this occuring is to override some of the mathjax extension logic in myst_parser/mathjax.py. But this needs updating for Mathjax 3, plus I also want to revisit the logic and check that this is the best approach.

I should be able to do this by next week though 👍

Cheers,
Chris

[chrisjsewell]

See also #362, executablebooks/meta#306 (comment), jupyter-book/jupyter-book#963

[chrisjsewell]

cc also @mgeier, in relation to: spatialaudio/nbsphinx#560 (thanks also for the work on mathjax3_config 👍)

[mgeier]

In my case (nbsphinx) it is quite annoying that I have to tell users to use myst_update_mathjax = False, it would be great if this could be avoided.

I've tried to solve the problem in the past by only parsing "math" elements: sphinx-doc/sphinx#5504.

However now with both MathJax2 and MathJax3 being around this got a little more complicated.

And BTW, please never use double dollars ($$) and don't recommend using them!

[chrisjsewell]

And BTW, please never use double dollars ($$) and don't recommend using them!

Why?

[mgeier]

https://tex.stackexchange.com/questions/503/why-is-preferable-to

For me, the main reason is that there should be only one obvious way to do block equations, and that's math environments.

I've seen several reports of confused users who used math environments nested within $$, which doesn't make sense in LaTeX. I think it would be best just to simply never ever use $$. Sadly, there are a lot of people who keep using them ... among them are you MyST people (https://myst-parser.readthedocs.io/en/latest/using/syntax.html#dollar-delimited-math), which is very unfortunate.

Probably related: #380

[chrisjsewell]

For me, the main reason is that there should be only one obvious way to do block equations, and that's math environments.

But then (a) you can't use them inline, and (b) a lot of people already use $$ in Jupyter Notebook, not least their actual creator 😬: jupyter-book/jupyter-book#1181

[chrisjsewell]

(but I do understand your viewpoint, and will have a think about it)

[mgeier]

Can you please clarify (a)?

And yes, (b) is unfortunate, but at least Fernando shows signs of changing to proper environments in jupyter-book/jupyter-book#1181 (comment).

[choldgraf]

My intuition is that if we deviate from $$ we are going to face a massive uphill battle, because almost every single Jupyter tutorial that I've seen uses $$ rather than the proper "math environments" syntax. At this point I feel like $$ syntax is a de-facto standard for math blocks. (not saying anything about whether it should be the case, just saying that this is effectively what people have standardized on)

[mgeier]

I personally will never use double dollars, and I won't recommend it in any material I'm writing.
I don't have to energy to try to widely convince people to do the same, but once in a while, like in this here issue discussion, I'm mentioning it in the hope to convince at least a few people to join me on the rightful path.

It would be great if MyST-Parser would support math environments out-of-the-box and never use double dollars, but if it keeps doing it, the world will not end because of it.

[cooperlees]

Hi all - What's the expected time frame on removing the version pinning? What are the main blockers to support sphinx >= 4.0.0? It seems only $$ right?

What are our options here? As someone who does not use $$ can we offer a pre version maybe that's unpinned while this is worked out for us bleeding edge people to use?

[chrisjsewell]

us bleeding edge people to use

You damn bleeding edge people 😆 Right I'll get this done on Monday 👍
while I have some attention, I would ask anyone interested to go comment on: sphinx-doc/sphinx#9040, because my PR there has been waiting to be merged for months, so its a similar frustration lol

[RuRo]

@chrisjsewell any updates?