Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sphinx 2.0 compatibility issue? #215

Closed
rsokl opened this issue Apr 28, 2019 · 45 comments · Fixed by #264
Closed

Sphinx 2.0 compatibility issue? #215

rsokl opened this issue Apr 28, 2019 · 45 comments · Fixed by #264
Milestone

Comments

@rsokl
Copy link

rsokl commented Apr 28, 2019

My readthedocs project has started failing to render my docs according to the numpydoc (v0.9.1) layout.


E.g. this is how my docs looked a couple of months ago:

image


And now my docs build to this:

image


It appears that this is concurrent with RTD upgrading its sphinx version from 1.8 to 2.0. Is this possibly the culprit? I am not sure how best to proceed in debugging this, so any guidance would be very much appreciated!

@rgommers
Copy link
Member

does Sphinx 2.0 + numpydoc 0.8.0 work for you, or is that broken as well?

@rgommers
Copy link
Member

also, can you link to the docstring for mygrad.add or paste it here?

@rsokl
Copy link
Author

rsokl commented Apr 28, 2019

I had been using numpydoc 0.8.0 and that version broke. I tried upgrading to 0.9.1, and saw no difference.

Here is the docstring:

def add(a, b, constant=False):
    """ ``f(a, b) -> a + b``

        Parameters
        ----------
        a : array_like

        b : array_like

        constant : bool, optional(default=False)
            If ``True``, the returned tensor is a constant (it
            does not back-propagate a gradient)

        Returns
        -------
        mygrad.Tensor
        
        Examples
        --------
        >>> import mygrad as mg
        >>> mg.add(1.0, 4.0)
        Tensor(5.0)
        >>> x1 = mg.arange(9.0).reshape((3, 3))
        >>> x2 = mg.arange(3.0)
        >>> mg.add(x1, x2)  # equivalent to `x1 + x2`
        Tensor([[  0.,   2.,   4.],
                [  3.,   5.,   7.],
                [  6.,   8.,  10.]])"""
    return Tensor._op(Add, a, b, constant=constant)

@rgommers
Copy link
Member

The indentation is wrong, you need to de-indent everything from Parameters onwards by 4 spaces. That should fix it. Sphinx probably started processing indentation a little differently.

@rsokl
Copy link
Author

rsokl commented Apr 28, 2019

Okay, thank you!

@rsokl rsokl closed this as completed Apr 28, 2019
@rgommers
Copy link
Member

I'll close this issue, because I don't think we can or want to add a workaround for this. If there's something else that's off please keep commenting or open a new issue of course.

@rsokl
Copy link
Author

rsokl commented Apr 28, 2019

Actually, I think indentation may not be the case. This function has the correct indentation level, and yet it still fails to render properly.

Edit: here is another docstring with proper indentation, but that fails to render

@rsokl rsokl reopened this Apr 28, 2019
@rgommers
Copy link
Member

Yes those docstrings look perfectly fine. Looks like a bigger issue.

For now downgrade Sphinx I guess. We'll have to investigate this.

@rgommers rgommers added this to the 1.0.0 milestone Apr 28, 2019
@rsokl
Copy link
Author

rsokl commented Apr 28, 2019

Unfortunately I have to use conda to build my RTD environment, and there is no way to keep RTD from overwriting my version of sphinx during their pip install for now.

Obviously this is an issue with RTD, and not numpydocs. But I might as well make note of it here in case others come here with the same issue.

@rgommers
Copy link
Member

Good to know, thanks. Can't you just conda install pip and then pip install sphinx==1.8.5 --force or something like that?

@rsokl
Copy link
Author

rsokl commented Apr 28, 2019

Unfortunately, no. They made a strange decision when it came to the conda pipeline, and run, rain or shine:

conda install --yes --quiet --name latest mock pillow sphinx sphinx_rtd_theme

After running your .yml script. Fortunately it looks like they are actively working on this.

@prisae
Copy link

prisae commented Apr 29, 2019

I can confirm this, same happens to my projects.

@prisae
Copy link

prisae commented Apr 29, 2019

I run a little test with numpy.linalg.leastsq to see a bit more. It looks like there are two issues, one is a change between numpydoc 0.7.0 and later, the other is with Sphinx 1.85 vs Sphinx 2.0 (I put the Sphinx- and numpy-versions in the copyright string, so they are visible in the screenshots of the docs).


Sphinx 1.8.5 plus numpydoc 0.7.0 looks like this:

Sphinx_185_numpydoc_07


Sphinx 1.8.5 plus numpydoc 0.8.0, 0.9.0, and 0.9.1 look like this:

Sphinx_185_numpydoc_091


Sphinx 2.0.0 plus numydoc 0.7.0, 0.8.0, 0.9.0, and 0.9.1 look like this:

Sphinx_2_numpydoc_091


Edit: numpydocc 0.7.0 looks also different than the others for Sphinx 2.0.0

Sphinx 2.0.0 plus numydoc 0.7.0 looks like this:

Sphinx_2_numpydoc_07


This example is just taken from a random numpy-function, so the problem is definitely a wider one affecting numpy/scipy itself at some point.

@prisae
Copy link

prisae commented Apr 29, 2019

This is the test-function I used, copied from numpy/linalg/linalg.py:

def lstsq(a, b, rcond="warn"):
    """

    From anaconda3/lib/python3.7/site-packages/numpy/linalg

    Parameters
    ----------
    a : (M, N) array_like
        "Coefficient" matrix.
    b : {(M,), (M, K)} array_like
        Ordinate or "dependent variable" values. If `b` is two-dimensional,
        the least-squares solution is calculated for each of the `K` columns
        of `b`.
    rcond : float, optional
        Cut-off ratio for small singular values of `a`.
        For the purposes of rank determination, singular values are treated
        as zero if they are smaller than `rcond` times the largest singular
        value of `a`.

    """
    pass

The whole minimal test-project is here: test.zip, where I built the docs with make singlehtml.

@prisae
Copy link

prisae commented Apr 29, 2019

It might be that the problem is related to the RTD theme. If I use the default Sphinx-theme, then there is no problem.

@larsoner
Copy link
Collaborator

FYI I had to add this to conf.py to get make singlehtml to run:

import os
import sys
sys.path.insert(0, os.path.dirname(__file__))

Agreed that commenting out html_theme = 'sphinx_rtd_theme' seems to fix things. I guess this is just how the sphinx_rtd_theme renders definition lists?

@prisae
Copy link

prisae commented Apr 29, 2019

sys.path.insert(0, os.path.dirname(__file__)) even though testnumpy in the same directory? I did not need this, what could be the reason that it runs on my machine but not on others?

@prisae
Copy link

prisae commented Apr 29, 2019

I tested now sphinx_rtd_theme all the way back to version 0.2.0, but it is always the same. So the change does not seem to come from there, but fromSphinx and/or from numpydoc.

@jorisvandenbossche
Copy link
Contributor

I think this is related to a change in sphinx 2.0 on how they handle field lists in the conversion to html.

I saw this in the pandas docs as well (pandas-dev/pandas#26058 (comment)), and already suspected that numpy/scipy will also encounter this shortly, but didn't have the time yet to further investigate.

@jnothman
Copy link
Member

jnothman commented Apr 29, 2019 via email

@jorisvandenbossche
Copy link
Contributor

As mentionec above by @prisae (#215 (comment)), there seems to be both a change in numpydoc as in sphinx.
For the sphinx change, I am seeing the following. With sphinx 1.8.5, you get something like this (for the Parameters section of the pandas.melt docstring):

<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><dl class="first last docutils">
<dt><strong>frame</strong> <span class="classifier-delimiter">:</span> <span class="classifier">DataFrame</span></dt>
<dd></dd>
<dt><strong>id_vars</strong> <span class="classifier-delimiter">:</span> <span class="classifier">tuple, list, or ndarray, optional</span></dt>
<dd><p class="first last">Column(s) to use as identifier variables.</p>
</dd>
<dt><strong>value_vars</strong> <span class="classifier-delimiter">:</span> <span class="classifier">tuple, list, or ndarray, optional</span></dt>
<dd><p class="first last">Column(s) to unpivot. If not specified, uses all columns that
are not set as <cite>id_vars</cite>.</p>
</dd>
...

while with sphinx 2.0, it is now:

<dl class="field-list simple">
<dt class="field-odd">Parameters</dt>
<dd class="field-odd"><dl class="simple">
<dt><strong>frame</strong><span class="classifier">DataFrame</span></dt><dd></dd>
<dt><strong>id_vars</strong><span class="classifier">tuple, list, or ndarray, optional</span></dt><dd><p>Column(s) to use as identifier variables.</p>
</dd>
<dt><strong>value_vars</strong><span class="classifier">tuple, list, or ndarray, optional</span></dt><dd><p>Column(s) to unpivot. If not specified, uses all columns that
are not set as <cite>id_vars</cite>.</p>
</dd>
<dt><strong>var_name</strong><span class="classifier">scalar</span></dt><dd><p>Name to use for the ‘variable’ column. If None it uses
<code class="docutils literal notranslate"><span class="pre">frame.columns.name</span></code> or ‘variable’.</p>
</dd>
...

So before, there was a html table with class="docutils field-list", and this was styled specifically in our custom css, which broke due to this change in the generated html. And the same happens in the scipy sphinx theme:

https://github.com/scipy/scipy-sphinx-theme/blob/f0d96ae2bf3b010ce53adadde1e38997497a513e/_theme/scipy/static/scipy.css_t#L111-L156

So even regardless of the numpydoc change, this is something that will need to be updated in custom css (or reported to sphinx, in case this was not intentional)

@stefanv
Copy link
Contributor

stefanv commented May 11, 2019

The issue was correctly identified by @jorisvandenbossche in pandas-dev/pandas#26058 (comment). The styling with the new HTML is a bit tricky, because there are no class labels added by Sphinx that help to disambiguate outer from inner cells etc.

@rgommers
Copy link
Member

Are there remaining issues? I really don't want to stay on Sphinx 1.8.5 forever :-/. If we're just talking about a lack of time, I'm happy to do a PR with @dstansby CSS.

It seems like @stefanv was saying it didn't fully fix the issue for scikit-image, however if you could create a PR and check that it fixes the original issue reported here, that would be very helpful.

@QuLogic
Copy link

QuLogic commented Mar 20, 2020

The scikit-image PR from above was eventually merged, so it seems like it's all CSS, and nothing for numydoc to do.

@jnothman
Copy link
Member

Do we need to document these CSS changes and then close this issue and release?

@prisae
Copy link

prisae commented May 20, 2020

Not sure that this should be the solution.

That would mean that each project, which uses numpydoc, has to manually do a fix in a custom css-style-file? And the ones that fail to realize this will have badly rendered docs?

Or did I miss something?

@prisae
Copy link

prisae commented May 20, 2020

(I see, or maybe each theme, not each numpydoc-user)

@jnothman
Copy link
Member

(I see, or maybe each theme, not each numpydoc-user)

Yes, so it would be more of a compatibility note than user docs.

@prisae
Copy link

prisae commented May 20, 2020

Great. I just realized that it was fixed in RTD-theme 15 days ago, readthedocs/sphinx_rtd_theme#838

Just tried (sphinx_rtd_theme=0.15.0rc1, not officially released yet) and works fine.

@larsoner
Copy link
Collaborator

Just tried (sphinx_rtd_theme=0.15.0rc1, not officially released yet) and works fine.

Great! @rsokl feel free to try it. Let's leave this open until ...

Do we need to document these CSS changes and then close this issue and release?

@jnothman I would say so. Are you up for doing the release or would you prefer someone else do it? I could find time next week.

@jnothman
Copy link
Member

jnothman commented May 24, 2020 via email

@rgommers
Copy link
Member

I can add both of you, please let me know your PyPI username

@larsoner
Copy link
Collaborator

Eric89GXL on PyPI

@rgommers
Copy link
Member

added, thanks @larsoner

@rgommers
Copy link
Member

added jnothman as well (checked on scikit-learn page)

@JulianFerry
Copy link

Great. I just realized that it was fixed in RTD-theme 15 days ago, readthedocs/sphinx_rtd_theme#838

Just tried (sphinx_rtd_theme=0.15.0rc1, not officially released yet) and works fine.

Can confirm that this works, thanks!

@jaladh-singhal
Copy link

Just tried (sphinx_rtd_theme=0.15.0rc1, not officially released yet) and works fine.

For anyone looking into it, it's actually 0.5.0.rc1 - to clear the confusion.

And yesterday, they have officially released 0.5.0 - so upgrading with pip or conda works!

@prisae
Copy link

prisae commented Jun 19, 2020

Thanks @jaladh-singhal for correcting my typo! Looks like 0.5.0.rc1 crossed with "15 days ago" in my comment to create a not-existing version...

@rsokl
Copy link
Author

rsokl commented Jun 19, 2020

I am still finding the same abberations in my docs after upgrading sphinx_rtd_theme to 0.5.0

Can anyone spot what I am doing wrong here?

RTD build log

Collecting numpy==1.16.2
  Downloading numpy-1.16.2-cp37-cp37m-manylinux1_x86_64.whl (17.3 MB)
Collecting numba==0.41.0
  Downloading numba-0.41.0-cp37-cp37m-manylinux1_x86_64.whl (3.2 MB)
Collecting llvmlite==0.32.1
  Downloading llvmlite-0.32.1-cp37-cp37m-manylinux1_x86_64.whl (20.2 MB)
Collecting sphinx==3.0.4
  Downloading Sphinx-3.0.4-py3-none-any.whl (2.8 MB)
Collecting numpydoc>=1.0.0
  Downloading numpydoc-1.0.0-py3-none-any.whl (47 kB)
Collecting sphinx-rtd-theme==0.5.0
  Downloading sphinx_rtd_theme-0.5.0-py2.py3-none-any.whl (10.8 MB)
Requirement already satisfied: Jinja2>=2.3 in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from sphinx==3.0.4->-r docs/requirements.txt (line 4)) (2.11.2)
Requirement already satisfied: docutils>=0.12 in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from sphinx==3.0.4->-r docs/requirements.txt (line 4)) (0.14)
Requirement already satisfied: snowballstemmer>=1.1 in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from sphinx==3.0.4->-r docs/requirements.txt (line 4)) (2.0.0)
Collecting sphinxcontrib-qthelp
  Downloading sphinxcontrib_qthelp-1.0.3-py2.py3-none-any.whl (90 kB)
Collecting sphinxcontrib-applehelp
  Downloading sphinxcontrib_applehelp-1.0.2-py2.py3-none-any.whl (121 kB)
Collecting sphinxcontrib-htmlhelp
  Downloading sphinxcontrib_htmlhelp-1.0.3-py2.py3-none-any.whl (96 kB)
Requirement already satisfied: babel>=1.3 in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from sphinx==3.0.4->-r docs/requirements.txt (line 4)) (2.8.0)
Requirement already satisfied: packaging in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from sphinx==3.0.4->-r docs/requirements.txt (line 4)) (20.4)
Collecting sphinxcontrib-devhelp
  Downloading sphinxcontrib_devhelp-1.0.2-py2.py3-none-any.whl (84 kB)
Requirement already satisfied: Pygments>=2.0 in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from sphinx==3.0.4->-r docs/requirements.txt (line 4)) (2.3.1)
Collecting sphinxcontrib-serializinghtml
  Downloading sphinxcontrib_serializinghtml-1.1.4-py2.py3-none-any.whl (89 kB)
Requirement already satisfied: imagesize in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from sphinx==3.0.4->-r docs/requirements.txt (line 4)) (1.2.0)
Collecting sphinxcontrib-jsmath
  Downloading sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl (5.1 kB)
Requirement already satisfied: requests>=2.5.0 in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from sphinx==3.0.4->-r docs/requirements.txt (line 4)) (2.24.0)
Requirement already satisfied: setuptools in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from sphinx==3.0.4->-r docs/requirements.txt (line 4)) (41.0.1)
Requirement already satisfied: alabaster<0.8,>=0.7 in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from sphinx==3.0.4->-r docs/requirements.txt (line 4)) (0.7.12)
Requirement already satisfied: MarkupSafe>=0.23 in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from Jinja2>=2.3->sphinx==3.0.4->-r docs/requirements.txt (line 4)) (1.1.1)
Requirement already satisfied: pytz>=2015.7 in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from babel>=1.3->sphinx==3.0.4->-r docs/requirements.txt (line 4)) (2020.1)
Requirement already satisfied: six in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from packaging->sphinx==3.0.4->-r docs/requirements.txt (line 4)) (1.15.0)
Requirement already satisfied: pyparsing>=2.0.2 in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from packaging->sphinx==3.0.4->-r docs/requirements.txt (line 4)) (2.4.7)
Requirement already satisfied: certifi>=2017.4.17 in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from requests>=2.5.0->sphinx==3.0.4->-r docs/requirements.txt (line 4)) (2020.4.5.2)
Requirement already satisfied: chardet<4,>=3.0.2 in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from requests>=2.5.0->sphinx==3.0.4->-r docs/requirements.txt (line 4)) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from requests>=2.5.0->sphinx==3.0.4->-r docs/requirements.txt (line 4)) (1.25.9)
Requirement already satisfied: idna<3,>=2.5 in /home/docs/checkouts/readthedocs.org/user_builds/mygrad/envs/latest/lib/python3.7/site-packages (from requests>=2.5.0->sphinx==3.0.4->-r docs/requirements.txt (line 4)) (2.9)
Installing collected packages: numpy, llvmlite, numba, sphinxcontrib-qthelp, sphinxcontrib-applehelp, sphinxcontrib-htmlhelp, sphinxcontrib-devhelp, sphinxcontrib-serializinghtml, sphinxcontrib-jsmath, sphinx, numpydoc, sphinx-rtd-theme
  Attempting uninstall: sphinx
    Found existing installation: Sphinx 1.8.5
    Uninstalling Sphinx-1.8.5:
      Successfully uninstalled Sphinx-1.8.5
  Attempting uninstall: sphinx-rtd-theme
    Found existing installation: sphinx-rtd-theme 0.4.3
    Uninstalling sphinx-rtd-theme-0.4.3:
      Successfully uninstalled sphinx-rtd-theme-0.4.3
Successfully installed llvmlite-0.32.1 numba-0.41.0 numpy-1.16.2 numpydoc-1.0.0 sphinx-3.0.4 sphinx-rtd-theme-0.5.0 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-1.0.3 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.4

@prisae
Copy link

prisae commented Jun 19, 2020

Your docs look good to me. What output would you expect?

Before the fix, it would have look like

constantbool, optional(default=False)

Now it looks like:

constant : bool, optional(default=False)

@rsokl
Copy link
Author

rsokl commented Jun 19, 2020

Ah, I was actually expecting the grey boxes

image

to be removed. I see that the main issue is indeed fixed now. Thank you!

@prisae
Copy link

prisae commented Jun 19, 2020

Indeed, this is just a change with the newer sphinx. The problem was that the : was removed, and the variable name was therefore merged with the description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.