Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sphinx build: catch warnings for style errors #208

Merged
merged 25 commits into from
Sep 6, 2020
Merged

Conversation

alex-l-kong
Copy link
Contributor

@alex-l-kong alex-l-kong commented Aug 31, 2020

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Addresses and closes #197. Currently, our Sphinx build only fails in the case of an error (ex. syntax error in conf.py). However, we need to make sure it fails if a docstring is malformed as well. The reason is because documentation will change overtime. Either new functions will be defined or existing functions will change. We don't want to commit broken documentation to master because that will appear malformed on readthedocs.

How did you implement your changes

From what I've seen, it looks like all we need is to add a certain command in a .readthedocs.yml file. If not, then I'll have to look more into it. If yes, there may be some more documentation/import-related issues we need to fix.

Remaining Issues

Is there a way we can get better Google-style docstring checks (ex. something equivalent to pylint for this)? It seems like the general consensus from the sample I've read online is that this is something that one shouldn't completely rely on ReadTheDocs to do that for you. Sounds kind of silly but it is what it is if this is really the case. That or we ask people to really test the documentation manually before requesting reviews/merging into master.

@alex-l-kong alex-l-kong changed the title Sphinx build with warnings for style errors Sphinx build: catch warnings for style errors Aug 31, 2020
@alex-l-kong
Copy link
Contributor Author

alex-l-kong commented Sep 2, 2020

With the new Makefile and .readthedocs.yml config attached, ReadTheDocs now will fail the build if a warning is raised (ex. one of your docstrings is misformatted).

Here's the problem: Sphinx is extremely sloppy when it comes to throwing warnings for Google-style documentation. As in, almost non-existent. I've updated the Remaining Issues section on top but essentially sphinx-build with Napoleon really sucked at catching several misformed docstrings. The only warnings I was able to consistently get were unexpected indentations caused by having one too many spaces in the args and returns list.

Regardless, suggestions are welcome.

@ngreenwald
Copy link
Member

Got it. May just be something we have to remember to check.
What causes an issue like this to happen? https://deepcell.readthedocs.io/en/master/API/deepcell.callbacks.html#deepcell.callbacks.RedirectModel
And would that be caught?

@alex-l-kong
Copy link
Contributor Author

Got it. May just be something we have to remember to check.
What causes an issue like this to happen? https://deepcell.readthedocs.io/en/master/API/deepcell.callbacks.html#deepcell.callbacks.RedirectModel
And would that be caught?

I looked at the docstring and what seems to screw it up is their attempt to include code. From what I've read, using ``` isn't the proper way to do this. You would need to do something like what is mentioned here: https://stackoverflow.com/questions/44577360/google-style-docstring-example-section-is-not-rendering-as-a-code-snippet.

As for the entire code block going off the page, if using the double colon doesn't already fix this, you may need to include additional formatting prior (ex. a pipe |) before each line to force a break. Not tried it out myself yet but I read somewhere that this was the Sphinx way to do it.

No, I do not think ReadTheDocs would catch this because it seems like all sphinx-build checks for are indentation issues, and there weren't any in the docstring which generated the example you showed. Even then, it's pretty bad at that. Perhaps @ackagel can comment further to see if there were any other warnings Sphinx was generating when he updated the docstrings, but on my end, that's the only I could find.

@ackagel
Copy link
Contributor

ackagel commented Sep 2, 2020

I didn't catch any particularly helpful warnings when I was reformatting the doc-strings.

That being said, there is a way we can roughly check for correct formatting and throw an error/warning otherwise. It should be possible to do some regex matching to verify the general structure of the docstrings, but this leaves argument's unchecked. Luckily, we can actually get the argument names programmatically and verify that they are all formatted correctly. This can be done in conf.py as follows:

import inspect


def check_format(app, what, name, obj, options, lines):
    if what == 'function':
        argnames = inspect.getargspec(obj)[0]
        if len(argnames) > 0:
            # loop through args+lines to find all mentions of each arg and assure one line matches w/ correct structure
            # otherwise break and raise error/warning
            # after looping over args, construct regex match for general structure
        else:
            # construct regex match w/o args requirement

        # compile and run the regex, throw error/warning if no match


def setup(app):
    app.connect('autodoc-process-docstring', check_format)

On the other hand, there's no non-overkill way to check for correct types, Returns, and Raises formatting. Regardless, I'd imagine strict argument format requirements will encourage people to correctly format their Returns and Raises.

@ackagel
Copy link
Contributor

ackagel commented Sep 2, 2020

finding this functionality in the sphinx docs was ironically difficult :)

@ngreenwald
Copy link
Member

ngreenwald commented Sep 2, 2020 via email

@alex-l-kong
Copy link
Contributor Author

alex-l-kong commented Sep 4, 2020

So from my other branch on context spatial setup, I discovered that part of the reason we may not have been getting errors is because we're not building our docstrings correctly on pushing to readthedocs. You have to rebuild them using something like this: readthedocs/readthedocs.org#1139 (comment). I've updated the Makefile to do this locally for you, but you'd have to push those changes in the _markdown folder for them to be visible on ReadTheDocs. This solution should at least prevent us from having to store the actual _markdown files, basically, hiding the sphinx-apidoc call in conf.py to allow readthedocs to build the documentation for us. If needed, as @ackagel suggested, we can also add a docstring check on top of that.

@ngreenwald
Copy link
Member

So in order for the build to fail, the user would have to build the docs locally and then push those as part of the PR?

@alex-l-kong
Copy link
Contributor Author

So in order for the build to fail, the user would have to build the docs locally and then push those as part of the PR?

To guarantee it, yes, especially if you added a new docstring. I haven't tried the new Makefile on an intentionally horrendous docstring yet but my hope is that sphinx-apidoc can actually catch misformatted docstrings better than sphinx-build.

@ngreenwald
Copy link
Member

ngreenwald commented Sep 4, 2020 via email

@alex-l-kong
Copy link
Contributor Author

alex-l-kong commented Sep 4, 2020

What currently happens when you modify a docstring? Is there a local version that gets updated and needs to be checked in? Or we don't currently store a local version of the docs, and this change will a) build a local version and b) have that local version get checked? I'm hesitant to require people to run a separate make command and then also check in additional documentation related files

So clarifications and corrections. When you push to readthedocs, currently all that is happening is that it runs sphinx-build for you on your most recent commit. sphinx-build is what actually builds the HTML from the .md files you have in your most recent commit (which in our case, we have to push ourselves because we have not configured readthedocs to create the .md files for us on each commit using sphinx-apidoc). Think of the .md files as sort of a map that helps sphinx-build locate each module, the functions within, and the associated docstrings.

To clarify further, when you run the Makefile on your machine, you generate local documentation that you can view using open _build/html/index.html in the docs folder. That's all cool...except when you push to GitHub, ReadTheDocs creates its own copy based on the built-in commands it uses. And by default, all it runs is sphinx-build. Meaning that if our Makefile has commands other than sphinx-build to generate updated .md files (which it does), they will not be seen unless you explicitly push the new .md files along with all your updated code. That way, sphinx-build can use the updated roadmap on each push to generate the documentation that will be seen by the public. Just remember that the Makefile is completely for local use and ReadTheDocs does not care to use it.

That's why we need to explicitly configure our conf.py so that it tells ReadTheDocs to run the sphinx-apidoc command to regenerate the .md roadmaps before it tries to build them. This will also have the side effect of us not having to store and push .md files to the repo ourselves, because ReadTheDocs will do it on its end. All we need to provide is an index.rst and a conf.py.

Unfortunately, it does not look like sphinx-apidoc actually checks the formatting of the docstrings when building the .md roadmaps. sphinx-build is the one that is supposed to do that, but the Napoleon extension, while great for parsing Google (and Numpy)-style docstrings, is also pretty bad at identifying ill-formatted ones from my experience. So we also will likely have to tell ReadTheDocs through conf.py to run a separate check outside of the warnings sphinx-build throws to identify these docstring offenders (see Adam's comment on how to do that).

@ngreenwald
Copy link
Member

ngreenwald commented Sep 5, 2020

Okay awesome, that all makes sense.

Think of the .md files as sort of a map that helps sphinx-build locate each module, the functions within, and the associated docstrings.

This is because we're telling it that we're using google style docstrings, correct? There's not any actual docstring content in the .md files, just instructions on how to appropriately locate them?

This means we'll be able to remove the formatting specific .md files that can be globally specified in the conf.py instead, and then it's just a question of writing the appropriate parsing logic to catch the errors the RTD isn't catching?

@alex-l-kong
Copy link
Contributor Author

alex-l-kong commented Sep 5, 2020

So with the new conf.py the build does indeed catch docstring errors as evidenced in the above build. There are a few more cases that need to be handled (ex. return values) but overall it's looking super nice. Helped me locate at least three major documentation oopsies in the args list. And we don't need the _markdown folder anymore.

Copy link
Member

@ngreenwald ngreenwald left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's great!

I feel bad for complaining about the coveralls UI: the UI for RTD failures is like 100x worse 😆

docs/conf.py Outdated Show resolved Hide resolved
@ngreenwald
Copy link
Member

ngreenwald commented Sep 5, 2020 via email

@alex-l-kong
Copy link
Contributor Author

Sounds good, we'll move forward doing that then.

ngreenwald
ngreenwald previously approved these changes Sep 5, 2020
Copy link
Member

@ngreenwald ngreenwald left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! I'm gonna wait till @ackagel takes a look since my ability to give useful feedback on this sphinx stuff is fairly limited. I'v put together an issue for the next tasks for documentation #213

Copy link
Contributor

@ackagel ackagel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fantastic! Just one optional comment on the argnames vs params checks.

docs/conf.py Outdated Show resolved Hide resolved
@ngreenwald ngreenwald merged commit 5542a46 into master Sep 6, 2020
@ngreenwald ngreenwald deleted the sphinx_style branch September 6, 2020 19:23
alex-l-kong added a commit that referenced this pull request Jan 14, 2021
* See if adding fail_on_warning in a readthedocs yml file does the trick

* Change to 4 spaces indent for .readthedocs.yml

* Update documentation: remove extraneous md files and change Makefile to fail if warning caught

* Intentionally put a mistake in the documentation to try and force error

* Get configuration right to ensure we fail on warning

* Fix version error (removed)

* Add an intentional indentation error (appeared on local make html) to see if readthedocs fails

* Fix error when trying to break spatial_analysis.py

* Change .readthedocs.yml

* Add more configuration

* See if changing to version 2 changes anything

* Now fix intentional issue and see if readthedocs passes now

* This should fix it...

* Fix docstring in test_utils and configure conf.py to ignore ark.md

* Try out new conf.py settings to see if Sphinx makes the documentation for us

* Re-add _static and _templates and update params in conf.py to build documentation in correct locations

* Fix regex to ignore testing files

* Update .gitignore to ensure people do not try to add the _markdown files generated if testing locally

* Configure conf.py to catch docstring errors, and intentionally push a bad build to see if ReadTheDocs actually catches them

* Fix argument-list errors caught by autodoc-process-docstring

* conf.py now works to catch basic return errors not caught by arglist checking

* conf.py now handles argument-less function docstring checks, and fixed bulleted list issue in test_utils.py

* Remove TODOs in conf.py

* Remove extra arguments checks, leave just 1 all-encompassing one
y2kbugger pushed a commit that referenced this pull request Jul 29, 2021
* See if adding fail_on_warning in a readthedocs yml file does the trick

* Change to 4 spaces indent for .readthedocs.yml

* Update documentation: remove extraneous md files and change Makefile to fail if warning caught

* Intentionally put a mistake in the documentation to try and force error

* Get configuration right to ensure we fail on warning

* Fix version error (removed)

* Add an intentional indentation error (appeared on local make html) to see if readthedocs fails

* Fix error when trying to break spatial_analysis.py

* Change .readthedocs.yml

* Add more configuration

* See if changing to version 2 changes anything

* Now fix intentional issue and see if readthedocs passes now

* This should fix it...

* Fix docstring in test_utils and configure conf.py to ignore ark.md

* Try out new conf.py settings to see if Sphinx makes the documentation for us

* Re-add _static and _templates and update params in conf.py to build documentation in correct locations

* Fix regex to ignore testing files

* Update .gitignore to ensure people do not try to add the _markdown files generated if testing locally

* Configure conf.py to catch docstring errors, and intentionally push a bad build to see if ReadTheDocs actually catches them

* Fix argument-list errors caught by autodoc-process-docstring

* conf.py now works to catch basic return errors not caught by arglist checking

* conf.py now handles argument-less function docstring checks, and fixed bulleted list issue in test_utils.py

* Remove TODOs in conf.py

* Remove extra arguments checks, leave just 1 all-encompassing one
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Identify syntax errors in sphinx/RTD
3 participants