-
Notifications
You must be signed in to change notification settings - Fork 41
ENH: Support remote data in doctest #137
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
I think simply skipping as per setting Supporting the equivalent of Some test cases to consider (just brainstorming here, don't take this as a command). These might not all be suitable in CI but should at least be manually tested during development:
.. doctest-remote-data::
>>> some_func() # doctest: +FLOAT_CMP
1.2345
>>> some_other_func_with_warnings() # doctest: +IGNORE_WARNINGS +ELLIPSIS
1.23...
.. doctest-remote-data::
>>> some_func_1() # doctest: +REMOTE_DATA
>>> some_func_2()
Thank you for tackling this! |
|
Hi @pllim, thank you for your suggestions. I have written a couple of test cases like you suggested, and I have been able to test for when the This is the command I call This is the error I get If I go by the error message, do I really need to query an actual remote source for my tests? |
|
To test how it is intended to be used, a lightweight remote data access might be nice. Something simple like using But to better debug your problem, it would be nice if you can first reproduce that exact same error in the CI here. It seems like you might need to install |
|
We have debugged this during our weekly call today. With dev version So, |
|
As for the current test failures, that you also asked on slack, I have a solution that works locally for the first option, please let me know if you want me to push it to this branch. But I'm equally happy if you rather want to figure it out yourself @tinumide (you can use the astropy config files as guidance) as there is a similar situation there. |
|
There are two more things that came into my mind while reviewing your tests:
|
|
p.s. Re: Test dependencies -- I am okay with defining |
|
No, I think one example should be skipping because of the remote data, but it should be remained skipped due you the "missing" dependency |
|
Ahhh... okay... maybe something clearer like |
Sure, sounds good to try out. |
I think raising an exception when remote data is accessed without using the directive would be a nice to have. I mean if the goal is to avoid spamming data providers as much as possible while testing, then the exception would come in handy in the case where a developer forgets to add the remote-data directive, or a new contributor is not sure of what should be done or is even completely unaware. Although I'm not sure of the likeliness of any of these events occurring. |
Also @bsipocz I think we can create issues for the new features such as the first point you've suggested here so that we can keep track of them. Or would you rather have them in this MR? |
pytest_doctestplus/plugin.py
Outdated
| if config.getoption('remote_data', 'none') != 'any': | ||
| matches = [re.match( | ||
| r'{}\s+doctest-remote-data\s*::(\s+.*)?'.format(comment_char), | ||
| last_line) for last_line in last_lines] | ||
|
|
||
| if len(matches) > 1: | ||
| match = matches[0] or matches[1] | ||
| else: | ||
| match = matches[0] | ||
|
|
||
| if match: | ||
| marker = match.group(1) | ||
| if (marker is None or | ||
| (marker.strip() == 'win32' and | ||
| sys.platform == 'win32')): | ||
| skip_next = True | ||
| continue |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You don't need the extra match that you copy-pasted from doctest-requires, as you don't need to capture the upstream dependency here.
| if config.getoption('remote_data', 'none') != 'any': | |
| matches = [re.match( | |
| r'{}\s+doctest-remote-data\s*::(\s+.*)?'.format(comment_char), | |
| last_line) for last_line in last_lines] | |
| if len(matches) > 1: | |
| match = matches[0] or matches[1] | |
| else: | |
| match = matches[0] | |
| if match: | |
| marker = match.group(1) | |
| if (marker is None or | |
| (marker.strip() == 'win32' and | |
| sys.platform == 'win32')): | |
| skip_next = True | |
| continue | |
| if config.getoption('remote_data', 'none') != 'any': | |
| matches = [re.match( | |
| r'{}\s+doctest-remote-data\s*::'.format(comment_char), | |
| last_line) for last_line in last_lines] | |
| if len(matches) > 1: | |
| match = matches[0] or matches[1] | |
| else: | |
| match = matches[0] | |
| if match: | |
| skip_next = True | |
| continue |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh I see
|
Yes, @tinumide , I agree about |
Alright, thanks. @astrofrog kindly take a look |
114a71b to
5024dad
Compare
|
Opening a new issue for the issue with the |
pllim
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please undo all the unnecessary blank lines across different files and fix the CI.
There also needs to be a clear warning that this new directive cannot be used together with IGNORE_WARNINGS.
0258596 to
0ee459b
Compare
I've rebased. |
pllim
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All the commits need to be squashed into one commit to remove from the history that vscode file that was accidentally added and then removed.
You might want to set your editor to automatically remove trailing whitespace. I see some of those in your diff, especially if you pasted code from your Python session.
We no longer has a test job that make sure pytest-doctestplus can still run without pytest-remotedata. Do we need that?
Would be nice if @ceb8 or @bsipocz could take this PR for a spin over at astroquery after the review comments are addressed but before merge.
README.rst
Outdated
| this way: | ||
|
|
||
| .. code-block:: python | ||
| .. doctest-remote-data:: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is good practice to add blank line after the directive because I think some of the directives out there actually require it (but I am not 100% sure).
| .. doctest-remote-data:: | |
| .. doctest-remote-data:: | |
README.rst
Outdated
| >>> from contextlib import closing | ||
| >>> from urllib.request import urlopen | ||
| >>> with closing(urlopen('https://www.astropy.org')) as remote: | ||
| ... remote.read() # doctest: +IGNORE_OUTPUT | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The doc in readme isn't tested, so you can simplify the example as follows without fear of introducing extra test dependencies:
| >>> from contextlib import closing | |
| >>> from urllib.request import urlopen | |
| >>> with closing(urlopen('https://www.astropy.org')) as remote: | |
| ... remote.read() # doctest: +IGNORE_OUTPUT | |
| >>> import requests | |
| >>> r = requests.get('https://www.astropy.org') |
tests/docs/skip_some.rst
Outdated
|
|
||
| This code won't run. So let's make sure that it gets skipped: | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove addition of trailing whitespace.
tests/docs/skip_some.rst
Outdated
| .. doctest-skip:: | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove addition of trailing whitespace.
tests/docs/skip_some.rst
Outdated
|
|
||
|
|
||
| Remote data block code sandwiched in block codes | ||
| =============================================== |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| =============================================== | |
| ================================================ |
tests/docs/skip_some.rst
Outdated
|
|
||
| This code block should work just fine:: | ||
|
|
||
| >>> 1 + 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How would we know if this is actually silently skipped too, say, because it thinks this code is still somehow part of remote date block above? That is, how are we sure the remote data block really ended before this line?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So a blank line within a doctest separates one chunk of code from another in the case of ..doctes-remote-data:: i.e the directive considers the chunk of code after a blank line as a separate chunk of code. That way we know that it does not control the chunk of code.
tests/docs/skip_some.rst
Outdated
|
|
||
| This code block should work just fine:: | ||
|
|
||
| >>> 1 + 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How would we know if this is actually silently skipped too, say, because it thinks this code is still somehow part of remote date block above? That is, how are we sure the remote data block really ended before this line?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I'm not sure how we could do that, as currently all the narrative docs testings are done as one test, if it's stopped somewhere the rest of the file is not being picked up. But then we have a tracelog with a failure. Whenever it's skipped, we have no easy way to see that the rest of the file was run as expected. Maybe add a failing one to the very bottom of the file? But then it's a failing test, and here we cannot really have it as an xfail, afaik.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, perhaps that test case is too complicated to be tested this way. If this is tested in the other unit tests, then we don't have to do it here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh, good point, there is indeed a better handle of these in the unittest content.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pllim - So if we run this file with --remote-data, then it of course stops with a fail on L98. As expected. So I'm not sure what to do with the file itself, maybe only include it in when --remote-data is not used, and test the working of the remote-data stuff separately, only in unit tests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe put those tests in a dedicated file that can be run with and without --remote-data ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what was the goal of having all those blocks that should fail ? Since the first failing test will fail the file anyway, we cannot test that the other blocks that should fail actually fail. So I guess we could test the expected failures in the unit tests, and in this file test blocks that should either pass or be skipped ?
tests/test_doctestplus.py
Outdated
| p = testdir.makefile( | ||
| '.rst', | ||
| """ | ||
| # This test should fail, but we skip it, thus the test should pass. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| # This test should fail, but we skip it, thus the test should pass. | |
| # This test should fail, but we skip it, thus the test should be skipped. |
| testdir.makeini( | ||
| """ | ||
| [pytest] | ||
| doctest_optionflags = ELLIPSIS |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the purpose of the extra doctest_optionflags = ELLIPSIS in this test?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was trying to test if the ELLIPSIS option works well with the .. doctest-remote-data:: directive, but I think I misunderstood it's function. The correct option here should be NORMALIZE_WHITESPACE
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pllim - we run into some cases where it seemed some of the default flags don't work nicely together, so as mentioned above the purpose was to test some of them.
Maybe the test is superfluous, then we can remove it later.
0ee459b to
d9b5ba4
Compare
saimn
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems mostly ok, just a few comments about the test comments which need some clarifications.
bsipocz
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should just wrap this up. I still don't know what the best approach with the skip_some.rst file, I like that it includes the remote-data related stuff, but then we cannot test it with remote-data enabled.
Otherwise the only other outstanding idea is to double up the tests, run one with remote-data one without and change only the expected outcome.
Otherwise, this seems to be working as expected with the astroquery PRs that I test it with, so would vote for merging it and making a quick release asap. If anything comes up, we can always fix it in a new release, after all this should not break anything but adds new functionality only.
| testdir.makeini( | ||
| """ | ||
| [pytest] | ||
| doctest_optionflags = ELLIPSIS |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pllim - we run into some cases where it seemed some of the default flags don't work nicely together, so as mentioned above the purpose was to test some of them.
Maybe the test is superfluous, then we can remove it later.
|
Re: tests -- I am okay with getting rid of those that don't work in the interest of wrapping this up and doing a release. We can always revisit if we encounter bugs in production. I am only mainly interested in this not breaking any existing functionality, especially for |
|
I can certainly do a test pr astropy core, but can't see anything that would break it. |
more descriptive comments in skip_some.rst Update tests/test_doctestplus.py Co-authored-by: Brigitta Sipőcz <[email protected]> added pytest-remotedata as a requirement to be installed by tox added version to pytest-remotedata composite directive test fails latest commit removed test for ignore warning directive test cleanup removed extra marker updated README and added changelog entry review corrections added vscode setting to .gitignore removed .vscode splitted test functions review corrections
|
@pllim - I tested this locally with astropy main, and all seems good, I see test failures, but I see them with the current doctestplus release, too. |
tests/docs/skip_some.rst
Outdated
|
|
||
| This code block should work just fine:: | ||
|
|
||
| >>> 1 + 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe put those tests in a dedicated file that can be run with and without --remote-data ?
Co-authored-by: Simon Conseil <[email protected]>
|
OK, so let's go ahead with this to avoid Friday EOB releases :) |
|
Thank you very much @tinumide! |
This is my attempt of resolving the issue of adding directive supporting the use of remote data in doctests.
The issue can be found here #2
EDIT: Fix #2
EDIT: Re-posted notes from the author below.
.. doctest-remote-data::that skips the next doctest chunk if--remote-datais not passed as an argument to the pytest command is included--remote-dataoption was not passed then check the chunk of strings provided by the result ofdoctest.DocTestParser.parsefor the new directives using regex