Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace use of test.test_asyncio.utils.TestCase with unittest.IsolatedAsyncioTestCase #93357

Closed
21 tasks
arhadthedev opened this issue May 30, 2022 · 4 comments
Closed
21 tasks

Replace use of test.test_asyncio.utils.TestCase with unittest.IsolatedAsyncioTestCase #93357

arhadthedev opened this issue May 30, 2022 · 4 comments
Labels
tests Tests in the Lib/test dir topic-asyncio type-feature A feature request or enhancement

Comments

@arhadthedev
Copy link
Member

arhadthedev commented May 30, 2022 •
edited by bedevere-bot
Loading

Feature or enhancement

Get rid of ad-hoc implementation of async tests by using all relatively new features of standard IsolatedAsyncIoTestCase. For this we need to:

  • replace all occurences of class ...(test_utils.TestCase): with class ...(unittest.IsolatedAsynIoTestCase):
  • remove manually created event loops in favor of the builtin one of IsolatedAsyncIoTestCase
  • make loop-dependend test cases async
  • replace multithreading with asyncio wherever possible
  • remove unused imports
  • remove the TestCase and classes it uses (588 -> 346 lines in test_asyncio/utils.py)

Pitch

  • test_asyncio.utils.TestCase provides no extra functionality over IsolatedAsyncioTestCase. Instead, it takes 200+ lines of code together with two more classes used nowhere else (TestSelector and TestLoop)
  • TestCase requires manual scheduling of tasks using loop.run_until_complete while IsolatedAsyncioTestCase allows await
  • TestCase was introduced on 2014 by c73701d while IsolatedAsynIoTestCase was introduced 5 years later by 4dd3e3f

Progress tracking

There are 83 cases that use test_asyncio.utils.TestCase
  • test/test_asyncio/test_base_events.py:class BaseEventTests(test_utils.TestCase):
  • test/test_asyncio/test_base_events.py:class BaseEventLoopTests(test_utils.TestCase):
  • test/test_asyncio/test_base_events.py:class BaseEventLoopWithSelectorTests(test_utils.TestCase):
  • test/test_asyncio/test_base_events.py:class BaseLoopSockSendfileTests(test_utils.TestCase):
  • test/test_asyncio/test_base_events.py:class TestSelectorUtils(test_utils.TestCase):
  • test/test_asyncio/test_events.py: test_utils.TestCase):
  • test/test_asyncio/test_events.py: test_utils.TestCase):
  • test/test_asyncio/test_events.py: test_utils.TestCase):
  • test/test_asyncio/test_events.py: test_utils.TestCase):
  • test/test_asyncio/test_events.py: test_utils.TestCase):
  • test/test_asyncio/test_events.py: test_utils.TestCase):
  • test/test_asyncio/test_events.py:class HandleTests(test_utils.TestCase):
  • test/test_asyncio/test_futures.py:class DuckTests(test_utils.TestCase):
  • test/test_asyncio/test_futures.py:class CFutureTests(BaseFutureTests, test_utils.TestCase):
  • test/test_asyncio/test_futures.py:class CSubFutureTests(BaseFutureTests, test_utils.TestCase):
  • test/test_asyncio/test_futures.py:class PyFutureTests(BaseFutureTests, test_utils.TestCase):
  • test/test_asyncio/test_futures.py: test_utils.TestCase):
  • test/test_asyncio/test_futures.py: test_utils.TestCase):
  • test/test_asyncio/test_futures.py: test_utils.TestCase):
  • test/test_asyncio/test_futures.py: test_utils.TestCase):
  • test/test_asyncio/test_futures.py: test_utils.TestCase):
  • test/test_asyncio/test_pep492.py:class BaseTest(test_utils.TestCase):
  • test/test_asyncio/test_proactor_events.py:class ProactorSocketTransportTests(test_utils.TestCase):
  • test/test_asyncio/test_proactor_events.py:class ProactorDatagramTransportTests(test_utils.TestCase):
  • test/test_asyncio/test_proactor_events.py:class BaseProactorEventLoopTests(test_utils.TestCase):
  • test/test_asyncio/test_proactor_events.py:class ProactorEventLoopUnixSockSendfileTests(test_utils.TestCase):
  • test/test_asyncio/test_selector_events.py:class BaseSelectorEventLoopTests(test_utils.TestCase):
  • test/test_asyncio/test_selector_events.py:class SelectorTransportTests(test_utils.TestCase):
  • test/test_asyncio/test_selector_events.py:class SelectorSocketTransportTests(test_utils.TestCase):
  • test/test_asyncio/test_selector_events.py:class SelectorSocketTransportBufferedProtocolTests(test_utils.TestCase):
  • test/test_asyncio/test_selector_events.py:class SelectorDatagramTransportTests(test_utils.TestCase):
  • test/test_asyncio/test_sendfile.py: test_utils.TestCase):
  • test/test_asyncio/test_sendfile.py: test_utils.TestCase):
  • test/test_asyncio/test_sendfile.py: test_utils.TestCase):
  • test/test_asyncio/test_sendfile.py: test_utils.TestCase):
  • test/test_asyncio/test_sendfile.py: test_utils.TestCase):
  • test/test_asyncio/test_sendfile.py: test_utils.TestCase):
  • test/test_asyncio/test_sock_lowlevel.py: test_utils.TestCase):
  • test/test_asyncio/test_sock_lowlevel.py: test_utils.TestCase):
  • test/test_asyncio/test_sock_lowlevel.py: test_utils.TestCase):
  • test/test_asyncio/test_sock_lowlevel.py: test_utils.TestCase):
  • test/test_asyncio/test_sock_lowlevel.py: test_utils.TestCase):
  • test/test_asyncio/test_sock_lowlevel.py: test_utils.TestCase):
  • test/test_asyncio/test_ssl.py:class TestSSL(test_utils.TestCase):
  • test/test_asyncio/test_sslproto.py:class SslProtoHandshakeTests(test_utils.TestCase):
  • test/test_asyncio/test_streams.py:class StreamTests(test_utils.TestCase):
  • test/test_asyncio/test_subprocess.py:class SubprocessTransportTests(test_utils.TestCase):
  • test/test_asyncio/test_subprocess.py: test_utils.TestCase):
  • test/test_asyncio/test_subprocess.py: test_utils.TestCase):
  • test/test_asyncio/test_subprocess.py: test_utils.TestCase):
  • test/test_asyncio/test_subprocess.py: test_utils.TestCase):
  • test/test_asyncio/test_subprocess.py: test_utils.TestCase):
  • test/test_asyncio/test_subprocess.py: class SubprocessProactorTests(SubprocessMixin, test_utils.TestCase):
  • test/test_asyncio/test_tasks.py: test_utils.TestCase):
  • test/test_asyncio/test_tasks.py:class CTask_CFuture_SubclassTests(BaseTaskTests, test_utils.TestCase):
  • test/test_asyncio/test_tasks.py:class CTaskSubclass_PyFuture_Tests(BaseTaskTests, test_utils.TestCase):
  • test/test_asyncio/test_tasks.py:class PyTask_CFutureSubclass_Tests(BaseTaskTests, test_utils.TestCase):
  • test/test_asyncio/test_tasks.py:class CTask_PyFuture_Tests(BaseTaskTests, test_utils.TestCase):
  • test/test_asyncio/test_tasks.py:class PyTask_CFuture_Tests(BaseTaskTests, test_utils.TestCase):
  • test/test_asyncio/test_tasks.py: test_utils.TestCase):
  • test/test_asyncio/test_tasks.py:class PyTask_PyFuture_SubclassTests(BaseTaskTests, test_utils.TestCase):
  • test/test_asyncio/test_tasks.py:class CTask_Future_Tests(test_utils.TestCase):
  • test/test_asyncio/test_tasks.py:class PyIntrospectionTests(test_utils.TestCase, BaseTaskIntrospectionTests):
  • test/test_asyncio/test_tasks.py:class CIntrospectionTests(test_utils.TestCase, BaseTaskIntrospectionTests):
  • test/test_asyncio/test_tasks.py:class PyCurrentLoopTests(BaseCurrentLoopTests, test_utils.TestCase):
  • test/test_asyncio/test_tasks.py:class CCurrentLoopTests(BaseCurrentLoopTests, test_utils.TestCase):
  • test/test_asyncio/test_tasks.py:class GenericTaskTests(test_utils.TestCase):
  • test/test_asyncio/test_tasks.py:class FutureGatherTests(GatherTestsBase, test_utils.TestCase):
  • test/test_asyncio/test_tasks.py:class CoroutineGatherTests(GatherTestsBase, test_utils.TestCase):
  • test/test_asyncio/test_tasks.py:class RunCoroutineThreadsafeTests(test_utils.TestCase):
  • test/test_asyncio/test_tasks.py:class SleepTests(test_utils.TestCase):
  • test/test_asyncio/test_tasks.py:class CompatibilityTests(test_utils.TestCase):
  • test/test_asyncio/test_unix_events.py:class SelectorEventLoopSignalTests(test_utils.TestCase):
  • test/test_asyncio/test_unix_events.py:class SelectorEventLoopUnixSocketTests(test_utils.TestCase):
  • test/test_asyncio/test_unix_events.py:class SelectorEventLoopUnixSockSendfileTests(test_utils.TestCase):
  • test/test_asyncio/test_unix_events.py:class UnixReadPipeTransportTests(test_utils.TestCase):
  • test/test_asyncio/test_unix_events.py:class UnixWritePipeTransportTests(test_utils.TestCase):
  • test/test_asyncio/test_unix_events.py:class SafeChildWatcherTests (ChildWatcherTestsMixin, test_utils.TestCase):
  • test/test_asyncio/test_unix_events.py:class FastChildWatcherTests (ChildWatcherTestsMixin, test_utils.TestCase):
  • test/test_asyncio/test_windows_events.py:class ProactorLoopCtrlC(test_utils.TestCase):
  • test/test_asyncio/test_windows_events.py:class ProactorMultithreading(test_utils.TestCase):
  • test/test_asyncio/test_windows_events.py:class ProactorTests(test_utils.TestCase):
  • test/test_asyncio/test_windows_events.py:class WinPolicyTests(test_utils.TestCase):

Linked PRs
@arhadthedev arhadthedev added the type-feature A feature request or enhancement label May 30, 2022
@AlexWaygood AlexWaygood added tests Tests in the Lib/test dir topic-asyncio labels May 30, 2022
@arhadthedev arhadthedev changed the title Replace use of test.test_asyncio.utils.TestCase with unittest.IsolatedAsynIoTestCase Replace use of test.test_asyncio.utils.TestCase with unittest.IsolatedAsyncioTestCase May 30, 2022
@arhadthedev arhadthedev mentioned this issue May 31, 2022
Merged
@sobolevn sobolevn mentioned this issue Sep 25, 2022
Closed
gvanrossum pushed a commit that referenced this issue Oct 4, 2022
@arhadthedev
gh-93357: Start porting asyncio server test cases to IsolatedAsyncioT…
ce8fc18 
…estCase (#93369)

Lay the foundation for further work in `asyncio.test_streams`.
@bedevere-bot bedevere-bot mentioned this issue Oct 5, 2022
Merged
gvanrossum pushed a commit that referenced this issue Oct 5, 2022
@arhadthedev
gh-93357: Port test cases to IsolatedAsyncioTestCase, part 2 (#97896)
09aea94 
This fixes the buildbots.
carljm added a commit to carljm/cpython that referenced this issue Oct 6, 2022
@carljm
Merge branch 'main' into dictwatch
4bb5d9d 
* main: (66 commits)
  pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879)
  docs(typing): add "see PEP 675" to LiteralString (python#97926)
  pythongh-97850: Remove all known instances of module_repr() (python#97876)
  I changed my surname early this year (python#96671)
  pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768)
  pythongh-91539: improve performance of get_proxies_environment  (python#91566)
  build(deps): bump actions/stale from 5 to 6 (python#97701)
  pythonGH-95172 Make the same version `versionadded` oneline (python#95172)
  pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073)
  pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774)
  pythongh-93357: Port test cases to IsolatedAsyncioTestCase, part 2 (python#97896)
  pythongh-95196: Disable incorrect pickling of the C implemented classmethod descriptors (pythonGH-96383)
  pythongh-97758: Fix a crash in getpath_joinpath() called without arguments (pythonGH-97759)
  pythongh-74696: Pass root_dir to custom archivers which support it (pythonGH-94251)
  pythongh-97661: Improve accuracy of sqlite3.Cursor.fetchone docs (python#97662)
  pythongh-87092: bring compiler code closer to a preprocessing-opt-assembler organisation (pythonGH-97644)
  pythonGH-96704: Add {Task,Handle}.get_context(), use it in call_exception_handler() (python#96756)
  pythongh-93738: Documentation C syntax (:c:type:`PyTypeObject*` -> :c:expr:`PyTypeObject*`) (python#97778)
  pythongh-97825: fix AttributeError when calling subprocess.check_output(input=None) with encoding or errors args (python#97826)
  Add re.VERBOSE flag documentation example (python#97678)
  ...
arhadthedev added a commit to arhadthedev/cpython that referenced this issue Oct 7, 2022
@arhadthedev
Revert "pythongh-93357: Port test cases to IsolatedAsyncioTestCase, p…
3713915 
…art 2 (python#97896)"

This reverts commit 09aea94.
arhadthedev added a commit to arhadthedev/cpython that referenced this issue Oct 7, 2022
@arhadthedev
Revert "pythongh-93357: Start porting asyncio server test cases to Is…
c17e930 
…olatedAsyncioTestCase (python#93369)"

This reverts commit ce8fc18.
@arhadthedev arhadthedev mentioned this issue Oct 7, 2022
Merged
gvanrossum pushed a commit that referenced this issue Oct 7, 2022
@arhadthedev
gh-97983: Revert "Lay the foundation for further work in asyncio.test…
f99bb20 
…_streams: port server cases to IsolatedAsyncioTestCase" (#98015)

This PR reverts gh-93369 and gh-97896 because they've made asyncio tests unstable. After these PRs were merged, random GitHub action jobs of random commits started to fail unrelated tests and test framework methods.

The reverting is necessary because such shrapnel failures are a symptom of some underlying bug that must be found and fixed first.

I had a hope that it's a server overload because we already have extremely rare disc access errors. However, one and a half day passed, and the failures continue to emerge both in PRs and commits.

Affected issue: gh-93357.
First reported in #97940 (comment).

* Revert "gh-93357: Port test cases to IsolatedAsyncioTestCase, part 2 (#97896)"

This reverts commit 09aea94.

* Revert "gh-93357: Start porting asyncio server test cases to IsolatedAsyncioTestCase (#93369)"

This reverts commit ce8fc18.
mpage pushed a commit to mpage/cpython that referenced this issue Oct 11, 2022
@arhadthedev @mpage
pythongh-93357: Start porting asyncio server test cases to IsolatedAs…
81fcd2c 
…yncioTestCase (python#93369)

Lay the foundation for further work in `asyncio.test_streams`.
mpage pushed a commit to mpage/cpython that referenced this issue Oct 11, 2022
@arhadthedev @mpage
pythongh-93357: Port test cases to IsolatedAsyncioTestCase, part 2 (p…
2d20cf5 
…ython#97896)

This fixes the buildbots.
mpage pushed a commit to mpage/cpython that referenced this issue Oct 11, 2022
@arhadthedev @mpage
pythongh-97983: Revert "Lay the foundation for further work in asynci…
e75ada1 
…o.test_streams: port server cases to IsolatedAsyncioTestCase" (python#98015)

This PR reverts pythongh-93369 and pythongh-97896 because they've made asyncio tests unstable. After these PRs were merged, random GitHub action jobs of random commits started to fail unrelated tests and test framework methods.

The reverting is necessary because such shrapnel failures are a symptom of some underlying bug that must be found and fixed first.

I had a hope that it's a server overload because we already have extremely rare disc access errors. However, one and a half day passed, and the failures continue to emerge both in PRs and commits.

Affected issue: pythongh-93357.
First reported in python#97940 (comment).

* Revert "pythongh-93357: Port test cases to IsolatedAsyncioTestCase, part 2 (python#97896)"

This reverts commit 09aea94.

* Revert "pythongh-93357: Start porting asyncio server test cases to IsolatedAsyncioTestCase (python#93369)"

This reverts commit ce8fc18.
@kumaraditya303
Copy link
Contributor

If we want to continue with this, then IMO this needs to be done early in the dev cycle so we have time to fixup the CI if it breaks. We certainly don't want to deal with unstable CI at the beta stage.

@arhadthedev
Copy link
Member Author

@kumaraditya303 You're right; I'll return to this issue this weekend and start with stabilization of reverted gh-98015.

@arhadthedev
Copy link
Member Author

Put on hold until gh-98219 is resolved (it probably will fix Altered environment on Windows and Env changed in buildbot in gh-97983).

@bedevere-bot bedevere-bot mentioned this issue Oct 30, 2022
Closed
@kumaraditya303
Copy link
Contributor

I'm going to just close this now, it is not the best use of our time to port these tests as it doesn't increase test coverage and chance of breaking CI is very high especially as asyncio is very flaky. When I'll remove child watcher I may port some tests to ease writing new tests but not other things.

@kumaraditya303 kumaraditya303 closed this as not planned Won't fix, can't repro, duplicate, stale Apr 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
tests Tests in the Lib/test dir topic-asyncio type-feature A feature request or enhancement
Projects
Unittest issues
Status: Done
asyncio
Status: Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@asvetlov @arhadthedev @kumaraditya303 @AlexWaygood