test_trace uses test_pprint

[serhiy-storchaka]

BPO	24215
Nosy	@terryjreedy, @abalkin, @larryhastings, @bitdancer, @serhiy-storchaka
Files	issue24215.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-05-17.05:55:50.073>
labels = ['type-bug', 'tests']
title = 'test_trace uses test_pprint'
updated_at = <Date 2018-06-03.15:47:58.591>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2018-06-03.15:47:58.591>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2015-05-17.05:55:50.073>
creator = 'serhiy.storchaka'
dependencies = []
files = ['39431']
hgrepos = []
issue_num = 24215
keywords = ['patch']
message_count = 10.0
messages = ['243386', '243416', '243582', '243584', '243588', '243592', '243594', '243614', '243679', '318561']
nosy_count = 6.0
nosy_names = ['terry.reedy', 'belopolsky', 'larry', 'r.david.murray', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24215'
versions = ['Python 3.4', 'Python 3.5']

Linked PRs

• gh-68403: Fix test_coverage in test_trace #108910
• [3.12] gh-68403: Fix test_coverage in test_trace (GH-108910) #109104
• [3.11] gh-68403: Fix test_coverage in test_trace (GH-108910) #109105

[serhiy-storchaka]

Converting test_pprint to be unittest discoverable had broke the test_trace that uses test_main from test_pprint.

======================================================================
ERROR: test_coverage (test.test_trace.TestCoverage)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/test_trace.py", line 312, in test_coverage
    self._coverage(tracer)
  File "/home/serhiy/py/cpython/Lib/test/test_trace.py", line 305, in _coverage
    tracer.run(cmd)
  File "/home/serhiy/py/cpython/Lib/trace.py", line 500, in run
    self.runctx(cmd, dict, dict)
  File "/home/serhiy/py/cpython/Lib/trace.py", line 508, in runctx
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
AttributeError: module 'test.test_pprint' has no attribute 'test_main'

======================================================================
ERROR: test_coverage_ignore (test.test_trace.TestCoverage)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/test_trace.py", line 327, in test_coverage_ignore
    self._coverage(tracer)
  File "/home/serhiy/py/cpython/Lib/test/test_trace.py", line 305, in _coverage
    tracer.run(cmd)
  File "/home/serhiy/py/cpython/Lib/trace.py", line 500, in run
    self.runctx(cmd, dict, dict)
  File "/home/serhiy/py/cpython/Lib/trace.py", line 508, in runctx
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
AttributeError: module 'test.test_pprint' has no attribute 'test_main'

---

The simplest way to fix the regression is to restore test_main in test_pprint. But for now test_pprint is much larger and slower than it was when test_trace was written. Perhaps it is not the best example for testing tracing. Are there better ideas?

[terryjreedy]

Since test files should no longer have test_main, test_trace should no longer look for one. (And indeed, test_trace itself should be converted, which looks trivial.) The offending line is
cmd='from test import test_pprint; test_pprint.test_main()'):
The use of 'test_main' is just a convenience; it could just as well be 'trace_main'. But there is no need to put anything extra in the traced file. The following works for me on on 3.4.3
cmd='import unittest, test.test_pprint;'
'unittest.main(test.test_pprint, exit=False)'):

This solves the problem of this issue, but the tracing takes about 10 of 14.5 seconds total, An alternative might be considered, but test_coverage is tied to the output from pprint run by unittest. Alexander, can you suggest an alternate target and output test that would still fulfill the intent of the test?

[bitdancer]

I've backed out the change set that broke the buildbots (see de9c43fabda6).

I can understand this changeset getting committed, since one doesn't normally expect a change in the tests to break other tests (though I myself do often run the full test suite anyway), but it is not acceptable that the breakage of the buildbots was not addressed quickly. The problem should have been fixed or the changeset backed out within hours of the problem being recognized.

[python-dev]

New changeset cf52756f19b6 by R David Murray in branch '3.4':
bpo-24215: also back out changeset that broke test_trace in 3.4.
https://hg.python.org/cpython/rev/cf52756f19b6

[serhiy-storchaka]

Sorry, I expected this issue be solved much faster.

Unfortunately Terry's recipe doesn't work. Here is a patch that reapplies changes that broke buildbots (additional tests for buildins and making test_pprint discoverable), and also fixes test_trace and makes it discoverable.

I found yet one issue with test_trace. It fails if run regrtest with the -m option (both patched and unpatched versions, there is no regression).

$ ./python -m test.regrtest test_trace
[1/1] test_trace
1 test OK.
$ ./python -m test.regrtest -m test_coverage test_trace
[1/1] test_trace
test test_trace failed -- Traceback (most recent call last):
  File "/home/serhiy/py/cpython-3.4/Lib/test/test_trace.py", line 313, in test_coverage
    self.assertTrue("pprint.cover" in files)
AssertionError: False is not true

1 test failed:
test_trace

It fails because run_unittest() is used in tested script. That is yet one argument against using test_pprint in test_trace.

[bitdancer]

If you are looking for a small test file to switch to, perhaps test_uu would be a good candidate. It isn't very likely to grow, since I don't think uu encoding is used much any more, so the likelyhood of someone caring about it is relatively low. But I haven't looked at what test_trace is using test_pprint for, so perhaps it isn't suitable.

[serhiy-storchaka]

There are issues with additional tests for uu.

The fact that test_trace fails with the -m option means that any file that runs unittest is not good example (unittest looks on sys.argv).

[terryjreedy]

My recipe worked for me, but I am not surprised that it might fail elsewhere. Is the specific purpose of the two problemmatic test_trace test to test that trace works with unittest? It did not seem like it to me. If not, test_trace could be decoupled from other tests and use a 'tracetest.py' file that defines one function to be traced.

In the meanwhile, I think you should disable the two tests in test_trace and apply the test_pprint changes, rather than hold test_pprint hostage.

[python-dev]

New changeset 2c074a8dd084 by Serhiy Storchaka in branch '3.4':
bpo-24215: Added tests for more builtin types in test_pprint.
https://hg.python.org/cpython/rev/2c074a8dd084

New changeset da711bdcc1bf by Serhiy Storchaka in branch 'default':
bpo-24215: Added tests for more builtin types in test_pprint.
https://hg.python.org/cpython/rev/da711bdcc1bf

[serhiy-storchaka]

There is different failure when run this test separately:

$ ./python -m test -v -m test_coverage test_trace
...

======================================================================
FAIL: test_coverage (test.test_trace.TestCoverage)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/test_trace.py", line 318, in test_coverage
    self.assertIn("pprint.cover", files)
AssertionError: 'pprint.cover' not found in ['warnings.cover', 'unittest.case.cover', 'test.support.__init__.cover', 'test.test_pprint.cover', 'trace.cover', 'platform.cover', 'weakref.cover', 'unittest.loader.cover', 'unittest.util.cover', 'copy.cover', 'unittest.runner.cover', 'unittest.result.cover', 'unittest.suite.cover', 'unittest.signals.cover', 'test.test_set.cover']

---