Consistently include test setup and teardown in duration measurement

[mvorisek]

fix #378, measure the whole TestCase::runBare() duration

also measure any test, not only the successful ones, measuring failed/errored tests is helpful especially for tests using IO or external services

[codecov]

Codecov Report

Attention: 7 lines in your changes are missing coverage. Please review.

Comparison is base (d94c5ad) 70.54% compared to head (e418bd4) 69.86%.

Files	Patch %	Lines
src/Extension.php	0.00%	7 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #380      +/-   ##
============================================
- Coverage     70.54%   69.86%   -0.68%     
  Complexity      110      110              
============================================
  Files            26       26              
  Lines           516      521       +5     
============================================
  Hits            364      364              
- Misses          152      157       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[localheinz]

@mvorisek

Two things:

• I do not believe this extension should report slow tests that have not passed. When I have a test suite where a test has not passed, I would fix the failing test first, and improve the speed of passing tests second.
• If we want to (somewhat) consistently report slow tests across phpunit/phpunit:^8.5.19, phpunit/phpunit:^9.0.0, and phpunit/phpunit:^10.0.0, then the extension should probably
  • start timing on PHPUnit\Event\Test\BeforeFirstTestMethodCalled
  • stop timing on PHPUnit\Event\Test\AfterTestMethodFinished

What do you think?

[mvorisek]

I do not believe this extension should report slow tests that have not passed. When I have a test suite where a test has not passed, I would fix the failing test first, and improve the speed of passing tests second.

I belive it should. When failed test failed fast, it is never reported. And in general, tests do not fail slower than when passed fully.

But if a failed test took longer than a maximum duration, it should be reported to make the developer aware there is also some slowness issue. Especially when run remotely/in CI.

If we want to (somewhat) consistently report slow tests across phpunit/phpunit:^8.5.19, phpunit/phpunit:^9.0.0, and phpunit/phpunit:^10.0.0, then the extension should probably

I am looking into the PHPUnit v8/9 hooks to make the behaviour consistent.

[mvorisek]

Rebased.

[localheinz]

@mvorisek

Don't worry about rebasing, I'm working on it and will finish it in a moment!

[localheinz]

Thank you, @mvorisek!