Request for Consistent Handler Restoration on expectOutputString and expectOutputRegex Failure

[KentarouTakeda]

Dear Sebastian Bergmann,

This pull request addresses an issue where error and exception handlers are not restored properly when expectOutputString or expectOutputRegex assertions fail. Below is a minimal code example to reproduce the behavior:

<?php

namespace Test;

use PHPUnit\Framework\TestCase;

class Test extends TestCase
{
    public $existsOriginalErrorHandler;
    public $existsOriginalExceptionHandler;

    // Assume that the error handler is changed within the test code.
    public function setUp(): void
    {
        $this->existsOriginalErrorHandler = (bool)set_error_handler(fn () => null);
        $this->existsOriginalExceptionHandler = (bool)set_exception_handler(fn () => null);
    }

    // Assume that the error handler changes are reliably restored to their original state in `tearDown()`.
    public function tearDown(): void
    {
        restore_exception_handler();
        restore_error_handler();
    }

    // In PHPUnit, tests typically start with this handler setting.
    public function test1()
    {
        $this->assertTrue($this->existsOriginalErrorHandler);
        $this->assertFalse($this->existsOriginalExceptionHandler);
    }

    public function test2()
    {
        // The same assertions as before will obviously succeed.
        $this->assertTrue($this->existsOriginalErrorHandler);
        $this->assertFalse($this->existsOriginalExceptionHandler);

        // This assertion is intentionally set to fail. This failure leads to:
        $this->expectOutputString('This test will be failed'); // Failure (intentional)
    }

    // The next test starts without `tearDown()` being executed,
    public function test3()
    {
        // The exact same assertions will fail here.
        $this->assertTrue($this->existsOriginalErrorHandler);
        $this->assertFalse($this->existsOriginalExceptionHandler); // Failure (unintentional)
    }
}

Although this may not be a "bug" per se, the inconsistent behavior makes it challenging for some frameworks to avoid PHPUnit's warning messages like "Test code or tested code ..." . It would be greatly appreciated if this issue could be addressed in PHPUnit to ensure consistent handler restoration.

Thank you for considering this request.

Best regards,
Kentaro Takeda

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.32%. Comparing base (d475be0) to head (84cf94e).
⚠️ Report is 2913 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #5842      +/-   ##
============================================
+ Coverage     89.28%   92.32%   +3.03%     
+ Complexity     6585     6555      -30     
============================================
  Files           693      699       +6     
  Lines         19937    19774     -163     
============================================
+ Hits          17801    18256     +455     
+ Misses         2136     1518     -618     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[staabm]

@KentarouTakeda could you please double check your reported issue still exist?
I have a feeling this was fixed in the meantime.

if it still does not produce the expected outcome, please provide describe in more detail why/what does not work.

thank you

[KentarouTakeda]

@staabm
The issue still persists. I tested with the two latest actively supported versions, 11.5.33 and 12.3.5, and confirmed the same behavior as in the initial report.

Let me explain the details. Please take a look at the test code I first presented. test1 and test3 contain exactly the same code. Nevertheless, test1 passes while test3 fails. In this case, the success of test1 is the correct behavior. The unintended failure of test3 is caused by the failure of test2 with expectOutputString(). This pull request aims to fix these behaviors.

The minimal reproduction code above demonstrates the issue, but this behavior also affects feature tests in many frameworks, including Laravel and Symfony, producing unintended test result reports for users.

For example, in Laravel, running the following test code:

<?php

namespace Tests\Feature;

use PHPUnit\Framework\Attributes\DataProvider;
use PHPUnit\Framework\Attributes\Test;
use Tests\TestCase;

// Important note: Here, `TestCase` extends the Laravel Framework's `TestCase`.
class ExampleTest extends TestCase
{
    #[Test]
    public function intentionalFailure()
    {
        $this->expectOutputString('This test will be failed'); // Failure (intentional)
    }

    #[Test]
    #[DataProvider('repeatProvider')]
    public function tenSuccessfulTests()
    {
        $this->assertTrue(true);
    }

    public static function repeatProvider()
    {
        return array_map(fn() => [], range(1, 10));
    }
}

produces the following result:

FRRRRRRRRRR                                                       11 / 11 (100%)

While the first test fails as expected, it is not expected that all subsequent tests are marked as Risky.

The detailed output includes messages like:

There was 1 risky test:

1) Tests\Feature\ExampleTest::tenSuccessfulTests
* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

* Test code or tested code removed error handlers other than its own

* Test code or tested code removed exception handlers other than its own

As shown above, in frameworks that manipulate error or exception handlers (including Laravel and Symfony), a failure in tests involving input/output can lead to user confusion due to misleading test reports.

This pull request fixes the problem where the failure of one test causes handler state inconsistencies that propagate to subsequent tests, resulting in them being incorrectly marked as Risky.

[Ernestopheles]

This was merely done by Github Copilot:
phpunit-5842-backport.patch

Hi,

we reproduced the issue described in PR #5842 in our Contao/Symfony project while running PHPUnit inside a DDEV container. Summary of our findings and testing:

• Environment: PHP 8.3 (8.3.25 in container), PHPUnit 12.4.1, Contao Kernel boot inside tests.
• Symptom: After a failing output assertion (expectOutputString/expectOutputRegex) PHPUnit left the global error/exception handler stacks in a different state, causing subsequent tests to be marked "risky" with messages like "Test code or tested code did not remove its own error handlers".
• Reproduction notes: We run a single functional test that boots the Contao Kernel and executes a KernelBrowser POST request. In our environment the kernel and some libraries manipulate handler stacks during boot, which made the problem visible.

What we tried:

• Applied the change suggested in this PR (remove the throw $e; in TestCase::performAssertionsOnOutput) locally as a vendor-backport and re-ran the failing test: this avoids re-throwing the assertion exception in the output-assertion path, and helps avoid leaving PHPUnit's handler state inconsistent.
• Additionally, we re-registered PHPUnit's ErrorHandler after kernel boot inside the test (set_error_handler(\PHPUnit\Runner\ErrorHandler::instance())) and implemented a PHPUnit-like snapshot/restore of the initial handler stacks to ensure they are restored exactly in finally().

Result:

• After the backport + the minimal test stabilization change, the failing-case no longer caused subsequent tests to be marked as risky in our DDEV environment. The single functional test ran and passed assertions; PHPUnit reported only Deprecation warnings.

Notes / Environment caveats:

• Process isolation (@runInSeparateProcess / PHP attributes) in our DDEV container produces set_mempolicy: Operation not permitted errors; therefore we used an in-process approach and re-registration of PHPUnit's handler as workaround.
• We created a small patch locally (see attached) that removes the throw $e; line. We consider this a low-risk change which mirrors the intent of PR Request for Consistent Handler Restoration on expectOutputString and expectOutputRegex Failure #5842.

Regards