Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GENERATE doesn't display which values caused test failures in the test report #2366

Open
fschuh opened this issue Feb 8, 2022 · 3 comments

Comments

@fschuh
Copy link

fschuh commented Feb 8, 2022

Description

The GENERATE macro runs a given test multiple times with different parameters, but the test report doesn't seem to provide context or information on which of those parameters are causing failures in the test.

Additional context

As an example, let's consider the following simple test:

    TEST_CASE("Test with generators")
    {
        int value = GENERATE(1, 3, 5);
        REQUIRE_FALSE(testFunction(value));
    }

If the test fails with generated parameter values 1 and 3, this is what the test report looks like:

-------------------------------------------------------------------------------
Test with generators
-------------------------------------------------------------------------------
C:\TestProject1\Tests1.cpp(131)
...............................................................................

C:\TestProject1\Tests1.cpp(134): FAILED:
  REQUIRE_FALSE( testFunction(value) )
with expansion:
  !true

-------------------------------------------------------------------------------
Test with generators
-------------------------------------------------------------------------------
C:\TestProject1\Tests1.cpp(131)
...............................................................................

C:\TestProject1\Tests1.cpp(134): FAILED:
  REQUIRE_FALSE( testFunction(value) )
with expansion:
  !true

-------------------------------------------------------------------------------

As we can see, there aren't any hints whatsoever that the values 1 and 3 were the reasons of those failures, while 5 was good.
If there isn't a way to have Catch2 display the generated parameters on the report, is this something that would be feasible to add?
Are there any reasons why this information doesn't show up in the test report?

@fschuh fschuh changed the title GENERATE doesn't display in the test report which values caused test failures GENERATE doesn't display which values caused test failures in the test report Feb 8, 2022
@neobrain
Copy link

neobrain commented Feb 9, 2022

This is a handy feature that I found myself looking for previously too. I don't know what's blocking Catch2 from supporting it currently, but a workaround is to use dynamic sections with manual for-loops:

    TEST_CASE("Test with generators")
    {
        for (int value : { 1, 3, 5 }) {
            DYNAMIC_SECTION("VALUE IS " << value) {
                REQUIRE_FALSE(testFunction(value));
            }
        }
    }

@fschuh
Copy link
Author

fschuh commented Feb 9, 2022

Another workaround is to use the CAPTURE macro.
However, that adds extra noise to the test code, and you have to remember to use it on every test.
It would be nice if we could get the values to show up in the report without having to add additional boilerplate to the test.

@tanlin2013
Copy link

Are there any updates for this feature, after a year?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants