suite: add support for user-defined test filtering

[renzoarreaza]

Summary

Adds support for users to provide extra filtering on which tests to run.

Changes

A new Exported function was needed in the suite package to support this.

func RunWithSkip(t *testing.T, suite TestingSuite, skip func(string, string) bool)

This function is the same as the existing Run function with an extra call to the skip function passed by the user.
I now let the Run function call this one.

Another change was to move the running of the SetupSuite to outside the iteration of the (test) methods.
If the skip function logs the skipped tests, this change makes sure all of these logs are together, at the top of the testrun, instead of having some skips, followed by the suite setup and more skips afterwards.

Motivation

The main motivation of this change is to allow for test registration. This allows tests to be skipped, as defined by the user, without repeatedly running the SetupTest/TeardownTest repeatedly for skipped tests.

Related

#1050 (issue) #1051 (PR)

[dolmen]

I see some value in the refactoring of Run that moves the suite setup out of the loop. It would be worth submitting it as a separate cleanup PR.

However at this point I don't understand the use case. So I didn't take time to review the test suite.

[dolmen]

The use case is not clear to me. I don't see why I would need this.

So far this is 👎 for me.

@renzoarreaza If you are serious about adding this feature, you'll have to convince us (Testify maintainers) better (first requirement), improve the provided documentation (2nd requirement). A runnable example would also be helpful as additional documentation (3rd requirement).

[renzoarreaza]

Thanks for the feedback, I will take a look. I have some ideas to improve it (apart from your comments) and I will work on a clearer example of the use case.

[renzoarreaza]

I have made some changes to the proposal.
The idea is the same, but the implementation has changed.
Instead of having RunWithSkip and a function that the user has to pass I've added a new interface that the user can optionally implement as part of their suite struct.

Motivation:

The goal of this proposal is for the user to have a way to define a filter for which tests should or shouldn't run.
(you could think of it as an extension of -testify.m)
I use testify for integration tests and SetupSuite and SetupTest are on the slower side.
Depending on how I build the src code, (daily/beta builds, specific feature flags enabled or not etc...) or on cli flags (e.g runLevel express, regular, extreme) I might want to not run some tests. Right now, the only way to achieve this is to have some checks at the beginning of the test and decide if s.T().Skip() needs to be called or not. This however results in SetupTest running unnecessarily.
These changes allow the user to skip tests at the collection phase and skipping the SetupTest for skipped tests (and even the SetupSuite is skipped if all tests are).

[dolmen]

@renzoarreaza I have just merged #1769 so a rebase would be welcome.

[brackendawson]

I think it would be better to make sure that calling testing.T.Skip in the setup function stops that function and prevents the test and the teardown form being run too.

Edit: Skip will stop both the rest of BeforeTest, and the test, but not the call to AfterTest. I raised #1781