Support for excluding subset of tests #478
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
This PR is a proposal/suggestion for allowing users to provide Test excludes as part of the
TestSuite. Its still work in progress PR and would like to understand if the community finds this useful.In one of our project where we use
kuttltest framework for our e2e tests. We have a scenario where we have to skip some subset of tests based on some predefined conditions.Example scenario:
We have to run all tests under
tests/e2edirectory except for the below testsWe couldn't find an easy way of doing that. We had to copy all the tests to a temporary location and had to remove the test sub directories (
rm -rf <test_sub_dir>) that needs to be skipped and run the tests from that temporary directory. This is not the ideal way of excluding tests. So I propose to add a new attribute inTestSuiteCR calledtestExcludeDirswhere user can specify the test sub directories they would like to exclude and run the remaining tests that are present in thetestDirs.This feature could also provide an easy way to exclude tests that are broken and needs to be fixed.
Note: If there are better ways of handling test excludes, without requiring this change, please let me know and I am open to use that approach.