Skip to content

Conversation

@anandf
Copy link

@anandf anandf commented Jul 11, 2023

What this PR does / why we need it:
This PR is a proposal/suggestion for allowing users to provide Test excludes as part of the TestSuite. Its still work in progress PR and would like to understand if the community finds this useful.

In one of our project where we use kuttl test framework for our e2e tests. We have a scenario where we have to skip some subset of tests based on some predefined conditions.

Example scenario:
We have to run all tests under tests/e2e directory except for the below tests

1-031_validate_toolchain 
1-085_validate_dynamic_plugin_installation 
1-036_validate_keycloak_resource_reqs 
1-038_validate_productized_images 
1-051-validate_csv_permissions 
1-073_validate_rhsso 
1-077_validate_disable_dex_removed 
1-090_validate_permissions

We couldn't find an easy way of doing that. We had to copy all the tests to a temporary location and had to remove the test sub directories (rm -rf <test_sub_dir>) that needs to be skipped and run the tests from that temporary directory. This is not the ideal way of excluding tests. So I propose to add a new attribute in TestSuite CR called testExcludeDirs where user can specify the test sub directories they would like to exclude and run the remaining tests that are present in the testDirs.

This feature could also provide an easy way to exclude tests that are broken and needs to be fixed.

Note: If there are better ways of handling test excludes, without requiring this change, please let me know and I am open to use that approach.

anandf added 2 commits July 11, 2023 20:43
Signed-off-by: Anand Francis Joseph <[email protected]>
@anandf anandf marked this pull request as draft July 11, 2023 17:00
@anandf anandf marked this pull request as ready for review July 13, 2023 12:19
@anandf anandf changed the title WIP: Support for excluding subset of tests Support for excluding subset of tests Jul 24, 2023
@porridge
Copy link
Member

Did #483 meet your need @anandf ?

@kensipe
Copy link
Member

kensipe commented Jan 31, 2024

check again @anandf if this PR is needed or if #483 solve it

@porridge
Copy link
Member

Closing, as #483 should have addressed this need.
Please reopen and explain if you disagree @anandf

@porridge porridge closed this Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants