Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion x-pack/test/api_integration/deployment_agnostic/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,10 +246,28 @@ node scripts/functional_test_runner --config x-pack/test/api_integration/deploym
```

## Tagging and Skipping the Tests
Since deployment-agnostic tests are designed to run both locally and on MKI/Cloud, we believe no extra tagging is required. If a test is not working on MKI/Cloud or both, there is most likely an issue with the FTR service or the configuration file it uses.
Since deployment-agnostic tests are designed to run both locally and on MKI/Cloud, we believe no extra tagging is required in general (read below for exceptions). If a test is not working on MKI/Cloud or both, there is most likely an issue with the FTR service or the configuration file it uses.

When a test fails on CI, automation will apply `.skip` to the top-level describe block. This means the test will be skipped in **both serverless and stateful environments**. If a test is unstable in a specific environment only, it is probably a sign that the test is not truly deployment-agnostic.

### Excluding a suite from test environments
As pointed out above, deployment agnostic tests should be designed to run in stateful and serverless, locally and in cloud (ECH, MKI). However, there are situations where a test suite should only run on a subset of these environments. **This should be an exception.**

Here are the supported suite labels to control execution in test environments:
* `skipStateful` - this will exclude the suite from **all stateful test runs, local and ECH**
* `skipCloud` - this will exclude the suite from **stateful cloud / ECH test runs**
* `skipServerless` - this will exclude the suite from **all serverless test runs, local and MKI**
* `skipMKI` - this will exclude the suite from **serverless cloud / MKI test runs**

Note that tags can not be applied to an arrow function suite like `describe('test suite', () => {`. Here's an example of how to apply a suite tag:
```ts
describe('test suite', function () {
// add a comment to explain why this suite is excluded from that test environment
this.tags(['skipMKI']);
[...]
});
```

## Migrating existing tests
If your tests align with the outlined criteria and requirements, you can migrate them to deployment-agnostic by following these steps:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,10 @@ export function createServerlessTestConfig<T extends DeploymentAgnosticCommonSer
},
testFiles: options.testFiles,
junit: options.junit,
suiteTags: options.suiteTags,
suiteTags: {
include: options.suiteTags?.include,
exclude: [...(options.suiteTags?.exclude || []), 'skipServerless'],
},
};
};
}
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,10 @@ export function createStatefulTestConfig<T extends DeploymentAgnosticCommonServi
// services can be customized, but must extend DeploymentAgnosticCommonServices
services: options.services || services,
junit: options.junit,
suiteTags: options.suiteTags,
suiteTags: {
include: options.suiteTags?.include,
exclude: [...(options.suiteTags?.exclude || []), 'skipStateful'],
},

esTestCluster: {
...xPackAPITestsConfig.get('esTestCluster'),
Expand Down
Loading