[8.x] [scout] extend config-discovery with CI validator (#214403)#215343
Merged
dmlemeshko merged 5 commits intoelastic:8.xfrom Mar 23, 2025
Merged
[8.x] [scout] extend config-discovery with CI validator (#214403)#215343dmlemeshko merged 5 commits intoelastic:8.xfrom
dmlemeshko merged 5 commits intoelastic:8.xfrom
Conversation
## Summary
More teams are adding Scout tests in their plugins, often as a PoC and
not stable yet for continuous execution.
We don't want to block it, but need a way to manage the scope of Scout
pipeline and be able to disable it quickly to unblock the Scout
development.
Since Scout is in active development and we need it to be simple and
quick as possible (we can iterate and improve later), we agreed with
Robert to disable tests by plugin:
```
ui_tests:
enabled:
- apm
- discover_enhanced
- maps
- observability_onboarding
disabled:
- *skipped_plugin*
```
When scout configuration is added to the new plugin, it will require to
update `.buildkite/scout_ci_config.yml` that is owned by `appex-qa`
team. If there is no intention to run Scout tests on CI, plugin name
should be added under `disabled` section.
**How to test locally:**
- Scout tests were added in `observability_onboarding` plugin, pipeline
will throw error
modify locally `.buildkite/scout_ci_config.yml`
```
ui_tests:
enabled:
- apm
- discover_enhanced
- maps
disabled:
```
run `node scripts/scout discover-playwright-configs --validate --save`
```
ERROR The following plugins are not registered in Scout CI config '.buildkite/scout_ci_config.yml'
- observability_onboarding
```
~~On CI annotation will be added to clarify the failure:~~
we decided to move validation to "Quick Checks", no need to annotate.
<img width="1583" alt="image"
src="https://github.com/user-attachments/assets/ed6b5778-74cb-4473-8218-b96239aab067"
/>
- `observability_onboarding` plugin is disabled, pipeline won't include
it (excluded in `scout_playwright_configs.json`)
modify locally `.buildkite/scout_ci_config.yml`
```
ui_tests:
enabled:
- apm
- discover_enhanced
- maps
disabled:
- observability_onboarding
```
run `node scripts/scout discover-playwright-configs --validate --save`
```
warn The following plugins are disabled in '.buildkite/scout_ci_config.yml' and will be excluded from CI run
- observability_onboarding
info Found Playwright config files in '4' plugins.
Saved '3' plugins to '/Users/dmle/github/kibana/.scout/test_configs/scout_playwright_configs.json'
```
(cherry picked from commit 05447fe)
# Conflicts:
# .buildkite/scripts/steps/checks/quick_checks.txt
Contributor
💚 Build Succeeded
Metrics [docs]
History
|
MadameSheema
approved these changes
Mar 23, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto8.x:Questions ?
Please refer to the Backport tool documentation