[scout] extend config-discovery with CI validator by dmlemeshko · Pull Request #214403 · elastic/kibana

dmlemeshko · 2025-03-13T14:53:18Z

Summary

More teams are adding Scout tests in their plugins, often as a PoC and not stable yet for continuous execution.
We don't want to block it, but need a way to manage the scope of Scout pipeline and be able to disable it quickly to unblock the Scout development.

Since Scout is in active development and we need it to be simple and quick as possible (we can iterate and improve later), we agreed with Robert to disable tests by plugin:

ui_tests:
  enabled:
    - apm
    - discover_enhanced
    - maps
    - observability_onboarding
  disabled:
    - *skipped_plugin*

When scout configuration is added to the new plugin, it will require to update .buildkite/scout_ci_config.yml that is owned by appex-qa team. If there is no intention to run Scout tests on CI, plugin name should be added under disabled section.

How to test locally:

Scout tests were added in observability_onboarding plugin, pipeline will throw error

modify locally .buildkite/scout_ci_config.yml

ui_tests:
  enabled:
    - apm
    - discover_enhanced
    - maps
  disabled:

run node scripts/scout discover-playwright-configs --validate --save

ERROR The following plugins are not registered in Scout CI config '.buildkite/scout_ci_config.yml'
      - observability_onboarding

~~On CI annotation will be added to clarify the failure:~~

we decided to move validation to "Quick Checks", no need to annotate.

observability_onboarding plugin is disabled, pipeline won't include it (excluded in scout_playwright_configs.json)

modify locally .buildkite/scout_ci_config.yml

ui_tests:
  enabled:
    - apm
    - discover_enhanced
    - maps
  disabled:
   - observability_onboarding

run node scripts/scout discover-playwright-configs --validate --save

 warn The following plugins are disabled in '.buildkite/scout_ci_config.yml' and will be excluded from CI run
      - observability_onboarding
 info Found Playwright config files in '4' plugins.
      Saved '3' plugins to '/Users/dmle/github/kibana/.scout/test_configs/scout_playwright_configs.json'

dmlemeshko · 2025-03-14T16:12:05Z

.buildkite/pipelines/on_merge_unsupported_ftrs.yml

+        - exit_status: 10
+          limit: 0


@elastic/kibana-operations if there is reserved status code for similar cases (intentional failure without a retry), I'm happy to replace 10 for consistency.

.buildkite/scripts/steps/test/scout_test_run_builder.sh

jbudz · 2025-03-14T18:52:16Z

.buildkite/scripts/steps/test/scout_test_run_builder.sh

-node scripts/scout discover-playwright-configs --save
+
+set +e;
+OUTPUT=$(node scripts/scout discover-playwright-configs --save --validate 2>&1)


When scout tests move to on-merge it may make sense to move this to quick checks. It'll fail CI quicker if there's issues

csr

First batch of comments (1 out of 2), expect more later today.

csr · 2025-03-17T09:06:22Z

.buildkite/scout_ci_config.yml

We mention in the PR description

When scout configuration is added to the new plugin, it will require to update .buildkite/scout_ci_config.yml that is owned by appex-qa team.

I understand that the CI will throw an error if the plugin is not included in the file, but perhaps this should be mentioned in the kbn/scout README as well, for clarity. What do you think?

Bonus 1 (optional!): the node scripts/scout discover-playwright-configs doesn't seem to be documented in the README. I understand it is mostly a command for the CI, but it could be useful to document it here, if developers want to do some troubleshooting and see how it works.

Bonus 2 (optional!): you know what would be really cool? That the ERROR The following plugins are not registered ... message on Buildkite would link to the relevant README kbn/scout section, so developers have 100% complete knowledge (and confidence) to correctly fix the error.

.buildkite/pipelines/on_merge_unsupported_ftrs.yml

csr

Looking good to me! I left some minor comments. This is a very good step forward for Scout, we will all benefit from enabling (and most importantly) disabling plugins on demand.

src/platform/packages/shared/kbn-scout/src/cli/config_discovery.ts

src/platform/packages/shared/kbn-scout/src/config/discovery/search_configs.ts

This reverts commit 994915a.

csr

Nice README + tests updates! LGTM.

                        *     .--.
                                / /  `
               +               | |
                      '         \ \__,
                  *          +   '--'  *
                      +   /\
         +              .'  '.   *
                *      /======\      +
                      ;:.  _   ;
                      |:. (_)  |
                      |:.  _   |
            +         |:. (_)  |          *
                      ;:.      ;
                    .' \:.    / `.
                   / .-'':._.'`-. \
                   |/    /||\    \|
             jgs _..--"""````"""--.._
           _.-'``                    ``'-._
         -'                                '-

elasticmachine · 2025-03-18T14:19:43Z

💚 Build Succeeded

Buildkite Build
Commit: 6c3f9e1

Metrics [docs]

✅ unchanged

History

💔 Build #284960 failed 5b1feb1
💚 Build #284177 succeeded ba4a4e1
💔 Build #284146 failed f0cb536
💔 Build #284115 failed 3afd691
💔 Build #284067 failed 0be3259
💔 Build #284014 failed 83e0c96

cc @dmlemeshko

kibanamachine · 2025-03-18T20:25:12Z

Starting backport for target branches: 8.x, 9.0

https://github.com/elastic/kibana/actions/runs/13932807865

kibanamachine · 2025-03-18T20:31:34Z

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts
❌	9.0	Backport failed because of merge conflicts You might need to backport the following PRs to 9.0: - Upgrade to Storybook 8 (#195148) - SKA: Update and breakdown x-pack/.gitignore (#212341)

Manual backport

To create the backport manually run:

node scripts/backport --pr 214403

Questions ?

Please refer to the Backport tool documentation

## Summary More teams are adding Scout tests in their plugins, often as a PoC and not stable yet for continuous execution. We don't want to block it, but need a way to manage the scope of Scout pipeline and be able to disable it quickly to unblock the Scout development. Since Scout is in active development and we need it to be simple and quick as possible (we can iterate and improve later), we agreed with Robert to disable tests by plugin: ``` ui_tests: enabled: - apm - discover_enhanced - maps - observability_onboarding disabled: - *skipped_plugin* ``` When scout configuration is added to the new plugin, it will require to update `.buildkite/scout_ci_config.yml` that is owned by `appex-qa` team. If there is no intention to run Scout tests on CI, plugin name should be added under `disabled` section. **How to test locally:** - Scout tests were added in `observability_onboarding` plugin, pipeline will throw error modify locally `.buildkite/scout_ci_config.yml` ``` ui_tests: enabled: - apm - discover_enhanced - maps disabled: ``` run `node scripts/scout discover-playwright-configs --validate --save` ``` ERROR The following plugins are not registered in Scout CI config '.buildkite/scout_ci_config.yml' - observability_onboarding ``` ~~On CI annotation will be added to clarify the failure:~~ we decided to move validation to "Quick Checks", no need to annotate. <img width="1583" alt="image" src="https://github.com/user-attachments/assets/ed6b5778-74cb-4473-8218-b96239aab067" /> - `observability_onboarding` plugin is disabled, pipeline won't include it (excluded in `scout_playwright_configs.json`) modify locally `.buildkite/scout_ci_config.yml` ``` ui_tests: enabled: - apm - discover_enhanced - maps disabled: - observability_onboarding ``` run `node scripts/scout discover-playwright-configs --validate --save` ``` warn The following plugins are disabled in '.buildkite/scout_ci_config.yml' and will be excluded from CI run - observability_onboarding info Found Playwright config files in '4' plugins. Saved '3' plugins to '/Users/dmle/github/kibana/.scout/test_configs/scout_playwright_configs.json' ``` (cherry picked from commit 05447fe) # Conflicts: # .github/CODEOWNERS

dmlemeshko · 2025-03-20T13:54:49Z

💚 All backports created successfully

Status	Branch	Result
✅	9.0

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

## Summary More teams are adding Scout tests in their plugins, often as a PoC and not stable yet for continuous execution. We don't want to block it, but need a way to manage the scope of Scout pipeline and be able to disable it quickly to unblock the Scout development. Since Scout is in active development and we need it to be simple and quick as possible (we can iterate and improve later), we agreed with Robert to disable tests by plugin: ``` ui_tests: enabled: - apm - discover_enhanced - maps - observability_onboarding disabled: - *skipped_plugin* ``` When scout configuration is added to the new plugin, it will require to update `.buildkite/scout_ci_config.yml` that is owned by `appex-qa` team. If there is no intention to run Scout tests on CI, plugin name should be added under `disabled` section. **How to test locally:** - Scout tests were added in `observability_onboarding` plugin, pipeline will throw error modify locally `.buildkite/scout_ci_config.yml` ``` ui_tests: enabled: - apm - discover_enhanced - maps disabled: ``` run `node scripts/scout discover-playwright-configs --validate --save` ``` ERROR The following plugins are not registered in Scout CI config '.buildkite/scout_ci_config.yml' - observability_onboarding ``` ~~On CI annotation will be added to clarify the failure:~~ we decided to move validation to "Quick Checks", no need to annotate. <img width="1583" alt="image" src="https://github.com/user-attachments/assets/ed6b5778-74cb-4473-8218-b96239aab067" /> - `observability_onboarding` plugin is disabled, pipeline won't include it (excluded in `scout_playwright_configs.json`) modify locally `.buildkite/scout_ci_config.yml` ``` ui_tests: enabled: - apm - discover_enhanced - maps disabled: - observability_onboarding ``` run `node scripts/scout discover-playwright-configs --validate --save` ``` warn The following plugins are disabled in '.buildkite/scout_ci_config.yml' and will be excluded from CI run - observability_onboarding info Found Playwright config files in '4' plugins. Saved '3' plugins to '/Users/dmle/github/kibana/.scout/test_configs/scout_playwright_configs.json' ``` (cherry picked from commit 05447fe) # Conflicts: # .buildkite/scripts/steps/checks/quick_checks.txt

dmlemeshko · 2025-03-20T13:57:34Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2025-03-20T20:48:57Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

## Summary More teams are adding Scout tests in their plugins, often as a PoC and not stable yet for continuous execution. We don't want to block it, but need a way to manage the scope of Scout pipeline and be able to disable it quickly to unblock the Scout development. Since Scout is in active development and we need it to be simple and quick as possible (we can iterate and improve later), we agreed with Robert to disable tests by plugin: ``` ui_tests: enabled: - apm - discover_enhanced - maps - observability_onboarding disabled: - *skipped_plugin* ``` When scout configuration is added to the new plugin, it will require to update `.buildkite/scout_ci_config.yml` that is owned by `appex-qa` team. If there is no intention to run Scout tests on CI, plugin name should be added under `disabled` section. **How to test locally:** - Scout tests were added in `observability_onboarding` plugin, pipeline will throw error modify locally `.buildkite/scout_ci_config.yml` ``` ui_tests: enabled: - apm - discover_enhanced - maps disabled: ``` run `node scripts/scout discover-playwright-configs --validate --save` ``` ERROR The following plugins are not registered in Scout CI config '.buildkite/scout_ci_config.yml' - observability_onboarding ``` ~~On CI annotation will be added to clarify the failure:~~ we decided to move validation to "Quick Checks", no need to annotate. <img width="1583" alt="image" src="https://github.com/user-attachments/assets/ed6b5778-74cb-4473-8218-b96239aab067" /> - `observability_onboarding` plugin is disabled, pipeline won't include it (excluded in `scout_playwright_configs.json`) modify locally `.buildkite/scout_ci_config.yml` ``` ui_tests: enabled: - apm - discover_enhanced - maps disabled: - observability_onboarding ``` run `node scripts/scout discover-playwright-configs --validate --save` ``` warn The following plugins are disabled in '.buildkite/scout_ci_config.yml' and will be excluded from CI run - observability_onboarding info Found Playwright config files in '4' plugins. Saved '3' plugins to '/Users/dmle/github/kibana/.scout/test_configs/scout_playwright_configs.json' ```

…15340) # Backport This will backport the following commits from `main` to `9.0`: - [[scout] extend config-discovery with CI validator (#214403)](#214403)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

kibanamachine · 2025-03-21T21:48:52Z

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

## Summary More teams are adding Scout tests in their plugins, often as a PoC and not stable yet for continuous execution. We don't want to block it, but need a way to manage the scope of Scout pipeline and be able to disable it quickly to unblock the Scout development. Since Scout is in active development and we need it to be simple and quick as possible (we can iterate and improve later), we agreed with Robert to disable tests by plugin: ``` ui_tests: enabled: - apm - discover_enhanced - maps - observability_onboarding disabled: - *skipped_plugin* ``` When scout configuration is added to the new plugin, it will require to update `.buildkite/scout_ci_config.yml` that is owned by `appex-qa` team. If there is no intention to run Scout tests on CI, plugin name should be added under `disabled` section. **How to test locally:** - Scout tests were added in `observability_onboarding` plugin, pipeline will throw error modify locally `.buildkite/scout_ci_config.yml` ``` ui_tests: enabled: - apm - discover_enhanced - maps disabled: ``` run `node scripts/scout discover-playwright-configs --validate --save` ``` ERROR The following plugins are not registered in Scout CI config '.buildkite/scout_ci_config.yml' - observability_onboarding ``` ~~On CI annotation will be added to clarify the failure:~~ we decided to move validation to "Quick Checks", no need to annotate. <img width="1583" alt="image" src="https://github.com/user-attachments/assets/ed6b5778-74cb-4473-8218-b96239aab067" /> - `observability_onboarding` plugin is disabled, pipeline won't include it (excluded in `scout_playwright_configs.json`) modify locally `.buildkite/scout_ci_config.yml` ``` ui_tests: enabled: - apm - discover_enhanced - maps disabled: - observability_onboarding ``` run `node scripts/scout discover-playwright-configs --validate --save` ``` warn The following plugins are disabled in '.buildkite/scout_ci_config.yml' and will be excluded from CI run - observability_onboarding info Found Playwright config files in '4' plugins. Saved '3' plugins to '/Users/dmle/github/kibana/.scout/test_configs/scout_playwright_configs.json' ```

…15343) # Backport This will backport the following commits from `main` to `8.x`: - [[scout] extend config-discovery with CI validator (#214403)](#214403)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

## Summary More teams are adding Scout tests in their plugins, often as a PoC and not stable yet for continuous execution. We don't want to block it, but need a way to manage the scope of Scout pipeline and be able to disable it quickly to unblock the Scout development. Since Scout is in active development and we need it to be simple and quick as possible (we can iterate and improve later), we agreed with Robert to disable tests by plugin: ``` ui_tests: enabled: - apm - discover_enhanced - maps - observability_onboarding disabled: - *skipped_plugin* ``` When scout configuration is added to the new plugin, it will require to update `.buildkite/scout_ci_config.yml` that is owned by `appex-qa` team. If there is no intention to run Scout tests on CI, plugin name should be added under `disabled` section. **How to test locally:** - Scout tests were added in `observability_onboarding` plugin, pipeline will throw error modify locally `.buildkite/scout_ci_config.yml` ``` ui_tests: enabled: - apm - discover_enhanced - maps disabled: ``` run `node scripts/scout discover-playwright-configs --validate --save` ``` ERROR The following plugins are not registered in Scout CI config '.buildkite/scout_ci_config.yml' - observability_onboarding ``` ~~On CI annotation will be added to clarify the failure:~~ we decided to move validation to "Quick Checks", no need to annotate. <img width="1583" alt="image" src="https://github.com/user-attachments/assets/ed6b5778-74cb-4473-8218-b96239aab067" /> - `observability_onboarding` plugin is disabled, pipeline won't include it (excluded in `scout_playwright_configs.json`) modify locally `.buildkite/scout_ci_config.yml` ``` ui_tests: enabled: - apm - discover_enhanced - maps disabled: - observability_onboarding ``` run `node scripts/scout discover-playwright-configs --validate --save` ``` warn The following plugins are disabled in '.buildkite/scout_ci_config.yml' and will be excluded from CI run - observability_onboarding info Found Playwright config files in '4' plugins. Saved '3' plugins to '/Users/dmle/github/kibana/.scout/test_configs/scout_playwright_configs.json' ```

dmlemeshko added 3 commits March 13, 2025 15:36

[scout] extend config-discovery with CI validator

631ac05

update script call

98192a1

unregister plugin

8fd961c

dmlemeshko requested a review from pheyos March 13, 2025 14:58

dmlemeshko added 7 commits March 13, 2025 16:51

skip pipeline retry when script fails

5daecb9

update pipeline retry order

83e0c96

add bk annotation with error message

0be3259

Update scout_test_run_builder.sh

3afd691

Update scout_test_run_builder.sh

f0cb536

re-enable discover_enhanced, update codeowners

da3123a

Merge branch 'main' into scout/add-ci-check-config

ba4a4e1

dmlemeshko self-assigned this Mar 14, 2025

dmlemeshko added release_note:skip Skip the PR/issue when compiling release notes v9.0.0 backport:version Backport to applied version labels test:scout v9.1.0 v8.19.0 labels Mar 14, 2025

dmlemeshko marked this pull request as ready for review March 14, 2025 16:09

dmlemeshko requested review from a team as code owners March 14, 2025 16:09

dmlemeshko commented Mar 14, 2025

View reviewed changes

.buildkite/scripts/steps/test/scout_test_run_builder.sh Outdated Show resolved Hide resolved

jbudz reviewed Mar 14, 2025

View reviewed changes

csr reviewed Mar 17, 2025

View reviewed changes

src/platform/packages/shared/kbn-scout/src/config/discovery/search_configs.ts Show resolved Hide resolved

dmlemeshko added 3 commits March 18, 2025 10:18

move validation to quick checks

27745ec

fix order for retry

559a6e3

revert annotation and remove validation

6ba3326

dmlemeshko requested a review from jbudz March 18, 2025 12:22

dmlemeshko added 3 commits March 18, 2025 13:42

add path to docs

f34ff56

Revert "intentionally unregister plugin"

8ab2177

This reverts commit 994915a.

Merge branch 'main' into scout/add-ci-check-config

6c3f9e1

csr approved these changes Mar 18, 2025

View reviewed changes

jbudz approved these changes Mar 18, 2025

View reviewed changes

dmlemeshko merged commit 05447fe into elastic:main Mar 18, 2025
10 checks passed

This was referenced Mar 18, 2025

[Streams] [Dedicated Grok UI] Package / highlight matched parts / suggestions #213278

Merged

SKA: Extract list of Kibana solutions into a dedicated package #213353

Merged

dmlemeshko mentioned this pull request Mar 20, 2025

[9.0] [scout] extend config-discovery with CI validator (#214403) #215340

Merged

dmlemeshko mentioned this pull request Mar 20, 2025

[8.x] [scout] extend config-discovery with CI validator (#214403) #215343

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 20, 2025

This was referenced Mar 21, 2025

[ResponseOps][Alerts] Create embeddable alerts table plugin #213909

Merged

initial @kbn/scout-security plugin #210433

Merged

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 23, 2025

Conversation

dmlemeshko commented Mar 13, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

dmlemeshko Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jbudz Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

csr left a comment

Choose a reason for hiding this comment

Uh oh!

csr Mar 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

csr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

csr left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 18, 2025

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

Uh oh!

kibanamachine commented Mar 18, 2025

Uh oh!

kibanamachine commented Mar 18, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

dmlemeshko commented Mar 20, 2025

💚 All backports created successfully

Questions ?

Uh oh!

dmlemeshko commented Mar 20, 2025

💚 All backports created successfully

Questions ?

Uh oh!

kibanamachine commented Mar 20, 2025

Uh oh!

kibanamachine commented Mar 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dmlemeshko commented Mar 13, 2025 •

edited by kibanamachine

Loading

jbudz Mar 14, 2025 •

edited

Loading