[scout] adding test helper @kbn/scout-oblt package and uptate onboarding tests#209761
Merged
dmlemeshko merged 25 commits intoelastic:mainfrom Feb 11, 2025
Merged
[scout] adding test helper @kbn/scout-oblt package and uptate onboarding tests#209761dmlemeshko merged 25 commits intoelastic:mainfrom
@kbn/scout-oblt package and uptate onboarding tests#209761dmlemeshko merged 25 commits intoelastic:mainfrom
Conversation
dmlemeshko
force-pushed
the
scout/add-oblt-scout-helper
branch
from
a3cd1fa to
1d8cf7e
Compare
dmlemeshko
changed the title
@kbn/scout-oblt package @kbn/scout-oblt package and uptate onboarding tests
dmlemeshko
commented
Feb 7, 2025
.github/CODEOWNERS
Outdated
| packages/kbn-repo-source-classifier-cli @elastic/kibana-operations | ||
| packages/kbn-scout @elastic/appex-qa | ||
| packages/kbn-scout-info @elastic/appex-qa | ||
| packages/kbn-scout-oblt @elastic/appex-qa |
Contributor
Author
There was a problem hiding this comment.
temporarily, we will need to decide on shared ownership across Oblt Teams
…ibana into scout/add-oblt-scout-helper
Contributor
💔 Build Failed
Failed CI StepsHistory
|
This reverts commit 69af69d.
mykolaharmash
approved these changes
Feb 11, 2025
Contributor
mykolaharmash
left a comment
mykolaharmash
left a comment
There was a problem hiding this comment.
Thank you @dmlemeshko! This is a great change, totally agree with your points about following the test pyramid and reducing the scope of e2e tests 🙌 Initially this code was just a conversion from Cypress without re-thinking the general approach, thank you for taking the time to put this in order!
Ikuni17
approved these changes
Feb 11, 2025
Contributor
💔 Build Failed
Failed CI StepsMetrics [docs]Public APIs missing comments
Any counts in public APIs
Public APIs missing exports
Unknown metric groupsAPI count
ESLint disabled line counts
Total ESLint disabled count
Unreferenced deprecated APIs
History
cc @dmlemeshko |
Contributor
|
Starting backport for target branches: 8.x, 9.0 https://github.com/elastic/kibana/actions/runs/13269037257 |
kibanamachine
added a commit
to kibanamachine/kibana
that referenced
this pull request
Feb 11, 2025
…rding tests (elastic#209761) ## Summary `@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test helpers specifically designed to test `Observability` applications in Kibana. All Oblt plugins should only import from `@kbn/scout-oblt` Its primary goal is to simplify the test development experience for teams working on `Observability` plugins by providing custom Playwright fixtures, page objects, and utilities tailored for Observability-related testing scenarios. Contributing: - when Fixture/Page Object is sharable across all Solutions and Platform (`fleetApi` fixture), it should be added in `@kbn/scout` - when Fixture/Page Object is Oblt-specific but is shared across tests under the multiple plugins (`OnboardingHome` page), it should be added in `@kbn/scout-oblt` - when Fixture/Page Object is only used in a single plugin (`onboarding` internal APIs ?), it should be added in this plugin. I also re-worked existing tests with few ideas in mind: - Scout is **e2e testing tool** and should target primary e2e test scenarios; We have _API integration tests_ to test multiple short scenarios for APIs behavior (response, status code) and _jest/React testing library_ to test components in isolation (elements rendering, fields validation). Doing all the testing with e2e tool like Playwright will dramatically affect cost efficiency and stability of tests, but also slows overall CI execution and PRs delivery. The goal is to follow testing pyramid and keep in mind its principles. - We on purpose spin up new browser context for each `test` block to make sure our **tests are independent**. Having too many short `test` blocks in the file significantly slows down the execution: every block triggers browser context, saml authentication, adding/removing Fleet integrations (each call up to 2 seconds) and other beforeEach/afterEach hooks. Real browser-based testing is expensive. It is not about putting every step into 1 `test` block, but also not a Jest unit-test-style design. When it is possible to group similar actions on the same page and if it is a part of the same user flow - we should do it. It also doesn't bring the testing value repeating the same UI steps multiple times in different scenarios. _Our CI costs are critical to cut when it is possible_ - Avoid **nesting describe** blocks: it complicates test readability and also complicates for CI bot to properly skip the failing block (it will skip the top level one). We encourage **Scout parallel test execution** based on running test spec files in multiple workers, not the `test` blocks within the same file. Having too many `test` blocks in the same file will be slowly run in the single thread and in case of flakiness, it means Team lose more test coverage than they probably expect. Before (**59** test blocks - **8-8.5 min** per distro): <img width="1709" alt="Screenshot 2025-02-08 at 18 01 40" src="https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab" /> After (**15** test blocks - **3.5-4 min** per distro): <img width="1578" alt="Screenshot 2025-02-10 at 18 14 42" src="https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120" /> For reviewers: updated tests are possible to run in 2 parallel workers against the same Kibana/ES instance and run time is dropping to **2.5-3 min** 🚀 . It is up to UX-Logs team to decide if you want to keep parallel run (new tests can be added either to parallel or sequential run) <img width="1578" alt="Screenshot 2025-02-11 at 12 14 30" src="https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41" /> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit bd13e82)
Contributor
💔 Some backports could not be created
Note: Successful backport PRs will be merged automatically after passing CI. Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Feb 11, 2025
…and uptate onboarding tests (#209761) (#210675) # Backport This will backport the following commits from `main` to `9.0`: - [[scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761)](#209761) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Dzmitry Lemechko","email":"dzmitry.lemechko@elastic.co"},"sourceCommit":{"committedDate":"2025-02-11T17:38:41Z","message":"[scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761)\n\n## Summary\r\n\r\n`@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test\r\nhelpers specifically designed to test `Observability` applications in\r\nKibana. All Oblt plugins should only import from `@kbn/scout-oblt`\r\n\r\nIts primary goal is to simplify the test development experience for\r\nteams working on `Observability` plugins by providing custom Playwright\r\nfixtures, page objects, and utilities tailored for Observability-related\r\ntesting scenarios.\r\n\r\nContributing:\r\n- when Fixture/Page Object is sharable across all Solutions and Platform\r\n(`fleetApi` fixture), it should be added in `@kbn/scout`\r\n- when Fixture/Page Object is Oblt-specific but is shared across tests\r\nunder the multiple plugins (`OnboardingHome` page), it should be added\r\nin `@kbn/scout-oblt`\r\n- when Fixture/Page Object is only used in a single plugin (`onboarding`\r\ninternal APIs ?), it should be added in this plugin.\r\n\r\nI also re-worked existing tests with few ideas in mind:\r\n- Scout is **e2e testing tool** and should target primary e2e test\r\nscenarios; We have _API integration tests_ to test multiple short\r\nscenarios for APIs behavior (response, status code) and _jest/React\r\ntesting library_ to test components in isolation (elements rendering,\r\nfields validation). Doing all the testing with e2e tool like Playwright\r\nwill dramatically affect cost efficiency and stability of tests, but\r\nalso slows overall CI execution and PRs delivery. The goal is to follow\r\ntesting pyramid and keep in mind its principles.\r\n- We on purpose spin up new browser context for each `test` block to\r\nmake sure our **tests are independent**. Having too many short `test`\r\nblocks in the file significantly slows down the execution: every block\r\ntriggers browser context, saml authentication, adding/removing Fleet\r\nintegrations (each call up to 2 seconds) and other beforeEach/afterEach\r\nhooks. Real browser-based testing is expensive. It is not about putting\r\nevery step into 1 `test` block, but also not a Jest unit-test-style\r\ndesign. When it is possible to group similar actions on the same page\r\nand if it is a part of the same user flow - we should do it. It also\r\ndoesn't bring the testing value repeating the same UI steps multiple\r\ntimes in different scenarios. _Our CI costs are critical to cut when it\r\nis possible_\r\n- Avoid **nesting describe** blocks: it complicates test readability and\r\nalso complicates for CI bot to properly skip the failing block (it will\r\nskip the top level one). We encourage **Scout parallel test execution**\r\nbased on running test spec files in multiple workers, not the `test`\r\nblocks within the same file. Having too many `test` blocks in the same\r\nfile will be slowly run in the single thread and in case of flakiness,\r\nit means Team lose more test coverage than they probably expect.\r\n\r\nBefore (**59** test blocks - **8-8.5 min** per distro):\r\n<img width=\"1709\" alt=\"Screenshot 2025-02-08 at 18 01 40\"\r\nsrc=\"https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab\"\r\n/>\r\n\r\nAfter (**15** test blocks - **3.5-4 min** per distro):\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-10 at 18 14 42\"\r\nsrc=\"https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120\"\r\n/>\r\n\r\nFor reviewers: updated tests are possible to run in 2 parallel workers\r\nagainst the same Kibana/ES instance and run time is dropping to **2.5-3\r\nmin** 🚀 . It is up to UX-Logs team to decide if you want to keep\r\nparallel run (new tests can be added either to parallel or sequential\r\nrun)\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-11 at 12 14 30\"\r\nsrc=\"https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41\"\r\n/>\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"bd13e829498032c07bf8490f770a563f34e9f856","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:version","test:scout","v9.1.0","v8.19.0"],"title":"[scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests","number":209761,"url":"https://github.com/elastic/kibana/pull/209761","mergeCommit":{"message":"[scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761)\n\n## Summary\r\n\r\n`@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test\r\nhelpers specifically designed to test `Observability` applications in\r\nKibana. All Oblt plugins should only import from `@kbn/scout-oblt`\r\n\r\nIts primary goal is to simplify the test development experience for\r\nteams working on `Observability` plugins by providing custom Playwright\r\nfixtures, page objects, and utilities tailored for Observability-related\r\ntesting scenarios.\r\n\r\nContributing:\r\n- when Fixture/Page Object is sharable across all Solutions and Platform\r\n(`fleetApi` fixture), it should be added in `@kbn/scout`\r\n- when Fixture/Page Object is Oblt-specific but is shared across tests\r\nunder the multiple plugins (`OnboardingHome` page), it should be added\r\nin `@kbn/scout-oblt`\r\n- when Fixture/Page Object is only used in a single plugin (`onboarding`\r\ninternal APIs ?), it should be added in this plugin.\r\n\r\nI also re-worked existing tests with few ideas in mind:\r\n- Scout is **e2e testing tool** and should target primary e2e test\r\nscenarios; We have _API integration tests_ to test multiple short\r\nscenarios for APIs behavior (response, status code) and _jest/React\r\ntesting library_ to test components in isolation (elements rendering,\r\nfields validation). Doing all the testing with e2e tool like Playwright\r\nwill dramatically affect cost efficiency and stability of tests, but\r\nalso slows overall CI execution and PRs delivery. The goal is to follow\r\ntesting pyramid and keep in mind its principles.\r\n- We on purpose spin up new browser context for each `test` block to\r\nmake sure our **tests are independent**. Having too many short `test`\r\nblocks in the file significantly slows down the execution: every block\r\ntriggers browser context, saml authentication, adding/removing Fleet\r\nintegrations (each call up to 2 seconds) and other beforeEach/afterEach\r\nhooks. Real browser-based testing is expensive. It is not about putting\r\nevery step into 1 `test` block, but also not a Jest unit-test-style\r\ndesign. When it is possible to group similar actions on the same page\r\nand if it is a part of the same user flow - we should do it. It also\r\ndoesn't bring the testing value repeating the same UI steps multiple\r\ntimes in different scenarios. _Our CI costs are critical to cut when it\r\nis possible_\r\n- Avoid **nesting describe** blocks: it complicates test readability and\r\nalso complicates for CI bot to properly skip the failing block (it will\r\nskip the top level one). We encourage **Scout parallel test execution**\r\nbased on running test spec files in multiple workers, not the `test`\r\nblocks within the same file. Having too many `test` blocks in the same\r\nfile will be slowly run in the single thread and in case of flakiness,\r\nit means Team lose more test coverage than they probably expect.\r\n\r\nBefore (**59** test blocks - **8-8.5 min** per distro):\r\n<img width=\"1709\" alt=\"Screenshot 2025-02-08 at 18 01 40\"\r\nsrc=\"https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab\"\r\n/>\r\n\r\nAfter (**15** test blocks - **3.5-4 min** per distro):\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-10 at 18 14 42\"\r\nsrc=\"https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120\"\r\n/>\r\n\r\nFor reviewers: updated tests are possible to run in 2 parallel workers\r\nagainst the same Kibana/ES instance and run time is dropping to **2.5-3\r\nmin** 🚀 . It is up to UX-Logs team to decide if you want to keep\r\nparallel run (new tests can be added either to parallel or sequential\r\nrun)\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-11 at 12 14 30\"\r\nsrc=\"https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41\"\r\n/>\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"bd13e829498032c07bf8490f770a563f34e9f856"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.x"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/209761","number":209761,"mergeCommit":{"message":"[scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761)\n\n## Summary\r\n\r\n`@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test\r\nhelpers specifically designed to test `Observability` applications in\r\nKibana. All Oblt plugins should only import from `@kbn/scout-oblt`\r\n\r\nIts primary goal is to simplify the test development experience for\r\nteams working on `Observability` plugins by providing custom Playwright\r\nfixtures, page objects, and utilities tailored for Observability-related\r\ntesting scenarios.\r\n\r\nContributing:\r\n- when Fixture/Page Object is sharable across all Solutions and Platform\r\n(`fleetApi` fixture), it should be added in `@kbn/scout`\r\n- when Fixture/Page Object is Oblt-specific but is shared across tests\r\nunder the multiple plugins (`OnboardingHome` page), it should be added\r\nin `@kbn/scout-oblt`\r\n- when Fixture/Page Object is only used in a single plugin (`onboarding`\r\ninternal APIs ?), it should be added in this plugin.\r\n\r\nI also re-worked existing tests with few ideas in mind:\r\n- Scout is **e2e testing tool** and should target primary e2e test\r\nscenarios; We have _API integration tests_ to test multiple short\r\nscenarios for APIs behavior (response, status code) and _jest/React\r\ntesting library_ to test components in isolation (elements rendering,\r\nfields validation). Doing all the testing with e2e tool like Playwright\r\nwill dramatically affect cost efficiency and stability of tests, but\r\nalso slows overall CI execution and PRs delivery. The goal is to follow\r\ntesting pyramid and keep in mind its principles.\r\n- We on purpose spin up new browser context for each `test` block to\r\nmake sure our **tests are independent**. Having too many short `test`\r\nblocks in the file significantly slows down the execution: every block\r\ntriggers browser context, saml authentication, adding/removing Fleet\r\nintegrations (each call up to 2 seconds) and other beforeEach/afterEach\r\nhooks. Real browser-based testing is expensive. It is not about putting\r\nevery step into 1 `test` block, but also not a Jest unit-test-style\r\ndesign. When it is possible to group similar actions on the same page\r\nand if it is a part of the same user flow - we should do it. It also\r\ndoesn't bring the testing value repeating the same UI steps multiple\r\ntimes in different scenarios. _Our CI costs are critical to cut when it\r\nis possible_\r\n- Avoid **nesting describe** blocks: it complicates test readability and\r\nalso complicates for CI bot to properly skip the failing block (it will\r\nskip the top level one). We encourage **Scout parallel test execution**\r\nbased on running test spec files in multiple workers, not the `test`\r\nblocks within the same file. Having too many `test` blocks in the same\r\nfile will be slowly run in the single thread and in case of flakiness,\r\nit means Team lose more test coverage than they probably expect.\r\n\r\nBefore (**59** test blocks - **8-8.5 min** per distro):\r\n<img width=\"1709\" alt=\"Screenshot 2025-02-08 at 18 01 40\"\r\nsrc=\"https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab\"\r\n/>\r\n\r\nAfter (**15** test blocks - **3.5-4 min** per distro):\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-10 at 18 14 42\"\r\nsrc=\"https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120\"\r\n/>\r\n\r\nFor reviewers: updated tests are possible to run in 2 parallel workers\r\nagainst the same Kibana/ES instance and run time is dropping to **2.5-3\r\nmin** 🚀 . It is up to UX-Logs team to decide if you want to keep\r\nparallel run (new tests can be added either to parallel or sequential\r\nrun)\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-11 at 12 14 30\"\r\nsrc=\"https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41\"\r\n/>\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"bd13e829498032c07bf8490f770a563f34e9f856"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
kapral18
added a commit
to agusruidiazgd/kibana
that referenced
this pull request
Feb 11, 2025
…on-206439 * main: (402 commits) [Search]: Fix Number type field to have correct property (elastic#210462) Change filter for rule monitoring gaps (elastic#209983) Update Logs Explorer deprecation messages (elastic#201307) [APM] Remove `error.id` in `getErrorGroupMainStatistics` query as it's not used (elastic#210613) [Embeddable] Fix presentation panel styles (elastic#210113) [ci] enable Scout reporter for on-merge-unsupported-ftrs (elastic#210627) [Fix][Synonyms UI]Add navigation link to the Detail breadcrumb. (elastic#209574) chore(dep): bump `store2` from `2.12.0` to `2.14.4` (elastic#210530) [scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (elastic#209761) [Cloud Security] Asset Inventory table flyout controls (elastic#208452) [ML] Fix model deployment check in file uploader (elastic#209585) Updates archive again (elastic#209828) [Security Solution] Added concurrency limits and request throttling to prebuilt rule routes (elastic#209551) [Search] [Onboarding] Update search api to use EventEmitter instead of Provider (elastic#209784) [maps] lazy load map actions (elastic#210252) [Cloud Security] Adding telemetry collection condition based on render condition (elastic#208758) [Solution nav] Use flyout for Stack Management in Search and Observability solutions (elastic#208632) [Search] Fix Add Inference Endpoint API call (elastic#210243) [Agentless Connectors] Integration overview panel (elastic#210222) [Lens] Restore dynamic colouring by value for Last value agg (elastic#209110) ...
This was referenced Feb 12, 2025
Contributor
Author
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
dmlemeshko
added a commit
to dmlemeshko/kibana
that referenced
this pull request
Feb 14, 2025
…rding tests (elastic#209761) ## Summary `@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test helpers specifically designed to test `Observability` applications in Kibana. All Oblt plugins should only import from `@kbn/scout-oblt` Its primary goal is to simplify the test development experience for teams working on `Observability` plugins by providing custom Playwright fixtures, page objects, and utilities tailored for Observability-related testing scenarios. Contributing: - when Fixture/Page Object is sharable across all Solutions and Platform (`fleetApi` fixture), it should be added in `@kbn/scout` - when Fixture/Page Object is Oblt-specific but is shared across tests under the multiple plugins (`OnboardingHome` page), it should be added in `@kbn/scout-oblt` - when Fixture/Page Object is only used in a single plugin (`onboarding` internal APIs ?), it should be added in this plugin. I also re-worked existing tests with few ideas in mind: - Scout is **e2e testing tool** and should target primary e2e test scenarios; We have _API integration tests_ to test multiple short scenarios for APIs behavior (response, status code) and _jest/React testing library_ to test components in isolation (elements rendering, fields validation). Doing all the testing with e2e tool like Playwright will dramatically affect cost efficiency and stability of tests, but also slows overall CI execution and PRs delivery. The goal is to follow testing pyramid and keep in mind its principles. - We on purpose spin up new browser context for each `test` block to make sure our **tests are independent**. Having too many short `test` blocks in the file significantly slows down the execution: every block triggers browser context, saml authentication, adding/removing Fleet integrations (each call up to 2 seconds) and other beforeEach/afterEach hooks. Real browser-based testing is expensive. It is not about putting every step into 1 `test` block, but also not a Jest unit-test-style design. When it is possible to group similar actions on the same page and if it is a part of the same user flow - we should do it. It also doesn't bring the testing value repeating the same UI steps multiple times in different scenarios. _Our CI costs are critical to cut when it is possible_ - Avoid **nesting describe** blocks: it complicates test readability and also complicates for CI bot to properly skip the failing block (it will skip the top level one). We encourage **Scout parallel test execution** based on running test spec files in multiple workers, not the `test` blocks within the same file. Having too many `test` blocks in the same file will be slowly run in the single thread and in case of flakiness, it means Team lose more test coverage than they probably expect. Before (**59** test blocks - **8-8.5 min** per distro): <img width="1709" alt="Screenshot 2025-02-08 at 18 01 40" src="https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab" /> After (**15** test blocks - **3.5-4 min** per distro): <img width="1578" alt="Screenshot 2025-02-10 at 18 14 42" src="https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120" /> For reviewers: updated tests are possible to run in 2 parallel workers against the same Kibana/ES instance and run time is dropping to **2.5-3 min** 🚀 . It is up to UX-Logs team to decide if you want to keep parallel run (new tests can be added either to parallel or sequential run) <img width="1578" alt="Screenshot 2025-02-11 at 12 14 30" src="https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41" /> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit bd13e82) # Conflicts: # .buildkite/scripts/steps/functional/scout_ui_tests.sh # .github/CODEOWNERS # x-pack/platform/packages/private/ml/kibana_theme/tsconfig.json # x-pack/platform/plugins/shared/logs_shared/public/containers/logs/log_highlights/index.ts # x-pack/solutions/observability/packages/kbn-scout-oblt/src/playwright/index.ts # x-pack/solutions/observability/packages/kbn-scout-oblt/src/playwright/page_objects/onboarding_home.ts # x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/README.md # x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/fixtures/index.ts # x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/fixtures/page_objects/index.ts # x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/fixtures/page_objects/onboarding_home.ts # x-pack/solutions/search/plugins/enterprise_search/public/applications/app_search/components/query_tester/i18n.ts
This was referenced Feb 14, 2025
dmlemeshko
added a commit
to dmlemeshko/kibana
that referenced
this pull request
Feb 17, 2025
…rding tests (elastic#209761) ## Summary `@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test helpers specifically designed to test `Observability` applications in Kibana. All Oblt plugins should only import from `@kbn/scout-oblt` Its primary goal is to simplify the test development experience for teams working on `Observability` plugins by providing custom Playwright fixtures, page objects, and utilities tailored for Observability-related testing scenarios. Contributing: - when Fixture/Page Object is sharable across all Solutions and Platform (`fleetApi` fixture), it should be added in `@kbn/scout` - when Fixture/Page Object is Oblt-specific but is shared across tests under the multiple plugins (`OnboardingHome` page), it should be added in `@kbn/scout-oblt` - when Fixture/Page Object is only used in a single plugin (`onboarding` internal APIs ?), it should be added in this plugin. I also re-worked existing tests with few ideas in mind: - Scout is **e2e testing tool** and should target primary e2e test scenarios; We have _API integration tests_ to test multiple short scenarios for APIs behavior (response, status code) and _jest/React testing library_ to test components in isolation (elements rendering, fields validation). Doing all the testing with e2e tool like Playwright will dramatically affect cost efficiency and stability of tests, but also slows overall CI execution and PRs delivery. The goal is to follow testing pyramid and keep in mind its principles. - We on purpose spin up new browser context for each `test` block to make sure our **tests are independent**. Having too many short `test` blocks in the file significantly slows down the execution: every block triggers browser context, saml authentication, adding/removing Fleet integrations (each call up to 2 seconds) and other beforeEach/afterEach hooks. Real browser-based testing is expensive. It is not about putting every step into 1 `test` block, but also not a Jest unit-test-style design. When it is possible to group similar actions on the same page and if it is a part of the same user flow - we should do it. It also doesn't bring the testing value repeating the same UI steps multiple times in different scenarios. _Our CI costs are critical to cut when it is possible_ - Avoid **nesting describe** blocks: it complicates test readability and also complicates for CI bot to properly skip the failing block (it will skip the top level one). We encourage **Scout parallel test execution** based on running test spec files in multiple workers, not the `test` blocks within the same file. Having too many `test` blocks in the same file will be slowly run in the single thread and in case of flakiness, it means Team lose more test coverage than they probably expect. Before (**59** test blocks - **8-8.5 min** per distro): <img width="1709" alt="Screenshot 2025-02-08 at 18 01 40" src="https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab" /> After (**15** test blocks - **3.5-4 min** per distro): <img width="1578" alt="Screenshot 2025-02-10 at 18 14 42" src="https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120" /> For reviewers: updated tests are possible to run in 2 parallel workers against the same Kibana/ES instance and run time is dropping to **2.5-3 min** 🚀 . It is up to UX-Logs team to decide if you want to keep parallel run (new tests can be added either to parallel or sequential run) <img width="1578" alt="Screenshot 2025-02-11 at 12 14 30" src="https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41" /> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit bd13e82) # Conflicts: # .buildkite/scripts/steps/functional/scout_ui_tests.sh # .github/CODEOWNERS # x-pack/solutions/observability/plugins/observability_onboarding/ui_tests/tests/custom_logs/install_elastic_agent.spec.ts
Contributor
Author
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
dmlemeshko
added a commit
that referenced
this pull request
Feb 17, 2025
… onboarding tests (#209761) (#211490) # Backport This will backport the following commits from `main` to `8.x`: - [[scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761)](#209761) <!--- Backport version: 9.6.4 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Dzmitry Lemechko","email":"dzmitry.lemechko@elastic.co"},"sourceCommit":{"committedDate":"2025-02-11T17:38:41Z","message":"[scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761)\n\n## Summary\r\n\r\n`@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test\r\nhelpers specifically designed to test `Observability` applications in\r\nKibana. All Oblt plugins should only import from `@kbn/scout-oblt`\r\n\r\nIts primary goal is to simplify the test development experience for\r\nteams working on `Observability` plugins by providing custom Playwright\r\nfixtures, page objects, and utilities tailored for Observability-related\r\ntesting scenarios.\r\n\r\nContributing:\r\n- when Fixture/Page Object is sharable across all Solutions and Platform\r\n(`fleetApi` fixture), it should be added in `@kbn/scout`\r\n- when Fixture/Page Object is Oblt-specific but is shared across tests\r\nunder the multiple plugins (`OnboardingHome` page), it should be added\r\nin `@kbn/scout-oblt`\r\n- when Fixture/Page Object is only used in a single plugin (`onboarding`\r\ninternal APIs ?), it should be added in this plugin.\r\n\r\nI also re-worked existing tests with few ideas in mind:\r\n- Scout is **e2e testing tool** and should target primary e2e test\r\nscenarios; We have _API integration tests_ to test multiple short\r\nscenarios for APIs behavior (response, status code) and _jest/React\r\ntesting library_ to test components in isolation (elements rendering,\r\nfields validation). Doing all the testing with e2e tool like Playwright\r\nwill dramatically affect cost efficiency and stability of tests, but\r\nalso slows overall CI execution and PRs delivery. The goal is to follow\r\ntesting pyramid and keep in mind its principles.\r\n- We on purpose spin up new browser context for each `test` block to\r\nmake sure our **tests are independent**. Having too many short `test`\r\nblocks in the file significantly slows down the execution: every block\r\ntriggers browser context, saml authentication, adding/removing Fleet\r\nintegrations (each call up to 2 seconds) and other beforeEach/afterEach\r\nhooks. Real browser-based testing is expensive. It is not about putting\r\nevery step into 1 `test` block, but also not a Jest unit-test-style\r\ndesign. When it is possible to group similar actions on the same page\r\nand if it is a part of the same user flow - we should do it. It also\r\ndoesn't bring the testing value repeating the same UI steps multiple\r\ntimes in different scenarios. _Our CI costs are critical to cut when it\r\nis possible_\r\n- Avoid **nesting describe** blocks: it complicates test readability and\r\nalso complicates for CI bot to properly skip the failing block (it will\r\nskip the top level one). We encourage **Scout parallel test execution**\r\nbased on running test spec files in multiple workers, not the `test`\r\nblocks within the same file. Having too many `test` blocks in the same\r\nfile will be slowly run in the single thread and in case of flakiness,\r\nit means Team lose more test coverage than they probably expect.\r\n\r\nBefore (**59** test blocks - **8-8.5 min** per distro):\r\n<img width=\"1709\" alt=\"Screenshot 2025-02-08 at 18 01 40\"\r\nsrc=\"https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab\"\r\n/>\r\n\r\nAfter (**15** test blocks - **3.5-4 min** per distro):\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-10 at 18 14 42\"\r\nsrc=\"https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120\"\r\n/>\r\n\r\nFor reviewers: updated tests are possible to run in 2 parallel workers\r\nagainst the same Kibana/ES instance and run time is dropping to **2.5-3\r\nmin** 🚀 . It is up to UX-Logs team to decide if you want to keep\r\nparallel run (new tests can be added either to parallel or sequential\r\nrun)\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-11 at 12 14 30\"\r\nsrc=\"https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41\"\r\n/>\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"bd13e829498032c07bf8490f770a563f34e9f856","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:version","test:scout","v9.1.0","v8.19.0"],"title":"[scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests","number":209761,"url":"https://github.com/elastic/kibana/pull/209761","mergeCommit":{"message":"[scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761)\n\n## Summary\r\n\r\n`@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test\r\nhelpers specifically designed to test `Observability` applications in\r\nKibana. All Oblt plugins should only import from `@kbn/scout-oblt`\r\n\r\nIts primary goal is to simplify the test development experience for\r\nteams working on `Observability` plugins by providing custom Playwright\r\nfixtures, page objects, and utilities tailored for Observability-related\r\ntesting scenarios.\r\n\r\nContributing:\r\n- when Fixture/Page Object is sharable across all Solutions and Platform\r\n(`fleetApi` fixture), it should be added in `@kbn/scout`\r\n- when Fixture/Page Object is Oblt-specific but is shared across tests\r\nunder the multiple plugins (`OnboardingHome` page), it should be added\r\nin `@kbn/scout-oblt`\r\n- when Fixture/Page Object is only used in a single plugin (`onboarding`\r\ninternal APIs ?), it should be added in this plugin.\r\n\r\nI also re-worked existing tests with few ideas in mind:\r\n- Scout is **e2e testing tool** and should target primary e2e test\r\nscenarios; We have _API integration tests_ to test multiple short\r\nscenarios for APIs behavior (response, status code) and _jest/React\r\ntesting library_ to test components in isolation (elements rendering,\r\nfields validation). Doing all the testing with e2e tool like Playwright\r\nwill dramatically affect cost efficiency and stability of tests, but\r\nalso slows overall CI execution and PRs delivery. The goal is to follow\r\ntesting pyramid and keep in mind its principles.\r\n- We on purpose spin up new browser context for each `test` block to\r\nmake sure our **tests are independent**. Having too many short `test`\r\nblocks in the file significantly slows down the execution: every block\r\ntriggers browser context, saml authentication, adding/removing Fleet\r\nintegrations (each call up to 2 seconds) and other beforeEach/afterEach\r\nhooks. Real browser-based testing is expensive. It is not about putting\r\nevery step into 1 `test` block, but also not a Jest unit-test-style\r\ndesign. When it is possible to group similar actions on the same page\r\nand if it is a part of the same user flow - we should do it. It also\r\ndoesn't bring the testing value repeating the same UI steps multiple\r\ntimes in different scenarios. _Our CI costs are critical to cut when it\r\nis possible_\r\n- Avoid **nesting describe** blocks: it complicates test readability and\r\nalso complicates for CI bot to properly skip the failing block (it will\r\nskip the top level one). We encourage **Scout parallel test execution**\r\nbased on running test spec files in multiple workers, not the `test`\r\nblocks within the same file. Having too many `test` blocks in the same\r\nfile will be slowly run in the single thread and in case of flakiness,\r\nit means Team lose more test coverage than they probably expect.\r\n\r\nBefore (**59** test blocks - **8-8.5 min** per distro):\r\n<img width=\"1709\" alt=\"Screenshot 2025-02-08 at 18 01 40\"\r\nsrc=\"https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab\"\r\n/>\r\n\r\nAfter (**15** test blocks - **3.5-4 min** per distro):\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-10 at 18 14 42\"\r\nsrc=\"https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120\"\r\n/>\r\n\r\nFor reviewers: updated tests are possible to run in 2 parallel workers\r\nagainst the same Kibana/ES instance and run time is dropping to **2.5-3\r\nmin** 🚀 . It is up to UX-Logs team to decide if you want to keep\r\nparallel run (new tests can be added either to parallel or sequential\r\nrun)\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-11 at 12 14 30\"\r\nsrc=\"https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41\"\r\n/>\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"bd13e829498032c07bf8490f770a563f34e9f856"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/210675","number":210675,"state":"MERGED","mergeCommit":{"sha":"187b2307220a1f54776b0b3b05d131b0e75b6e03","message":"[9.0] [scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761) (#210675)\n\n# Backport\n\nThis will backport the following commits from `main` to `9.0`:\n- [[scout] adding test helper `@kbn/scout-oblt` package and\nuptate onboarding tests\n(#209761)](https://github.com/elastic/kibana/pull/209761)\n\n<!--- Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT [{\"author\":{\"name\":\"Dzmitry\nLemechko\",\"email\":\"dzmitry.lemechko@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2025-02-11T17:38:41Z\",\"message\":\"[scout]\nadding test helper `@kbn/scout-oblt` package and uptate onboarding tests\n(#209761)\\n\\n## Summary\\r\\n\\r\\n`@kbn/scout-oblt` is a test library that\nextends `@kbn/scout` with test\\r\\nhelpers specifically designed to test\n`Observability` applications in\\r\\nKibana. All Oblt plugins should only\nimport from `@kbn/scout-oblt`\\r\\n\\r\\nIts primary goal is to simplify the\ntest development experience for\\r\\nteams working on `Observability`\nplugins by providing custom Playwright\\r\\nfixtures, page objects, and\nutilities tailored for Observability-related\\r\\ntesting\nscenarios.\\r\\n\\r\\nContributing:\\r\\n- when Fixture/Page Object is\nsharable across all Solutions and Platform\\r\\n(`fleetApi` fixture), it\nshould be added in `@kbn/scout`\\r\\n- when Fixture/Page Object is\nOblt-specific but is shared across tests\\r\\nunder the multiple plugins\n(`OnboardingHome` page), it should be added\\r\\nin `@kbn/scout-oblt`\\r\\n-\nwhen Fixture/Page Object is only used in a single plugin\n(`onboarding`\\r\\ninternal APIs ?), it should be added in this\nplugin.\\r\\n\\r\\nI also re-worked existing tests with few ideas in\nmind:\\r\\n- Scout is **e2e testing tool** and should target primary e2e\ntest\\r\\nscenarios; We have _API integration tests_ to test multiple\nshort\\r\\nscenarios for APIs behavior (response, status code) and\n_jest/React\\r\\ntesting library_ to test components in isolation\n(elements rendering,\\r\\nfields validation). Doing all the testing with\ne2e tool like Playwright\\r\\nwill dramatically affect cost efficiency and\nstability of tests, but\\r\\nalso slows overall CI execution and PRs\ndelivery. The goal is to follow\\r\\ntesting pyramid and keep in mind its\nprinciples.\\r\\n- We on purpose spin up new browser context for each\n`test` block to\\r\\nmake sure our **tests are independent**. Having too\nmany short `test`\\r\\nblocks in the file significantly slows down the\nexecution: every block\\r\\ntriggers browser context, saml authentication,\nadding/removing Fleet\\r\\nintegrations (each call up to 2 seconds) and\nother beforeEach/afterEach\\r\\nhooks. Real browser-based testing is\nexpensive. It is not about putting\\r\\nevery step into 1 `test` block,\nbut also not a Jest unit-test-style\\r\\ndesign. When it is possible to\ngroup similar actions on the same page\\r\\nand if it is a part of the\nsame user flow - we should do it. It also\\r\\ndoesn't bring the testing\nvalue repeating the same UI steps multiple\\r\\ntimes in different\nscenarios. _Our CI costs are critical to cut when it\\r\\nis\npossible_\\r\\n- Avoid **nesting describe** blocks: it complicates test\nreadability and\\r\\nalso complicates for CI bot to properly skip the\nfailing block (it will\\r\\nskip the top level one). We encourage **Scout\nparallel test execution**\\r\\nbased on running test spec files in\nmultiple workers, not the `test`\\r\\nblocks within the same file. Having\ntoo many `test` blocks in the same\\r\\nfile will be slowly run in the\nsingle thread and in case of flakiness,\\r\\nit means Team lose more test\ncoverage than they probably expect.\\r\\n\\r\\nBefore (**59** test blocks -\n**8-8.5 min** per distro):\\r\\n<img width=\\\"1709\\\" alt=\\\"Screenshot\n2025-02-08 at 18 01\n40\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab\\\"\\r\\n/>\\r\\n\\r\\nAfter\n(**15** test blocks - **3.5-4 min** per distro):\\r\\n<img width=\\\"1578\\\"\nalt=\\\"Screenshot 2025-02-10 at 18 14\n42\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120\\\"\\r\\n/>\\r\\n\\r\\nFor\nreviewers: updated tests are possible to run in 2 parallel\nworkers\\r\\nagainst the same Kibana/ES instance and run time is dropping\nto **2.5-3\\r\\nmin** 🚀 . It is up to UX-Logs team to decide if you want\nto keep\\r\\nparallel run (new tests can be added either to parallel or\nsequential\\r\\nrun)\\r\\n<img width=\\\"1578\\\" alt=\\\"Screenshot 2025-02-11 at\n12 14\n30\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41\\\"\\r\\n/>\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by:\nkibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\",\"sha\":\"bd13e829498032c07bf8490f770a563f34e9f856\",\"branchLabelMapping\":{\"^v9.1.0$\":\"main\",\"^v8.19.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:skip\",\"v9.0.0\",\"backport:version\",\"test:scout\",\"v9.1.0\",\"v8.19.0\"],\"title\":\"[scout]\nadding test helper `@kbn/scout-oblt` package and uptate onboarding\ntests\",\"number\":209761,\"url\":\"https://github.com/elastic/kibana/pull/209761\",\"mergeCommit\":{\"message\":\"[scout]\nadding test helper `@kbn/scout-oblt` package and uptate onboarding tests\n(#209761)\\n\\n## Summary\\r\\n\\r\\n`@kbn/scout-oblt` is a test library that\nextends `@kbn/scout` with test\\r\\nhelpers specifically designed to test\n`Observability` applications in\\r\\nKibana. All Oblt plugins should only\nimport from `@kbn/scout-oblt`\\r\\n\\r\\nIts primary goal is to simplify the\ntest development experience for\\r\\nteams working on `Observability`\nplugins by providing custom Playwright\\r\\nfixtures, page objects, and\nutilities tailored for Observability-related\\r\\ntesting\nscenarios.\\r\\n\\r\\nContributing:\\r\\n- when Fixture/Page Object is\nsharable across all Solutions and Platform\\r\\n(`fleetApi` fixture), it\nshould be added in `@kbn/scout`\\r\\n- when Fixture/Page Object is\nOblt-specific but is shared across tests\\r\\nunder the multiple plugins\n(`OnboardingHome` page), it should be added\\r\\nin `@kbn/scout-oblt`\\r\\n-\nwhen Fixture/Page Object is only used in a single plugin\n(`onboarding`\\r\\ninternal APIs ?), it should be added in this\nplugin.\\r\\n\\r\\nI also re-worked existing tests with few ideas in\nmind:\\r\\n- Scout is **e2e testing tool** and should target primary e2e\ntest\\r\\nscenarios; We have _API integration tests_ to test multiple\nshort\\r\\nscenarios for APIs behavior (response, status code) and\n_jest/React\\r\\ntesting library_ to test components in isolation\n(elements rendering,\\r\\nfields validation). Doing all the testing with\ne2e tool like Playwright\\r\\nwill dramatically affect cost efficiency and\nstability of tests, but\\r\\nalso slows overall CI execution and PRs\ndelivery. The goal is to follow\\r\\ntesting pyramid and keep in mind its\nprinciples.\\r\\n- We on purpose spin up new browser context for each\n`test` block to\\r\\nmake sure our **tests are independent**. Having too\nmany short `test`\\r\\nblocks in the file significantly slows down the\nexecution: every block\\r\\ntriggers browser context, saml authentication,\nadding/removing Fleet\\r\\nintegrations (each call up to 2 seconds) and\nother beforeEach/afterEach\\r\\nhooks. Real browser-based testing is\nexpensive. It is not about putting\\r\\nevery step into 1 `test` block,\nbut also not a Jest unit-test-style\\r\\ndesign. When it is possible to\ngroup similar actions on the same page\\r\\nand if it is a part of the\nsame user flow - we should do it. It also\\r\\ndoesn't bring the testing\nvalue repeating the same UI steps multiple\\r\\ntimes in different\nscenarios. _Our CI costs are critical to cut when it\\r\\nis\npossible_\\r\\n- Avoid **nesting describe** blocks: it complicates test\nreadability and\\r\\nalso complicates for CI bot to properly skip the\nfailing block (it will\\r\\nskip the top level one). We encourage **Scout\nparallel test execution**\\r\\nbased on running test spec files in\nmultiple workers, not the `test`\\r\\nblocks within the same file. Having\ntoo many `test` blocks in the same\\r\\nfile will be slowly run in the\nsingle thread and in case of flakiness,\\r\\nit means Team lose more test\ncoverage than they probably expect.\\r\\n\\r\\nBefore (**59** test blocks -\n**8-8.5 min** per distro):\\r\\n<img width=\\\"1709\\\" alt=\\\"Screenshot\n2025-02-08 at 18 01\n40\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab\\\"\\r\\n/>\\r\\n\\r\\nAfter\n(**15** test blocks - **3.5-4 min** per distro):\\r\\n<img width=\\\"1578\\\"\nalt=\\\"Screenshot 2025-02-10 at 18 14\n42\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120\\\"\\r\\n/>\\r\\n\\r\\nFor\nreviewers: updated tests are possible to run in 2 parallel\nworkers\\r\\nagainst the same Kibana/ES instance and run time is dropping\nto **2.5-3\\r\\nmin** 🚀 . It is up to UX-Logs team to decide if you want\nto keep\\r\\nparallel run (new tests can be added either to parallel or\nsequential\\r\\nrun)\\r\\n<img width=\\\"1578\\\" alt=\\\"Screenshot 2025-02-11 at\n12 14\n30\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41\\\"\\r\\n/>\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by:\nkibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\",\"sha\":\"bd13e829498032c07bf8490f770a563f34e9f856\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"9.0\",\"8.x\"],\"targetPullRequestStates\":[{\"branch\":\"9.0\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"main\",\"label\":\"v9.1.0\",\"branchLabelMappingKey\":\"^v9.1.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/209761\",\"number\":209761,\"mergeCommit\":{\"message\":\"[scout]\nadding test helper `@kbn/scout-oblt` package and uptate onboarding tests\n(#209761)\\n\\n## Summary\\r\\n\\r\\n`@kbn/scout-oblt` is a test library that\nextends `@kbn/scout` with test\\r\\nhelpers specifically designed to test\n`Observability` applications in\\r\\nKibana. All Oblt plugins should only\nimport from `@kbn/scout-oblt`\\r\\n\\r\\nIts primary goal is to simplify the\ntest development experience for\\r\\nteams working on `Observability`\nplugins by providing custom Playwright\\r\\nfixtures, page objects, and\nutilities tailored for Observability-related\\r\\ntesting\nscenarios.\\r\\n\\r\\nContributing:\\r\\n- when Fixture/Page Object is\nsharable across all Solutions and Platform\\r\\n(`fleetApi` fixture), it\nshould be added in `@kbn/scout`\\r\\n- when Fixture/Page Object is\nOblt-specific but is shared across tests\\r\\nunder the multiple plugins\n(`OnboardingHome` page), it should be added\\r\\nin `@kbn/scout-oblt`\\r\\n-\nwhen Fixture/Page Object is only used in a single plugin\n(`onboarding`\\r\\ninternal APIs ?), it should be added in this\nplugin.\\r\\n\\r\\nI also re-worked existing tests with few ideas in\nmind:\\r\\n- Scout is **e2e testing tool** and should target primary e2e\ntest\\r\\nscenarios; We have _API integration tests_ to test multiple\nshort\\r\\nscenarios for APIs behavior (response, status code) and\n_jest/React\\r\\ntesting library_ to test components in isolation\n(elements rendering,\\r\\nfields validation). Doing all the testing with\ne2e tool like Playwright\\r\\nwill dramatically affect cost efficiency and\nstability of tests, but\\r\\nalso slows overall CI execution and PRs\ndelivery. The goal is to follow\\r\\ntesting pyramid and keep in mind its\nprinciples.\\r\\n- We on purpose spin up new browser context for each\n`test` block to\\r\\nmake sure our **tests are independent**. Having too\nmany short `test`\\r\\nblocks in the file significantly slows down the\nexecution: every block\\r\\ntriggers browser context, saml authentication,\nadding/removing Fleet\\r\\nintegrations (each call up to 2 seconds) and\nother beforeEach/afterEach\\r\\nhooks. Real browser-based testing is\nexpensive. It is not about putting\\r\\nevery step into 1 `test` block,\nbut also not a Jest unit-test-style\\r\\ndesign. When it is possible to\ngroup similar actions on the same page\\r\\nand if it is a part of the\nsame user flow - we should do it. It also\\r\\ndoesn't bring the testing\nvalue repeating the same UI steps multiple\\r\\ntimes in different\nscenarios. _Our CI costs are critical to cut when it\\r\\nis\npossible_\\r\\n- Avoid **nesting describe** blocks: it complicates test\nreadability and\\r\\nalso complicates for CI bot to properly skip the\nfailing block (it will\\r\\nskip the top level one). We encourage **Scout\nparallel test execution**\\r\\nbased on running test spec files in\nmultiple workers, not the `test`\\r\\nblocks within the same file. Having\ntoo many `test` blocks in the same\\r\\nfile will be slowly run in the\nsingle thread and in case of flakiness,\\r\\nit means Team lose more test\ncoverage than they probably expect.\\r\\n\\r\\nBefore (**59** test blocks -\n**8-8.5 min** per distro):\\r\\n<img width=\\\"1709\\\" alt=\\\"Screenshot\n2025-02-08 at 18 01\n40\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab\\\"\\r\\n/>\\r\\n\\r\\nAfter\n(**15** test blocks - **3.5-4 min** per distro):\\r\\n<img width=\\\"1578\\\"\nalt=\\\"Screenshot 2025-02-10 at 18 14\n42\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120\\\"\\r\\n/>\\r\\n\\r\\nFor\nreviewers: updated tests are possible to run in 2 parallel\nworkers\\r\\nagainst the same Kibana/ES instance and run time is dropping\nto **2.5-3\\r\\nmin** 🚀 . It is up to UX-Logs team to decide if you want\nto keep\\r\\nparallel run (new tests can be added either to parallel or\nsequential\\r\\nrun)\\r\\n<img width=\\\"1578\\\" alt=\\\"Screenshot 2025-02-11 at\n12 14\n30\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41\\\"\\r\\n/>\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by:\nkibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\",\"sha\":\"bd13e829498032c07bf8490f770a563f34e9f856\"}},{\"branch\":\"8.x\",\"label\":\"v8.19.0\",\"branchLabelMappingKey\":\"^v8.19.0$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>"}},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/209761","number":209761,"mergeCommit":{"message":"[scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761)\n\n## Summary\r\n\r\n`@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test\r\nhelpers specifically designed to test `Observability` applications in\r\nKibana. All Oblt plugins should only import from `@kbn/scout-oblt`\r\n\r\nIts primary goal is to simplify the test development experience for\r\nteams working on `Observability` plugins by providing custom Playwright\r\nfixtures, page objects, and utilities tailored for Observability-related\r\ntesting scenarios.\r\n\r\nContributing:\r\n- when Fixture/Page Object is sharable across all Solutions and Platform\r\n(`fleetApi` fixture), it should be added in `@kbn/scout`\r\n- when Fixture/Page Object is Oblt-specific but is shared across tests\r\nunder the multiple plugins (`OnboardingHome` page), it should be added\r\nin `@kbn/scout-oblt`\r\n- when Fixture/Page Object is only used in a single plugin (`onboarding`\r\ninternal APIs ?), it should be added in this plugin.\r\n\r\nI also re-worked existing tests with few ideas in mind:\r\n- Scout is **e2e testing tool** and should target primary e2e test\r\nscenarios; We have _API integration tests_ to test multiple short\r\nscenarios for APIs behavior (response, status code) and _jest/React\r\ntesting library_ to test components in isolation (elements rendering,\r\nfields validation). Doing all the testing with e2e tool like Playwright\r\nwill dramatically affect cost efficiency and stability of tests, but\r\nalso slows overall CI execution and PRs delivery. The goal is to follow\r\ntesting pyramid and keep in mind its principles.\r\n- We on purpose spin up new browser context for each `test` block to\r\nmake sure our **tests are independent**. Having too many short `test`\r\nblocks in the file significantly slows down the execution: every block\r\ntriggers browser context, saml authentication, adding/removing Fleet\r\nintegrations (each call up to 2 seconds) and other beforeEach/afterEach\r\nhooks. Real browser-based testing is expensive. It is not about putting\r\nevery step into 1 `test` block, but also not a Jest unit-test-style\r\ndesign. When it is possible to group similar actions on the same page\r\nand if it is a part of the same user flow - we should do it. It also\r\ndoesn't bring the testing value repeating the same UI steps multiple\r\ntimes in different scenarios. _Our CI costs are critical to cut when it\r\nis possible_\r\n- Avoid **nesting describe** blocks: it complicates test readability and\r\nalso complicates for CI bot to properly skip the failing block (it will\r\nskip the top level one). We encourage **Scout parallel test execution**\r\nbased on running test spec files in multiple workers, not the `test`\r\nblocks within the same file. Having too many `test` blocks in the same\r\nfile will be slowly run in the single thread and in case of flakiness,\r\nit means Team lose more test coverage than they probably expect.\r\n\r\nBefore (**59** test blocks - **8-8.5 min** per distro):\r\n<img width=\"1709\" alt=\"Screenshot 2025-02-08 at 18 01 40\"\r\nsrc=\"https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab\"\r\n/>\r\n\r\nAfter (**15** test blocks - **3.5-4 min** per distro):\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-10 at 18 14 42\"\r\nsrc=\"https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120\"\r\n/>\r\n\r\nFor reviewers: updated tests are possible to run in 2 parallel workers\r\nagainst the same Kibana/ES instance and run time is dropping to **2.5-3\r\nmin** 🚀 . It is up to UX-Logs team to decide if you want to keep\r\nparallel run (new tests can be added either to parallel or sequential\r\nrun)\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-11 at 12 14 30\"\r\nsrc=\"https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41\"\r\n/>\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"bd13e829498032c07bf8490f770a563f34e9f856"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@kbn/scout-obltis a test library that extends@kbn/scoutwith test helpers specifically designed to testObservabilityapplications in Kibana. All Oblt plugins should only import from@kbn/scout-obltIts primary goal is to simplify the test development experience for teams working on
Observabilityplugins by providing custom Playwright fixtures, page objects, and utilities tailored for Observability-related testing scenarios.Contributing:
fleetApifixture), it should be added in@kbn/scoutOnboardingHomepage), it should be added in@kbn/scout-obltonboardinginternal APIs ?), it should be added in this plugin.I also re-worked existing tests with few ideas in mind:
testblock to make sure our tests are independent. Having too many shorttestblocks in the file significantly slows down the execution: every block triggers browser context, saml authentication, adding/removing Fleet integrations (each call up to 2 seconds) and other beforeEach/afterEach hooks. Real browser-based testing is expensive. It is not about putting every step into 1testblock, but also not a Jest unit-test-style design. When it is possible to group similar actions on the same page and if it is a part of the same user flow - we should do it. It also doesn't bring the testing value repeating the same UI steps multiple times in different scenarios. Our CI costs are critical to cut when it is possibletestblocks within the same file. Having too manytestblocks in the same file will be slowly run in the single thread and in case of flakiness, it means Team lose more test coverage than they probably expect.Before (59 test blocks - 8-8.5 min per distro):
After (15 test blocks - 3.5-4 min per distro):
For reviewers: updated tests are possible to run in 2 parallel workers against the same Kibana/ES instance and run time is dropping to 2.5-3 min 🚀 . It is up to UX-Logs team to decide if you want to keep parallel run (new tests can be added either to parallel or sequential run)
