[9.0] [scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761)#210675
Merged
kibanamachine merged 1 commit intoelastic:9.0from Feb 11, 2025
Conversation
…rding tests (elastic#209761) ## Summary `@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test helpers specifically designed to test `Observability` applications in Kibana. All Oblt plugins should only import from `@kbn/scout-oblt` Its primary goal is to simplify the test development experience for teams working on `Observability` plugins by providing custom Playwright fixtures, page objects, and utilities tailored for Observability-related testing scenarios. Contributing: - when Fixture/Page Object is sharable across all Solutions and Platform (`fleetApi` fixture), it should be added in `@kbn/scout` - when Fixture/Page Object is Oblt-specific but is shared across tests under the multiple plugins (`OnboardingHome` page), it should be added in `@kbn/scout-oblt` - when Fixture/Page Object is only used in a single plugin (`onboarding` internal APIs ?), it should be added in this plugin. I also re-worked existing tests with few ideas in mind: - Scout is **e2e testing tool** and should target primary e2e test scenarios; We have _API integration tests_ to test multiple short scenarios for APIs behavior (response, status code) and _jest/React testing library_ to test components in isolation (elements rendering, fields validation). Doing all the testing with e2e tool like Playwright will dramatically affect cost efficiency and stability of tests, but also slows overall CI execution and PRs delivery. The goal is to follow testing pyramid and keep in mind its principles. - We on purpose spin up new browser context for each `test` block to make sure our **tests are independent**. Having too many short `test` blocks in the file significantly slows down the execution: every block triggers browser context, saml authentication, adding/removing Fleet integrations (each call up to 2 seconds) and other beforeEach/afterEach hooks. Real browser-based testing is expensive. It is not about putting every step into 1 `test` block, but also not a Jest unit-test-style design. When it is possible to group similar actions on the same page and if it is a part of the same user flow - we should do it. It also doesn't bring the testing value repeating the same UI steps multiple times in different scenarios. _Our CI costs are critical to cut when it is possible_ - Avoid **nesting describe** blocks: it complicates test readability and also complicates for CI bot to properly skip the failing block (it will skip the top level one). We encourage **Scout parallel test execution** based on running test spec files in multiple workers, not the `test` blocks within the same file. Having too many `test` blocks in the same file will be slowly run in the single thread and in case of flakiness, it means Team lose more test coverage than they probably expect. Before (**59** test blocks - **8-8.5 min** per distro): <img width="1709" alt="Screenshot 2025-02-08 at 18 01 40" src="https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab" /> After (**15** test blocks - **3.5-4 min** per distro): <img width="1578" alt="Screenshot 2025-02-10 at 18 14 42" src="https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120" /> For reviewers: updated tests are possible to run in 2 parallel workers against the same Kibana/ES instance and run time is dropping to **2.5-3 min** 🚀 . It is up to UX-Logs team to decide if you want to keep parallel run (new tests can be added either to parallel or sequential run) <img width="1578" alt="Screenshot 2025-02-11 at 12 14 30" src="https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41" /> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit bd13e82)
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
💚 Build Succeeded
Metrics [docs]Public APIs missing comments
Any counts in public APIs
Public APIs missing exports
Unknown metric groupsAPI count
ESLint disabled line counts
Total ESLint disabled count
Unreferenced deprecated APIs
cc @dmlemeshko |
dmlemeshko
added a commit
that referenced
this pull request
Feb 17, 2025
… onboarding tests (#209761) (#211490) # Backport This will backport the following commits from `main` to `8.x`: - [[scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761)](#209761) <!--- Backport version: 9.6.4 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Dzmitry Lemechko","email":"dzmitry.lemechko@elastic.co"},"sourceCommit":{"committedDate":"2025-02-11T17:38:41Z","message":"[scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761)\n\n## Summary\r\n\r\n`@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test\r\nhelpers specifically designed to test `Observability` applications in\r\nKibana. All Oblt plugins should only import from `@kbn/scout-oblt`\r\n\r\nIts primary goal is to simplify the test development experience for\r\nteams working on `Observability` plugins by providing custom Playwright\r\nfixtures, page objects, and utilities tailored for Observability-related\r\ntesting scenarios.\r\n\r\nContributing:\r\n- when Fixture/Page Object is sharable across all Solutions and Platform\r\n(`fleetApi` fixture), it should be added in `@kbn/scout`\r\n- when Fixture/Page Object is Oblt-specific but is shared across tests\r\nunder the multiple plugins (`OnboardingHome` page), it should be added\r\nin `@kbn/scout-oblt`\r\n- when Fixture/Page Object is only used in a single plugin (`onboarding`\r\ninternal APIs ?), it should be added in this plugin.\r\n\r\nI also re-worked existing tests with few ideas in mind:\r\n- Scout is **e2e testing tool** and should target primary e2e test\r\nscenarios; We have _API integration tests_ to test multiple short\r\nscenarios for APIs behavior (response, status code) and _jest/React\r\ntesting library_ to test components in isolation (elements rendering,\r\nfields validation). Doing all the testing with e2e tool like Playwright\r\nwill dramatically affect cost efficiency and stability of tests, but\r\nalso slows overall CI execution and PRs delivery. The goal is to follow\r\ntesting pyramid and keep in mind its principles.\r\n- We on purpose spin up new browser context for each `test` block to\r\nmake sure our **tests are independent**. Having too many short `test`\r\nblocks in the file significantly slows down the execution: every block\r\ntriggers browser context, saml authentication, adding/removing Fleet\r\nintegrations (each call up to 2 seconds) and other beforeEach/afterEach\r\nhooks. Real browser-based testing is expensive. It is not about putting\r\nevery step into 1 `test` block, but also not a Jest unit-test-style\r\ndesign. When it is possible to group similar actions on the same page\r\nand if it is a part of the same user flow - we should do it. It also\r\ndoesn't bring the testing value repeating the same UI steps multiple\r\ntimes in different scenarios. _Our CI costs are critical to cut when it\r\nis possible_\r\n- Avoid **nesting describe** blocks: it complicates test readability and\r\nalso complicates for CI bot to properly skip the failing block (it will\r\nskip the top level one). We encourage **Scout parallel test execution**\r\nbased on running test spec files in multiple workers, not the `test`\r\nblocks within the same file. Having too many `test` blocks in the same\r\nfile will be slowly run in the single thread and in case of flakiness,\r\nit means Team lose more test coverage than they probably expect.\r\n\r\nBefore (**59** test blocks - **8-8.5 min** per distro):\r\n<img width=\"1709\" alt=\"Screenshot 2025-02-08 at 18 01 40\"\r\nsrc=\"https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab\"\r\n/>\r\n\r\nAfter (**15** test blocks - **3.5-4 min** per distro):\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-10 at 18 14 42\"\r\nsrc=\"https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120\"\r\n/>\r\n\r\nFor reviewers: updated tests are possible to run in 2 parallel workers\r\nagainst the same Kibana/ES instance and run time is dropping to **2.5-3\r\nmin** 🚀 . It is up to UX-Logs team to decide if you want to keep\r\nparallel run (new tests can be added either to parallel or sequential\r\nrun)\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-11 at 12 14 30\"\r\nsrc=\"https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41\"\r\n/>\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"bd13e829498032c07bf8490f770a563f34e9f856","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:version","test:scout","v9.1.0","v8.19.0"],"title":"[scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests","number":209761,"url":"https://github.com/elastic/kibana/pull/209761","mergeCommit":{"message":"[scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761)\n\n## Summary\r\n\r\n`@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test\r\nhelpers specifically designed to test `Observability` applications in\r\nKibana. All Oblt plugins should only import from `@kbn/scout-oblt`\r\n\r\nIts primary goal is to simplify the test development experience for\r\nteams working on `Observability` plugins by providing custom Playwright\r\nfixtures, page objects, and utilities tailored for Observability-related\r\ntesting scenarios.\r\n\r\nContributing:\r\n- when Fixture/Page Object is sharable across all Solutions and Platform\r\n(`fleetApi` fixture), it should be added in `@kbn/scout`\r\n- when Fixture/Page Object is Oblt-specific but is shared across tests\r\nunder the multiple plugins (`OnboardingHome` page), it should be added\r\nin `@kbn/scout-oblt`\r\n- when Fixture/Page Object is only used in a single plugin (`onboarding`\r\ninternal APIs ?), it should be added in this plugin.\r\n\r\nI also re-worked existing tests with few ideas in mind:\r\n- Scout is **e2e testing tool** and should target primary e2e test\r\nscenarios; We have _API integration tests_ to test multiple short\r\nscenarios for APIs behavior (response, status code) and _jest/React\r\ntesting library_ to test components in isolation (elements rendering,\r\nfields validation). Doing all the testing with e2e tool like Playwright\r\nwill dramatically affect cost efficiency and stability of tests, but\r\nalso slows overall CI execution and PRs delivery. The goal is to follow\r\ntesting pyramid and keep in mind its principles.\r\n- We on purpose spin up new browser context for each `test` block to\r\nmake sure our **tests are independent**. Having too many short `test`\r\nblocks in the file significantly slows down the execution: every block\r\ntriggers browser context, saml authentication, adding/removing Fleet\r\nintegrations (each call up to 2 seconds) and other beforeEach/afterEach\r\nhooks. Real browser-based testing is expensive. It is not about putting\r\nevery step into 1 `test` block, but also not a Jest unit-test-style\r\ndesign. When it is possible to group similar actions on the same page\r\nand if it is a part of the same user flow - we should do it. It also\r\ndoesn't bring the testing value repeating the same UI steps multiple\r\ntimes in different scenarios. _Our CI costs are critical to cut when it\r\nis possible_\r\n- Avoid **nesting describe** blocks: it complicates test readability and\r\nalso complicates for CI bot to properly skip the failing block (it will\r\nskip the top level one). We encourage **Scout parallel test execution**\r\nbased on running test spec files in multiple workers, not the `test`\r\nblocks within the same file. Having too many `test` blocks in the same\r\nfile will be slowly run in the single thread and in case of flakiness,\r\nit means Team lose more test coverage than they probably expect.\r\n\r\nBefore (**59** test blocks - **8-8.5 min** per distro):\r\n<img width=\"1709\" alt=\"Screenshot 2025-02-08 at 18 01 40\"\r\nsrc=\"https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab\"\r\n/>\r\n\r\nAfter (**15** test blocks - **3.5-4 min** per distro):\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-10 at 18 14 42\"\r\nsrc=\"https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120\"\r\n/>\r\n\r\nFor reviewers: updated tests are possible to run in 2 parallel workers\r\nagainst the same Kibana/ES instance and run time is dropping to **2.5-3\r\nmin** 🚀 . It is up to UX-Logs team to decide if you want to keep\r\nparallel run (new tests can be added either to parallel or sequential\r\nrun)\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-11 at 12 14 30\"\r\nsrc=\"https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41\"\r\n/>\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"bd13e829498032c07bf8490f770a563f34e9f856"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/210675","number":210675,"state":"MERGED","mergeCommit":{"sha":"187b2307220a1f54776b0b3b05d131b0e75b6e03","message":"[9.0] [scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761) (#210675)\n\n# Backport\n\nThis will backport the following commits from `main` to `9.0`:\n- [[scout] adding test helper `@kbn/scout-oblt` package and\nuptate onboarding tests\n(#209761)](https://github.com/elastic/kibana/pull/209761)\n\n<!--- Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT [{\"author\":{\"name\":\"Dzmitry\nLemechko\",\"email\":\"dzmitry.lemechko@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2025-02-11T17:38:41Z\",\"message\":\"[scout]\nadding test helper `@kbn/scout-oblt` package and uptate onboarding tests\n(#209761)\\n\\n## Summary\\r\\n\\r\\n`@kbn/scout-oblt` is a test library that\nextends `@kbn/scout` with test\\r\\nhelpers specifically designed to test\n`Observability` applications in\\r\\nKibana. All Oblt plugins should only\nimport from `@kbn/scout-oblt`\\r\\n\\r\\nIts primary goal is to simplify the\ntest development experience for\\r\\nteams working on `Observability`\nplugins by providing custom Playwright\\r\\nfixtures, page objects, and\nutilities tailored for Observability-related\\r\\ntesting\nscenarios.\\r\\n\\r\\nContributing:\\r\\n- when Fixture/Page Object is\nsharable across all Solutions and Platform\\r\\n(`fleetApi` fixture), it\nshould be added in `@kbn/scout`\\r\\n- when Fixture/Page Object is\nOblt-specific but is shared across tests\\r\\nunder the multiple plugins\n(`OnboardingHome` page), it should be added\\r\\nin `@kbn/scout-oblt`\\r\\n-\nwhen Fixture/Page Object is only used in a single plugin\n(`onboarding`\\r\\ninternal APIs ?), it should be added in this\nplugin.\\r\\n\\r\\nI also re-worked existing tests with few ideas in\nmind:\\r\\n- Scout is **e2e testing tool** and should target primary e2e\ntest\\r\\nscenarios; We have _API integration tests_ to test multiple\nshort\\r\\nscenarios for APIs behavior (response, status code) and\n_jest/React\\r\\ntesting library_ to test components in isolation\n(elements rendering,\\r\\nfields validation). Doing all the testing with\ne2e tool like Playwright\\r\\nwill dramatically affect cost efficiency and\nstability of tests, but\\r\\nalso slows overall CI execution and PRs\ndelivery. The goal is to follow\\r\\ntesting pyramid and keep in mind its\nprinciples.\\r\\n- We on purpose spin up new browser context for each\n`test` block to\\r\\nmake sure our **tests are independent**. Having too\nmany short `test`\\r\\nblocks in the file significantly slows down the\nexecution: every block\\r\\ntriggers browser context, saml authentication,\nadding/removing Fleet\\r\\nintegrations (each call up to 2 seconds) and\nother beforeEach/afterEach\\r\\nhooks. Real browser-based testing is\nexpensive. It is not about putting\\r\\nevery step into 1 `test` block,\nbut also not a Jest unit-test-style\\r\\ndesign. When it is possible to\ngroup similar actions on the same page\\r\\nand if it is a part of the\nsame user flow - we should do it. It also\\r\\ndoesn't bring the testing\nvalue repeating the same UI steps multiple\\r\\ntimes in different\nscenarios. _Our CI costs are critical to cut when it\\r\\nis\npossible_\\r\\n- Avoid **nesting describe** blocks: it complicates test\nreadability and\\r\\nalso complicates for CI bot to properly skip the\nfailing block (it will\\r\\nskip the top level one). We encourage **Scout\nparallel test execution**\\r\\nbased on running test spec files in\nmultiple workers, not the `test`\\r\\nblocks within the same file. Having\ntoo many `test` blocks in the same\\r\\nfile will be slowly run in the\nsingle thread and in case of flakiness,\\r\\nit means Team lose more test\ncoverage than they probably expect.\\r\\n\\r\\nBefore (**59** test blocks -\n**8-8.5 min** per distro):\\r\\n<img width=\\\"1709\\\" alt=\\\"Screenshot\n2025-02-08 at 18 01\n40\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab\\\"\\r\\n/>\\r\\n\\r\\nAfter\n(**15** test blocks - **3.5-4 min** per distro):\\r\\n<img width=\\\"1578\\\"\nalt=\\\"Screenshot 2025-02-10 at 18 14\n42\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120\\\"\\r\\n/>\\r\\n\\r\\nFor\nreviewers: updated tests are possible to run in 2 parallel\nworkers\\r\\nagainst the same Kibana/ES instance and run time is dropping\nto **2.5-3\\r\\nmin** 🚀 . It is up to UX-Logs team to decide if you want\nto keep\\r\\nparallel run (new tests can be added either to parallel or\nsequential\\r\\nrun)\\r\\n<img width=\\\"1578\\\" alt=\\\"Screenshot 2025-02-11 at\n12 14\n30\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41\\\"\\r\\n/>\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by:\nkibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\",\"sha\":\"bd13e829498032c07bf8490f770a563f34e9f856\",\"branchLabelMapping\":{\"^v9.1.0$\":\"main\",\"^v8.19.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:skip\",\"v9.0.0\",\"backport:version\",\"test:scout\",\"v9.1.0\",\"v8.19.0\"],\"title\":\"[scout]\nadding test helper `@kbn/scout-oblt` package and uptate onboarding\ntests\",\"number\":209761,\"url\":\"https://github.com/elastic/kibana/pull/209761\",\"mergeCommit\":{\"message\":\"[scout]\nadding test helper `@kbn/scout-oblt` package and uptate onboarding tests\n(#209761)\\n\\n## Summary\\r\\n\\r\\n`@kbn/scout-oblt` is a test library that\nextends `@kbn/scout` with test\\r\\nhelpers specifically designed to test\n`Observability` applications in\\r\\nKibana. All Oblt plugins should only\nimport from `@kbn/scout-oblt`\\r\\n\\r\\nIts primary goal is to simplify the\ntest development experience for\\r\\nteams working on `Observability`\nplugins by providing custom Playwright\\r\\nfixtures, page objects, and\nutilities tailored for Observability-related\\r\\ntesting\nscenarios.\\r\\n\\r\\nContributing:\\r\\n- when Fixture/Page Object is\nsharable across all Solutions and Platform\\r\\n(`fleetApi` fixture), it\nshould be added in `@kbn/scout`\\r\\n- when Fixture/Page Object is\nOblt-specific but is shared across tests\\r\\nunder the multiple plugins\n(`OnboardingHome` page), it should be added\\r\\nin `@kbn/scout-oblt`\\r\\n-\nwhen Fixture/Page Object is only used in a single plugin\n(`onboarding`\\r\\ninternal APIs ?), it should be added in this\nplugin.\\r\\n\\r\\nI also re-worked existing tests with few ideas in\nmind:\\r\\n- Scout is **e2e testing tool** and should target primary e2e\ntest\\r\\nscenarios; We have _API integration tests_ to test multiple\nshort\\r\\nscenarios for APIs behavior (response, status code) and\n_jest/React\\r\\ntesting library_ to test components in isolation\n(elements rendering,\\r\\nfields validation). Doing all the testing with\ne2e tool like Playwright\\r\\nwill dramatically affect cost efficiency and\nstability of tests, but\\r\\nalso slows overall CI execution and PRs\ndelivery. The goal is to follow\\r\\ntesting pyramid and keep in mind its\nprinciples.\\r\\n- We on purpose spin up new browser context for each\n`test` block to\\r\\nmake sure our **tests are independent**. Having too\nmany short `test`\\r\\nblocks in the file significantly slows down the\nexecution: every block\\r\\ntriggers browser context, saml authentication,\nadding/removing Fleet\\r\\nintegrations (each call up to 2 seconds) and\nother beforeEach/afterEach\\r\\nhooks. Real browser-based testing is\nexpensive. It is not about putting\\r\\nevery step into 1 `test` block,\nbut also not a Jest unit-test-style\\r\\ndesign. When it is possible to\ngroup similar actions on the same page\\r\\nand if it is a part of the\nsame user flow - we should do it. It also\\r\\ndoesn't bring the testing\nvalue repeating the same UI steps multiple\\r\\ntimes in different\nscenarios. _Our CI costs are critical to cut when it\\r\\nis\npossible_\\r\\n- Avoid **nesting describe** blocks: it complicates test\nreadability and\\r\\nalso complicates for CI bot to properly skip the\nfailing block (it will\\r\\nskip the top level one). We encourage **Scout\nparallel test execution**\\r\\nbased on running test spec files in\nmultiple workers, not the `test`\\r\\nblocks within the same file. Having\ntoo many `test` blocks in the same\\r\\nfile will be slowly run in the\nsingle thread and in case of flakiness,\\r\\nit means Team lose more test\ncoverage than they probably expect.\\r\\n\\r\\nBefore (**59** test blocks -\n**8-8.5 min** per distro):\\r\\n<img width=\\\"1709\\\" alt=\\\"Screenshot\n2025-02-08 at 18 01\n40\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab\\\"\\r\\n/>\\r\\n\\r\\nAfter\n(**15** test blocks - **3.5-4 min** per distro):\\r\\n<img width=\\\"1578\\\"\nalt=\\\"Screenshot 2025-02-10 at 18 14\n42\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120\\\"\\r\\n/>\\r\\n\\r\\nFor\nreviewers: updated tests are possible to run in 2 parallel\nworkers\\r\\nagainst the same Kibana/ES instance and run time is dropping\nto **2.5-3\\r\\nmin** 🚀 . It is up to UX-Logs team to decide if you want\nto keep\\r\\nparallel run (new tests can be added either to parallel or\nsequential\\r\\nrun)\\r\\n<img width=\\\"1578\\\" alt=\\\"Screenshot 2025-02-11 at\n12 14\n30\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41\\\"\\r\\n/>\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by:\nkibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\",\"sha\":\"bd13e829498032c07bf8490f770a563f34e9f856\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"9.0\",\"8.x\"],\"targetPullRequestStates\":[{\"branch\":\"9.0\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"main\",\"label\":\"v9.1.0\",\"branchLabelMappingKey\":\"^v9.1.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/209761\",\"number\":209761,\"mergeCommit\":{\"message\":\"[scout]\nadding test helper `@kbn/scout-oblt` package and uptate onboarding tests\n(#209761)\\n\\n## Summary\\r\\n\\r\\n`@kbn/scout-oblt` is a test library that\nextends `@kbn/scout` with test\\r\\nhelpers specifically designed to test\n`Observability` applications in\\r\\nKibana. All Oblt plugins should only\nimport from `@kbn/scout-oblt`\\r\\n\\r\\nIts primary goal is to simplify the\ntest development experience for\\r\\nteams working on `Observability`\nplugins by providing custom Playwright\\r\\nfixtures, page objects, and\nutilities tailored for Observability-related\\r\\ntesting\nscenarios.\\r\\n\\r\\nContributing:\\r\\n- when Fixture/Page Object is\nsharable across all Solutions and Platform\\r\\n(`fleetApi` fixture), it\nshould be added in `@kbn/scout`\\r\\n- when Fixture/Page Object is\nOblt-specific but is shared across tests\\r\\nunder the multiple plugins\n(`OnboardingHome` page), it should be added\\r\\nin `@kbn/scout-oblt`\\r\\n-\nwhen Fixture/Page Object is only used in a single plugin\n(`onboarding`\\r\\ninternal APIs ?), it should be added in this\nplugin.\\r\\n\\r\\nI also re-worked existing tests with few ideas in\nmind:\\r\\n- Scout is **e2e testing tool** and should target primary e2e\ntest\\r\\nscenarios; We have _API integration tests_ to test multiple\nshort\\r\\nscenarios for APIs behavior (response, status code) and\n_jest/React\\r\\ntesting library_ to test components in isolation\n(elements rendering,\\r\\nfields validation). Doing all the testing with\ne2e tool like Playwright\\r\\nwill dramatically affect cost efficiency and\nstability of tests, but\\r\\nalso slows overall CI execution and PRs\ndelivery. The goal is to follow\\r\\ntesting pyramid and keep in mind its\nprinciples.\\r\\n- We on purpose spin up new browser context for each\n`test` block to\\r\\nmake sure our **tests are independent**. Having too\nmany short `test`\\r\\nblocks in the file significantly slows down the\nexecution: every block\\r\\ntriggers browser context, saml authentication,\nadding/removing Fleet\\r\\nintegrations (each call up to 2 seconds) and\nother beforeEach/afterEach\\r\\nhooks. Real browser-based testing is\nexpensive. It is not about putting\\r\\nevery step into 1 `test` block,\nbut also not a Jest unit-test-style\\r\\ndesign. When it is possible to\ngroup similar actions on the same page\\r\\nand if it is a part of the\nsame user flow - we should do it. It also\\r\\ndoesn't bring the testing\nvalue repeating the same UI steps multiple\\r\\ntimes in different\nscenarios. _Our CI costs are critical to cut when it\\r\\nis\npossible_\\r\\n- Avoid **nesting describe** blocks: it complicates test\nreadability and\\r\\nalso complicates for CI bot to properly skip the\nfailing block (it will\\r\\nskip the top level one). We encourage **Scout\nparallel test execution**\\r\\nbased on running test spec files in\nmultiple workers, not the `test`\\r\\nblocks within the same file. Having\ntoo many `test` blocks in the same\\r\\nfile will be slowly run in the\nsingle thread and in case of flakiness,\\r\\nit means Team lose more test\ncoverage than they probably expect.\\r\\n\\r\\nBefore (**59** test blocks -\n**8-8.5 min** per distro):\\r\\n<img width=\\\"1709\\\" alt=\\\"Screenshot\n2025-02-08 at 18 01\n40\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab\\\"\\r\\n/>\\r\\n\\r\\nAfter\n(**15** test blocks - **3.5-4 min** per distro):\\r\\n<img width=\\\"1578\\\"\nalt=\\\"Screenshot 2025-02-10 at 18 14\n42\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120\\\"\\r\\n/>\\r\\n\\r\\nFor\nreviewers: updated tests are possible to run in 2 parallel\nworkers\\r\\nagainst the same Kibana/ES instance and run time is dropping\nto **2.5-3\\r\\nmin** 🚀 . It is up to UX-Logs team to decide if you want\nto keep\\r\\nparallel run (new tests can be added either to parallel or\nsequential\\r\\nrun)\\r\\n<img width=\\\"1578\\\" alt=\\\"Screenshot 2025-02-11 at\n12 14\n30\\\"\\r\\nsrc=\\\"https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41\\\"\\r\\n/>\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by:\nkibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\",\"sha\":\"bd13e829498032c07bf8490f770a563f34e9f856\"}},{\"branch\":\"8.x\",\"label\":\"v8.19.0\",\"branchLabelMappingKey\":\"^v8.19.0$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>"}},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/209761","number":209761,"mergeCommit":{"message":"[scout] adding test helper `@kbn/scout-oblt` package and uptate onboarding tests (#209761)\n\n## Summary\r\n\r\n`@kbn/scout-oblt` is a test library that extends `@kbn/scout` with test\r\nhelpers specifically designed to test `Observability` applications in\r\nKibana. All Oblt plugins should only import from `@kbn/scout-oblt`\r\n\r\nIts primary goal is to simplify the test development experience for\r\nteams working on `Observability` plugins by providing custom Playwright\r\nfixtures, page objects, and utilities tailored for Observability-related\r\ntesting scenarios.\r\n\r\nContributing:\r\n- when Fixture/Page Object is sharable across all Solutions and Platform\r\n(`fleetApi` fixture), it should be added in `@kbn/scout`\r\n- when Fixture/Page Object is Oblt-specific but is shared across tests\r\nunder the multiple plugins (`OnboardingHome` page), it should be added\r\nin `@kbn/scout-oblt`\r\n- when Fixture/Page Object is only used in a single plugin (`onboarding`\r\ninternal APIs ?), it should be added in this plugin.\r\n\r\nI also re-worked existing tests with few ideas in mind:\r\n- Scout is **e2e testing tool** and should target primary e2e test\r\nscenarios; We have _API integration tests_ to test multiple short\r\nscenarios for APIs behavior (response, status code) and _jest/React\r\ntesting library_ to test components in isolation (elements rendering,\r\nfields validation). Doing all the testing with e2e tool like Playwright\r\nwill dramatically affect cost efficiency and stability of tests, but\r\nalso slows overall CI execution and PRs delivery. The goal is to follow\r\ntesting pyramid and keep in mind its principles.\r\n- We on purpose spin up new browser context for each `test` block to\r\nmake sure our **tests are independent**. Having too many short `test`\r\nblocks in the file significantly slows down the execution: every block\r\ntriggers browser context, saml authentication, adding/removing Fleet\r\nintegrations (each call up to 2 seconds) and other beforeEach/afterEach\r\nhooks. Real browser-based testing is expensive. It is not about putting\r\nevery step into 1 `test` block, but also not a Jest unit-test-style\r\ndesign. When it is possible to group similar actions on the same page\r\nand if it is a part of the same user flow - we should do it. It also\r\ndoesn't bring the testing value repeating the same UI steps multiple\r\ntimes in different scenarios. _Our CI costs are critical to cut when it\r\nis possible_\r\n- Avoid **nesting describe** blocks: it complicates test readability and\r\nalso complicates for CI bot to properly skip the failing block (it will\r\nskip the top level one). We encourage **Scout parallel test execution**\r\nbased on running test spec files in multiple workers, not the `test`\r\nblocks within the same file. Having too many `test` blocks in the same\r\nfile will be slowly run in the single thread and in case of flakiness,\r\nit means Team lose more test coverage than they probably expect.\r\n\r\nBefore (**59** test blocks - **8-8.5 min** per distro):\r\n<img width=\"1709\" alt=\"Screenshot 2025-02-08 at 18 01 40\"\r\nsrc=\"https://github.com/user-attachments/assets/5fd65a1c-85f9-4594-9dae-3f8e99a005ab\"\r\n/>\r\n\r\nAfter (**15** test blocks - **3.5-4 min** per distro):\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-10 at 18 14 42\"\r\nsrc=\"https://github.com/user-attachments/assets/6846898f-7dd2-4f6b-8bc5-d06741b0b120\"\r\n/>\r\n\r\nFor reviewers: updated tests are possible to run in 2 parallel workers\r\nagainst the same Kibana/ES instance and run time is dropping to **2.5-3\r\nmin** 🚀 . It is up to UX-Logs team to decide if you want to keep\r\nparallel run (new tests can be added either to parallel or sequential\r\nrun)\r\n<img width=\"1578\" alt=\"Screenshot 2025-02-11 at 12 14 30\"\r\nsrc=\"https://github.com/user-attachments/assets/e94113f2-d7f1-470e-a6d5-cb5154d99c41\"\r\n/>\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"bd13e829498032c07bf8490f770a563f34e9f856"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto9.0:Questions ?
Please refer to the Backport tool documentation