[Search] Replace flaky FTR tests with Scout tests for Search Homepage by saikatsarkar056 · Pull Request #248381 · elastic/kibana

saikatsarkar056 · 2026-01-09T04:53:34Z

Summary

This PR migrates the FTR (Functional Test Runner) tests for the Search Homepage to Scout tests.

Background

The existing FTR tests for Search Homepage have been experiencing flakiness issues (see #226572), with several tests being skipped. This migration to Scout provides a more stable and maintainable test infrastructure.

How to Run Search Homepage Scout Tests

Prerequisites

Start the appropriate server first:

For Stateful (ESS)

node scripts/scout.js start-server --stateful

For Serverless

node scripts/scout.js start-server --serverless=es

Run all Search Homepage tests with browser visible

npx playwright test --config x-pack/solutions/search/plugins/search_homepage/test/scout/ui/parallel.playwright.config.ts --project local --headed --grep "Search Homepage"

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
Flaky Test Runner was used on any tests changed
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

See some risk examples
...

- Create search_homepage.spec.ts with tests migrated from FTR - Add embedded console methods to homepage page object - Use uiSettings to disable V2 flag for testing V1 homepage - Tests: embedded dev console, page load verification

saikatsarkar056 · 2026-01-09T18:52:03Z

Flaky test runner: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/10391/summary

kibanamachine · 2026-01-09T20:06:10Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#10390

[❌] x-pack/solutions/search/plugins/search_homepage/test/scout/ui/parallel.playwright.config.ts: 0/200 tests passed.

see run history

kibanamachine · 2026-01-09T20:08:20Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#10391

[❌] x-pack/solutions/search/plugins/search_homepage/test/scout/ui/parallel.playwright.config.ts: 0/200 tests passed.

see run history

saikatsarkar056 · 2026-01-09T23:56:53Z

Flaky test runner was failing due to an invalid CLI flag. This PR fixes the issue.

saikatsarkar056 · 2026-01-10T21:13:59Z

@elasticmachine merge upstream

kibanamachine · 2026-01-10T21:59:42Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#10408

[✅] x-pack/solutions/search/plugins/search_homepage/test/scout/ui/parallel.playwright.config.ts: 15/15 tests passed.

see run history

saikatsarkar056 · 2026-01-12T02:13:01Z

@elasticmachine merge upstream

elasticmachine · 2026-01-12T02:13:03Z

There are no new commits on the base branch.

efegurkan · 2026-01-12T10:31:12Z

...olutions/search/plugins/search_homepage/test/scout/ui/parallel_tests/search_homepage.spec.ts

+
+test.describe('Search Homepage (V1)', { tag: ['@ess', '@svlSearch'] }, () => {
+  // Disable the V2 homepage to test the old (V1) homepage
+  test.beforeAll(async ({ kbnClient, esClient }) => {


Hey @saikatsarkar056

We are not going to re-enable v1 homepage. I think the ticket was not ready 100% 😬 But what we should do is instead, take a look at the flows from skipped homepage tests and implement similar ones on the new homepage where it makes sense.

@efegurkan You are right. Let me modify this PR to cover only relevant changes needed for new homepage.

…a into scout_for_homepage

saikatsarkar056 · 2026-01-12T21:51:42Z

@efegurkan Could you please review the PR for another round? Thank you.

kibanamachine · 2026-01-12T22:29:13Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#10432

[✅] x-pack/solutions/search/plugins/search_homepage/test/scout/ui/parallel.playwright.config.ts: 15/15 tests passed.

see run history

saikatsarkar056 · 2026-01-13T14:59:08Z

@elasticmachine merge upstream

…a into scout_for_homepage

kibanamachine · 2026-01-13T19:35:52Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#10452

[✅] x-pack/solutions/search/plugins/search_homepage/test/scout/ui/parallel.playwright.config.ts: 15/15 tests passed.

see run history

saikatsarkar056 · 2026-01-14T15:17:29Z

/ci

saikatsarkar056 · 2026-01-14T15:37:21Z

@elasticmachine merge upstream

elasticmachine · 2026-01-14T15:37:24Z

expected head sha didn’t match current head ref.

saikatsarkar056 · 2026-01-15T16:10:22Z

@elasticmachine merge upstream

elasticmachine · 2026-01-15T18:10:10Z

💚 Build Succeeded

Buildkite Build
Commit: 3428850

Metrics [docs]

✅ unchanged

History

💔 Build #381661 failed 790a2ef
💔 Build #381566 failed ff46dc1
💔 Build #381532 failed af3c014
💔 Build #381509 failed 433d1ea
💛 Build #381112 was flaky 433d1ea
💔 Build #380996 failed ef62f7e

cc @saikatsarkar056

saikatsarkar056 added 7 commits January 8, 2026 15:51

Add scout tests for Search Homepage

c0944c8

Add Scout test for old Search Homepage (V1)

422ad19

- Create search_homepage.spec.ts with tests migrated from FTR - Add embedded console methods to homepage page object - Use uiSettings to disable V2 flag for testing V1 homepage - Tests: embedded dev console, page load verification

Add more tests

0c5f05e

Refactor the tests

847374c

Add test for security section

3d71e22

Add test for Create index

1399162

Fix tests for serverless env

99bd0a3

saikatsarkar056 requested a review from a team as a code owner January 9, 2026 04:53

saikatsarkar056 marked this pull request as draft January 9, 2026 04:53

saikatsarkar056 added 3 commits January 9, 2026 11:12

Refactor code

b8d1fdf

Refactor code

88412cb

Remove old FTR tests

4e56aa8

saikatsarkar056 changed the title ~~Scout for homepage~~ [Search] Replace flaky FTR tests with Scout tests for Search Homepage Jan 9, 2026

saikatsarkar056 self-assigned this Jan 9, 2026

saikatsarkar056 added backport:skip This PR does not require backporting v9.4.0 Team:Search release_note:skip Skip the PR/issue when compiling release notes labels Jan 9, 2026

saikatsarkar056 marked this pull request as ready for review January 9, 2026 18:49

saikatsarkar056 added 2 commits January 9, 2026 12:48

Remove unused code

9aad7f6

Merge branch 'main' of github.com:elastic/kibana into scout_for_homepage

45a49b8

Merge branch 'main' into scout_for_homepage

b694e0e

efegurkan reviewed Jan 12, 2026

View reviewed changes

saikatsarkar056 added 6 commits January 12, 2026 12:24

Remove tests for old search page

dba3ad5

Merge branch 'scout_for_homepage' of github.com:saikatsarkar056/kiban…

64e64df

…a into scout_for_homepage

Add similar tests for viewer

c40f0f1

Add tests for testing body links

d2ca387

Add tests for getting_started section

a3f05ea

Make the tests runnable in stack as well

b72664e

saikatsarkar056 requested a review from efegurkan January 12, 2026 21:44

Remove unused code

da15645

elasticmachine and others added 4 commits January 13, 2026 15:59

Merge branch 'main' into scout_for_homepage

ef62f7e

Check the expected urls for BodyLinks

b270fa5

Merge branch 'scout_for_homepage' of github.com:saikatsarkar056/kiban…

433d1ea

…a into scout_for_homepage

Merge branch 'main' of github.com:elastic/kibana into scout_for_homepage

88be805

efegurkan approved these changes Jan 14, 2026

View reviewed changes

saikatsarkar056 enabled auto-merge (squash) January 14, 2026 15:00

auto-merge was automatically disabled January 14, 2026 15:21
Pull Request is not mergeable

saikatsarkar056 added 3 commits January 14, 2026 08:39

Merge branch 'main' of github.com:elastic/kibana into scout_for_homepage

af3c014

Merge branch 'main' of github.com:elastic/kibana into scout_for_homepage

ff46dc1

Merge branch 'main' of github.com:elastic/kibana into scout_for_homepage

790a2ef

Merge branch 'main' into scout_for_homepage

3428850

saikatsarkar056 merged commit 36b6632 into elastic:main Jan 15, 2026
13 checks passed

Conversation

saikatsarkar056 commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Background

How to Run Search Homepage Scout Tests

Prerequisites

For Stateful (ESS)

For Serverless

Run all Search Homepage tests with browser visible

Checklist

Identify risks

Uh oh!

saikatsarkar056 commented Jan 9, 2026

Uh oh!

kibanamachine commented Jan 9, 2026

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#10390

Uh oh!

kibanamachine commented Jan 9, 2026

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#10391

Uh oh!

saikatsarkar056 commented Jan 9, 2026

Uh oh!

saikatsarkar056 commented Jan 10, 2026

Uh oh!

kibanamachine commented Jan 10, 2026

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#10408

Uh oh!

saikatsarkar056 commented Jan 12, 2026

Uh oh!

elasticmachine commented Jan 12, 2026

Uh oh!

efegurkan Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

saikatsarkar056 Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

saikatsarkar056 commented Jan 12, 2026

Uh oh!

kibanamachine commented Jan 12, 2026

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#10432

Uh oh!

saikatsarkar056 commented Jan 13, 2026

Uh oh!

kibanamachine commented Jan 13, 2026

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#10452

Uh oh!

saikatsarkar056 commented Jan 14, 2026

Uh oh!

saikatsarkar056 commented Jan 14, 2026

Uh oh!

elasticmachine commented Jan 14, 2026

Uh oh!

saikatsarkar056 commented Jan 15, 2026

Uh oh!

elasticmachine commented Jan 15, 2026

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

saikatsarkar056 commented Jan 9, 2026 •

edited

Loading