[scout] use project deps as global hooks for parallel tests

[dmlemeshko]

Summary

Currently we are using globalSetup script in configuration file to ingest Elasticsearch data before running the tests in parallel against the same ES/Kibana instances.

This approach doesn't work well when you need to adjust globalSetup logic based on specific condition, e.g. configuration file defining where servers are hosted, its credentials, etc.

Not only global hook, but ScoutConfig fixture expects an argument to define where servers configuration is defined:

kibana/packages/kbn-scout/src/playwright/fixtures/worker/core_fixtures.ts

Lines 65 to 75 in cd502ac

	config: [
	({ log }, use, testInfo) => {
	const configName = 'local';
	const projectUse = testInfo.project.use as ScoutTestOptions;
	const serversConfigDir = projectUse.serversConfigDir;
	const configInstance = createScoutConfig(serversConfigDir, configName, log);
	use(configInstance);
	},
	{ scope: 'worker' },
	],

testInfo is how Playwright exposes currently running configuration in a form of project interface: projects can be used to group tests, e.g. for specific envs or browsers.

Unfortunately testInfo is not exposed in global scripts, because in Playwright project design globalSetup scripts are run before multiple projects and projects can have its own setup hooks via dependencies:

    {
      name: 'setup',
      testMatch: /global.setup\.ts/,
    },
    {
      name: 'local',
      use: { ...devices['Desktop Chrome'], configName: 'local' },
      dependencies: 'setup',
    },

We already use project API to get serversConfigDir path, where we plan to store local and cloud server configurations. This PR proposes to define projects as local and cloud (maybe even separate cloud-mki, cloud-ech) as a way to provide playwright information about servers configuration.

Advantages:

1. we can re-use existing fixtures as-is, without adding custom exported helper functions for ES data ingestion
2. project dependency is displayed as setup in Playwright report
3. way better and simpler design for consumers:

import { globalSetupHook } from '@kbn/scout';

globalSetupHook('Ingest data to Elasticsearch', async ({ esArchiver, log }) => {
  // add archives to load, if needed
  const archives = [
    testData.ES_ARCHIVES.LOGSTASH,
  ];

  log.debug('[setup] loading test data (only if indexes do not exist)...');
  for (const archive of archives) {
    await esArchiver.loadIfNeeded(archive);
  }
});

4. it is supported by VSCode Playwright plugin

I find it extremely useful because you don't need to change env var when you want to switch b/w local or cloud run, all the configurations are loaded automatically and you just tick the checkbox!

Disadvantages:

1. it is important to run playwright test with --project flag to use the proper configuration
2. we have to define how projects are used for local and cloud configuration, and make sure it meets requirements of multiple teams. We can expose a way to pass custom project definitions in createPlaywrightConfig function, but it might complicate the support effort when every Team has too many custom projects.
3. project term is something we can't change and might be confusing
4. Since it is a Playwright feature, we might not have consistency with API tests runner under Scout

For reviewers:

Playing with it locally might give a better understanding about the pros/cons, especially with IDE playwright plugin installed.

Running servers with tests:

node scripts/scout.js run-tests --serverless=oblt --testTarget=local --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts

node scripts/scout.js run-tests --serverless=oblt --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts

Running test only requires passing project argument:

npx playwright test --project=local --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/playwright.config.ts

npx playwright test --project=local --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/parallel.playwright.config.ts

[dmlemeshko]

For MKI run we will need just to generate json and update --project flag

[dolaru]

Implementation LGTM 👍🏽

The only thing I'm not a fan of is the clash in terminology with the word project. That could easily be confused with a serverless project.

A couple of options to make this better would be:

• increase the specificity of the term to make it clear this is for Playwright (e.g. pw-project / pwProject)
• change the term used to something like target/testTarget to eliminate ambiguity

[dmlemeshko]

A couple of options to make this better would be:

• increase the specificity of the term to make it clear this is for Playwright (e.g. pw-project / pwProject)
• change the term used to something like target/testTarget to eliminate ambiguity

@dolaru I agree, it is confusing. Sadly we can only change it for scripts/scout, if we merge it running tests directly with playwright will require smth like

npx playwright test --config path/to/config --project local --grep=@svlSearch

Still if we agree on unified project names, it will be less confusing:

- local
- cloud-mki
- cloud-ech

for each of the defined projects there should be a corresponding json file, ideally same name cloud-mki.json where consumer puts hosts/credentials.

Some consumers might say "I want to run test agaist both Search and Oblt serverless projects without changing json back and forth": should we then have smth like

- local
- cloud-mki-es
- cloud-mki-oblt
- cloud-mki-security
- cloud-ech

for platform and limit it to just 3 options within Playwright config when it is in specific solution?

As for node scripts/scout run-tests I agree with you on target / testTarget, that would be pretty clear

[mykolaharmash]

Onboarding changes LGTM, thank you!

[dmlemeshko]

each project will override default configName to pass source of servers configuration ('local.json', 'cloud-mki', 'cloud-ech', etc)

[jennypavlova]

Sharing the error I am getting when running the apm UI tests also here for visibility:

[jennypavlova]

Nice, thank you!

[dmlemeshko]

Verified locally:

node scripts/scout run-tests --config x-pack/solutions/observability/plugins/apm/ui_tests/parallel.playwright.config.ts --stateful

node scripts/scout start-server --serverless=oblt
npx playwright test --config x-pack/solutions/observability/plugins/apm/ui_tests/parallel.playwright.config.ts --project local --grep @svlOblt

[jennypavlova]

LGTM 💯
✅ The APM UI tests are now passing locally using
node scripts/scout.js start-server --stateful
and
npx playwright test --config x-pack/solutions/observability/plugins/apm/ui_tests/parallel.playwright.config.ts --project=local --grep @ess

Thank you for all the fixes!

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: c8ec61a

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/scout	102	97	-5
@kbn/scout-oblt	74	65	-9
total			-14

Unknown metric groups

API count

id	before	after	diff
@kbn/scout	393	388	-5
@kbn/scout-oblt	365	356	-9
total			-14

ESLint disabled line counts

id	before	after	diff
apm	66	65	-1
discoverEnhanced	3	2	-1
total			-2

Total ESLint disabled count

id	before	after	diff
apm	79	78	-1
discoverEnhanced	3	2	-1
total			-2

History

• 💔 Build #283335 failed 089e756
• 💚 Build #283185 succeeded bf1af52
• 💚 Build #282484 succeeded c44c2ec
• 💔 Build #282051 failed 2784c92
• 💚 Build #281858 succeeded 78750fc

cc @dmlemeshko

[kibanamachine]

Starting backport for target branches: 8.x, 9.0

https://github.com/elastic/kibana/actions/runs/13830858430

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts
❌	9.0	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 211409

Questions ?

Please refer to the Backport tool documentation

[dmlemeshko]

💚 All backports created successfully

Status	Branch	Result
✅	9.0

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[kibanamachine]

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

[kibanamachine]

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

[dmlemeshko]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation