Skip to content

[kbn-test] improve run_check_ftr_configs_cli script #188854

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dmlemeshko merged 5 commits into from
Jul 22, 2024
Merged

[kbn-test] improve run_check_ftr_configs_cli script #188854

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d106ee9
[kbn-test] improve run_check_ftr_configs_cli script
dmlemeshko Jul 22, 2024
c871371
Merge branch 'main' into kbn-test/improve-ftr-configs-checker
dmlemeshko Jul 22, 2024
1f861d8
update err message
dmlemeshko Jul 22, 2024
d1f1f03
Merge branch 'kbn-test/improve-ftr-configs-checker' of github.com:dml…
dmlemeshko Jul 22, 2024
ac7d17f
Merge branch 'main' into kbn-test/improve-ftr-configs-checker
dmlemeshko Jul 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .buildkite/ftr_oblt_serverless_configs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
disabled:
# Base config files, only necessary to inform config finding script
- x-pack/test_serverless/functional/test_suites/observability/cypress/oblt_config.base.ts

# Cypress configs, for now these are still run manually
- x-pack/test_serverless/functional/test_suites/observability/cypress/config_headless.ts
Expand Down
3 changes: 3 additions & 0 deletions .buildkite/ftr_oblt_stateful_configs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ disabled:
- x-pack/plugins/observability_solution/profiling/e2e/ftr_config_runner.ts
- x-pack/plugins/observability_solution/profiling/e2e/ftr_config.ts

#FTR configs
- x-pack/plugins/observability_solution/uptime/e2e/config.ts
Copy link
Copy Markdown
Contributor Author

@dmlemeshko dmlemeshko Jul 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shahzad31 could you confirm if this config should be disabled and not run on kibana CI?

Copy link
Copy Markdown
Contributor

@shahzad31 shahzad31 Jul 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be run, why do we want to disable it?

Copy link
Copy Markdown
Contributor Author

@dmlemeshko dmlemeshko Jul 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It got lost a while ago and we didn't run it. I just fixed a script and it pointed the configs that are not in manifest file. It was one if them

Copy link
Copy Markdown
Contributor

@shahzad31 shahzad31 Jul 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it, let's hope tests aren't broken since :)

Copy link
Copy Markdown
Contributor Author

@dmlemeshko dmlemeshko Jul 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

[kbn-test/improve-ftr-configs-checker][~/github/kibana]$ node scripts/functional_tests.js --config x-pack/plugins/observability_solution/uptime/e2e/config.ts
 warn ❗️❗️❗️
 warn ❗️❗️❗️
 warn ❗️❗️❗️
 warn    Don't forget to use `node scripts/build_kibana_platform_plugins` to build plugins you plan on testing
 warn ❗️❗️❗️
 warn ❗️❗️❗️
 warn ❗️❗️❗️
ERROR UNHANDLED ERROR
ERROR Error: No tests defined.
          at FunctionalTestRunner.runHarness (functional_test_runner.ts:233:15)
          at FunctionalTestRunner.getTestStats (functional_test_runner.ts:153:23)
          at checkForEnabledTestsInFtrConfig (run_ftr.ts:44:27)
          at run_tests.ts:72:61
          at tooling_log.ts:84:18
          at runTests (run_tests.ts:64:5)
          at description (cli.ts:24:7)
          at run.ts:73:10
          at withProcRunner (with_proc_runner.ts:29:5)
          at run (run.ts:71:5)

Somehow there are no tests defined in config. I will keep it as disabled. If there should be some tests in config, I think code owners should wok on it separately and re-enable config

Copy link
Copy Markdown
Contributor

@shahzad31 shahzad31 Jul 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's executed via a custom script

in uptime plugin, you can do

node scripts/e2e.js

Copy link
Copy Markdown
Contributor Author

@dmlemeshko dmlemeshko Jul 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, then it is correct to keep it under disabled: we enable only the configs that are run classical way through scripts/functional_tests


# Elastic Synthetics configs
- x-pack/plugins/observability_solution/uptime/e2e/uptime/synthetics_run.ts
- x-pack/plugins/observability_solution/synthetics/e2e/config.ts
Expand Down
2 changes: 2 additions & 0 deletions .buildkite/ftr_platform_stateful_configs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ disabled:
- x-pack/test/functional_with_es_ssl/config.base.ts
- x-pack/test/api_integration/config.ts
- x-pack/test/fleet_api_integration/config.base.ts
- x-pack/test/functional_basic/apps/ml/config.base.ts
- x-pack/test/functional_basic/apps/transform/config.base.ts

# QA suites that are run out-of-band
- x-pack/test/stack_functional_integration/configs/config.stack_functional_integration_base.js
Expand Down
2 changes: 2 additions & 0 deletions .buildkite/ftr_security_serverless_configs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
disabled:
# Base config files, only necessary to inform config finding script
- x-pack/test_serverless/functional/test_suites/security/cypress/security_config.base.ts
- x-pack/test_serverless/functional/test_suites/security/cypress/cypress.config.ts
Copy link
Copy Markdown
Contributor Author

@dmlemeshko dmlemeshko Jul 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MadameSheema could you confirm this config should be disabled (not run on kibana CI) ?

- x-pack/test/security_solution_api_integration/config/serverless/config.base.ts
- x-pack/test/security_solution_api_integration/config/serverless/config.base.essentials.ts
- x-pack/test/security_solution_api_integration/config/serverless/config.base.edr_workflows.ts
Expand Down
1 change: 1 addition & 0 deletions .buildkite/ftr_security_stateful_configs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ disabled:
- x-pack/test/security_solution_api_integration/config/ess/config.base.edr_workflows.trial.ts
- x-pack/test/security_solution_api_integration/config/ess/config.base.edr_workflows.ts
- x-pack/test/security_solution_api_integration/config/ess/config.base.basic.ts
- x-pack/test/security_solution_api_integration/config/ess/config.base.trial.ts
- x-pack/test/security_solution_endpoint/configs/config.base.ts
- x-pack/test/security_solution_endpoint/config.base.ts
- x-pack/test/security_solution_endpoint_api_int/config.base.ts
Expand Down
39 changes: 33 additions & 6 deletions packages/kbn-test/src/functional_test_runner/lib/config/run_check_ftr_configs_cli.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,13 +62,36 @@ export async function runCheckFtrConfigsCli() {
return false;
}

if (file.match(/jest.config.(t|j)s$/)) {
if (file.match(/(jest(\.integration)?)\.config\.(t|j)s$/)) {
return false;
}

return readFileSync(file)
.toString()
.match(/(testRunner)|(testFiles)/);
if (file.match(/mocks.ts$/)) {
return false;
}

const fileContent = readFileSync(file).toString();

if (fileContent.match(/(testRunner)|(testFiles)/)) {
// test config
return true;
}

if (fileContent.match(/(describe)|(defineCypressConfig)/)) {
// test file or Cypress config
return false;
}

// FTR config file should have default export
try {
// eslint-disable-next-line @typescript-eslint/no-var-requires
const exports = require(file);
const defaultExport = exports.__esModule ? exports.default : exports;
return !!defaultExport;
} catch (err) {
log.debug(`Failed to load file: ${err.message}`);
return false;
}
});

const { allFtrConfigs, manifestPaths } = getAllFtrConfigsAndManifests();
Expand All @@ -77,10 +100,14 @@ export async function runCheckFtrConfigsCli() {
if (invalid.length) {
const invalidList = invalid.map((path) => Path.relative(REPO_ROOT, path)).join('\n - ');
log.error(
`The following files look like FTR configs which are not listed in one of manifest files:\nstateful: ${manifestPaths.stateful}\nserverless: ${manifestPaths.serverless}\n - ${invalidList}`
`The following files look like FTR configs which are not listed in one of manifest files:\n${invalidList}\n
Make sure to add your new FTR config to the correct manifest file.\n
Stateful tests:\n${(manifestPaths.stateful as string[]).join('\n')}\n
Serverless tests:\n${(manifestPaths.serverless as string[]).join('\n')}
`
);
throw createFailError(
`Please add the listed paths to the correct manifest file. If it's not an FTR config, you can add it to the IGNORED_PATHS in ${THIS_REL} or contact #kibana-operations`
`Please add the listed paths to the correct manifest files. If it's not an FTR config, you can add it to the IGNORED_PATHS in ${THIS_REL} or contact #kibana-operations`
);
}
},
Expand Down