[kbn-test] improve run_check_ftr_configs_cli script#188854
[kbn-test] improve run_check_ftr_configs_cli script#188854dmlemeshko merged 5 commits intoelastic:mainfrom
Conversation
| disabled: | ||
| # Base config files, only necessary to inform config finding script | ||
| - x-pack/test_serverless/functional/test_suites/security/cypress/security_config.base.ts | ||
| - x-pack/test_serverless/functional/test_suites/security/cypress/cypress.config.ts |
There was a problem hiding this comment.
@MadameSheema could you confirm this config should be disabled (not run on kibana CI) ?
| - x-pack/plugins/observability_solution/profiling/e2e/ftr_config.ts | ||
|
|
||
| #FTR configs | ||
| - x-pack/plugins/observability_solution/uptime/e2e/config.ts |
There was a problem hiding this comment.
@shahzad31 could you confirm if this config should be disabled and not run on kibana CI?
There was a problem hiding this comment.
it should be run, why do we want to disable it?
There was a problem hiding this comment.
It got lost a while ago and we didn't run it. I just fixed a script and it pointed the configs that are not in manifest file. It was one if them
There was a problem hiding this comment.
got it, let's hope tests aren't broken since :)
There was a problem hiding this comment.
[kbn-test/improve-ftr-configs-checker][~/github/kibana]$ node scripts/functional_tests.js --config x-pack/plugins/observability_solution/uptime/e2e/config.ts
warn ❗️❗️❗️
warn ❗️❗️❗️
warn ❗️❗️❗️
warn Don't forget to use `node scripts/build_kibana_platform_plugins` to build plugins you plan on testing
warn ❗️❗️❗️
warn ❗️❗️❗️
warn ❗️❗️❗️
ERROR UNHANDLED ERROR
ERROR Error: No tests defined.
at FunctionalTestRunner.runHarness (functional_test_runner.ts:233:15)
at FunctionalTestRunner.getTestStats (functional_test_runner.ts:153:23)
at checkForEnabledTestsInFtrConfig (run_ftr.ts:44:27)
at run_tests.ts:72:61
at tooling_log.ts:84:18
at runTests (run_tests.ts:64:5)
at description (cli.ts:24:7)
at run.ts:73:10
at withProcRunner (with_proc_runner.ts:29:5)
at run (run.ts:71:5)
Somehow there are no tests defined in config. I will keep it as disabled. If there should be some tests in config, I think code owners should wok on it separately and re-enable config
There was a problem hiding this comment.
it's executed via a custom script
in uptime plugin, you can do
node scripts/e2e.js
There was a problem hiding this comment.
ok, then it is correct to keep it under disabled: we enable only the configs that are run classical way through scripts/functional_tests
dmlemeshko
added
release_note:skip
|
/ci |
…emeshko/kibana into kbn-test/improve-ftr-configs-checker
|
/ci |
wayneseymour
left a comment
wayneseymour
left a comment
There was a problem hiding this comment.
Ran successfully against my local
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]Unknown metric groupsESLint disabled line counts
Total ESLint disabled count
To update your PR or re-run it, just comment with: |
## Summary Follow-up to #188854 On CI script is taking >1 min, while before it was taking seconds. Probably because 80+ files were loaded as potential FTR configs. I adjusted regular expressions to minimize the amount of files that we need to load to validate if it is FTR config or not. In this CI run script took 20s (part of `Quick Checks` group)
6 participants
Summary
Follow-up to #188825
@crespocarlos reported that some Oblt configs were missing after #187440
I was using
node scripts/check_ftr_configs.jsto validate I did not miss anything and decided to debug the script.We had a pretty strict config file content validation like
testRunner|testFiles, that was skipping some FTR configs likex-pack/test/apm_api_integration/basic/config.tsI extended file content check to look for default export function and also skip test/suite or Cypress-own config files.
In the end 7 FTR configs were discovered, but only 2 are with tests. I will ask owners to confirm if it should be enabled/disabled. Script run output: