[Functional Tests] Use @kbn/test on Kibana CI by rhoboat · Pull Request #18967 · elastic/kibana

rhoboat · 2018-05-10T00:02:37Z

Addresses #18593

Background

We currently use the @kbn/test package on CI in one place: running X-Pack Functional, API, and SAML API tests. These run in succession with a single script (node scripts/functional_tests from x-pack/).

We'd like to use the package to run OSS Kibana tests on CI as well. Currently we run API tests in the test:api grunt task, and Selenium (functional) tests in the jenkins:selenium grunt task. We can unify these or keep them separate. (See #18593 for high-level details.)

Running from source/snapshot

One of the key requirements:

ES should always run from source when on CI. It should run from snapshot by default (for dev).

Now pass in --esFrom=<source|snapshot> default: snapshot

Check for invalid cli args
Replace api tests
Replace selenium tests
Jest tests

How to test:

Unit tests

There are Jest tests.

Integration tests

Look at the test run outputs in Jenkins.

Search for Running "run:functional_tests" (run) task in multijob-selenium task
Search for Running "run:api_integration_tests" (run) task in multijob-intake task

Then in the console output, just take a look at how Elasticsearch is started up (from source on CI), how Kibana is run with cli args, and how functional test runner runs.

Manual testing in development

# bail option not a boolean
node scripts/functional_tests --config test/functional/config.js --esFrom snapshot --debug --grep management --bail=peanut -- --server.maxPayloadBytes=1648576

# esFrom option not valid
node scripts/functional_tests --config test/functional/config.js --esFrom butter --debug --grep management --bail=peanut -- --server.maxPayloadBytes=1648576

# config option not a string
node scripts/functional_tests --config --esFrom source --debug --grep management --bail=peanut -- --server.maxPayloadBytes=1648576

# works with esFrom = source
node scripts/functional_tests --config test/functional/config.js --esFrom source --debug --grep management --bail -- --server.maxPayloadBytes=1648576

# server.<option> options not valid before the --
node scripts/functional_tests --config test/functional/config.js --esFrom source --debug --server.maxPayloadBytes=1648576 --grep management --bail=peanut -- --server.maxPayloadBytes=1648576

# works with multiple config options
node scripts/functional_tests --config test/functional/config.js --config test/api_integration/config.js --esFrom snapshot --debug --grep management --bail -- --server.maxPayloadBytes=1648576

# works with no config options
node scripts/functional_tests --esFrom snapshot --debug --grep management --bail -- --server.maxPayloadBytes=1648576

rhoboat · 2018-05-10T00:46:10Z

tasks/config/run.js

Remove test:api related code.

Replace it with a call to @kbn/test code, scoped to just run api tests.

Run api_integration_tests using the scripts/functional_tests (which uses @kbn/test

Pass in just the config for api_integration tests

jenkins_config.js is built on the original with just a change to run ES from source

Alternative way to do this:

Call runTests directly from @kbn/test, though I'm not sure how to do this. If it needs to be done in a command-line way, then we already have it in scripts/functional_tests.

Allow test/api_integration/config.js to take in a from.

Or allow scripts/functional_tests to take in a --from.

This approach seems worth trying, for comparison.

rhoboat · 2018-05-10T01:45:37Z

@tylersmalley @azasypkin @spalger @epixa What do you think so far? I.e. Which of the 3 alternatives would you prefer?

elasticmachine · 2018-05-10T02:35:35Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

rhoboat · 2018-05-10T12:48:41Z

jenkins test this

elasticmachine · 2018-05-10T13:57:32Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

spalger · 2018-05-10T14:57:56Z

What about a fourth option, the CI environment variable is always set when we run in CI, so we could just update https://github.com/elastic/kibana/blob/master/test/common/config.js#L21

    esTestCluster: {
      license: 'oss',
-     from: 'snapshot',
+     from: process.env.CI ? 'source' : 'snapshot',
      serverArgs: [
      ],
    },

spalger · 2018-05-10T15:48:38Z

Actually, relying on the CI environment variable makes it pretty tricky to run elasticsearch from source intentionally, and will require a good amount of digging into the code to figure out how.

I'm personally a fan of the --es-from command line flag, mostly because I like that it's documented in --help and that command line flags are normally how I tweak the behavior of a command line tool.

I just ran the x-pack/scripts/functional_tests_server script and was surprised by my inability to choose where es was built from. I tried --help, but it didn't offer any, and I would prefer that it did.

tylersmalley · 2018-05-10T18:49:09Z

I am also +1 for the command line argument. Running from source is not something that we just want to do for Jenkins, it's also if you are working off an ES feature branch.

elasticmachine · 2018-05-17T14:11:41Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-17T17:42:15Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

rhoboat · 2018-05-17T20:13:25Z

jenkins test this

elasticmachine · 2018-05-17T21:23:48Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-18T19:22:03Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-21T16:15:43Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-21T17:59:47Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-21T19:24:06Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

spalger · 2018-05-21T19:28:54Z

packages/kbn-test/src/es/es_test_cluster.js

These changes seems really redundant.

spalger · 2018-05-21T19:31:18Z

packages/kbn-test/src/functional_tests/cli/run_tests_cli.js

Why not just use options['es-from'] || 'snapshot'? This ternary will allow passing --es-from=foo and interpret it as --es-from=snapshot because any none "source" value is ignored.

elasticmachine · 2018-05-21T20:54:46Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-21T22:42:04Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

rhoboat · 2018-05-21T22:54:17Z

jenkins, test this

elasticmachine · 2018-05-22T00:03:50Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

rhoboat · 2018-05-22T00:05:02Z

hmm. this keeps failing

23:50:16 │ proc [ftr] └- ✖ fail: "management test large number of fields test_huge data should have expected number of fields"
23:50:16 │ proc [ftr] │ tryForTime timeout: Error: tryForTime timeout: Error: tryForTime timeout: [POST http://localhost:9515/session/ba2732fe4fefc2e1444fba16e5f86da7/element / {"using":"css selector","value":"[data-test-subj~="tab-count-indexedFields"]"}] no such element: Unable to locate element: {"method":"css selector","selector":"[data-test-subj~="tab-count-indexedFields"]"}

rhoboat · 2018-05-22T00:05:12Z

jenkins test this

elasticmachine · 2018-05-22T01:10:28Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-22T02:29:24Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-22T12:54:34Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

rhoboat · 2018-06-20T19:55:54Z

@kobelb @azasypkin I've made the default log level debug for the tasks in tasks/config/run.js

elasticmachine · 2018-06-20T20:04:08Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

rhoboat · 2018-06-20T20:19:25Z

@azasypkin yes, indeed, was waiting for that dev.mode flag removal. 🎉

elasticmachine · 2018-06-20T21:18:49Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

rhoboat · 2018-06-22T17:44:21Z

@kobelb I took care of your comments, i think you're out on vacation...

rhoboat · 2018-06-22T19:44:09Z

jenkins test this

elasticmachine · 2018-06-22T20:47:02Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-06-25T15:32:36Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-06-25T16:46:28Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-06-25T20:34:11Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

rhoboat · 2018-06-26T15:06:36Z

jenkins, test this

elasticmachine · 2018-06-26T16:17:54Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

azasypkin

LGTM, looks great! I've run all possible functional/integration tests I could think of in Kibana and x-pack, everything worked flawlessly.

azasypkin · 2018-06-26T18:49:00Z

packages/kbn-test/src/functional_tests/cli/run_tests/__snapshots__/args.test.js.snap

+  --help                         Display this menu and exit. 
+  --config <file>                Pass in a config. Can pass in multiple configs. 
+  --esFrom <snapshot|source>     Build Elasticsearch from source or run from snapshot. Default: snapshot
+  --kibana-install-dir <dir>     Run Kibana from existing install directory instead of from source. 


nit: it's the only parameter that uses dashes now, maybe we can make it --kibanaInsallDir just for consistency?

kobelb

LGTM - the improved test coverage is great!

* Replace test:api with @kbn/test runTests * Improve CLI help menu 🆘 * Use --es-from * Replace jenkins:selenium with kbn-test * Validate cli args, fixing test in the process * Clean up some stuff * Code review fixes * Explanation for collectCliArgs * Remove exit codes, they're useless anyway. * Make markdown vis test pass with dev_mode setting * Tests * Remove unneeded export * Code review: move console logging up to cli.js * Code review: refactor startServers and runTests to take single options collection * Code review: Remove all things I am sure we do not use * Improve tests * Code review fixes * Pass created log to runFtr, runElasticsearch, runKibanaServer * Update --es-from option to --esFrom

* [Functional Tests] Use @kbn/test on Kibana CI (#18967) * Replace test:api with @kbn/test runTests * Improve CLI help menu 🆘 * Use --es-from * Replace jenkins:selenium with kbn-test * Validate cli args, fixing test in the process * Clean up some stuff * Code review fixes * Explanation for collectCliArgs * Remove exit codes, they're useless anyway. * Make markdown vis test pass with dev_mode setting * Tests * Remove unneeded export * Code review: move console logging up to cli.js * Code review: refactor startServers and runTests to take single options collection * Code review: Remove all things I am sure we do not use * Improve tests * Code review fixes * Pass created log to runFtr, runElasticsearch, runKibanaServer * Update --es-from option to --esFrom * Let dev server run from snapshot by default

* [Functional Tests] Use @kbn/test on Kibana CI (#18967) * Replace test:api with @kbn/test runTests * Improve CLI help menu 🆘 * Use --es-from * Replace jenkins:selenium with kbn-test * Validate cli args, fixing test in the process * Clean up some stuff * Code review fixes * Explanation for collectCliArgs * Remove exit codes, they're useless anyway. * Make markdown vis test pass with dev_mode setting * Tests * Remove unneeded export * Code review: move console logging up to cli.js * Code review: refactor startServers and runTests to take single options collection * Code review: Remove all things I am sure we do not use * Improve tests * Code review fixes * Pass created log to runFtr, runElasticsearch, runKibanaServer * Update --es-from option to --esFrom * Still need dev_mode setting in 6.3

rhoboat changed the title ~~Replace test:api with @kbn/test runTests~~ [Functional Tests] Use @kbn/test on Kibana CI May 10, 2018

rhoboat commented May 10, 2018

View reviewed changes

kimjoar mentioned this pull request May 17, 2018

Integration tests for new platform #12699

Closed

13 tasks

spalger reviewed May 21, 2018

View reviewed changes

packages/kbn-test/src/es/es_test_cluster.js Outdated

Copy link

Contributor

spalger May 21, 2018 •

edited

Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes seems really redundant.

spalger reviewed May 21, 2018

View reviewed changes

rhoboat added the WIP Work in progress label May 22, 2018

Code review fixes

e16024e

Pass created log to runFtr, runElasticsearch, runKibanaServer

f12aa46

This was referenced Jun 25, 2018

Add pluggable panel action tests #20163

Merged

see if this command is bing run on jenkins #20210

Closed

archanid added 2 commits June 25, 2018 14:17

Update --es-from option to --esFrom

47dcc07

Merge branch 'master' into archanid-18593

c888b65

azasypkin approved these changes Jun 26, 2018

View reviewed changes

kobelb approved these changes Jun 26, 2018

View reviewed changes

rhoboat merged commit b03243f into elastic:master Jun 26, 2018

rhoboat added the backport pending label Jun 28, 2018

rhoboat deleted the archanid-18593 branch June 28, 2018 16:56

rhoboat mentioned this pull request Jul 1, 2018

Run functional test dev server from snapshot by default #20312

Merged

rhoboat mentioned this pull request Jul 23, 2018

Deprecate test:ui commands #21092

Merged

spalger removed backport pending labels Sep 11, 2018

Conversation

rhoboat commented May 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Running from source/snapshot

How to test:

Unit tests

Integration tests

Manual testing in development

Uh oh!

rhoboat May 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rhoboat commented May 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented May 10, 2018

💔 Build Failed

Uh oh!

rhoboat commented May 10, 2018

Uh oh!

elasticmachine commented May 10, 2018

💔 Build Failed

Uh oh!

spalger commented May 10, 2018

Uh oh!

spalger commented May 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tylersmalley commented May 10, 2018

Uh oh!

elasticmachine commented May 17, 2018

💔 Build Failed

Uh oh!

elasticmachine commented May 17, 2018

💔 Build Failed

Uh oh!

rhoboat commented May 17, 2018

Uh oh!

elasticmachine commented May 17, 2018

💔 Build Failed

Uh oh!

elasticmachine commented May 18, 2018

💔 Build Failed

Uh oh!

elasticmachine commented May 21, 2018

💔 Build Failed

Uh oh!

elasticmachine commented May 21, 2018

💔 Build Failed

Uh oh!

elasticmachine commented May 21, 2018

💚 Build Succeeded

Uh oh!

spalger May 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

spalger May 21, 2018

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented May 21, 2018

💔 Build Failed

Uh oh!

elasticmachine commented May 21, 2018

💔 Build Failed

Uh oh!

rhoboat commented May 21, 2018

Uh oh!

elasticmachine commented May 22, 2018

💔 Build Failed

Uh oh!

rhoboat commented May 22, 2018

Uh oh!

rhoboat commented May 22, 2018

Uh oh!

elasticmachine commented May 22, 2018

💔 Build Failed

Uh oh!

rhoboat commented May 10, 2018 •

edited

Loading

rhoboat May 10, 2018 •

edited

Loading

rhoboat commented May 10, 2018 •

edited

Loading

spalger commented May 10, 2018 •

edited

Loading

spalger May 21, 2018 •

edited

Loading