[ci][ftr][kbn/test] split up CI jobs#25838
Merged
spalger merged 6 commits intoelastic:masterfrom Nov 20, 2018
Merged
Conversation
Contributor
|
Pinging @elastic/kibana-operations |
spalger
force-pushed
the
implement/ci-groups
branch
from
e138493 to
d456549
Compare
This comment has been minimized.
This comment has been minimized.
spalger
force-pushed
the
implement/ci-groups
branch
from
e3f492b to
8128877
Compare
Contributor
💚 Build Succeeded |
spalger
force-pushed
the
implement/ci-groups
branch
from
8128877 to
8055ca3
Compare
Contributor
💚 Build Succeeded |
Contributor
💚 Build Succeeded |
Contributor
💚 Build Succeeded |
Contributor
💚 Build Succeeded |
Member
|
Much better! |
tylersmalley
approved these changes
Nov 20, 2018
spalger
pushed a commit
to spalger/kibana
that referenced
this pull request
Nov 20, 2018
* [ci][ftr][kbn/test] split up CI jobs * [ci] run --assert-none-excluded in ci groups, before builds * [ftr] improve error message when excluded tests found * [ci] document other places the ciGroups live
spalger
pushed a commit
that referenced
this pull request
Nov 20, 2018
* [ci][ftr][kbn/test] split up CI jobs (#25838) * [ci][ftr][kbn/test] split up CI jobs * [ci] run --assert-none-excluded in ci groups, before builds * [ftr] improve error message when excluded tests found * [ci] document other places the ciGroups live * [grunt] fix --kibana-install-dir value
Contributor
Author
|
6.x/6.6: c64fab0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #22359, thank you @LeeDr for kicking off this effort!
Fixes #22041
Our current CI jobs take about 2 hours to complete, which is long, and even worse when they have to be re-run because of a flaky test failure. Since the Kibana project just keep growing, and functional testing is a very useful way to test our software, we need to find a way to scale our tests. This PR splits the existing 3 CI jobs into 20 to help with that.
Each job sets up its own environment, many build Kibana in the process, and then run a small chunk of the tests that our old jobs did.
Just like the current jobs it takes about 15 minutes to bootstrap the environment. Because we have so many jobs now we are at an increased risk of observing transient network failures while installing node modules, so to prevent this issue we are working with infra to bake the node_modules, es, and optimizer caches into the testing images. This helps us decrease the risk of network related failures, and helps speed things up a bit.
In the future we hope to decrease the redundancy in the CI jobs do, but setting up relationships between jobs is a difficult process that will take time to figure out.
All of our non-ftr tests run, along with the functional tests that don't run against built versions of Kibana, run in the
kibana-intakeorx-pack-intakejobs (linting, license checks, api_integration, etc.). The other job are for tests run by the functional test runner that execute against a built version of Kibana.The functional test runner knows which tests to run in which group using suite tags which we introduced in #25021. Tests in
test/functionalcan use tagsciGroup1throughciGroup12to assign a suite to a specific job. Tests inx-pack/test/*can use the tagsciGroup1throughciGroup6. If you use a tag that is not supported or forget to tag your tests the CI job will refuse to run.The jobs are somewhat balanced right now, so when adding a new set of tests please consider the execution time of the other tests and avoid adding your tests to a group that is already slower than others.