Skip to content

Conversation

@brianseeders
Copy link
Contributor

@brianseeders brianseeders commented May 8, 2020

Some builds in CI have been failing randomly by hitting the 2h 15m timeout. It doesn't happen on master, but happens once or twice a day or so on 7.*.

It turned out to be the upgrade_assistant tests. Sometimes, no reserved users exist after doing the initial data load. This would be 6.x data since it's for the upgrade assistant, which would explain why this issue is not affecting master (it has 7.x data for 8.0).

When no reserved users were found, kibana would never fully initialize, and a few errors would just repeat until the entire job timed out.

This PR immediately throws an error when this happens. This won't fix the main problem, but it will at least turn our timeouts into failures and give us some better data.

I was able to re-produce the problem and see this code working in the flaky test suite runner here: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/443/execution/node/411/log/

@brianseeders brianseeders added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.9.0 v7.6.3 v7.7.1 v7.8.1 labels May 8, 2020
@brianseeders
Copy link
Contributor Author

@elasticmachine merge upstream

@brianseeders brianseeders added the Team:Operations Kibana-Operations Team label May 11, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations (Team:Operations)

@brianseeders brianseeders marked this pull request as ready for review May 11, 2020 19:13
@brianseeders brianseeders requested a review from a team as a code owner May 11, 2020 19:13
Copy link
Contributor

@spalger spalger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@brianseeders
Copy link
Contributor Author

@elasticmachine merge upstream

@spalger
Copy link
Contributor

spalger commented May 11, 2020

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release_note:skip Skip the PR/issue when compiling release notes Team:Operations Kibana-Operations Team v7.6.3 v7.7.1 v7.8.1 v7.9.0 v8.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants