Skip to content

Conversation

@lizozom
Copy link
Contributor

@lizozom lizozom commented Jul 9, 2020

Summary

Improve test stability

Backported here #71211

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@lizozom lizozom added v8.0.0 Team:AppArch release_note:skip Skip the PR/issue when compiling release notes v7.9.0 labels Jul 9, 2020
@lizozom lizozom self-assigned this Jul 9, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app-arch (Team:AppArch)

@lizozom lizozom requested a review from kertal July 9, 2020 14:51
@timroes
Copy link
Contributor

timroes commented Jul 9, 2020

Since the test is now skipped on master, this needs to merge upstream and unskip it again.

@lizozom
Copy link
Contributor Author

lizozom commented Jul 9, 2020

@elasticmachine merge upstream

@lizozom
Copy link
Contributor Author

lizozom commented Jul 11, 2020

@elasticmachine merge upstream

Copy link
Member

@kertal kertal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix looks good to flaky test runner: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/582/testReport/ 👍
Could you modify the title and description of this PR to be a bit more descriptive? Many thx. I bet it's the long weekend but I totally forgot what bug fix 7198 was 🧠

const esArchiver = getService('esArchiver');
const retry = getService('retry');

// Flaky: https://github.com/elastic/kibana/issues/71216
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can remove this comment now

@lizozom lizozom merged commit 79df8e0 into elastic:master Jul 13, 2020
@lizozom lizozom changed the title improve bugfix 7198 test stability improve bugfix 7198 (null tests in discover) test stability Jul 13, 2020
@lizozom lizozom removed the v7.9.0 label Jul 13, 2020
@kibanamachine
Copy link
Contributor

💛 Build succeeded, but was flaky


Test Failures

Kibana Pipeline / kibana-xpack-agent / Kibana Embedded in iframe with X-Pack Security.x-pack/test/functional_embedded/tests/iframe_embedded·ts.Kibana embedded in iframe should open Kibana for logged-in user

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 5 times on tracked branches: https://github.com/elastic/kibana/issues/70928

[00:00:00]       │
[00:00:00]         └-: Kibana embedded
[00:00:00]           └-> "before all" hook
[00:00:00]           └-: in iframe
[00:00:00]             └-> "before all" hook
[00:00:00]             └-> should open Kibana for logged-in user
[00:00:00]               └-> "before each" hook: global before each
[00:00:00]               │ debg TestSubjects.exists(headerGlobalNav)
[00:00:00]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="headerGlobalNav"]') with timeout=2500
[00:00:02]               │ debg --- retry.tryForTime error: [data-test-subj="headerGlobalNav"] is not displayed
[00:00:03]               │ debg navigating to login url: https://localhost:6131/login
[00:00:03]               │ debg navigate to: https://localhost:6131/login
[00:00:03]               │ debg browser[INFO] https://localhost:6131/login?_t=1594626075031 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:03]               │
[00:00:03]               │ debg browser[INFO] https://localhost:6131/bundles/app/core/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:03]               │ debg ... sleep(700) start
[00:00:04]               │ debg ... sleep(700) end
[00:00:04]               │ debg returned from get, calling refresh
[00:00:04]               │ debg browser[INFO] https://localhost:6131/login?_t=1594626075031 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:04]               │
[00:00:04]               │ debg browser[INFO] https://localhost:6131/bundles/app/core/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:04]               │ debg currentUrl = https://localhost:6131/login
[00:00:04]               │          appUrl = https://localhost:6131/login
[00:00:04]               │ debg TestSubjects.find(kibanaChrome)
[00:00:04]               │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:00:08]               │ debg browser[INFO] https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js 452:106112 "INFO: 2020-07-13T07:41:18Z
[00:00:08]               │        Adding connection to https://localhost:6131/elasticsearch
[00:00:08]               │
[00:00:08]               │      "
[00:00:08]               │ debg ... sleep(501) start
[00:00:08]               │ debg ... sleep(501) end
[00:00:08]               │ debg in navigateTo url = https://localhost:6131/login
[00:00:08]               │ debg TestSubjects.exists(statusPageContainer)
[00:00:08]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:00:11]               │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:00:12]               │ debg Waiting for Login Form to appear.
[00:00:12]               │ debg Waiting up to 100000ms for login form...
[00:00:12]               │ debg TestSubjects.exists(loginForm)
[00:00:12]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="loginForm"]') with timeout=2500
[00:00:12]               │ debg TestSubjects.setValue(loginUsername, elastic)
[00:00:12]               │ debg TestSubjects.click(loginUsername)
[00:00:12]               │ debg Find.clickByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:00:12]               │ debg Find.findByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:00:12]               │ debg TestSubjects.setValue(loginPassword, changeme)
[00:00:12]               │ debg TestSubjects.click(loginPassword)
[00:00:12]               │ debg Find.clickByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:00:12]               │ debg Find.findByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:00:12]               │ debg TestSubjects.click(loginSubmit)
[00:00:12]               │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:00:12]               │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:00:12]               │ debg Waiting for login result, expected: undefined.
[00:00:12]               │ debg Waiting up to 20000ms for logout button visible...
[00:00:12]               │ debg TestSubjects.exists(userMenuButton)
[00:00:12]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenuButton"]') with timeout=2500
[00:00:15]               │ debg browser[INFO] https://localhost:6131/app/home 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:15]               │
[00:00:15]               │ debg browser[INFO] https://localhost:6131/bundles/app/core/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:15]               │ debg browser[INFO] https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js 452:106112 "INFO: 2020-07-13T07:41:26Z
[00:00:15]               │        Adding connection to https://localhost:6131/elasticsearch
[00:00:15]               │
[00:00:15]               │      "
[00:00:15]               │ debg --- retry.tryForTime error: [data-test-subj="userMenuButton"] is not displayed
[00:00:16]               │ proc [kibana]  error  [07:41:28.065] [error][client][connection] Error: 140600031983488:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:
[00:00:16]               │ proc [kibana] 
[00:00:16]               │ERROR browser[SEVERE] http://localhost:6131/api/_newsfeed-FTS-external-service-simulators/kibana/v8.0.0-SNAPSHOT.json - Failed to load resource: net::ERR_EMPTY_RESPONSE
[00:00:16]               │ERROR browser[SEVERE] https://localhost:6131/34549/bundles/plugin/newsfeed/newsfeed.plugin.js 0:19107 TypeError: Failed to fetch
[00:00:16]               │          at Fetch._callee3$ (https://localhost:6131/34549/bundles/core/core.entry.js:34:105174)
[00:00:16]               │          at l (https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155138)
[00:00:16]               │          at Generator._invoke (https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:154891)
[00:00:16]               │          at Generator.forEach.e.<computed> [as throw] (https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155495)
[00:00:16]               │          at fetch_asyncGeneratorStep (https://localhost:6131/34549/bundles/core/core.entry.js:34:99267)
[00:00:16]               │          at _throw (https://localhost:6131/34549/bundles/core/core.entry.js:34:99675)
[00:00:16]               │ debg TestSubjects.exists(userMenuButton)
[00:00:16]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenuButton"]') with timeout=2500
[00:00:16]               │ debg TestSubjects.exists(userMenu)
[00:00:16]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"]') with timeout=2500
[00:00:19]               │ debg --- retry.tryForTime error: [data-test-subj="userMenu"] is not displayed
[00:00:19]               │ debg TestSubjects.click(userMenuButton)
[00:00:19]               │ debg Find.clickByCssSelector('[data-test-subj="userMenuButton"]') with timeout=10000
[00:00:19]               │ debg Find.findByCssSelector('[data-test-subj="userMenuButton"]') with timeout=10000
[00:00:19]               │ debg Waiting up to 20000ms for user menu opened...
[00:00:19]               │ debg TestSubjects.exists(userMenu)
[00:00:19]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"]') with timeout=2500
[00:00:19]               │ debg TestSubjects.exists(userMenu > logoutLink)
[00:00:19]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"] [data-test-subj="logoutLink"]') with timeout=2500
[00:00:20]               │ debg TestSubjects.find(iframe_embedded)
[00:00:20]               │ debg Find.findByCssSelector('[data-test-subj="iframe_embedded"]') with timeout=10000
[00:00:20]               │ debg browser[INFO] https://localhost:6131/app/home 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:20]               │
[00:00:20]               │ debg browser[INFO] https://localhost:6131/bundles/app/core/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:20]               │ debg TestSubjects.exists(headerGlobalNav)
[00:00:20]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="headerGlobalNav"]') with timeout=2500
[00:00:22]               │ debg browser[INFO] https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js 452:106112 "INFO: 2020-07-13T07:41:33Z
[00:00:22]               │        Adding connection to https://localhost:6131/elasticsearch
[00:00:22]               │
[00:00:22]               │      "
[00:00:22]               │ debg --- retry.tryForTime error: [data-test-subj="headerGlobalNav"] is not displayed
[00:00:23]               │ proc [kibana]  error  [07:41:35.080] [error][client][connection] Error: 140600031983488:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:
[00:00:23]               │ proc [kibana] 
[00:00:23]               │ info Taking screenshot "/dev/shm/workspace/kibana/x-pack/test/functional_embedded/screenshots/failure/Kibana embedded in iframe should open Kibana for logged-in user.png"
[00:00:23]               │ERROR browser[SEVERE] http://localhost:6131/api/_newsfeed-FTS-external-service-simulators/kibana/v8.0.0-SNAPSHOT.json - Failed to load resource: net::ERR_EMPTY_RESPONSE
[00:00:23]               │ERROR browser[SEVERE] https://localhost:6131/34549/bundles/plugin/newsfeed/newsfeed.plugin.js 0:19107 TypeError: Failed to fetch
[00:00:23]               │          at Fetch._callee3$ (https://localhost:6131/34549/bundles/core/core.entry.js:34:105174)
[00:00:23]               │          at l (https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155138)
[00:00:23]               │          at Generator._invoke (https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:154891)
[00:00:23]               │          at Generator.forEach.e.<computed> [as throw] (https://localhost:6131/34549/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155495)
[00:00:23]               │          at fetch_asyncGeneratorStep (https://localhost:6131/34549/bundles/core/core.entry.js:34:99267)
[00:00:23]               │          at _throw (https://localhost:6131/34549/bundles/core/core.entry.js:34:99675)
[00:00:23]               │ info Current URL is: https://localhost:6131/iframe_embedded
[00:00:23]               │ info Saving page source to: /dev/shm/workspace/kibana/x-pack/test/functional_embedded/failure_debug/html/Kibana embedded in iframe should open Kibana for logged-in user.html
[00:00:23]               └- ✖ fail: Kibana embedded in iframe should open Kibana for logged-in user
[00:00:23]               │      Error: expected true to equal false
[00:00:23]               │       at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
[00:00:23]               │       at Assertion.be.Assertion.equal (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:227:8)
[00:00:23]               │       at Assertion.be (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:69:22)
[00:00:23]               │       at Context.it (test/functional_embedded/tests/iframe_embedded.ts:39:33)
[00:00:23]               │ 
[00:00:23]               │ 

Stack Trace

Error: expected true to equal false
    at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.be.Assertion.equal (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:227:8)
    at Assertion.be (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:69:22)
    at Context.it (test/functional_embedded/tests/iframe_embedded.ts:39:33)

Build metrics

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

gmmorris added a commit to gmmorris/kibana that referenced this pull request Jul 13, 2020
* master: (78 commits)
  Bump lodash package version (elastic#71392)
  refactor: 💡 use allow-list in AppArch codebase (elastic#71400)
  improve bugfix 7198 test stability (elastic#71250)
  [Security Solution][Ingest Manager][Endpoint] Optional ingest manager (elastic#71198)
  [Metrics UI] Round metric threshold time buckets to nearest unit (elastic#71172)
  [Security Solution][Endpoint] Policy creation callback fixes + Improved error handling in user manifest loop (elastic#71269)
  [Security Solution] Allow to configure Event Renderers settings (elastic#69693)
  Fix a11y keyboard overlay (elastic#71214)
  [APM] UI text updates (elastic#71333)
  [Logs UI] Limit `extendDatemath` to valid ranges (elastic#71113)
  [SIEM] fix tooltip of notes (elastic#71342)
  address index templates feedback (elastic#71353)
  Upgrade EUI to v26.3.1 (elastic#70243)
  [build] Creates Linux aarch64 archive (elastic#69165)
  [SIEM][Detection Engine] Fixes skipped tests (elastic#71347)
  [SIEM][Detection Engine][Lists] Adds read_privileges route for lists and list items
  [kbn/optimizer] implement "requiredBundles" property of KP plugins (elastic#70911)
  [Security Solution][Exceptions] - Exceptions modal pt 2 (elastic#70886)
  [ML] DF Analytics:  stop status polling when job stopped (elastic#71159)
  [SIEM][CASE] IBM Resilient Connector (elastic#66385)
  ...
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jul 15, 2020
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

2 similar comments
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

18 similar comments
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71250 or prevent reminders by adding the backport:skip label.

@lizozom lizozom added the backport:skip This PR does not require backporting label Aug 17, 2020
@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Aug 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting release_note:skip Skip the PR/issue when compiling release notes v8.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants