Skip to content

[ML] Adds security_linux and security_windows Modules #85065

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Dec 10, 2020
Merged

[ML] Adds security_linux and security_windows Modules #85065

merged 12 commits into from
Dec 10, 2020

Conversation

@randomuserid
Copy link
Contributor

@randomuserid randomuserid commented Dec 4, 2020 •
edited by blaklaybul
Loading

Summary

Adds the files for a 2 new modules - security_linux and security_windows - for use within the Security app, containing the job and datafeed configuration files that support multi-index, multi-pipeline analyses across beats and agents that collect linux and windows data.

Contains:

  • 2 Module manifest.json
  • ML Job configurations for 14 jobs (job ids still tbd):
    • security_linux:
      • v2_rare_process_by_host_linux_ecs
      • v2_linux_rare_metadata_user
      • v2_linux_rare_metadata_process
      • v2_linux_anomalous_user_name_ecs
      • v2_linux_anomalous_process_all_hosts_ecs
      • v2_linux_anomalous_network_port_activity_ecs
    • security_windows:
      • v2_rare_process_by_host_windows_ecs
      • v2_windows_anomalous_network_activity_ecs
      • v2_windows_anomalous_path_activity_ecs
      • v2_windows_anomalous_process_all_hosts_ecs
      • v2_windows_anomalous_process_creation
      • v2_windows_anomalous_user_name_ecs
      • v2_windows_rare_metadata_process
      • v2_windows_rare_metadata_user
  • corresponding datafeed configurations
  • Logo

initial commit
e08ddf2 
refactored multi-index, multi-pipeline jobs for 7.11. These are new modules that will live alongside the existing jobs.
@randomuserid randomuserid requested a review from a team as a code owner December 4, 2020 20:10
@elasticmachine
Copy link
Contributor

elasticmachine commented Dec 4, 2020

Pinging @elastic/ml-ui (:ml)

@randomuserid randomuserid self-assigned this Dec 4, 2020
@spong
Copy link
Member

spong commented Dec 4, 2020

Would you be willing to add the two new modules here @randomuserid? This will ensure they show up within the Security Solution ML UI 🙂

https://github.com/elastic/kibana/blob/765c2d1ad3308a3c3af50f8d67b80579aeb13a9a/x-pack/plugins/security_solution/public/common/components/ml_popover/ml_modules.tsx

Update ml_modules.tsx
010974f 
added new module names to the list
@randomuserid randomuserid requested review from a team as code owners December 4, 2020 22:33
@spong
Copy link
Member

spong commented Dec 4, 2020

For your failing tests, add your new modules here:

kibana/x-pack/test/api_integration/apis/ml/modules/get_module.ts

Lines 13 to 35 in 6d12c68

const moduleIds = [
'apache_ecs',
'apm_jsbase',
'apm_nodejs',
'apm_transaction',
'auditbeat_process_docker_ecs',
'auditbeat_process_hosts_ecs',
'logs_ui_analysis',
'logs_ui_categories',
'metricbeat_system_ecs',
'metrics_ui_hosts',
'metrics_ui_k8s',
'nginx_ecs',
'sample_data_ecommerce',
'sample_data_weblogs',
'siem_auditbeat',
'siem_auditbeat_auth',
'siem_cloudtrail',
'siem_packetbeat',
'siem_winlogbeat',
'siem_winlogbeat_auth',
'uptime_heartbeat',
];

FTR Failure:

Update get_module.ts
bc8ed98 
added new module names
@randomuserid
Copy link
Contributor Author

randomuserid commented Dec 4, 2020

Would you be willing to add the two new modules here @randomuserid? This will ensure they show up within the Security Solution ML UI 🙂

https://github.com/elastic/kibana/blob/765c2d1ad3308a3c3af50f8d67b80579aeb13a9a/x-pack/plugins/security_solution/public/common/components/ml_popover/ml_modules.tsx

done

spong
spong reviewed Dec 4, 2020
View reviewed changes
@spong spong added the v8.0.0 label Dec 5, 2020
@pheyos
Copy link
Member

pheyos commented Dec 7, 2020

Looks like our auditbeat dataset is also recognized for the security_linux module:

└- ✖ fail: apis Machine Learning modules module recognizer lists matching modules for auditbeat dataset
│       Error: Expected matching module ids for index 'ft_module_auditbeat' to be 'auditbeat_process_hosts_ecs,siem_auditbeat' (got 'auditbeat_process_hosts_ecs,security_linux,siem_auditbeat')
│       + expected - actual
│ 
│        [
│          "auditbeat_process_hosts_ecs"
│       -  "security_linux"
│          "siem_auditbeat"
│        ]

If that's the expected behavior, you can add it to the expected moduleIds in the recognizer test file.

@randomuserid
Copy link
Contributor Author

randomuserid commented Dec 7, 2020

Looks like our auditbeat dataset is also recognized for the security_linux module:

└- ✖ fail: apis Machine Learning modules module recognizer lists matching modules for auditbeat dataset
│       Error: Expected matching module ids for index 'ft_module_auditbeat' to be 'auditbeat_process_hosts_ecs,siem_auditbeat' (got 'auditbeat_process_hosts_ecs,security_linux,siem_auditbeat')
│       + expected - actual
│ 
│        [
│          "auditbeat_process_hosts_ecs"
│       -  "security_linux"
│          "siem_auditbeat"
│        ]

If that's the expected behavior, you can add it to the expected moduleIds in the recognizer test file.

I'm not familiar enough with these tests to understand this comment. The security_linux module was tested with both endpoint and auditbeat data although it should work with any Linux events in ECS format. Expected result is the jobs in the module can make use of any such data including data from beat / endpoint releases at the time of this writing.

@pheyos
Copy link
Member

pheyos commented Dec 7, 2020

I'm not familiar enough with these tests to understand this comment. The security_linux module was tested with both endpoint and auditbeat data although it should work with any Linux events in ECS format. Expected result is the jobs in the module can make use of any such data including data from beat / endpoint releases at the time of this writing.

The recognizer tests verifies the modules that are detected for a dataset. This is what the UI would offer when the user selects a dataset for job creation in the ML UI. In this case, the auditbeat dataset would bring up three module cards in the UI:

  • auditbeat_process_hosts_ecs
  • security_linux
  • siem_auditbeat

This should only happen if the detected module is running ok on the selected dataset. But your description sounds like the new security_linux module should run on auditbeat data, so I think it's fine to have the module recognized there. But maybe @blaklaybul knows more?

@peteharverson
Copy link
Contributor

peteharverson commented Dec 7, 2020

I suspect the primary method of creating the security module jobs will be through the Security plugin UI, but if the user goes in via the ML job wizard using the auditbeat data set, then I agree with @pheyos that it's fine that we now match against three modules and show the three cards for:

  • auditbeat_process_hosts_ecs
  • security_linux
  • siem_auditbeat

@blaklaybul
Copy link
Contributor

blaklaybul commented Dec 7, 2020

Given what @peteharverson and @pheyos pointed out, we will want to make the titles and descriptions as clear as possible. Currently, an index pattern that contains endpoint and auditbeat data (for example) will show the following cards:

image

Security Linux and Security Windows should have colons after Security to match the other module names. And if we intend to include the warning message in Security: Linux we should consider adding the same message to Security: Windows.

@peteharverson peteharverson added the Feature:Anomaly Detection ML anomaly detection label Dec 8, 2020
Craig added 2 commits December 8, 2020 14:21
manifest fixes
a73b72d 
added colon char to the module name and shortened the description
additon to description
b2e3100 
after talking with the security team today, adding this suggested text to the beginning of the description so it will tend to be visible to the user:
"This is a new refactored job which works on ECS compatible events across multiple indices."
@panbalag
Copy link

panbalag commented Dec 8, 2020

Screenshots from 14 jobs processing records, detecting anomalies, viewing anomalies and custom urls to security app.

downsampled1

downsampled2

linux1

windows

single cell

custom-urls

@blaklaybul blaklaybul changed the title 7.11 Security ML Modules [ML] Adds security_linux and security_windows Modules Dec 8, 2020
peteharverson
peteharverson reviewed Dec 9, 2020
View reviewed changes
@randomuserid randomuserid removed their assignment Dec 9, 2020
peteharverson
peteharverson reviewed Dec 9, 2020
View reviewed changes
influencers
f400f84 
changes to the metadata jobs to make influencers identical to the originals
blaklaybul
blaklaybul reviewed Dec 9, 2020
View reviewed changes
Copy link
Contributor

@blaklaybul blaklaybul left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated influencer lists match 👍 where have we settled on naming? Do we still want to use a v2 prefix or should we go with a more descriptive indicator, such as endpoint ?

@randomuserid
Copy link
Contributor Author

randomuserid commented Dec 10, 2020 •
edited
Loading

updated influencer lists match 👍 where have we settled on naming? Do we still want to use a v2 prefix or should we go with a more descriptive indicator, such as endpoint ?

Talked with the security team again on Tuesday and I added a sentence to the description, at their suggestion, rather than changing names. The prefix "endpoint" is not ideal because that could give the impression they are meant for some particular endpoint events when what we want people to take away is the fact that they are highly portable across data sources and not intended for any particular data type. After a number of conversations with the security team, no one has yet identified a naming convention that anyone likes. At their suggestion, I added the first sentence to the description instead which is expected to be more informative to users than any naming convention conceived this far.

peteharverson
peteharverson approved these changes Dec 10, 2020
View reviewed changes
Copy link
Contributor

@peteharverson peteharverson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested latest edits and LGTM

@panbalag
Copy link

panbalag commented Dec 10, 2020 •
edited
Loading

Verified the latest edits.

job1

job2

job3

job4

@panbalag panbalag self-requested a review December 10, 2020 13:30
panbalag
panbalag approved these changes Dec 10, 2020
View reviewed changes
Copy link

@panbalag panbalag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@randomuserid randomuserid requested a review from spong December 10, 2020 13:40
spong
spong approved these changes Dec 10, 2020
View reviewed changes
Copy link
Member

@spong spong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for wiring this up with the security app as well @randomuserid! 🙂

change for security app
8e8372a 
changes to two datafeeds needed for logic in the Security app - added the suffix "_ecs" to two ids.
@spong
Copy link
Member

spong commented Dec 10, 2020

In testing with @randomuserid we noticed the following two jobs wouldn't start within the Security UI as their datafeed ID's did not match their job names (were missing the _ecs postfix). The Security UI currently relies on this commonality between job ID and datafeed ID.

  • v2_linux_anomalous_network_port_activity_ecs
  • v2_linux_anomalous_user_name_ecs

@randomuserid updated the datafeed ID's to match in 8e8372a, and I pulled the latest and verified functionality in both the Security UI and ML UI. Thanks for pairing on this Craig! 🙂

@kibanamachine
Copy link
Contributor

kibanamachine commented Dec 10, 2020

💛 Build succeeded, but was flaky

Test Failures

Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/ml/data_frame_analytics/feature_importance·ts.machine learning data frame analytics total feature importance panel and decision path popover binary classification job should display the feature importance decision path in the data grid

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: machine learning
[00:00:00]           └-> "before all" hook
[00:00:00]           └-> "before all" hook
[00:00:00]             │ debg creating role ft_ml_source
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added role [ft_ml_source]
[00:00:00]             │ debg creating role ft_ml_source_readonly
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added role [ft_ml_source_readonly]
[00:00:00]             │ debg creating role ft_ml_dest
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added role [ft_ml_dest]
[00:00:00]             │ debg creating role ft_ml_dest_readonly
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added role [ft_ml_dest_readonly]
[00:00:00]             │ debg creating role ft_ml_ui_extras
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added role [ft_ml_ui_extras]
[00:00:00]             │ debg creating role ft_default_space_ml_all
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added role [ft_default_space_ml_all]
[00:00:00]             │ debg creating role ft_default_space1_ml_all
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added role [ft_default_space1_ml_all]
[00:00:00]             │ debg creating role ft_all_spaces_ml_all
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added role [ft_all_spaces_ml_all]
[00:00:00]             │ debg creating role ft_default_space_ml_read
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added role [ft_default_space_ml_read]
[00:00:00]             │ debg creating role ft_default_space1_ml_read
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added role [ft_default_space1_ml_read]
[00:00:00]             │ debg creating role ft_all_spaces_ml_read
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added role [ft_all_spaces_ml_read]
[00:00:00]             │ debg creating role ft_default_space_ml_none
[00:00:00]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added role [ft_default_space_ml_none]
[00:00:00]             │ debg creating user ft_ml_poweruser
[00:00:00]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added user [ft_ml_poweruser]
[00:00:00]             │ debg created user ft_ml_poweruser
[00:00:00]             │ debg creating user ft_ml_poweruser_spaces
[00:00:00]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added user [ft_ml_poweruser_spaces]
[00:00:00]             │ debg created user ft_ml_poweruser_spaces
[00:00:00]             │ debg creating user ft_ml_poweruser_space1
[00:00:00]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added user [ft_ml_poweruser_space1]
[00:00:00]             │ debg created user ft_ml_poweruser_space1
[00:00:00]             │ debg creating user ft_ml_poweruser_all_spaces
[00:00:00]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added user [ft_ml_poweruser_all_spaces]
[00:00:00]             │ debg created user ft_ml_poweruser_all_spaces
[00:00:00]             │ debg creating user ft_ml_viewer
[00:00:01]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added user [ft_ml_viewer]
[00:00:01]             │ debg created user ft_ml_viewer
[00:00:01]             │ debg creating user ft_ml_viewer_spaces
[00:00:01]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added user [ft_ml_viewer_spaces]
[00:00:01]             │ debg created user ft_ml_viewer_spaces
[00:00:01]             │ debg creating user ft_ml_viewer_space1
[00:00:01]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added user [ft_ml_viewer_space1]
[00:00:01]             │ debg created user ft_ml_viewer_space1
[00:00:01]             │ debg creating user ft_ml_viewer_all_spaces
[00:00:01]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added user [ft_ml_viewer_all_spaces]
[00:00:01]             │ debg created user ft_ml_viewer_all_spaces
[00:00:01]             │ debg creating user ft_ml_unauthorized
[00:00:01]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added user [ft_ml_unauthorized]
[00:00:01]             │ debg created user ft_ml_unauthorized
[00:00:01]             │ debg creating user ft_ml_unauthorized_spaces
[00:00:01]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] added user [ft_ml_unauthorized_spaces]
[00:00:01]             │ debg created user ft_ml_unauthorized_spaces
[00:39:42]           └-: data frame analytics
[00:39:42]             └-> "before all" hook
[00:44:30]             └-: total feature importance panel and decision path popover
[00:44:30]               └-> "before all" hook
[00:44:30]               └-> "before all" hook
[00:44:30]                 │ debg applying update to kibana config: {"dateFormat:tz":"UTC"}
[00:44:31]                 │ debg SecurityPage.forceLogout
[00:44:31]                 │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=100
[00:44:31]                 │ debg --- retry.tryForTime error: .login-form is not displayed
[00:44:32]                 │ debg Redirecting to /logout to force the logout
[00:44:32]                 │ debg Waiting on the login form to appear
[00:44:32]                 │ debg Waiting for Login Page to appear.
[00:44:32]                 │ debg Waiting up to 100000ms for login page...
[00:44:32]                 │ debg browser[INFO] http://localhost:6151/logout?_t=1607622869444 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:44:32]                 │
[00:44:32]                 │ debg browser[INFO] http://localhost:6151/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:44:32]                 │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=2500
[00:44:34]                 │ debg browser[INFO] http://localhost:6151/login?msg=LOGGED_OUT 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:44:34]                 │
[00:44:34]                 │ debg browser[INFO] http://localhost:6151/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:44:34]                 │ debg --- retry.tryForTime error: .login-form is not displayed
[00:44:35]                 │ERROR browser[SEVERE] http://localhost:6151/internal/spaces/_active_space - Failed to load resource: the server responded with a status of 401 (Unauthorized)
[00:44:35]                 │ debg browser[INFO] http://localhost:6151/38796/bundles/core/core.entry.js 12:194556 "Detected an unhandled Promise rejection.
[00:44:35]                 │      Error: Unauthorized"
[00:44:35]                 │ERROR browser[SEVERE] http://localhost:6151/38796/bundles/core/core.entry.js 5:3002 
[00:44:35]                 │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=2500
[00:44:36]                 │ debg TestSubjects.exists(loginForm)
[00:44:36]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="loginForm"]') with timeout=2500
[00:44:36]                 │ debg Waiting for Login Form to appear.
[00:44:36]                 │ debg Waiting up to 100000ms for login form...
[00:44:36]                 │ debg TestSubjects.exists(loginForm)
[00:44:36]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="loginForm"]') with timeout=2500
[00:44:36]                 │ debg TestSubjects.setValue(loginUsername, ft_ml_poweruser)
[00:44:36]                 │ debg TestSubjects.click(loginUsername)
[00:44:36]                 │ debg Find.clickByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:44:36]                 │ debg Find.findByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:44:36]                 │ debg TestSubjects.setValue(loginPassword, mlp001)
[00:44:36]                 │ debg TestSubjects.click(loginPassword)
[00:44:36]                 │ debg Find.clickByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:44:36]                 │ debg Find.findByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:44:36]                 │ debg TestSubjects.click(loginSubmit)
[00:44:36]                 │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:44:36]                 │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:44:36]                 │ debg Waiting for login result, expected: chrome.
[00:44:36]                 │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"] .app-wrapper:not(.hidden-chrome)') with timeout=20000
[00:44:36]                 │ proc [kibana]   log   [17:54:33.823] [info][plugins][routes][security] Logging in with provider "basic" (basic)
[00:44:38]                 │ debg browser[INFO] http://localhost:6151/app/home 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:44:38]                 │
[00:44:38]                 │ debg browser[INFO] http://localhost:6151/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:44:38]                 │ debg Finished login process currentUrl = http://localhost:6151/app/home#/
[00:44:38]                 │ debg Waiting up to 20000ms for logout button visible...
[00:44:38]                 │ debg TestSubjects.exists(userMenuButton)
[00:44:38]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenuButton"]') with timeout=2500
[00:44:38]                 │ debg TestSubjects.exists(userMenu)
[00:44:38]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"]') with timeout=2500
[00:44:41]                 │ debg --- retry.tryForTime error: [data-test-subj="userMenu"] is not displayed
[00:44:41]                 │ debg TestSubjects.click(userMenuButton)
[00:44:41]                 │ debg Find.clickByCssSelector('[data-test-subj="userMenuButton"]') with timeout=10000
[00:44:41]                 │ debg Find.findByCssSelector('[data-test-subj="userMenuButton"]') with timeout=10000
[00:44:41]                 │ debg TestSubjects.exists(userMenu)
[00:44:41]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"]') with timeout=120000
[00:44:41]                 │ debg TestSubjects.exists(userMenu > logoutLink)
[00:44:41]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"] [data-test-subj="logoutLink"]') with timeout=2500
[00:44:41]                 │ info [ml/ihp_outlier] Loading "mappings.json"
[00:44:41]                 │ info [ml/ihp_outlier] Loading "data.json.gz"
[00:44:41]                 │ info [ml/ihp_outlier] Skipped restore for existing index "ft_ihp_outlier"
[00:44:41]                 │ debg Searching for 'index-pattern' with title 'ft_ihp_outlier'...
[00:44:41]                 │ debg  > Found 'f10b6660-3b0a-11eb-964c-3b9b3d3fb8a8'
[00:44:41]                 │ debg Index pattern with title 'ft_ihp_outlier' already exists. Nothing to create.
[00:44:41]                 │ debg Creating data frame analytic job with id 'ihp_fi_binary_1607620196901' ...
[00:44:41]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [.ml-config] creating index, cause [auto(bulk api)], templates [.ml-config], shards [1]/[1]
[00:44:41]                 │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] updating number_of_replicas to [0] for indices [.ml-config]
[00:44:41]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [.ml-annotations-6] creating index, cause [api], templates [], shards [1]/[1]
[00:44:41]                 │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] updating number_of_replicas to [0] for indices [.ml-annotations-6]
[00:44:42]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [.ml-config/Fnlb5sAbQM27s_qlucFMrg] update_mapping [_doc]
[00:44:42]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [.ml-notifications-000001] creating index, cause [auto(bulk api)], templates [.ml-notifications-000001], shards [1]/[1]
[00:44:42]                 │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] updating number_of_replicas to [0] for indices [.ml-notifications-000001]
[00:44:42]                 │ debg Waiting up to 5000ms for 'ihp_fi_binary_1607620196901' to exist...
[00:44:42]                 │ debg Fetching data frame analytics job 'ihp_fi_binary_1607620196901'...
[00:44:42]                 │ debg > DFA job fetched.
[00:44:42]                 │ debg > DFA job created.
[00:44:42]                 │ debg Starting data frame analytics job 'ihp_fi_binary_1607620196901'...
[00:44:42]                 │ info [o.e.x.m.a.TransportStartDataFrameAnalyticsAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_binary_1607620196901] Starting data frame analytics from state [null]
[00:44:42]                 │ info [o.e.x.c.m.u.MlIndexAndAlias] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] About to create first concrete index [.ml-state-000001] with alias [.ml-state-write]
[00:44:42]                 │ debg > DFA job started.
[00:44:42]                 │ debg Waiting up to 60000ms for 'ihp_fi_binary_1607620196901' to have training_docs_count > 0...
[00:44:42]                 │ debg Fetching data frame analytics job stats for job ihp_fi_binary_1607620196901...
[00:44:42]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [.ml-state-000001] creating index, cause [api], templates [.ml-state], shards [1]/[1]
[00:44:42]                 │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] updating number_of_replicas to [0] for indices [.ml-state-000001]
[00:44:42]                 │ debg > DFA job stats fetched.
[00:44:42]                 │ debg --- retry.waitForWithTimeout error: expected data frame analytics job 'ihp_fi_binary_1607620196901' to have training_docs_count > 0 (got 0)
[00:44:42]                 │ info [o.e.x.c.m.u.MlIndexAndAlias] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] About to create first concrete index [.ml-stats-000001] with alias [.ml-stats-write]
[00:44:42]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [.ml-stats-000001] creating index, cause [api], templates [.ml-stats], shards [1]/[1]
[00:44:42]                 │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] updating number_of_replicas to [0] for indices [.ml-stats-000001]
[00:44:43]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] moving index [.ml-state-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [ml-size-based-ilm-policy]
[00:44:43]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] moving index [.ml-stats-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [ml-size-based-ilm-policy]
[00:44:43]                 │ info [o.e.x.m.d.DataFrameAnalyticsManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_binary_1607620196901] Creating destination index [user-ihp_fi_binary_1607620196901]
[00:44:43]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] moving index [.ml-state-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [ml-size-based-ilm-policy]
[00:44:43]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [user-ihp_fi_binary_1607620196901] creating index, cause [api], templates [], shards [1]/[1]
[00:44:43]                 │ info [o.e.x.m.d.DataFrameAnalyticsManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_binary_1607620196901] Started reindexing
[00:44:43]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] moving index [.ml-stats-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [ml-size-based-ilm-policy]
[00:44:43]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] moving index [.ml-state-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [ml-size-based-ilm-policy]
[00:44:43]                 │ info [o.e.x.i.IndexLifecycleTransition] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] moving index [.ml-stats-000001] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [ml-size-based-ilm-policy]
[00:44:43]                 │ debg Fetching data frame analytics job stats for job ihp_fi_binary_1607620196901...
[00:44:43]                 │ debg > DFA job stats fetched.
[00:44:43]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:43]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_binary_1607620196901] Started loading data
[00:44:43]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_binary_1607620196901] Started analyzing
[00:44:43]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_binary_1607620196901] Waiting for result processor to complete
[00:44:43]                 │ debg Fetching data frame analytics job stats for job ihp_fi_binary_1607620196901...
[00:44:43]                 │ debg > DFA job stats fetched.
[00:44:43]                 │ debg Waiting up to 120000ms for analytics state to be stopped...
[00:44:43]                 │ debg Fetching analytics state for job ihp_fi_binary_1607620196901
[00:44:43]                 │ debg Fetching data frame analytics job stats for job ihp_fi_binary_1607620196901...
[00:44:43]                 │ debg > DFA job stats fetched.
[00:44:43]                 │ debg --- retry.waitForWithTimeout error: expected analytics state to be stopped but got analyzing
[00:44:44]                 │ debg Fetching analytics state for job ihp_fi_binary_1607620196901
[00:44:44]                 │ debg Fetching data frame analytics job stats for job ihp_fi_binary_1607620196901...
[00:44:44]                 │ debg > DFA job stats fetched.
[00:44:44]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:44]                 │ debg Fetching analytics state for job ihp_fi_binary_1607620196901
[00:44:44]                 │ debg Fetching data frame analytics job stats for job ihp_fi_binary_1607620196901...
[00:44:44]                 │ debg > DFA job stats fetched.
[00:44:44]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:45]                 │ debg Fetching analytics state for job ihp_fi_binary_1607620196901
[00:44:45]                 │ debg Fetching data frame analytics job stats for job ihp_fi_binary_1607620196901...
[00:44:45]                 │ debg > DFA job stats fetched.
[00:44:45]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:45]                 │ debg Fetching analytics state for job ihp_fi_binary_1607620196901
[00:44:45]                 │ debg Fetching data frame analytics job stats for job ihp_fi_binary_1607620196901...
[00:44:45]                 │ debg > DFA job stats fetched.
[00:44:45]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:46]                 │ debg Fetching analytics state for job ihp_fi_binary_1607620196901
[00:44:46]                 │ debg Fetching data frame analytics job stats for job ihp_fi_binary_1607620196901...
[00:44:46]                 │ debg > DFA job stats fetched.
[00:44:46]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:46]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [.ml-inference-000003] creating index, cause [auto(bulk api)], templates [.ml-inference-000003], shards [1]/[1]
[00:44:46]                 │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] updating number_of_replicas to [0] for indices [.ml-inference-000003]
[00:44:46]                 │ info [o.e.x.m.d.p.ChunkedTrainedModelPersister] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_binary_1607620196901] finished storing trained model with id [ihp_fi_binary_1607620196901-1607622884079]
[00:44:46]                 │ info [o.e.x.m.d.p.AnalyticsResultProcessor] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_binary_1607620196901] Started writing results
[00:44:46]                 │ debg Fetching analytics state for job ihp_fi_binary_1607620196901
[00:44:46]                 │ debg Fetching data frame analytics job stats for job ihp_fi_binary_1607620196901...
[00:44:47]                 │ debg > DFA job stats fetched.
[00:44:47]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:47]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_binary_1607620196901] Result processor has completed
[00:44:47]                 │ info [o.e.x.m.d.i.InferenceRunner] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_binary_1607620196901] Started inference on test data against model [ihp_fi_binary_1607620196901-1607622884079]
[00:44:47]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_binary_1607620196901] Closing process
[00:44:47]                 │ info [o.e.x.m.p.l.CppLogMessageHandler] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_binary_1607620196901] [data_frame_analyzer/267134] [Main.cc@241] [{"name":"E_DFTPMEstimatedPeakMemoryUsage","description":"The upfront estimate of the peak memory training the predictive model would use","value":6040456}
[00:44:47]                 │      ,{"name":"E_DFTPMPeakMemoryUsage","description":"The peak memory training the predictive model used","value":323050}
[00:44:47]                 │      ,{"name":"E_DFTPMTimeToTrain","description":"The time it took to train the predictive model","value":2756}
[00:44:47]                 │      ,{"name":"E_DFTPMTrainedForestNumberTrees","description":"The total number of trees in the trained forest","value":4}
[00:44:47]                 │      ]
[00:44:47]                 │ info [o.e.x.m.p.AbstractNativeProcess] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_binary_1607620196901] State output finished
[00:44:47]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_binary_1607620196901] Closed process
[00:44:47]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_binary_1607620196901] Marking task completed
[00:44:47]                 │ debg Fetching analytics state for job ihp_fi_binary_1607620196901
[00:44:47]                 │ debg Fetching data frame analytics job stats for job ihp_fi_binary_1607620196901...
[00:44:47]                 │ debg > DFA job stats fetched.
[00:44:47]                 │ info [ml/ihp_outlier] Loading "mappings.json"
[00:44:47]                 │ info [ml/ihp_outlier] Loading "data.json.gz"
[00:44:47]                 │ info [ml/ihp_outlier] Skipped restore for existing index "ft_ihp_outlier"
[00:44:47]                 │ debg Searching for 'index-pattern' with title 'ft_ihp_outlier'...
[00:44:47]                 │ debg  > Found 'f10b6660-3b0a-11eb-964c-3b9b3d3fb8a8'
[00:44:47]                 │ debg Index pattern with title 'ft_ihp_outlier' already exists. Nothing to create.
[00:44:47]                 │ debg Creating data frame analytic job with id 'ihp_fi_multi_1607620196901' ...
[00:44:48]                 │ debg Waiting up to 5000ms for 'ihp_fi_multi_1607620196901' to exist...
[00:44:48]                 │ debg Fetching data frame analytics job 'ihp_fi_multi_1607620196901'...
[00:44:48]                 │ debg > DFA job fetched.
[00:44:48]                 │ debg > DFA job created.
[00:44:48]                 │ debg Starting data frame analytics job 'ihp_fi_multi_1607620196901'...
[00:44:48]                 │ info [o.e.x.m.a.TransportStartDataFrameAnalyticsAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_multi_1607620196901] Starting data frame analytics from state [null]
[00:44:48]                 │ debg > DFA job started.
[00:44:48]                 │ debg Waiting up to 60000ms for 'ihp_fi_multi_1607620196901' to have training_docs_count > 0...
[00:44:48]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:48]                 │ debg > DFA job stats fetched.
[00:44:48]                 │ debg --- retry.waitForWithTimeout error: expected data frame analytics job 'ihp_fi_multi_1607620196901' to have training_docs_count > 0 (got 0)
[00:44:48]                 │ info [o.e.x.m.d.DataFrameAnalyticsManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_multi_1607620196901] Creating destination index [user-ihp_fi_multi_1607620196901]
[00:44:48]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [user-ihp_fi_multi_1607620196901] creating index, cause [api], templates [], shards [1]/[1]
[00:44:49]                 │ info [o.e.x.m.d.DataFrameAnalyticsManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_multi_1607620196901] Started reindexing
[00:44:49]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:49]                 │ debg > DFA job stats fetched.
[00:44:49]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:49]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_multi_1607620196901] Started loading data
[00:44:49]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_multi_1607620196901] Started analyzing
[00:44:49]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_multi_1607620196901] Waiting for result processor to complete
[00:44:49]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:49]                 │ debg > DFA job stats fetched.
[00:44:49]                 │ debg Waiting up to 120000ms for analytics state to be stopped...
[00:44:49]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:49]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:49]                 │ debg > DFA job stats fetched.
[00:44:49]                 │ debg --- retry.waitForWithTimeout error: expected analytics state to be stopped but got analyzing
[00:44:50]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:50]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:50]                 │ debg > DFA job stats fetched.
[00:44:50]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:50]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:50]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:50]                 │ debg > DFA job stats fetched.
[00:44:50]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:51]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:51]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:51]                 │ debg > DFA job stats fetched.
[00:44:51]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:51]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:51]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:52]                 │ debg > DFA job stats fetched.
[00:44:52]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:52]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:52]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:52]                 │ debg > DFA job stats fetched.
[00:44:52]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:53]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:53]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:53]                 │ debg > DFA job stats fetched.
[00:44:53]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:53]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:53]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:53]                 │ debg > DFA job stats fetched.
[00:44:53]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:54]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:54]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:54]                 │ debg > DFA job stats fetched.
[00:44:54]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:54]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:54]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:54]                 │ debg > DFA job stats fetched.
[00:44:54]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:55]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:55]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:55]                 │ debg > DFA job stats fetched.
[00:44:55]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:55]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:55]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:55]                 │ debg > DFA job stats fetched.
[00:44:55]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:56]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:56]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:56]                 │ debg > DFA job stats fetched.
[00:44:56]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:56]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:56]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:56]                 │ debg > DFA job stats fetched.
[00:44:56]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:57]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:57]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:57]                 │ debg > DFA job stats fetched.
[00:44:57]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:57]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:57]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:57]                 │ debg > DFA job stats fetched.
[00:44:57]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:58]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:58]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:58]                 │ debg > DFA job stats fetched.
[00:44:58]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:58]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:58]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:58]                 │ debg > DFA job stats fetched.
[00:44:58]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:59]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:59]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:59]                 │ debg > DFA job stats fetched.
[00:44:59]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:44:59]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:44:59]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:44:59]                 │ debg > DFA job stats fetched.
[00:44:59]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:00]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:00]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:00]                 │ debg > DFA job stats fetched.
[00:45:00]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:00]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:00]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:00]                 │ debg > DFA job stats fetched.
[00:45:00]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:01]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:01]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:01]                 │ debg > DFA job stats fetched.
[00:45:01]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:01]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:01]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:01]                 │ debg > DFA job stats fetched.
[00:45:01]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:02]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:02]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:02]                 │ debg > DFA job stats fetched.
[00:45:02]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:02]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:02]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:02]                 │ debg > DFA job stats fetched.
[00:45:02]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:03]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:03]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:03]                 │ debg > DFA job stats fetched.
[00:45:03]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:03]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:03]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:03]                 │ debg > DFA job stats fetched.
[00:45:03]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:04]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:04]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:04]                 │ debg > DFA job stats fetched.
[00:45:04]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:04]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:04]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:04]                 │ debg > DFA job stats fetched.
[00:45:04]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:05]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:05]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:05]                 │ debg > DFA job stats fetched.
[00:45:05]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:05]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:05]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:05]                 │ debg > DFA job stats fetched.
[00:45:05]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:06]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:06]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:06]                 │ debg > DFA job stats fetched.
[00:45:06]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:06]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:06]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:06]                 │ debg > DFA job stats fetched.
[00:45:06]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:07]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:07]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:07]                 │ debg > DFA job stats fetched.
[00:45:07]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:07]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:07]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:07]                 │ debg > DFA job stats fetched.
[00:45:07]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:08]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:08]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:08]                 │ debg > DFA job stats fetched.
[00:45:08]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:08]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:08]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:08]                 │ debg > DFA job stats fetched.
[00:45:08]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:09]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:09]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:09]                 │ debg > DFA job stats fetched.
[00:45:09]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:09]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:09]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:09]                 │ debg > DFA job stats fetched.
[00:45:09]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:10]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:10]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:10]                 │ debg > DFA job stats fetched.
[00:45:10]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:10]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:10]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:10]                 │ debg > DFA job stats fetched.
[00:45:10]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:11]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:11]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:11]                 │ debg > DFA job stats fetched.
[00:45:11]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:11]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:11]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:11]                 │ debg > DFA job stats fetched.
[00:45:11]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:12]                 │ info [o.e.x.m.d.p.ChunkedTrainedModelPersister] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_multi_1607620196901] finished storing trained model with id [ihp_fi_multi_1607620196901-1607622909425]
[00:45:12]                 │ info [o.e.x.m.d.p.AnalyticsResultProcessor] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_multi_1607620196901] Started writing results
[00:45:12]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:12]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:12]                 │ debg > DFA job stats fetched.
[00:45:12]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:12]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_multi_1607620196901] Result processor has completed
[00:45:12]                 │ info [o.e.x.m.d.i.InferenceRunner] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_multi_1607620196901] Started inference on test data against model [ihp_fi_multi_1607620196901-1607622909425]
[00:45:12]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:12]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:12]                 │ debg > DFA job stats fetched.
[00:45:12]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:13]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_multi_1607620196901] Closing process
[00:45:13]                 │ info [o.e.x.m.p.l.CppLogMessageHandler] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_multi_1607620196901] [data_frame_analyzer/267666] [Main.cc@241] [{"name":"E_DFTPMEstimatedPeakMemoryUsage","description":"The upfront estimate of the peak memory training the predictive model would use","value":9613320}
[00:45:13]                 │      ,{"name":"E_DFTPMPeakMemoryUsage","description":"The peak memory training the predictive model used","value":1492919}
[00:45:13]                 │      ,{"name":"E_DFTPMTimeToTrain","description":"The time it took to train the predictive model","value":21678}
[00:45:13]                 │      ,{"name":"E_DFTPMTrainedForestNumberTrees","description":"The total number of trees in the trained forest","value":35}
[00:45:13]                 │      ]
[00:45:13]                 │ info [o.e.x.m.p.AbstractNativeProcess] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_multi_1607620196901] State output finished
[00:45:13]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_multi_1607620196901] Closed process
[00:45:13]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [ihp_fi_multi_1607620196901] Marking task completed
[00:45:13]                 │ debg Fetching analytics state for job ihp_fi_multi_1607620196901
[00:45:13]                 │ debg Fetching data frame analytics job stats for job ihp_fi_multi_1607620196901...
[00:45:13]                 │ debg > DFA job stats fetched.
[00:45:13]                 │ info [ml/egs_regression] Loading "mappings.json"
[00:45:13]                 │ info [ml/egs_regression] Loading "data.json.gz"
[00:45:13]                 │ info [ml/egs_regression] Skipped restore for existing index "ft_egs_regression"
[00:45:13]                 │ debg Searching for 'index-pattern' with title 'ft_egs_regression'...
[00:45:13]                 │ debg  > Found '2c196770-3b10-11eb-964c-3b9b3d3fb8a8'
[00:45:13]                 │ debg Index pattern with title 'ft_egs_regression' already exists. Nothing to create.
[00:45:13]                 │ debg Creating data frame analytic job with id 'egs_fi_reg_1607620196901' ...
[00:45:13]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [.ml-config/Fnlb5sAbQM27s_qlucFMrg] update_mapping [_doc]
[00:45:14]                 │ debg Waiting up to 5000ms for 'egs_fi_reg_1607620196901' to exist...
[00:45:14]                 │ debg Fetching data frame analytics job 'egs_fi_reg_1607620196901'...
[00:45:14]                 │ debg > DFA job fetched.
[00:45:14]                 │ debg > DFA job created.
[00:45:14]                 │ debg Starting data frame analytics job 'egs_fi_reg_1607620196901'...
[00:45:14]                 │ info [o.e.x.m.a.TransportStartDataFrameAnalyticsAction] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [egs_fi_reg_1607620196901] Starting data frame analytics from state [null]
[00:45:14]                 │ debg > DFA job started.
[00:45:14]                 │ debg Waiting up to 60000ms for 'egs_fi_reg_1607620196901' to have training_docs_count > 0...
[00:45:14]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:14]                 │ debg > DFA job stats fetched.
[00:45:14]                 │ debg --- retry.waitForWithTimeout error: expected data frame analytics job 'egs_fi_reg_1607620196901' to have training_docs_count > 0 (got 0)
[00:45:14]                 │ info [o.e.x.m.d.DataFrameAnalyticsManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [egs_fi_reg_1607620196901] Creating destination index [user-egs_fi_reg_1607620196901]
[00:45:14]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [user-egs_fi_reg_1607620196901] creating index, cause [api], templates [], shards [1]/[1]
[00:45:15]                 │ info [o.e.x.m.d.DataFrameAnalyticsManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [egs_fi_reg_1607620196901] Started reindexing
[00:45:15]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:15]                 │ debg > DFA job stats fetched.
[00:45:15]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:15]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [egs_fi_reg_1607620196901] Started loading data
[00:45:15]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [egs_fi_reg_1607620196901] Started analyzing
[00:45:15]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [egs_fi_reg_1607620196901] Waiting for result processor to complete
[00:45:15]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:15]                 │ debg > DFA job stats fetched.
[00:45:15]                 │ debg Waiting up to 120000ms for analytics state to be stopped...
[00:45:15]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:15]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:15]                 │ debg > DFA job stats fetched.
[00:45:15]                 │ debg --- retry.waitForWithTimeout error: expected analytics state to be stopped but got analyzing
[00:45:16]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:16]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:16]                 │ debg > DFA job stats fetched.
[00:45:16]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:16]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:16]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:16]                 │ debg > DFA job stats fetched.
[00:45:16]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:17]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:17]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:17]                 │ debg > DFA job stats fetched.
[00:45:17]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:17]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:17]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:18]                 │ debg > DFA job stats fetched.
[00:45:18]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:18]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:18]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:18]                 │ debg > DFA job stats fetched.
[00:45:18]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:19]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:19]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:19]                 │ debg > DFA job stats fetched.
[00:45:19]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:19]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:19]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:19]                 │ debg > DFA job stats fetched.
[00:45:19]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:20]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:20]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:20]                 │ debg > DFA job stats fetched.
[00:45:20]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:20]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:20]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:20]                 │ debg > DFA job stats fetched.
[00:45:20]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:21]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:21]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:21]                 │ debg > DFA job stats fetched.
[00:45:21]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:21]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:21]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:21]                 │ debg > DFA job stats fetched.
[00:45:21]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:22]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:22]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:22]                 │ debg > DFA job stats fetched.
[00:45:22]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:22]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:22]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:22]                 │ debg > DFA job stats fetched.
[00:45:22]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:23]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:23]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:23]                 │ debg > DFA job stats fetched.
[00:45:23]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:23]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:23]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:23]                 │ debg > DFA job stats fetched.
[00:45:23]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:24]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:24]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:24]                 │ debg > DFA job stats fetched.
[00:45:24]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:24]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:24]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:24]                 │ debg > DFA job stats fetched.
[00:45:24]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:25]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:25]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:25]                 │ debg > DFA job stats fetched.
[00:45:25]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:25]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:25]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:25]                 │ debg > DFA job stats fetched.
[00:45:25]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:26]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:26]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:26]                 │ debg > DFA job stats fetched.
[00:45:26]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:26]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:26]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:26]                 │ debg > DFA job stats fetched.
[00:45:26]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:27]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:27]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:27]                 │ debg > DFA job stats fetched.
[00:45:27]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:27]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:27]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:27]                 │ debg > DFA job stats fetched.
[00:45:27]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:28]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:28]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:28]                 │ debg > DFA job stats fetched.
[00:45:28]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:28]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:28]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:28]                 │ debg > DFA job stats fetched.
[00:45:28]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:29]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:29]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:29]                 │ debg > DFA job stats fetched.
[00:45:29]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:29]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:29]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:29]                 │ debg > DFA job stats fetched.
[00:45:29]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:30]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:30]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:30]                 │ debg > DFA job stats fetched.
[00:45:30]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:30]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:30]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:30]                 │ debg > DFA job stats fetched.
[00:45:30]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:31]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:31]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:31]                 │ debg > DFA job stats fetched.
[00:45:31]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:31]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:31]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:31]                 │ debg > DFA job stats fetched.
[00:45:31]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:32]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:32]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:32]                 │ debg > DFA job stats fetched.
[00:45:32]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:32]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:32]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:32]                 │ debg > DFA job stats fetched.
[00:45:32]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:33]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:33]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:33]                 │ debg > DFA job stats fetched.
[00:45:33]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:33]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:33]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:33]                 │ debg > DFA job stats fetched.
[00:45:33]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:34]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:34]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:34]                 │ debg > DFA job stats fetched.
[00:45:34]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:34]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:34]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:34]                 │ debg > DFA job stats fetched.
[00:45:34]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:35]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:35]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:35]                 │ debg > DFA job stats fetched.
[00:45:35]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:35]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:35]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:35]                 │ debg > DFA job stats fetched.
[00:45:35]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:36]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:36]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:36]                 │ debg > DFA job stats fetched.
[00:45:36]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:36]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:36]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:36]                 │ debg > DFA job stats fetched.
[00:45:36]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:37]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:37]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:37]                 │ debg > DFA job stats fetched.
[00:45:37]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:37]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:37]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:37]                 │ debg > DFA job stats fetched.
[00:45:37]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:38]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:38]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:38]                 │ debg > DFA job stats fetched.
[00:45:38]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:38]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:38]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:38]                 │ debg > DFA job stats fetched.
[00:45:38]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:39]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:39]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:39]                 │ debg > DFA job stats fetched.
[00:45:39]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:39]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:39]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:39]                 │ debg > DFA job stats fetched.
[00:45:39]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:40]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:40]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:40]                 │ debg > DFA job stats fetched.
[00:45:40]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:40]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:40]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:40]                 │ debg > DFA job stats fetched.
[00:45:40]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:41]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:41]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:41]                 │ debg > DFA job stats fetched.
[00:45:41]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:41]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:41]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:41]                 │ debg > DFA job stats fetched.
[00:45:41]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:42]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:42]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:42]                 │ debg > DFA job stats fetched.
[00:45:42]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:42]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:42]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:42]                 │ debg > DFA job stats fetched.
[00:45:42]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:43]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:43]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:43]                 │ debg > DFA job stats fetched.
[00:45:43]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:43]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:43]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:43]                 │ debg > DFA job stats fetched.
[00:45:43]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:44]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:44]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:44]                 │ debg > DFA job stats fetched.
[00:45:44]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:45]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:45]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:45]                 │ debg > DFA job stats fetched.
[00:45:45]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:45]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:45]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:45]                 │ debg > DFA job stats fetched.
[00:45:45]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:46]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:46]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:46]                 │ debg > DFA job stats fetched.
[00:45:46]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:46]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:46]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:46]                 │ debg > DFA job stats fetched.
[00:45:46]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:47]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:47]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:47]                 │ debg > DFA job stats fetched.
[00:45:47]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:47]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:47]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:47]                 │ debg > DFA job stats fetched.
[00:45:47]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:48]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:48]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:48]                 │ debg > DFA job stats fetched.
[00:45:48]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:48]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:48]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:48]                 │ debg > DFA job stats fetched.
[00:45:48]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:49]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:49]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:49]                 │ debg > DFA job stats fetched.
[00:45:49]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:49]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:49]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:49]                 │ debg > DFA job stats fetched.
[00:45:49]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:50]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:50]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:50]                 │ debg > DFA job stats fetched.
[00:45:50]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:50]                 │ info [o.e.x.m.d.p.ChunkedTrainedModelPersister] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [egs_fi_reg_1607620196901] finished storing trained model with id [egs_fi_reg_1607620196901-1607622947727]
[00:45:50]                 │ info [o.e.x.m.d.p.AnalyticsResultProcessor] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [egs_fi_reg_1607620196901] Started writing results
[00:45:50]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:50]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:50]                 │ debg > DFA job stats fetched.
[00:45:50]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:50]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [egs_fi_reg_1607620196901] Result processor has completed
[00:45:50]                 │ info [o.e.x.m.d.i.InferenceRunner] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [egs_fi_reg_1607620196901] Started inference on test data against model [egs_fi_reg_1607620196901-1607622947727]
[00:45:51]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:51]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:51]                 │ debg > DFA job stats fetched.
[00:45:51]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:51]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:51]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:51]                 │ debg > DFA job stats fetched.
[00:45:51]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:52]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:52]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:52]                 │ debg > DFA job stats fetched.
[00:45:52]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:52]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:52]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:52]                 │ debg > DFA job stats fetched.
[00:45:52]                 │ debg --- retry.waitForWithTimeout failed again with the same message...
[00:45:53]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [egs_fi_reg_1607620196901] Closing process
[00:45:53]                 │ info [o.e.x.m.p.l.CppLogMessageHandler] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [egs_fi_reg_1607620196901] [data_frame_analyzer/269158] [Main.cc@241] [{"name":"E_DFTPMEstimatedPeakMemoryUsage","description":"The upfront estimate of the peak memory training the predictive model would use","value":7582364}
[00:45:53]                 │      ,{"name":"E_DFTPMPeakMemoryUsage","description":"The peak memory training the predictive model used","value":6784368}
[00:45:53]                 │      ,{"name":"E_DFTPMTimeToTrain","description":"The time it took to train the predictive model","value":33999}
[00:45:53]                 │      ,{"name":"E_DFTPMTrainedForestNumberTrees","description":"The total number of trees in the trained forest","value":637}
[00:45:53]                 │      ]
[00:45:53]                 │ info [o.e.x.m.p.AbstractNativeProcess] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [egs_fi_reg_1607620196901] State output finished
[00:45:53]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [egs_fi_reg_1607620196901] Closed process
[00:45:53]                 │ info [o.e.x.m.d.p.AnalyticsProcessManager] [kibana-ci-immutable-debian-tests-xxl-1607618179260424959] [egs_fi_reg_1607620196901] Marking task completed
[00:45:53]                 │ debg Fetching analytics state for job egs_fi_reg_1607620196901
[00:45:53]                 │ debg Fetching data frame analytics job stats for job egs_fi_reg_1607620196901...
[00:45:53]                 │ debg > DFA job stats fetched.
[00:45:53]               └-: binary classification job
[00:45:53]                 └-> "before all" hook
[00:45:53]                 └-> "before all" hook
[00:45:53]                   │ debg navigating to ml url: http://localhost:6151/app/ml
[00:45:53]                   │ debg navigate to: http://localhost:6151/app/ml
[00:45:53]                   │ debg browser[INFO] http://localhost:6151/app/ml?_t=1607622950434 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:45:53]                   │
[00:45:53]                   │ debg browser[INFO] http://localhost:6151/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:45:53]                   │ debg ... sleep(700) start
[00:45:54]                   │ debg ... sleep(700) end
[00:45:54]                   │ debg returned from get, calling refresh
[00:45:54]                   │ debg browser[INFO] http://localhost:6151/app/ml?_t=1607622950434 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:45:54]                   │
[00:45:54]                   │ debg browser[INFO] http://localhost:6151/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:45:54]                   │ debg currentUrl = http://localhost:6151/app/ml
[00:45:54]                   │          appUrl = http://localhost:6151/app/ml
[00:45:54]                   │ debg TestSubjects.find(kibanaChrome)
[00:45:54]                   │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:45:56]                   │ debg ... sleep(501) start
[00:45:56]                   │ debg ... sleep(501) end
[00:45:56]                   │ debg in navigateTo url = http://localhost:6151/app/ml/overview
[00:45:56]                   │ debg --- retry.try error: URL changed, waiting for it to settle
[00:45:57]                   │ debg ... sleep(501) start
[00:45:57]                   │ debg ... sleep(501) end
[00:45:57]                   │ debg in navigateTo url = http://localhost:6151/app/ml/overview
[00:45:57]                   │ debg TestSubjects.exists(statusPageContainer)
[00:45:57]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:46:00]                   │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:46:00]                   │ debg TestSubjects.exists(mlApp)
[00:46:00]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlApp"]') with timeout=2000
[00:46:00]                   │ debg TestSubjects.click(~mlMainTab & ~dataFrameAnalytics)
[00:46:00]                   │ debg Find.clickByCssSelector('[data-test-subj~="mlMainTab"][data-test-subj~="dataFrameAnalytics"]') with timeout=10000
[00:46:00]                   │ debg Find.findByCssSelector('[data-test-subj~="mlMainTab"][data-test-subj~="dataFrameAnalytics"]') with timeout=10000
[00:46:01]                   │ debg TestSubjects.exists(~mlMainTab & ~dataFrameAnalytics & ~selected)
[00:46:01]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="mlMainTab"][data-test-subj~="dataFrameAnalytics"][data-test-subj~="selected"]') with timeout=120000
[00:46:01]                   │ debg TestSubjects.exists(mlPageDataFrameAnalytics)
[00:46:01]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlPageDataFrameAnalytics"]') with timeout=120000
[00:46:01]                   │ debg TestSubjects.exists(~mlAnalyticsTable)
[00:46:01]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="mlAnalyticsTable"]') with timeout=60000
[00:46:01]                   │ debg TestSubjects.exists(mlAnalyticsTable loaded)
[00:46:01]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlAnalyticsTable loaded"]') with timeout=30000
[00:46:01]                   │ debg TestSubjects.exists(~mlAnalyticsTable > ~row-ihp_fi_binary_1607620196901 > mlAnalyticsJobViewButton)
[00:46:01]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="mlAnalyticsTable"] [data-test-subj~="row-ihp_fi_binary_1607620196901"] [data-test-subj="mlAnalyticsJobViewButton"]') with timeout=120000
[00:46:01]                   │ debg TestSubjects.click(~mlAnalyticsTable > ~row-ihp_fi_binary_1607620196901 > mlAnalyticsJobViewButton)
[00:46:01]                   │ debg Find.clickByCssSelector('[data-test-subj~="mlAnalyticsTable"] [data-test-subj~="row-ihp_fi_binary_1607620196901"] [data-test-subj="mlAnalyticsJobViewButton"]') with timeout=10000
[00:46:01]                   │ debg Find.findByCssSelector('[data-test-subj~="mlAnalyticsTable"] [data-test-subj~="row-ihp_fi_binary_1607620196901"] [data-test-subj="mlAnalyticsJobViewButton"]') with timeout=10000
[00:46:01]                   │ debg TestSubjects.exists(mlPageDataFrameAnalyticsExploration)
[00:46:01]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlPageDataFrameAnalyticsExploration"]') with timeout=20000
[00:46:02]                 └-> should display the total feature importance in the results view
[00:46:02]                   └-> "before each" hook: global before each
[00:46:02]                   │ debg TestSubjects.exists(mlDFExpandableSection-FeatureImportanceSummary)
[00:46:02]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlDFExpandableSection-FeatureImportanceSummary"]') with timeout=120000
[00:46:03]                   │ debg TestSubjects.exists(mlTotalFeatureImportanceChart)
[00:46:03]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlTotalFeatureImportanceChart"]') with timeout=5000
[00:46:03]                   └- ✓ pass  (1.5s) "machine learning data frame analytics total feature importance panel and decision path popover binary classification job should display the total feature importance in the results view"
[00:46:03]                 └-> should display the feature importance decision path in the data grid
[00:46:03]                   └-> "before each" hook: global before each
[00:46:03]                   │ debg TestSubjects.exists(mlExplorationDataGrid loaded)
[00:46:03]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlExplorationDataGrid loaded"]') with timeout=5000
[00:46:03]                   │ debg TestSubjects.findAll(mlExplorationDataGrid loaded > dataGridRow)
[00:46:03]                   │ debg Find.allByCssSelector('[data-test-subj="mlExplorationDataGrid loaded"] [data-test-subj="dataGridRow"]') with timeout=10000
[00:46:03]                   │ debg TestSubjects.findAll(mlExplorationDataGrid loaded > dataGridRow)
[00:46:03]                   │ debg Find.allByCssSelector('[data-test-subj="mlExplorationDataGrid loaded"] [data-test-subj="dataGridRow"]') with timeout=10000
[00:46:03]                   │ debg TestSubjects.find(mlExplorationDataGrid loaded > dataGridRow)
[00:46:03]                   │ debg Find.findByCssSelector('[data-test-subj="mlExplorationDataGrid loaded"] [data-test-subj="dataGridRow"]') with timeout=10000
[00:46:04]                   │ debg TestSubjects.exists(mlDFADecisionPathPopover)
[00:46:04]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlDFADecisionPathPopover"]') with timeout=120000
[00:46:07]                   │ debg --- retry.tryForTime error: [data-test-subj="mlDFADecisionPathPopover"] is not displayed
[00:46:10]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:13]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:16]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:19]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:22]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:25]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:28]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:31]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:34]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:37]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:40]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:43]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:46]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:49]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:52]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:55]                   │ debg --- retry.tryForTime failed again with the same message...
[00:46:58]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:01]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:04]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:07]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:10]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:13]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:16]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:19]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:22]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:25]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:29]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:32]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:35]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:38]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:41]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:44]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:47]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:50]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:53]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:56]                   │ debg --- retry.tryForTime failed again with the same message...
[00:47:59]                   │ debg --- retry.tryForTime failed again with the same message...
[00:48:02]                   │ debg --- retry.tryForTime failed again with the same message...
[00:48:05]                   │ debg --- retry.tryForTime failed again with the same message...
[00:48:05]                   │ info Taking screenshot "/dev/shm/workspace/parallel/5/kibana/x-pack/test/functional/screenshots/failure/machine learning data frame analytics total feature importance panel and decision path popover binary classification job should display the feature importance decision path in the data grid.png"
[00:48:06]                   │ info Current URL is: http://localhost:6151/app/ml/data_frame_analytics/exploration?_g=(ml%3A(analysisType%3Aclassification%2CjobId%3Aihp_fi_binary_1607620196901))
[00:48:06]                   │ info Saving page source to: /dev/shm/workspace/parallel/5/kibana/x-pack/test/functional/failure_debug/html/machine learning data frame analytics total feature importance panel and decision path popover binary classification job should display the feature importance decision path in the data grid.html
[00:48:06]                   └- ✖ fail: machine learning data frame analytics total feature importance panel and decision path popover binary classification job should display the feature importance decision path in the data grid
[00:48:06]                   │      Error: expected testSubject(mlDFADecisionPathPopover) to exist
[00:48:06]                   │       at TestSubjects.existOrFail (/dev/shm/workspace/parallel/5/kibana/test/functional/services/common/test_subjects.ts:62:15)
[00:48:06]                   │       at Object.openFeatureImportanceDecisionPathPopover (test/functional/services/ml/data_frame_analytics_results.ts:98:7)
[00:48:06]                   │       at Context.<anonymous> (test/functional/apps/ml/data_frame_analytics/feature_importance.ts:204:11)
[00:48:06]                   │       at Object.apply (/dev/shm/workspace/parallel/5/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)
[00:48:06]                   │ 
[00:48:06]                   │

Stack Trace

Error: expected testSubject(mlDFADecisionPathPopover) to exist
    at TestSubjects.existOrFail (/dev/shm/workspace/parallel/5/kibana/test/functional/services/common/test_subjects.ts:62:15)
    at Object.openFeatureImportanceDecisionPathPopover (test/functional/services/ml/data_frame_analytics_results.ts:98:7)
    at Context.<anonymous> (test/functional/apps/ml/data_frame_analytics/feature_importance.ts:204:11)
    at Object.apply (/dev/shm/workspace/parallel/5/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 8.3MB 8.3MB +36.0B

Distributable file count

id before after diff
default 47750 47782 +32

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@randomuserid randomuserid merged commit 313d85e into master Dec 10, 2020
@spong spong mentioned this pull request Dec 10, 2020
Merged
spong added a commit that referenced this pull request Dec 10, 2020
@spong @randomuserid @pheyos
[ML] Adds security_linux and security_windows Modules (#85065) (#85637)
9fc09cb 
* initial commit

refactored multi-index, multi-pipeline jobs for 7.11. These are new modules that will live alongside the existing jobs.

* Update ml_modules.tsx

added new module names to the list

* Update get_module.ts

added new module names

* Linter fixes

* Order matters

* manifest fixes

added colon char to the module name and shortened the description

* additon to description

after talking with the security team today, adding this suggested text to the beginning of the description so it will tend to be visible to the user:
"This is a new refactored job which works on ECS compatible events across multiple indices."

* Adjust module recognizer test for auditbeat dataset

* influencers

changes to the metadata jobs to make influencers identical to the originals

* change for security app

changes to two datafeeds needed for logic in the Security app - added the suffix "_ecs" to two ids.

Co-authored-by: Garrett Spong <[email protected]>
Co-authored-by: Robert Oskamp <[email protected]>

Co-authored-by: Craig Chamberlain <[email protected]>
Co-authored-by: Robert Oskamp <[email protected]>
@pheyos pheyos mentioned this pull request Dec 14, 2020
Merged
@lcawl lcawl mentioned this pull request Dec 24, 2020
Merged
@lcawl lcawl mentioned this pull request Jan 29, 2021
Merged
@spong spong mentioned this pull request Mar 10, 2021
Closed
@spong spong deleted the ml-711 branch May 13, 2021 14:07
This was referenced Jan 13, 2022
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@spong spong spong approved these changes

@peteharverson peteharverson peteharverson approved these changes

+2 more reviewers

@blaklaybul blaklaybul blaklaybul left review comments

@panbalag panbalag panbalag approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Feature:Anomaly Detection ML anomaly detection :ml release_note:enhancement v7.11.0 v8.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

9 participants

@randomuserid @elasticmachine @spong @pheyos @peteharverson @blaklaybul @panbalag @kibanamachine