[Logs UI] Generalize ML module management #50662

weltenwort · 2019-11-14T15:18:26Z

Summary

This abstracts the specific job details out of the ML module management hooks to enable re-use with the upcoming categorization module.

closes #50322

Implementation notes

I tried to apply the following methodology:

hooks in containers/log_analysis have been made agnostic to the specific ML module
the log-entry-rate-specific hooks wrap/utilize the agnostic hooks and have been moved to pages/logs/log_entry_rate
the module specifics are defined in a moduleDescriptor data structure (i.e. logEntryRateModuleDescriptor)
the current source configuration is encapsulated in a ModuleSourceConfiguration
log-entry-rate-specific components have been moved to pages/logs/log_entry_rate (mostly a rename from pages/logs/log_analysis)
components useful for different ML modules to come have been moved to components/logging

The resulting relationships between the resulting components and hooks on the coarse scale could be described with the following graph:

 pages/logs/log_entry_rate                                                             containers/logging/log_analysis
+---------------------------------------------------------------------------------+   +------------------------------+
|                                                                                 |   |                              |
|                                                                                 |   |                              |
|  +------------------+          +-----------------------+                        |   |  +----------------------+    |
|  | LogEntryRatePage +----------> useLogEntryRateModule +-------------------------------> useLogAnalysisModule |    |
|  +--------+---------+ provides +---^---^---^---+---+---+ uses                   |   |  +--+-------------------+    |
|           |                        |   |   |   |   |                            |   |     |                        |
|           |                        |   |   |   |   |                            |   |     |   +-----------------+  |
|           |                        |   |   |   |   |                            |   |     +---> useModuleStatus |  |
|           |                        |   |   |   | +-v-------------------------+  |   |         +-----------------+  |
|           |                        |   |   |   | | moduleSourceConfiguration |  |   |                              |
|           |                        |   |   |   | +---------------------------+  |   |  +------------------------+  |
|           |                        |   |   |   |                                |   |  | callGetMlModuleAPI     |  |
|           |                        |   |   | +-v----------------------------+   |   |  | callSetupMlModuleAPI   |  |
|           |                        |   |   | | logEntryRateModuleDescriptor +----------> callJobsSummaryAPI     |  |
|           |                        |   |   | +------------------------------+   |   |  | callDeleteJobs         |  |
|           |      accesses (context)|   |   |                                    |   |  | callValidateIndicesAPI |  |
|           |                        |   |   |                                    |   |  +------------------------+  |
|  +--------v----------------+       |   |   |                                    |   |                              |
|  | LogEntryRatePageContent +-------+   |   |                                    |   |                              |
|  +--------+----------------+           |   |                                    |   |                              |
|           |                            |   |                                    |   |                              |
|           |   +------------------------+-+ |                                    |   |  +-----------------------+   |
|           +---> LogEntryRateSetupContent +---------------------------------------------> useAnalysisSetupState |   |
|           |   +--------------------------+ |                                    |   |  +-----------------------+   |
|           |                                |                                    |   |                              |
|           |   +----------------------------+-+   +------------------------+     |   |                              |
|           +---> LogEntryRateResultsContent   +---> useLogEntryRateResults |     |   +------------------------------+
|           |   +------------------------------+   +------------------------+     |
|           |                                                                     |
|           |   +--------------------------------+                                |
|           +---> LogEntryRateUnavailableContent |                                |
|               +--------------------------------+                                |
|                                                                                 |
|                                                                                 |
+---------------------------------------------------------------------------------+

Checklist

This was checked for cross-browser compatibility, including a check against IE11
~~[ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~
~~[ ] Documentation was added for features that require explanation or tutorials~~
~~[ ] Unit or functional tests were updated or added to match the most common scenarios~~
~~[ ] This was checked for keyboard-only and screenreader accessibility~~

elasticmachine · 2019-11-14T15:18:28Z

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

elasticmachine · 2019-11-14T16:37:51Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 4d0243dd0579c59a9ad12b149cbbd1476e5e86e9

elasticmachine · 2019-11-19T23:15:18Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 3344f32

elasticmachine · 2019-11-20T20:12:46Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 32075b6

elasticmachine · 2019-11-21T20:17:54Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 02aa0e2

…e-management

elasticmachine · 2019-11-25T14:16:57Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: b0a0c51

elasticmachine · 2019-11-25T16:13:31Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: df3bc37cb79405438eccc7b8165ba6fa4a217dd0

…e-management

elasticmachine · 2019-11-26T00:49:38Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 4b4268f9bd1098b4c93bf90779dc71dfe20b54f3

elasticmachine · 2019-11-26T16:54:19Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: ab364c7dd947120c3492859a9f7603c1c8a925d8

elasticmachine · 2019-11-27T20:25:33Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 1b4de9ce57cbe587d7c5be3f1668cb3439950b3a

elasticmachine · 2019-11-28T17:53:06Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: adb3ea2

elasticmachine · 2019-11-28T19:26:57Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: c293ec6

weltenwort · 2019-11-28T19:55:47Z

jenkins, test this again

elasticmachine · 2019-11-28T21:18:32Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: c293ec6

weltenwort · 2019-12-03T16:05:54Z

@elasticmachine update branch

…e-management

elasticmachine · 2019-12-03T17:18:33Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 94485fc

…e-management

weltenwort · 2019-12-09T20:55:47Z

Looks unrelated:

[00:31:41]                 └- ✖ fail: "Visualize feature controls security global visualize all privileges allow saving via the saved query management component popover with no saved query loaded"

weltenwort · 2019-12-09T20:55:55Z

@elasticmachine update branch

…e-management

Kerry350

Nice work 👍 This cleaned things up a lot, even with just the one use of generalised parts currently. The new directory structures look good too. I have left a couple of really minor comments.

Kerry350 · 2019-12-10T11:13:24Z

x-pack/legacy/plugins/infra/public/containers/logs/log_analysis/log_analysis_module_status.tsx

+      ...accumulatedJobStatus,
+      [jobType]: 'unknown',
+    }),
+    {} as Record<JobType, JobStatus>


Small nitpick: Could we use a generic type argument over the casting? (jobTypes.reduce<Record<JobType, JobStatus>>)?

That's what I usually prefer too, but in this case it doesn't work because the empty default object is not assignable to that record type:

Argument of type '{}' is not assignable to parameter of type 'Record<JobType, JobStatus>'.

Kerry350 · 2019-12-10T11:16:45Z

x-pack/legacy/plugins/infra/public/containers/logs/log_analysis/log_analysis_module_status.tsx

+            ...accumulatedJobStatus,
+            [jobType]: 'initializing',
+          }),
+          {} as Record<JobType, JobStatus>


Same here, generic type argument over casting?

same as above 😉

Kerry350 · 2019-12-10T11:21:57Z

x-pack/legacy/plugins/infra/public/pages/logs/log_entry_rate/page_providers.tsx

+
+import { useSourceContext } from '../../../containers/source';
+import { useKibanaSpaceId } from '../../../utils/use_kibana_space_id';
+// import { LogEntryRateJobsProvider } from './use_log_entry_rate_jobs';


Can this be deleted?

…e-management

elasticmachine · 2019-12-10T18:00:11Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 099de5c

History

💚 Build #14366 succeeded 3b7ab54
💔 Build #14329 failed 4c7832c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

weltenwort · 2019-12-10T18:27:36Z

ℹ️ backport to 7.x is waiting for #52613

This abstracts the specific job details out of the ML module management hooks to enable re-use with the upcoming categorization module. closes elastic#50322

…le-sql-highlighting * 'master' of github.com:elastic/kibana: (56 commits) Migrate url shortener service (elastic#50896) Re-enable datemath in from/to canvas timelion args (elastic#52159) [Logs + Metrics UI] Remove eslint exceptions (elastic#50979) [Logs + Metrics UI] Add missing headers in Logs & metrics (elastic#52405) [ML] API integration tests - initial tests for bucket span estimator (elastic#52636) [Watcher] New Platform (NP) Migration (elastic#50908) Decouple Authorization subsystem from Legacy API. (elastic#52638) [APM] Fix some warnings logged in APM tests (elastic#52487) [ui/public/utils] Delete unused base_object & find_by_param (elastic#52500) [ui/public/utils] Move items into ui/vis (elastic#52615) fix newlines in kbn-analytics build script Add top level examples folder and command to run, `--run-examples`. (elastic#52027) feat(NA): add trap for SIGINT in the git precommit hook (elastic#52662) [DOCS] Updtes description of elasticsearch.requestHeadersWhitelist (elastic#52675) [Telemetry/Pulse] Updates advanced settings text for usage data (elastic#52657) [SIEM][Detection Engine] Adds the default name space to the end of the signals index [Logs UI] Generalize ML module management (elastic#50662) Removing stateful saved object finder (elastic#52166) Shim oss telemetry (elastic#51168) [Reporting/Screenshots] Do not fail the report if request is aborted (elastic#52344) ... # Conflicts: # src/legacy/core_plugins/console/public/legacy.ts # src/legacy/core_plugins/console/public/np_ready/application/models/legacy_core_editor/mode/elasticsearch_sql_highlight_rules.ts # src/legacy/core_plugins/console/public/np_ready/lib/autocomplete/components/full_request_component.ts # src/legacy/core_plugins/console/public/quarantined/src/sense_editor/row_parser.js

This abstracts the specific job details out of the ML module management hooks to enable re-use with the upcoming categorization module. closes elastic#50322

Backports the following commits to 7.x: - [Logs UI] Generalize ML module management (#50662)

weltenwort added v8.0.0 Feature:Logs UI Logs UI feature Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services release_note:skip Skip the PR/issue when compiling release notes v7.6.0 labels Nov 14, 2019

weltenwort self-assigned this Nov 14, 2019

weltenwort added 3 commits November 19, 2019 16:08

Rename log rate page components to be more specific

cc8035e

Introduce specific useLogEntryRateJobs hook

a2d319e

Move log rate result hooks into the pages dirs

3344f32

weltenwort force-pushed the logs-ui-categorization-generalize-ml-module-management branch from 4d0243d to 3344f32 Compare November 19, 2019 22:08

weltenwort added 2 commits November 20, 2019 12:04

Move ml link button to common components dir

222df2a

Remove most job specifics from setup state hook

32075b6

weltenwort added 2 commits November 21, 2019 17:06

Parameterize api call helpers with job types

4aea65e

Move job overrides out of the generic jobs hook

02aa0e2

Merge branch 'master' into logs-ui-categorization-generalize-ml-modul…

b0a0c51

…e-management

Merge branch 'master' into logs-ui-categorization-generalize-ml-modul…

e7c4166

…e-management

weltenwort force-pushed the logs-ui-categorization-generalize-ml-module-management branch from df3bc37 to 4b4268f Compare November 25, 2019 23:29

weltenwort added 2 commits November 28, 2019 14:52

Further abstract module management and validation

1d16d06

Rename telemetry keys for the log rate tab

adb3ea2

weltenwort force-pushed the logs-ui-categorization-generalize-ml-module-management branch from 1b4de9c to adb3ea2 Compare November 28, 2019 16:34

weltenwort marked this pull request as ready for review November 28, 2019 23:23

weltenwort requested a review from a team as a code owner November 28, 2019 23:23

weltenwort requested a review from Kerry350 November 28, 2019 23:24

weltenwort added the review label Nov 28, 2019

weltenwort mentioned this pull request Nov 29, 2019

[Logs UI] Add setup screen for the categorization tab #51905

Merged

4 tasks

Merge branch 'master' into logs-ui-categorization-generalize-ml-modul…

94485fc

…e-management

Merge branch 'master' into logs-ui-categorization-generalize-ml-modul…

4c7832c

…e-management

Merge branch 'master' into logs-ui-categorization-generalize-ml-modul…

3b7ab54

…e-management

Kerry350 approved these changes Dec 10, 2019

View reviewed changes

weltenwort added 2 commits December 10, 2019 15:03

Merge branch 'master' into logs-ui-categorization-generalize-ml-modul…

a3acaab

…e-management

Remove left-over comment

099de5c

weltenwort merged commit cf28280 into elastic:master Dec 10, 2019

weltenwort deleted the logs-ui-categorization-generalize-ml-module-management branch December 10, 2019 18:21

weltenwort mentioned this pull request Dec 11, 2019

[7.x] [Logs UI] Generalize ML module management (#50662) #52730

Closed

weltenwort mentioned this pull request Dec 11, 2019

[7.x] [Logs UI] Generalize ML module management (#50662) #52777

Merged

weltenwort added a commit that referenced this pull request Dec 11, 2019

[7.x] [Logs UI] Generalize ML module management (#50662) (#52777)

737ce28

Backports the following commits to 7.x: - [Logs UI] Generalize ML module management (#50662)

[Logs UI] Generalize ML module management #50662

[Logs UI] Generalize ML module management #50662

Uh oh!

Conversation

weltenwort commented Nov 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Implementation notes

Checklist

Uh oh!

elasticmachine commented Nov 14, 2019

Uh oh!

elasticmachine commented Nov 14, 2019

💔 Build Failed

Uh oh!

elasticmachine commented Nov 19, 2019

💔 Build Failed

Uh oh!

elasticmachine commented Nov 20, 2019

💔 Build Failed

Uh oh!

elasticmachine commented Nov 21, 2019

💔 Build Failed

Uh oh!

elasticmachine commented Nov 25, 2019

💔 Build Failed

Uh oh!

elasticmachine commented Nov 25, 2019

💚 Build Succeeded

Uh oh!

elasticmachine commented Nov 26, 2019

💔 Build Failed

Uh oh!

elasticmachine commented Nov 26, 2019

💚 Build Succeeded

Uh oh!

elasticmachine commented Nov 27, 2019

💚 Build Succeeded

Uh oh!

elasticmachine commented Nov 28, 2019

💚 Build Succeeded

Uh oh!

elasticmachine commented Nov 28, 2019

💔 Build Failed

Uh oh!

weltenwort commented Nov 28, 2019

Uh oh!

elasticmachine commented Nov 28, 2019

💚 Build Succeeded

Uh oh!

weltenwort commented Dec 3, 2019

Uh oh!

elasticmachine commented Dec 3, 2019

💚 Build Succeeded

Uh oh!

weltenwort commented Dec 9, 2019

Uh oh!

weltenwort commented Dec 9, 2019

Uh oh!

Kerry350 left a comment

Choose a reason for hiding this comment

Uh oh!

Kerry350 Dec 10, 2019

Choose a reason for hiding this comment

Uh oh!

weltenwort Dec 10, 2019

Choose a reason for hiding this comment

Uh oh!

Kerry350 Dec 10, 2019

Choose a reason for hiding this comment

Uh oh!

weltenwort Dec 10, 2019

Choose a reason for hiding this comment

Uh oh!

Kerry350 Dec 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Dec 10, 2019

💚 Build Succeeded

weltenwort commented Nov 14, 2019 •

edited

Loading

Kerry350 Dec 10, 2019 •

edited

Loading