[Logs UI] Store Logs UI settings in a dedicated infrastructure-monitoring-log-view saved object

[weltenwort]

📝 Summary

This replaces the concept of "log sources" (which was used the shared "source configurations" internally) with "log views", which are separate saved objects.

closes #120924
closes #100057

🆕 New and changed concepts

To fulfill the ACs of #120924 this PR introduces a few new concepts and rearranges the architecture slightly to follow the platform conventions more closely:

New infrastructure-monitoring-log-view saved object type

The new infrastructure-monitoring-log-view saved object type is an extracted sub-set of the log-specific properties from the old shared infrastructure-ui-source saved object type. Simulaneously, the index_pattern log index reference type has been renamed to data_view.

New LogViewsService

(server-side implementation, client-side implementation)

A new LogViewsService mediates access to the new LogViewsClient on the server and client (see below). As recommended in the platform architecture guidelines it mimicks the plugin's lifecycle:

• During server setup() it exposes a registerInternalLogView function to replace the old registerInternalSourceConfiguration.
• During server start() it exposes getClient() and getScopedClient() functions to get access to LogViewsClient instances.
• During client start() it exposes a client property to get access to a pre-scoped LogViewsClient.

Jest mocks for the service are provided alongside the implementation files.

New LogViewsClient

(server-side implementation, client-side implementation)

The LogViewsClient abstracts read and write access to the underlying saved objects as described in the lookup order in #120924.

On the server it uses the data.search, data_views, saved_object, and InfraSource services to facilitate get, put and resolve operations. The rest of the server-side infra plugin (e.g. the HTTP routes and the log rate metrics code) as well as other plugins should use it to access log views. To that end it's available via the logViews.getClient() and logViews.getScopedClient() functions on the infra plugins start contract. The start contract is made available in the legacy dependency injection libs as getStartServices().

On the client it uses the http, data.search, and data_views services to access the log view HTTP routes and resolve the log view configurations. The rest of the client-side infra plugin (e.g. the React hierarchy) as well as other plugins should use it to access log views. To that that end it's available via the logViews.client property on the infra plugin start contract and injected into the infra plugins KibanaContextForPlugin. In some special cases it's also useful to directly instantiate it if a reference to the start contract cannot be obtained (currently the case in the shared log stream component, which will be improved in a follow-up step).

It offers the following set of features:

• Get a log view (taking the saved object, internal definitions and source config backwards-compatibility into account)
• Resolve a log view (particularly its data view references)
• Put a log view
• Only on the client, get a log view's index status (missing, empty or available)

Jest mocks for the client are provided alongside the implementation files.

New LogViewProvider, useLogViewContext, and useLogView hooks

(client-side implementation)

The new useLogView hook integrates the LogViewsClient given via the argument into the React render lifecycle. It replace the useLogSource hook. The constate-generated LogViewsProvider is available in the Logs UI's page hierarchy, so the useLogViewContext hook can be used to access the hook state and functions without initiating multiple (potentially expensive) requests. Its API mimics the predecessor's with all its quirks, so it should be cleaned up in a follow-up task.

A jest mock for the hook is provided alongside the implementation file.

New /logViews/:logViewId HTTP routes

(common type definitions, HTTP API type definitions)

The new /logViews/:logViewId HTTP route allows the client-side client to communicate with the server-side implementation. It supports the GET method for reading and the PUT method for writing log view attributes. Since it's not a public API it should normally not be used directly, but only via the client-side LogViewsClient. Direct access should be reserved for debugging situations.

Self-sufficient log threshold rule expression

The client-side log threshold rule now wraps the lazily loaded parameter expression component into a context provider. That means it's not longer dependent on the surrounding context and can therefore access the infra plugin's LogViewsClient instance even in the common rule management UI. This pattern would be applicable for the metrics rules as well.

Additional notes

• While most usages of "log source" in variable names have been replaced, some usages (like sourceId component props) remain un-renamed to limit the scope of the PR.
• I restructured the log stream storybook stories such that the mdx file contains as little TS code as possible. By moving them into separate files they can be covered by the compiler.

➡️ Follow-up tasks

• [Logs UI] Migrate usages of defineInternalSourceConfiguration to defineInternalLogView #129080
• [Logs UI] Export completely self-contained log stream component #129084
• Clean up the useLogView internal state and API.

[weltenwort]

@Kerry350 Thanks for taking on the review of this sizable PR ❤️

I removed the TODO comment and aligned the route registration to match the common practice of the other routes.

[Kerry350]

I think I've tested out everything and couldn't find any problems. The UI works as expected (including things like ML job configs). I tried out the various paths, e.g. loading straight from the new saved object vs loading from the InfraSources lib.

I also verified multiple log views (I know we don't expose it, but I checked the underlying mechanism). That worked great too.

I created that with the API directly, and then also tested that out in a few ways.

Nice work 🙌

[Kerry350]

✅

[Kerry350]

Took this for another spin with the changes to default and usage of IDs:

Singular:

Multiples using import / export with newly generated IDs:

For all of Fallback mechanism and Singular LogView and multiple LogViews:

• Stream ✅
• Creation of ML jobs (config) ✅
• ML pages ✅
• Observability overview ✅
• Alerts ✅

Resolution worked as expected, using the last updated view as the default.

As an example, here the import stripped two of the columns so there was only timestamp in use, this became the default view as expected.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: c5f3db5
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
infra	940	941	+1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
infra	25	28	+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
infra	1002.0KB	1005.7KB	+3.7KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
infra	3	5	+2

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
infra	89.8KB	88.1KB	-1.7KB

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
infrastructure-monitoring-log-view	-	2	+2

Unknown metric groups

API count

id	before	after	diff
infra	28	31	+3

ESLint disabled line counts

id	before	after	diff
infra	130	129	-1

References to deprecated APIs

id	before	after	diff
infra	11	2	-9

Total ESLint disabled count

id	before	after	diff
infra	137	136	-1

History

• 💚 Build #35176 succeeded 86d5a99
• 💚 Build #34963 succeeded 06e85d6
• 💔 Build #34942 failed 8239a38
• 💚 Build #34537 succeeded 1c50e36
• 💚 Build #34105 succeeded e738380
• 💚 Build #33487 succeeded 484b6fc

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @weltenwort

[afharo]

SO type_registrations test LGTM!

[Kerry350]

LGTM 👍

Thanks for persevering with the back and forth on this PR.

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 125014 or prevent reminders by adding the backport:skip label.