[Logs Overview] Overview component (iteration 1) by weltenwort · Pull Request #191899 · elastic/kibana

weltenwort · 2024-09-02T09:52:16Z

📝 Summary

This introduces a "Logs Overview" component for use in solution UIs behind a feature flag.

🏁 closes: [Logs Overview] Enhanced logs component for solution UIs #190848
🐞 affected by: change_point agg fails with IllegalArgumentException while indexing an array elasticsearch#112805

🏴‍☠️ Feature flag

The new logs overview is currently visible in the "Logs" tab of APM services when the feature flag observability:newLogsOverview is enabled in the advanced settings:

🎨 Previews

In APM UI

In the hosts UI

No log entries

⚠️ Known issues

Since this uses the change_point Elasticsearch aggregation extensively it is affected by change_point agg fails with IllegalArgumentException while indexing an array elasticsearch#112805.

❓ Open questions

See: #191899 (comment)

The change types returned by the change_point agg have some limitations:
- Some of them are hard to interpret (i.e. non-stationary)
- They are sensitive to slight changes in the distribution. A change can flap between a step, a spike and a distribution_change when just a few buckets are different. So it might be advisable to use the change point buckets detected and re-interpret their meaning using the histogram.
It's debatable whether a lexicographical sort order of the change types is a good idea. We might want to come up with more useful sorting weights.
Since the changes types shown in the design mock-ups don't line up with reality the assignment of the colors is unclear. Currently no types' badges are colored.

🕵️‍♀️ Review notes

This adds XState v5 as an additional dependency to start the transition. It can be imported as xstate5. Similarly, @xstate5/react is available.

Release Note

Add experimental logs overview to the observability hosts and service overviews

obltmachine · 2024-09-02T09:52:28Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

weltenwort · 2024-09-05T16:54:37Z

@patpscal @LucaWintergerst @alex-fedotyev There are a few question that arose while implementing this. I would appreciate your input on them:

Location of the "open in discover" link

The designs show the "open in discover" link to the right of the time picker. In most places that this component would be used, though, the page already has a time picker as part of the page layout. Am I correct assuming that we don't want to duplicate that time picker as it would be confusing to the user? If so, the layout of that part of the page is not under our control and therefore inserting the "open in discover" button in that location would be tricky.

Do we want to look for an alternative location that would work with those pages too?

Colors of the "change type" badges

We need to map the technical change types to the change types shown in the UI. From looking at the design the mapping is not obvious to me, so I would appreciate your input. These are the technical types that we're currently handling in the code:

none: no change, but also not rare
rare: too few documents for a change point detection, but we detect the "first occurrence"
spike / dip: the aggregation detects temporary deviations from an otherwise constant metric
step: the aggregation detects a point in time with different constant metrics before and after
trend: the aggregation detects a trend within the metric with an accompanying correlation coefficient (between time and the metric)
distribution: the aggregation detects a point in time at which the kind of distribution changes
other: the aggregation calls this "non-stationary", but the docs don't specify what it means
unknown: a catch-all in case Elasticsearch returns something we're not prepared to handle

Sorting by "change type"

What sorting criteria do we want to use to sort the "Change type" column by? Just sorting the text of the types mentioned above alphabetically is probably not the most useful. We probably want a custom sort order that reflects what we consider to be "important". Have you put any thought into that or should I come up with something?

alex-fedotyev · 2024-09-05T17:50:27Z

@weltenwort - I am still confused with the description of the change types, trying to map those to the simple terms.
Do you have a sense about change types and examples?
I have been always thinking about the changes in a way of "new pattern", significantly increased rate of existing pattern, significantly decreased rate of a pattern, rare messages.
Could you give me other examples to think about or maybe map the current change type to the ones I proposed?

LucaWintergerst · 2024-09-05T18:32:05Z

I'll leave the open in discover button decision to someone else.

Colors
I think I'll need to get hands on with a draft of it first to make a call here.
This also feels like something we can adjust later on, it should not block us for now.

Sorting
What is our default sort order? Are we using the p_value yet? I'm not sure if this was discussed before. I think we should include it too. It might even make sense for us to convert it into something more readable on a log scale to have a "confidence" column instead of the more cryptic p_value
Sorting on the type of change is a good question. I think alphabetical is not the worst idea, as I might want to check for specific types of changes, so I could for example sort, then scroll to where the first "rare" value begins and then see all other rare patterns next to it

alex-fedotyev · 2024-09-05T20:56:16Z

@LucaWintergerst - default sorting is going to be by "Change at" datetime field which will allow to show the changes above the systemic patterns, and also make it chronological.

I wonder if we should do the opposite and never expose the p-value in any shape, and also make the "Change type" to be just the inline badge and not the column.
This way we will leave true value to be driven by the AI to determine the impact of those changes, which I bet will be better than p-value or change type... in the end, impact can be high for either rare event or a change in the rate of the pattern, everything depends on the content and context.

elasticmachine · 2024-10-03T18:03:31Z

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

Dosant

packages/kbn-management/settings/setting_ids/index.ts lgtm

cauemarcondes

LGTM.

cauemarcondes · 2024-10-04T09:59:04Z

+  if (environment === ENVIRONMENT_ALL.value) {
+    return [serviceNameFilter];
+  } else {
+    return [
+      {
+        bool: {
+          filter: [
+            serviceNameFilter,
+            {
+              term: {
+                [SERVICE_ENVIRONMENT]: environment,
+              },
+            },
+          ],
+        },
+      },
+      {
+        bool: {
+          filter: [serviceNameFilter],
+          must_not: [
+            {
+              exists: {
+                field: SERVICE_ENVIRONMENT,
+              },
+            },
+          ],
+        },
+      },
+    ];
+  }
+}


Can you simplify this by using environmentQuery?

hm. not sure. I modeled this after the KQL query used for the old logs stream, which includes both the logs with the given environment and those not having an environment at all. The environmentQuery function doesn't do that. Can you help me understand whether a change in the query semantics would be desirable here?

afharo

Telemetry changes LGTM

Kerry350

Amazing work 👏

(Approving based on the iterative improvements that will come and tests are TODO currently).

…dable-iteration-1

weltenwort · 2024-10-09T14:30:03Z

@elasticmachine merge upstream

elasticmachine · 2024-10-09T16:07:29Z

💚 Build Succeeded

Buildkite Build
Commit: c6d26db
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-191899-c6d26db8c88c

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`datasetQuality`	224	225	+1
`infra`	1508	1509	+1
`logsExplorer`	562	563	+1
`logsShared`	228	298	+70
`observabilityLogsExplorer`	203	204	+1
`observabilityOnboarding`	255	256	+1
total			+75

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/apm-synthtrace-client`	208	223	+15
`@kbn/management-settings-ids`	142	143	+1
`@kbn/observability-logs-overview`	-	27	+27
`@kbn/xstate-utils`	13	14	+1
`logsShared`	281	284	+3
total			+47

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`apm`	3.4MB	3.4MB	+1.7KB
`infra`	1.6MB	1.6MB	+636.0B
`logsShared`	140.5KB	327.3KB	+186.8KB
total			+189.2KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`@kbn/apm-synthtrace-client`	33	36	+3
`@kbn/observability-logs-overview`	-	3	+3
`logsShared`	32	34	+2
total			+8

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`logsShared`	173.3KB	174.5KB	+1.1KB

Unknown metric groups

API count

id	before	after	diff
`@kbn/apm-synthtrace-client`	208	223	+15
`@kbn/management-settings-ids`	144	145	+1
`@kbn/observability-logs-overview`	-	29	+29
`@kbn/xstate-utils`	13	14	+1
`logsShared`	310	313	+3
total			+49

async chunk count

id	before	after	diff
`logsShared`	14	15	+1

ESLint disabled line counts

id	before	after	diff
`@kbn/xstate-utils`	0	1	+1

Total ESLint disabled count

id	before	after	diff
`@kbn/xstate-utils`	0	1	+1

History

💔 Build #240657 failed 4940642

cc @weltenwort

kibanamachine · 2024-10-09T17:22:08Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11259931384

kibanamachine · 2024-10-09T17:26:41Z

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 191899

Questions ?

Please refer to the Backport tool documentation

weltenwort · 2024-10-09T17:38:52Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

This introduces a "Logs Overview" component for use in solution UIs behind a feature flag. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kerry Gallagher <471693+Kerry350@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit 15bccdf) # Conflicts: # .github/CODEOWNERS # src/plugins/telemetry/schema/oss_plugins.json

This reverts commit 15bccdf.

Ikuni17 · 2024-10-09T18:44:28Z

This was reverted with a31b16e because xstate package is not working with Storybook build (on-merge logs). When reopening the PR, the label ci:build-storybooks will build Storybook in CI for testing.

weltenwort · 2024-10-09T18:47:17Z

Good to know, thank you. Sounds like that label should be the default, then?

This is a re-submission of #191899, which was reverted due to a storybook build problem. This introduces a "Logs Overview" component for use in solution UIs behind a feature flag. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kerry Gallagher <471693+Kerry350@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…#195673) This is a re-submission of elastic#191899, which was reverted due to a storybook build problem. This introduces a "Logs Overview" component for use in solution UIs behind a feature flag. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kerry Gallagher <471693+Kerry350@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit 0caea22) # Conflicts: # .github/CODEOWNERS # src/plugins/telemetry/schema/oss_plugins.json

…195673) (#195742) # Backport This will backport the following commits from `main` to `8.x`: - [[Logs Overview] Overview component (iteration 1) (attempt 2) (#195673)](#195673)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)

weltenwort added 2 commits August 20, 2024 18:11

Add rare logs scenario

b3a1af8

Add empty logs_overview package

0a43169

weltenwort added Feature:Logs UI Logs UI feature Team:obs-onboarding Observability Onboarding Team labels Sep 2, 2024

weltenwort self-assigned this Sep 2, 2024

weltenwort added 6 commits September 2, 2024 15:57

Add gaussian distribution to the unstructured logs scenario

d373e59

Add support for xstate5

278365e

Add xstate5 console inspector

1f4bd0f

Add cateogorization service

76ea2a5

Add categories grid

0528e8b

Enable translation linting for the package

0b7fc2c

weltenwort force-pushed the logs-enhanced-embeddable-iteration-1 branch from 8e940d2 to 0b7fc2c Compare September 2, 2024 16:07

weltenwort added 13 commits September 2, 2024 16:22

Improve typing of the sorting config

b428bd4

Improve default column widths

5be73af

Add loading screen with cancellation

e0af1bb

Merge branch 'main' into logs-enhanced-embeddable-iteration-1

c70a37a

Bump XState 5 version

169df23

Start cleaning up names and introducing the overview component

901bee3

Provide a self-contained overview via logs_shared

1023d69

Fix unnecessary uiSettings dependency

0d35529

Fix some refactoring mistakes

d551da4

Memoize time overview props

47153c8

Update zod import

dc8ea70

Add service filters and error reporting

3793877

Fix fetching remaining categories

c69a83b

botelastic Bot added ci:project-deploy-observability Create an Observability project Team:obs-ux-infra_services - DEPRECATED DEPRECATED - Use Team:obs-presentation. labels Oct 3, 2024

Dosant approved these changes Oct 4, 2024

View reviewed changes

cauemarcondes approved these changes Oct 4, 2024

View reviewed changes

afharo approved these changes Oct 4, 2024

View reviewed changes

Kerry350 approved these changes Oct 7, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/main' into logs-enhanced-embed…

4940642

…dable-iteration-1

weltenwort added the release_note:feature Makes this part of the condensed release notes label Oct 9, 2024

delanni approved these changes Oct 9, 2024

View reviewed changes

elasticmachine and others added 2 commits October 9, 2024 16:30

Merge branch 'main' into logs-enhanced-embeddable-iteration-1

46f8e9e

Use timeField parameter instead of hard-coded field name

c6d26db

weltenwort merged commit 15bccdf into elastic:main Oct 9, 2024

kibanamachine added the v9.0.0 label Oct 9, 2024

weltenwort mentioned this pull request Oct 9, 2024

[8.x] [Logs Overview] Overview component (iteration 1) (#191899) #195660

Closed

Ikuni17 added a commit that referenced this pull request Oct 9, 2024

Revert "[Logs Overview] Overview component (iteration 1) (#191899)"

a31b16e

This reverts commit 15bccdf.

Ikuni17 added the reverted label Oct 9, 2024

weltenwort mentioned this pull request Oct 9, 2024

[Logs Overview] Overview component (iteration 1) (attempt 2) #195673

Merged

smith added the apm:review label Oct 10, 2024

Conversation

weltenwort commented Sep 2, 2024 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Summary

🏴‍☠️ Feature flag

🎨 Previews

In APM UI

In the hosts UI

No log entries

⚠️ Known issues

❓ Open questions

🕵️‍♀️ Review notes

Release Note

Uh oh!

obltmachine commented Sep 2, 2024

🤖 GitHub comments

Uh oh!

weltenwort commented Sep 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Location of the "open in discover" link

Colors of the "change type" badges

Sorting by "change type"

Uh oh!

alex-fedotyev commented Sep 5, 2024

Uh oh!

LucaWintergerst commented Sep 5, 2024

Uh oh!

alex-fedotyev commented Sep 5, 2024

Uh oh!

elasticmachine commented Oct 3, 2024

Uh oh!

Dosant left a comment

Choose a reason for hiding this comment

Uh oh!

cauemarcondes left a comment

Choose a reason for hiding this comment

Uh oh!

cauemarcondes Oct 4, 2024

Choose a reason for hiding this comment

Uh oh!

weltenwort Oct 9, 2024

Choose a reason for hiding this comment

Uh oh!

afharo left a comment

Choose a reason for hiding this comment

Uh oh!

Kerry350 left a comment

Choose a reason for hiding this comment

Uh oh!

weltenwort commented Oct 9, 2024

Uh oh!

elasticmachine commented Oct 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Public APIs missing exports

Page load bundle

API count

async chunk count

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

kibanamachine commented Oct 9, 2024

Uh oh!

kibanamachine commented Oct 9, 2024

💔 All backports failed

Manual backport

Questions ?

Uh oh!

weltenwort commented Oct 9, 2024

💚 All backports created successfully

Questions ?

Uh oh!

Ikuni17 commented Oct 9, 2024

Uh oh!

weltenwort commented Oct 9, 2024

weltenwort commented Sep 2, 2024 •

edited by kibanamachine

Loading

weltenwort commented Sep 5, 2024 •

edited

Loading

elasticmachine commented Oct 9, 2024 •

edited

Loading