Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions explore-analyze/discover/run-pattern-analysis-discover.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@ description: Detect patterns in unstructured data with pattern analysis in Disco

Pattern analysis in **Discover** helps you find patterns in unstructured log messages by performing categorization analysis on text fields. It creates categories based on message structures, displays their distribution in a chart, and provides example documents for each pattern. This makes it easier to identify common log patterns, filter out noise, and focus on anomalous messages during troubleshooting.

:::{tip}
:applies_to: { stack: preview 9.4, serverless: preview }
You can also detect patterns from {{esql}} using the [`CATEGORIZE`](elasticsearch://reference/query-languages/esql/functions-operators/grouping-functions/categorize.md) grouping function. Discover renders the results as expandable groups with row actions. Refer to [](/explore-analyze/discover/try-esql.md#esql-cascade-layout).
:::

Pattern analysis works on any text field. This example uses the [sample web logs data](../index.md#gs-get-data-into-kibana), or you can use your own log data.

1. Go to **Discover**.
Expand Down
72 changes: 72 additions & 0 deletions explore-analyze/discover/try-esql.md
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,78 @@ When you save your edits, the control is updated for your query.
:::


## View grouped results from a STATS query [esql-cascade-layout]
```{applies_to}
stack: preview 9.4
serverless: preview
```

When your {{esql}} query uses a [`STATS BY`](elasticsearch://reference/query-languages/esql/commands/stats-by.md) clause with a single grouping field, **Discover** displays the results as expandable groups instead of a flat table. Each row represents one unique value of the grouping field, and you can expand it to inspect the underlying documents without leaving the query. The results count above the table reports the number of groups instead of the number of documents.

:::{image} /explore-analyze/images/discover-esql-cascade-overview.png
:alt: Grouped results layout in Discover, with one row expanded to show underlying documents
:screenshot:
:::

The grouped layout activates when the `BY` clause contains a single field reference or a single [`CATEGORIZE`](elasticsearch://reference/query-languages/esql/functions-operators/grouping-functions/categorize.md) call. Other grouping functions like `BUCKET` or `TBUCKET`, and queries that group by more than one field (for example, `BY clientip, extension`), keep the standard flat results table.

### Pattern rendering

When the grouping field uses [`CATEGORIZE`](elasticsearch://reference/query-languages/esql/functions-operators/grouping-functions/categorize.md), each row title shows the detected pattern with token highlighting, so you can scan repeated message structures at a glance. For example:

```esql
FROM kibana_sample_data_logs
| STATS Count = COUNT(*) BY Pattern = CATEGORIZE(message)
| SORT Count DESC
```

% RESTORE FOR 9.5 - start
% The block below documents the inline SPARKLINE rendering. SPARKLINE has been
% deferred from 9.4 to 9.5. When SPARKLINE ships in a release build, restore
% this block, rename the subsection back to "Pattern and sparkline rendering",
% uncomment the screenshot directive, and update the screenshot if needed.
% Tracked in: https://github.com/elastic/docs-content/issues/6215
%
% When the query also computes a [`SPARKLINE`](elasticsearch://reference/query-languages/esql/functions-operators/aggregation-functions/sparkline.md) over time, the resulting array is rendered as an inline sparkline next to the row aggregates. For example, the following query categorizes log messages and renders a per-pattern sparkline:
%
% ```esql
% FROM kibana_sample_data_logs
% | WHERE @timestamp <= ?_tend AND @timestamp > ?_tstart
% | STATS Count = COUNT(*),
% Sparkline = SPARKLINE(COUNT(*), @timestamp, 40, ?_tstart, ?_tend)
% BY Pattern = CATEGORIZE(message)
% | SORT Count DESC
% ```
%
% On larger data sets, add a [`SAMPLE`](elasticsearch://reference/query-languages/esql/commands/sample.md) command before `STATS` to keep the categorization fast, and divide `COUNT(*)` by the same sample fraction to keep the counts representative. For example, `SAMPLE 0.001` followed by `Count = COUNT(*) / 0.001`.
%
% :::{image} /explore-analyze/images/discover-esql-cascade-pattern-sparkline.png
% :alt: A grouped row showing a CATEGORIZE pattern with token highlighting and an inline sparkline
% :screenshot:
% :::
% RESTORE FOR 9.5 - end

::::{tip}
Pattern detection on text fields is also available outside {{esql}} from the **Patterns** tab in Discover's classic mode. Refer to [](/explore-analyze/discover/run-pattern-analysis-discover.md).
::::

### Grouped row actions

Select the {icon}`boxes_vertical` actions button on any group row to:

- **Copy to clipboard**: copy the group's value.
- **Filter in**: append a `WHERE` clause to your query that keeps only documents matching this group.
- **Filter out**: append a `WHERE` clause that excludes documents matching this group.
- **Open in new tab**: open the documents in this group in a new Discover tab, with a query scoped to that group.

**Filter in** and **Filter out** are disabled when the grouping field is not filterable.

### Opt out of the grouped layout

When the grouped layout activates, the regular results table toolbar is replaced with a {icon}`flask` **Group by** button. The button shows the number of active groupings as a badge.

The grouping field is preselected from your `STATS BY` clause. Open the **Group by** menu and select **none** to fall back to the standard flat results table and bring back the regular toolbar.

## Refine an {{esql}} query by interacting with the results table

Certain interactions with the results table of your {{esql}} query in Discover apply additional filters to your query. When hovering over a value cell, contextual options appear:
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading