diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/change_point.md b/docs/reference/query-languages/esql/_snippets/commands/layout/change_point.md index 86f5d186475e5..17c811677589f 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/change_point.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/change_point.md @@ -9,13 +9,13 @@ The `CHANGE_POINT` command requires a [platinum license](https://www.elastic.co/ `CHANGE_POINT` detects spikes, dips, and change points in a metric. -**Syntax** +## Syntax ```esql CHANGE_POINT value [ON key] [AS type_name, pvalue_name] ``` -**Parameters** +## Parameters `value` : The column with the metric in which you want to detect a change point. @@ -29,7 +29,7 @@ CHANGE_POINT value [ON key] [AS type_name, pvalue_name] `pvalue_name` : The name of the output column with the p-value that indicates how extreme the change point is. If not specified, `pvalue` is used. -**Description** +## Description `CHANGE_POINT` detects spikes, dips, and change points in a metric. The command adds columns to the table with the change point type and p-value, that indicates how extreme the change point is @@ -46,9 +46,9 @@ The possible change point types are: There must be at least 22 values for change point detection. Fewer than 1,000 is preferred. :::: -**Examples** +## Examples -The following example shows the detection of a step change: +The following example detects a step change in a metric: :::{include} ../examples/change_point.csv-spec/changePointForDocs.md ::: diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/completion.md b/docs/reference/query-languages/esql/_snippets/commands/layout/completion.md index 3fe5d5373117e..7274a08672122 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/completion.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/completion.md @@ -51,7 +51,7 @@ Best practices: :::: ::::: -**Syntax** +## Syntax ::::{applies-switch} @@ -73,7 +73,7 @@ COMPLETION [column =] prompt WITH my_inference_endpoint :::: -**Parameters** +## Parameters `column` : (Optional) The name of the output column containing the LLM's response. @@ -88,7 +88,7 @@ COMPLETION [column =] prompt WITH my_inference_endpoint : The ID of the [inference endpoint](docs-content://explore-analyze/elastic-inference/inference-api.md) to use for the task. The inference endpoint must be configured with the `completion` task type. -**Description** +## Description The `COMPLETION` command provides a general-purpose interface for text generation tasks using a Large Language Model (LLM) in ES|QL. @@ -103,13 +103,13 @@ including: - Content rewriting - Creative generation -**Requirements** +## Requirements To use this command, you must deploy your LLM model in Elasticsearch as an [inference endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put) with the task type `completion`. -#### Handling timeouts +### Handling timeouts `COMPLETION` commands may time out when processing large datasets or complex prompts. The default timeout is 10 minutes, but you can increase this limit if necessary. @@ -141,9 +141,13 @@ If you don't want to increase the timeout limit, try the following: * Configure your HTTP client's response timeout (Refer to [HTTP client configuration](/reference/elasticsearch/configuration-reference/networking-settings.md#_http_client_configuration)) -**Examples** +## Examples -Use the default column name (results stored in `completion` column): +The following examples show common `COMPLETION` patterns. + +### Use the default output column name + +If no column name is specified, the response is stored in `completion`: ```esql ROW question = "What is Elasticsearch?" @@ -155,7 +159,9 @@ ROW question = "What is Elasticsearch?" |------------------------|-------------------------------------------| | What is Elasticsearch? | A distributed search and analytics engine | -Specify the output column (results stored in `answer` column): +### Specify the output column name + +Use `column =` to assign the response to a named column: ```esql ROW question = "What is Elasticsearch?" @@ -167,7 +173,9 @@ ROW question = "What is Elasticsearch?" | --- | --- | | What is Elasticsearch? | A distributed search and analytics engine | -Summarize the top 10 highest-rated movies using a prompt: +### Summarize documents with a prompt + +Use `CONCAT` to build a prompt from field values before calling `COMPLETION`: ```esql FROM movies diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/dissect.md b/docs/reference/query-languages/esql/_snippets/commands/layout/dissect.md index da9eebcbfc7eb..da3262eb6d702 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/dissect.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/dissect.md @@ -5,17 +5,17 @@ stack: ga `DISSECT` enables you to [extract structured data out of a string](/reference/query-languages/esql/esql-process-data-with-dissect-grok.md). -**Syntax** +## Syntax ```esql DISSECT input "pattern" [APPEND_SEPARATOR=""] ``` -**Parameters** +## Parameters `input` -: The column that contains the string you want to structure. If the column has -multiple values, `DISSECT` will process each value. +: The column that contains the string you want to structure. + If the column has multiple values, `DISSECT` will process each value. `pattern` : A [dissect pattern](/reference/query-languages/esql/esql-process-data-with-dissect-grok.md#esql-dissect-patterns). @@ -25,21 +25,26 @@ multiple values, `DISSECT` will process each value. `` : A string used as the separator between appended values, when using the [append modifier](/reference/query-languages/esql/esql-process-data-with-dissect-grok.md#esql-append-modifier). -**Description** +## Description `DISSECT` enables you to [extract structured data out of a string](/reference/query-languages/esql/esql-process-data-with-dissect-grok.md). `DISSECT` matches the string against a delimiter-based pattern, and extracts the specified keys as columns. Refer to [Process data with `DISSECT`](/reference/query-languages/esql/esql-process-data-with-dissect-grok.md#esql-process-data-with-dissect) for the syntax of dissect patterns. -**Examples** +## Examples -The following example parses a string that contains a timestamp, some text, and -an IP address: +The following examples show how to parse and convert structured strings with `DISSECT`. + +### Parse a structured string + +Parse a string that contains a timestamp, some text, and an IP address: :::{include} ../examples/docs.csv-spec/basicDissect.md ::: +### Convert output to a non-string type + By default, `DISSECT` outputs keyword string columns. To convert to another type, use [Type conversion functions](/reference/query-languages/esql/functions-operators/type-conversion-functions.md): diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/drop.md b/docs/reference/query-languages/esql/_snippets/commands/layout/drop.md index 32aabf25278c9..aadbc45d357ae 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/drop.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/drop.md @@ -5,21 +5,28 @@ stack: ga The `DROP` processing command removes one or more columns. -**Syntax** +## Syntax ```esql DROP columns ``` -**Parameters** +## Parameters `columns` : A comma-separated list of columns to remove. Supports wildcards. -**Examples** +## Examples + +The following examples show how to remove columns by name and by pattern. + +### Drop a column by name :::{include} ../examples/drop.csv-spec/height.md ::: + +### Drop columns matching a wildcard pattern + Rather than specify each column by name, you can use wildcards to drop all columns with a name that matches a pattern: diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/enrich.md b/docs/reference/query-languages/esql/_snippets/commands/layout/enrich.md index 854f7d5285481..80d3d6bd790e3 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/enrich.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/enrich.md @@ -6,13 +6,22 @@ stack: ga `ENRICH` enables you to add data from existing indices as new columns using an enrich policy. -**Syntax** +::::{tip} +Consider using `LOOKUP JOIN` instead of `ENRICH` for your use case. + +Learn more: + +- [`LOOKUP JOIN` overview](/reference/query-languages/esql/esql-lookup-join.md) +- [`LOOKUP JOIN` command reference](/reference/query-languages/esql/commands/lookup-join.md) +:::: + +## Syntax ```esql ENRICH policy [ON match_field] [WITH [new_name1 = ]field1, [new_name2 = ]field2, ...] ``` -**Parameters** +## Parameters `policy` : The name of the enrich policy. @@ -38,16 +47,17 @@ ENRICH policy [ON match_field] [WITH [new_name1 = ]field1, [new_name2 = ]field2, enrich field is renamed. `new_nameX` -: Enables you to change the name of the column that’s added for each of the enrich +: Enables you to change the name of the column that's added for each of the enrich fields. Defaults to the enrich field name. If a column has the same name as the new name, it will be discarded. If a name (new or original) occurs more than once, only the rightmost duplicate creates a new column. -**Description** +## Description `ENRICH` enables you to add data from existing indices as new columns using an -enrich policy. Refer to [Data enrichment](/reference/query-languages/esql/esql-enrich-data.md) +enrich policy. +Refer to [Data enrichment](/reference/query-languages/esql/esql-enrich-data.md) for information about setting up a policy. :::{image} /reference/query-languages/images/esql-enrich.png @@ -58,26 +68,31 @@ for information about setting up a policy. Before you can use `ENRICH`, you need to [create and execute an enrich policy](/reference/query-languages/esql/esql-enrich-data.md#esql-set-up-enrich-policy). :::: +In case of name collisions, the newly created columns will override existing columns. -**Examples** +## Examples -The following example uses the `languages_policy` enrich policy to add a new -column for each enrich field defined in the policy. The match is performed using -the `match_field` defined in the [enrich policy](/reference/query-languages/esql/esql-enrich-data.md#esql-enrich-policy) and -requires that the input table has a column with the same name (`language_code` -in this example). `ENRICH` will look for records in th -[enrich index](/reference/query-languages/esql/esql-enrich-data.md#esql-enrich-index) -based on the match field value. +The following examples show common `ENRICH` patterns. + +### Use the default match field + +`ENRICH` looks for records in the [enrich index](/reference/query-languages/esql/esql-enrich-data.md#esql-enrich-index) +using the `match_field` defined in the [enrich policy](/reference/query-languages/esql/esql-enrich-data.md#esql-enrich-policy). +The input table must have a column with the same name (`language_code` in this example): :::{include} ../examples/enrich.csv-spec/enrich.md ::: +### Match on a different field using ON + To use a column with a different name than the `match_field` defined in the policy as the match field, use `ON `: :::{include} ../examples/enrich.csv-spec/enrich_on.md ::: +### Select specific enrich fields using WITH + By default, each of the enrich fields defined in the policy is added as a column. To explicitly select the enrich fields that are added, use `WITH , , ...`: @@ -85,10 +100,9 @@ column. To explicitly select the enrich fields that are added, use :::{include} ../examples/enrich.csv-spec/enrich_with.md ::: -You can rename the columns that are added using `WITH new_name=`: +### Rename enrich fields using WITH + +Rename the columns that are added using `WITH new_name=`: :::{include} ../examples/enrich.csv-spec/enrich_rename.md ::: - -In case of name collisions, the newly created columns will override existing -columns. diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/eval.md b/docs/reference/query-languages/esql/_snippets/commands/layout/eval.md index a8a39adec2dc5..c3c50163b3735 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/eval.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/eval.md @@ -6,13 +6,13 @@ stack: ga The `EVAL` processing command enables you to append new columns with calculated values. -**Syntax** +## Syntax ```esql EVAL [column1 =] value1[, ..., [columnN =] valueN] ``` -**Parameters** +## Parameters `columnX` : The column name. @@ -24,23 +24,31 @@ EVAL [column1 =] value1[, ..., [columnN =] valueN] [function](/reference/query-languages/esql/esql-functions-operators.md#esql-functions). Can use columns defined left of this one. -**Description** +## Description The `EVAL` processing command enables you to append new columns with calculated values. `EVAL` supports various functions for calculating values. Refer to [Functions](/reference/query-languages/esql/esql-functions-operators.md#esql-functions) for more information. -**Examples** +## Examples + +The following examples show common `EVAL` patterns. + +### Append a calculated column :::{include} ../examples/eval.csv-spec/eval.md ::: +### Overwrite an existing column + If the specified column already exists, the existing column will be dropped, and the new column will be appended to the table: :::{include} ../examples/eval.csv-spec/evalReplace.md ::: +### Use an expression as the column name + Specifying the output column name is optional. If not specified, the new column name is equal to the expression. The following query adds a column named `height*3.281`: @@ -48,6 +56,8 @@ name is equal to the expression. The following query adds a column named :::{include} ../examples/eval.csv-spec/evalUnnamedColumn.md ::: +### Reference an auto-named column in a subsequent command + Because this name contains special characters, [it needs to be quoted](/reference/query-languages/esql/esql-syntax.md#esql-identifiers) with backticks (```) when using it in subsequent commands: diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/fork.md b/docs/reference/query-languages/esql/_snippets/commands/layout/fork.md index 40f3b0ca1641d..33a88db493140 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/fork.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/fork.md @@ -6,52 +6,65 @@ stack: preview 9.1.0 The `FORK` processing command creates multiple execution branches to operate on the same input data and combines the results in a single output table. -**Syntax** +## Syntax ```esql FORK ( ) ( ) ... ( ) ``` -**Description** +## Description The `FORK` processing command creates multiple execution branches to operate on the same input data and combines the results in a single output table. A discriminator column (`_fork`) is added to identify which branch each row came from. Together with the [`FUSE`](/reference/query-languages/esql/commands/fuse.md) command, `FORK` enables hybrid search to combine and score results from multiple queries. To learn more about using {{esql}} for search, refer to [ES|QL for search](docs-content://solutions/search/esql-for-search.md). -**Branch identification:** -- The `_fork` column identifies each branch with values like `fork1`, `fork2`, `fork3` -- Values correspond to the order branches are defined -- `fork1` always indicates the first branch - -**Column handling:** -- `FORK` branches can output different columns -- Columns with the same name must have the same data type across all branches -- Missing columns are filled with `null` values - -**Row ordering:** -- `FORK` preserves row order within each branch -- Rows from different branches may be interleaved -- Use `SORT _fork` to group results by branch - ::::{note} `FORK` branches default to `LIMIT 1000` if no `LIMIT` is provided. :::: -**Limitations** +## Output behavior + +`FORK` merges its branch outputs into a single table, adding a `_fork` discriminator +column to each row to indicate which branch it came from. + +### Branch identification + +The `_fork` column takes values like `fork1`, `fork2`, `fork3`, corresponding to the +order branches are defined. + +### Column handling + +Branches can output different columns. Columns with the same name must have the same +data type across all branches; missing columns are filled with `null` values. + +### Row ordering + +Row order is preserved within each branch, but rows from different branches may be +interleaved. Use `SORT _fork` to group results by branch. + +## Limitations - `FORK` supports at most 8 execution branches. - In versions older than 9.3.0 using remote cluster references and `FORK` is not supported. - Using more than one `FORK` command in a query is not supported. -**Examples** +## Examples -In the following example, each `FORK` branch returns one row. -Notice how `FORK` adds a `_fork` column that indicates which row the branch originates from: +The following examples show how to run parallel branches and combine their results. + +### Run two branches and identify rows by branch + +Each `FORK` branch returns one row. `FORK` adds a `_fork` column that indicates +which branch each row came from: :::{include} ../examples/fork.csv-spec/simpleFork.md +::: + +### Return row count alongside top results -The next example, returns total number of rows that match the query along with -the top five rows sorted by score. +Returns the total number of rows that match the query along with +the top five rows sorted by score: :::{include} ../examples/fork.csv-spec/simpleForkWithStats.md +::: diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/from.md b/docs/reference/query-languages/esql/_snippets/commands/layout/from.md index 6a5d6c2387d19..ecd68d5768560 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/from.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/from.md @@ -6,13 +6,17 @@ stack: ga The `FROM` source command returns a table with data from a data stream, index, or alias. -**Syntax** +::::{tip} +For time series data, use the [`TS`](/reference/query-languages/esql/commands/ts.md) source command instead of `FROM`. `TS` enables time series aggregation functions and is optimized for processing time series indices. +:::: + +## Syntax ```esql FROM index_pattern [METADATA fields] ``` -**Parameters** +## Parameters `index_pattern` : A list of indices, data streams or aliases. Supports wildcards and date math. @@ -20,7 +24,7 @@ FROM index_pattern [METADATA fields] `fields` : A comma-separated list of [metadata fields](/reference/query-languages/esql/esql-metadata-fields.md) to retrieve. -**Description** +## Description The `FROM` source command returns a table with data from a data stream, index, or alias. Each row in the resulting table represents a document. Each column @@ -44,13 +48,19 @@ FROM employees :::: -**Examples** +## Examples + +The following examples show common `FROM` patterns. + +### Query an index ```esql FROM employees ``` -You can use [date math](/reference/elasticsearch/rest-apis/api-conventions.md#api-date-math-index-names) to refer to indices, aliases +### Use date math in index names + +Use [date math](/reference/elasticsearch/rest-apis/api-conventions.md#api-date-math-index-names) to refer to indices, aliases, and data streams. This can be useful for time series data, for example to access today’s index: @@ -58,6 +68,8 @@ today’s index: FROM ``` +### Query multiple indices + Use comma-separated lists or wildcards to [query multiple data streams, indices, or aliases](/reference/query-languages/esql/esql-multi-index.md): @@ -65,6 +77,8 @@ Use comma-separated lists or wildcards to FROM employees-00001,other-employees-* ``` +### Query remote clusters + Use the format `:` to [query data streams and indices on remote clusters](/reference/query-languages/esql/esql-cross-clusters.md): @@ -72,6 +86,8 @@ Use the format `:` to FROM cluster_one:employees-00001,cluster_two:other-employees-* ``` +### Include metadata fields + Use the optional `METADATA` directive to enable [metadata fields](/reference/query-languages/esql/esql-metadata-fields.md): @@ -79,6 +95,8 @@ Use the optional `METADATA` directive to enable FROM employees METADATA _id ``` +### Escape index names with special characters + Use enclosing double quotes (`"`) or three enclosing double quotes (`"""`) to escape index names that contain special characters: diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/fuse.md b/docs/reference/query-languages/esql/_snippets/commands/layout/fuse.md index 2b926423f1906..2f2dc88ce48a0 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/fuse.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/fuse.md @@ -76,10 +76,11 @@ When `fuse_method` is `LINEAR`, `options` supports the following parameters: ## Examples +The following examples use `FORK` to run parallel queries and `FUSE` to merge the results. + ### Use RRF -In the following example, we use the [`FORK`](/reference/query-languages/esql/commands/fork.md) command to run two different queries: a lexical and a semantic query. -We then use `FUSE` to merge the results (applies `RRF` by default): +Run a lexical and a semantic query in parallel with `FORK`, then merge with `FUSE` (applies `RRF` by default): ```esql FROM books METADATA _id, _index, _score # Include document ID, index name, and relevance score diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/grok.md b/docs/reference/query-languages/esql/_snippets/commands/layout/grok.md index 313eb8c2ce502..501ad9b68cbcb 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/grok.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/grok.md @@ -5,13 +5,13 @@ stack: ga `GROK` enables you to [extract structured data out of a string](/reference/query-languages/esql/esql-process-data-with-dissect-grok.md). -**Syntax** +## Syntax ```esql GROK input "pattern" ``` -**Parameters** +## Parameters `input` : The column that contains the string you want to structure. If the column has @@ -22,7 +22,7 @@ GROK input "pattern" If a field name is used more than once, a multi-valued column will be created with one value per each occurrence of the field name. -**Description** +## Description `GROK` enables you to [extract structured data out of a string](/reference/query-languages/esql/esql-process-data-with-dissect-grok.md). `GROK` matches the string against patterns, based on regular expressions, @@ -30,14 +30,19 @@ and extracts the specified patterns as columns. Refer to [Process data with `GROK`](/reference/query-languages/esql/esql-process-data-with-dissect-grok.md#esql-process-data-with-grok) for the syntax of grok patterns. -**Examples** +## Examples -The following example parses a string that contains a timestamp, an IP address, -an email address, and a number: +The following examples show how to parse, type-convert, and handle multivalued output with `GROK`. + +### Parse a structured string + +Parse a string that contains a timestamp, an IP address, an email address, and a number: :::{include} ../examples/docs.csv-spec/basicGrok.md ::: +### Convert int or float types with :type suffix + By default, `GROK` outputs keyword string columns. `int` and `float` types can be converted by appending `:type` to the semantics in the pattern. For example `{NUMBER:num:int}`: @@ -45,11 +50,15 @@ be converted by appending `:type` to the semantics in the pattern. For example :::{include} ../examples/docs.csv-spec/grokWithConversionSuffix.md ::: +### Convert other types using type conversion functions + For other type conversions, use [Type conversion functions](/reference/query-languages/esql/functions-operators/type-conversion-functions.md): :::{include} ../examples/docs.csv-spec/grokWithToDatetime.md ::: +### Create a multivalued column with duplicate field names + If a field name is used more than once, `GROK` creates a multi-valued column: :::{include} ../examples/docs.csv-spec/grokWithDuplicateFieldNames.md diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/inlinestats-by.md b/docs/reference/query-languages/esql/_snippets/commands/layout/inlinestats-by.md index f980ca200b87e..df8f4d70b4a06 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/inlinestats-by.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/inlinestats-by.md @@ -9,7 +9,7 @@ are appended as new columns to the input rows. The command is identical to [`STATS`](/reference/query-languages/esql/commands/stats-by.md) except that it preserves all the columns from the input table. -**Syntax** +## Syntax ```esql INLINE STATS [column1 =] expression1 [WHERE boolean_expression1][, @@ -20,7 +20,7 @@ INLINE STATS [column1 =] expression1 [WHERE boolean_expression1][, [grouping_nameN = ] grouping_expressionN]] ``` -**Parameters** +## Parameters `columnX` : The name by which the aggregated value is returned. If omitted, the name is @@ -43,7 +43,7 @@ Individual `null` values are skipped when computing aggregations. :::: -**Description** +## Description The `INLINE STATS` processing command groups rows according to a common value (also known as the grouping key), specified after `BY`, and calculates one or more @@ -69,38 +69,42 @@ The following [grouping functions](/reference/query-languages/esql/functions-ope * [`TBUCKET`](/reference/query-languages/esql/functions-operators/grouping-functions/tbucket.md) -**Examples** +## Examples -The following example shows how to calculate a statistic on one column and group -by the values of another column. +The following examples show common `INLINE STATS` patterns. + +### Group by a column + +Calculate a statistic on one column and group by the values of another: + +:::{include} ../examples/inlinestats.csv-spec/max-salary.md +::: :::{note} The `languages` column moves to the last position in the output table because it is a column overridden by the `INLINE STATS` command (it's the grouping key) and it is the last column defined by it. ::: -:::{include} ../examples/inlinestats.csv-spec/max-salary.md -::: +### Aggregate without grouping -The following example shows how to calculate an aggregation over the entire dataset -by omitting `BY`. The order of the existing columns is preserved and a new column -with the calculated maximum salary value is added as the last column: +Omit `BY` to apply aggregations over the entire dataset. The order of the existing +columns is preserved and the new column is added last: :::{include} ../examples/inlinestats.csv-spec/max-salary-without-by.md ::: -The following example shows how to calculate multiple aggregations with multiple grouping keys: +### Use multiple aggregations and grouping keys :::{include} ../examples/inlinestats.csv-spec/multi-agg-multi-grouping.md ::: -The following example shows how to filter which rows are used for each aggregation, using the `WHERE` clause: +### Filter rows per aggregation with WHERE :::{include} ../examples/inlinestats.csv-spec/avg-salaries-where.md ::: -**Limitations** +## Limitations - The [`CATEGORIZE`](/reference/query-languages/esql/functions-operators/grouping-functions/categorize.md) grouping function is not currently supported. - You cannot currently use [`LIMIT`](/reference/query-languages/esql/commands/limit.md) (explicit or implicit) before `INLINE STATS`, because this can lead to unexpected results. diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/keep.md b/docs/reference/query-languages/esql/_snippets/commands/layout/keep.md index 2c074712e3151..5026b60b55840 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/keep.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/keep.md @@ -6,20 +6,20 @@ stack: ga The `KEEP` processing command enables you to specify what columns are returned and the order in which they are returned. -**Syntax** +## Syntax ```esql KEEP columns ``` -**Parameters** +## Parameters `columns` : A comma-separated list of columns to keep. Supports wildcards. See below for the behavior in case an existing column matches multiple given wildcards or column names. -**Description** +## Description The `KEEP` processing command enables you to specify what columns are returned and the order in which they are returned. @@ -33,42 +33,50 @@ Fields are added in the order they appear. If one field matches multiple express If a field matches two expressions with the same precedence, the rightmost expression wins. -Refer to the examples for illustrations of these precedence rules. +## Examples -**Examples** +The following examples show how to select columns and illustrate the wildcard precedence rules. -The columns are returned in the specified order: +### Specify column order + +Columns are returned in the order they are listed: :::{include} ../examples/docs.csv-spec/keep.md ::: +### Use wildcards to select columns + Rather than specify each column by name, you can use wildcards to return all columns with a name that matches a pattern: :::{include} ../examples/docs.csv-spec/keepWildcard.md ::: -The asterisk wildcard (`*`) by itself translates to all columns that do not -match the other arguments. +### Control ordering with wildcard and specific columns -This query will first return all columns with a name +The asterisk wildcard (`*`) by itself translates to all columns that do not +match the other arguments. This query returns all columns with a name that starts with `h`, followed by all other columns: :::{include} ../examples/docs.csv-spec/keepDoubleWildcard.md ::: -The following examples show how precedence rules work when a field name matches multiple expressions. +### Exact name takes precedence over wildcards Complete field name has precedence over wildcard expressions: :::{include} ../examples/docs.csv-spec/keepCompleteName.md ::: -Wildcard expressions have the same priority, but last one wins (despite being less specific): +### Rightmost wildcard wins when priority is equal + +Wildcard expressions have the same priority, but the rightmost one wins (despite being less specific): :::{include} ../examples/docs.csv-spec/keepWildcardPrecedence.md ::: +### Bare wildcard has lowest priority + A simple wildcard expression `*` has the lowest precedence. Output order is determined by the other arguments: diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/limit.md b/docs/reference/query-languages/esql/_snippets/commands/layout/limit.md index c0646dd782114..267eeea22c7d9 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/limit.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/limit.md @@ -3,28 +3,27 @@ serverless: ga stack: ga ``` -The `LIMIT` processing command enables you to limit the number of rows that are -returned. +The `LIMIT` processing command limits the number of rows returned. -**Syntax** +## Syntax ```esql LIMIT max_number_of_rows ``` -**Parameters** +## Parameters `max_number_of_rows` : The maximum number of rows to return. -**Description** +## Description + +Use the `LIMIT` processing command to limit the number of rows returned. -The `LIMIT` processing command enables you to limit the number of rows that are -returned. :::{include} ../../common/result-set-size-limitation.md ::: -**Example** +## Example :::{include} ../examples/limit.csv-spec/basic.md ::: diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/lookup-join.md b/docs/reference/query-languages/esql/_snippets/commands/layout/lookup-join.md index 45b784acda6f1..9552725b5a69d 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/lookup-join.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/lookup-join.md @@ -9,14 +9,14 @@ and analysis workflows. Refer to [the high-level landing page](../../../../esql/esql-lookup-join.md) for an overview of the `LOOKUP JOIN` command, including use cases, prerequisites, and current limitations. -**Syntax** +## Syntax ```esql FROM | LOOKUP JOIN ON ``` -**Parameters** +## Parameters `` : The name of the lookup index. This must be a specific index name or alias. Wildcards and remote cluster prefixes are not supported. If the query source includes remote indices, the lookup index must exist on all involved clusters. Indices used for lookups must be configured with the [`lookup` index mode](/reference/elasticsearch/index-settings/index-modules.md#index-mode-setting). @@ -30,7 +30,7 @@ FROM : If using join on a single field or a field list, the fields used must exist in both your current query results and in the lookup index. If the fields contains multi-valued entries, those entries will not match anything (the added fields will contain `null` for those rows). -**Description** +## Description The `LOOKUP JOIN` command adds new columns to your {{esql}} query results table by finding documents in a lookup index that share the same @@ -50,10 +50,13 @@ For important information about using `LOOKUP JOIN`, refer to [Usage notes](../. :::{include} ../types/lookup-join.md ::: -**Examples** +## Examples -**IP Threat correlation**: This query would allow you to see if any source -IPs match known malicious addresses. +The following examples show common `LOOKUP JOIN` use cases. + +### IP threat correlation + +Check whether source IPs match known malicious addresses: :::{include} ../examples/docs-lookup-join.csv-spec/lookupJoinSourceIp.md ::: @@ -63,26 +66,23 @@ To filter only for those rows that have a matching `threat_list` entry, use `WHE :::{include} ../examples/docs-lookup-join.csv-spec/lookupJoinSourceIpWhere.md ::: -**Host metadata correlation**: This query pulls in environment or -ownership details for each host to correlate with your metrics data. +### Host metadata correlation + +Pull in environment or ownership details for each host to correlate with metrics data: :::{include} ../examples/docs-lookup-join.csv-spec/lookupJoinHostNameTwice.md ::: -**Service ownership mapping**: This query would show logs with the owning -team or escalation information for faster triage and incident response. +### Service ownership mapping + +Show logs alongside the owning team or escalation information for faster triage: :::{include} ../examples/docs-lookup-join.csv-spec/lookupJoinServiceId.md ::: -`LOOKUP JOIN` is generally faster when there are fewer rows to join -with. {{esql}} will try and perform any `WHERE` clause before the -`LOOKUP JOIN` where possible. +### Filter before LOOKUP JOIN -The following two examples will have the same results. One has the -`WHERE` clause before and the other after the `LOOKUP JOIN`. It does not -matter how you write your query, our optimizer will move the filter -before the lookup when possible. +`LOOKUP JOIN` is generally faster when there are fewer rows to join with. {{esql}} will try to perform any `WHERE` clause before the `LOOKUP JOIN` where possible. The following two queries produce the same results. One filters before the join, the other after. The optimizer will push the filter before the lookup when possible: :::{include} ../examples/lookup-join.csv-spec/filterOnLeftSide.md ::: diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/mv_expand.md b/docs/reference/query-languages/esql/_snippets/commands/layout/mv_expand.md index ce615913ff322..f1a978e2f36cf 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/mv_expand.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/mv_expand.md @@ -6,13 +6,13 @@ stack: preview The `MV_EXPAND` processing command expands multivalued columns into one row per value, duplicating other columns. -**Syntax** +## Syntax ```esql MV_EXPAND column ``` -**Parameters** +## Parameters `column` : The multivalued column to expand. @@ -23,7 +23,7 @@ preceding `SORT`s. To guarantee a certain ordering, place a `SORT` after any `MV_EXPAND`s. :::: -**Example** +## Example :::{include} ../examples/mv_expand.csv-spec/simple.md ::: diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/rename.md b/docs/reference/query-languages/esql/_snippets/commands/layout/rename.md index 3b3648add7b9e..72a0932b9e9ed 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/rename.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/rename.md @@ -5,7 +5,7 @@ stack: ga The `RENAME` processing command renames one or more columns. -**Syntax** +## Syntax ```esql RENAME old_name1 AS new_name1[, ..., old_nameN AS new_nameN] @@ -21,7 +21,7 @@ RENAME new_name1 = old_name1[, ..., new_nameN = old_nameN] Both syntax options can be used interchangeably but we recommend sticking to one for consistency and readability. :::: -**Parameters** +## Parameters `old_nameX` : The name of a column you want to rename. @@ -31,24 +31,26 @@ Both syntax options can be used interchangeably but we recommend sticking to one the existing column is dropped. If multiple columns are renamed to the same name, all but the rightmost column with the same new name are dropped. -**Description** +## Description The `RENAME` processing command renames one or more columns. If a column with the new name already exists, it will be replaced by the new column. A `RENAME` with multiple column renames is equivalent to multiple sequential `RENAME` commands. -**Examples** +## Examples + +### Rename a single column :::{include} ../examples/docs.csv-spec/rename.md ::: -Multiple columns can be renamed with a single `RENAME` command: +### Rename multiple columns in one command :::{include} ../examples/docs.csv-spec/renameMultipleColumns.md ::: -With multiple `RENAME` commands: +The same result can be achieved with multiple `RENAME` commands: :::{include} ../examples/docs.csv-spec/renameMultipleColumnsDifferentCommands.md ::: diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/rerank.md b/docs/reference/query-languages/esql/_snippets/commands/layout/rerank.md index bfc16a9fb4524..3b4e77829d800 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/rerank.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/rerank.md @@ -52,13 +52,13 @@ FROM books :::: -**Syntax** +## Syntax ```esql RERANK [column =] query ON field [, field, ...] [WITH { "inference_id" : "my_inference_endpoint" }] ``` -**Parameters** +## Parameters `column` : (Optional) The name of the output column containing the reranked scores. @@ -80,7 +80,7 @@ the [inference endpoint](docs-content://explore-analyze/elastic-inference/infere to use for the task. The inference endpoint must be configured with the `rerank` task type. -**Description** +## Description The `RERANK` command uses an inference model to compute a new relevance score for an initial set of documents, directly within your ES|QL queries. @@ -95,14 +95,14 @@ When using `RERANK` with a multivalue column, each value is ranked individually. The score column is then assigned the maximum score resulting from ranking the individual values. -**Requirements** +## Requirements To use this command, you must deploy your reranking model in Elasticsearch as an [inference endpoint](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put) with the task type `rerank`. -#### Handling timeouts +### Handling timeouts `RERANK` commands may time out when processing large datasets or complex queries. The default timeout is 10 minutes, but you can increase this limit if @@ -144,27 +144,24 @@ If you don't want to increase the timeout limit, try the following: * Configure your HTTP client's response timeout (Refer to [HTTP client configuration](/reference/elasticsearch/configuration-reference/networking-settings.md#_http_client_configuration)) -**Examples** - -Rerank search results using a simple query and a single field: +## Examples +### Rerank with a single field :::{include} ../examples/rerank.csv-spec/simple-query.md ::: -Rerank search results using a query and multiple fields, and store the new score -in a column named `rerank_score`: +### Rerank with multiple fields and a custom score column :::{include} ../examples/rerank.csv-spec/two-queries.md ::: -Combine the original score with the reranked score: +### Combine original score with reranked score :::{include} ../examples/rerank.csv-spec/combine.md ::: -Rerank using snippets extracted from the document with the `TOP_SNIPPETS` -function: +### Rerank using document snippets :::{include} ../examples/rerank.csv-spec/rerank-top-snippets.md ::: diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/row.md b/docs/reference/query-languages/esql/_snippets/commands/layout/row.md index ce28abccfae90..fc10514bb6fae 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/row.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/row.md @@ -6,13 +6,13 @@ stack: ga The `ROW` source command produces a row with one or more columns with values that you specify. This can be useful for testing. -**Syntax** +## Syntax ```esql ROW column1 = value1[, ..., columnN = valueN] ``` -**Parameters** +## Parameters `columnX` : The column name. @@ -22,17 +22,23 @@ ROW column1 = value1[, ..., columnN = valueN] : The value for the column. Can be a literal, an expression, or a [function](/reference/query-languages/esql/esql-functions-operators.md#esql-functions). -**Examples** +## Examples + +The following examples demonstrate common `ROW` patterns. + +### Create a row with literal values :::{include} ../examples/row.csv-spec/example.md ::: -Use square brackets to create multi-value columns: +### Create multivalued columns + +Use square brackets to create a column with multiple values: :::{include} ../examples/row.csv-spec/multivalue.md ::: -`ROW` supports the use of [functions](/reference/query-languages/esql/esql-functions-operators.md#esql-functions): +### Use functions as values :::{include} ../examples/row.csv-spec/function.md ::: diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/sample.md b/docs/reference/query-languages/esql/_snippets/commands/layout/sample.md index f0ae9e0d37cff..ea26b19edd2f7 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/sample.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/sample.md @@ -5,18 +5,18 @@ stack: preview 9.1.0 The `SAMPLE` command samples a fraction of the table rows. -**Syntax** +## Syntax ```esql SAMPLE probability ``` -**Parameters** +## Parameters `probability` : The probability that a row is included in the sample. The value must be between 0 and 1, exclusive. -**Examples** +## Examples :::{include} ../examples/sample.csv-spec/sampleForDocs.md ::: diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/show.md b/docs/reference/query-languages/esql/_snippets/commands/layout/show.md index 0083282e68610..405d9d3221de2 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/show.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/show.md @@ -6,18 +6,18 @@ stack: ga The `SHOW` source command returns information about the deployment and its capabilities. -**Syntax** +## Syntax ```esql SHOW item ``` -**Parameters** +## Parameters `item` : Can only be `INFO`. -**Examples** +## Examples Use `SHOW INFO` to return the deployment’s version, build date and hash. diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/sort.md b/docs/reference/query-languages/esql/_snippets/commands/layout/sort.md index 63223a0dfa15b..a028f7f4047ad 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/sort.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/sort.md @@ -5,18 +5,18 @@ stack: ga The `SORT` processing command sorts a table on one or more columns. -**Syntax** +## Syntax ```esql SORT column1 [ASC/DESC][NULLS FIRST/NULLS LAST][, ..., columnN [ASC/DESC][NULLS FIRST/NULLS LAST]] ``` -**Parameters** +## Parameters `columnX` : The column to sort on. -**Description** +## Description The `SORT` processing command sorts a table on one or more columns. @@ -34,22 +34,26 @@ an ascending sort order, `null` values are sorted last, and with a descending sort order, `null` values are sorted first. You can change that by providing `NULLS FIRST` or `NULLS LAST`. -**Examples** +## Examples + +The following examples show how to control sort order, tie-breaking, and null placement. + +### Sort in default ascending order :::{include} ../examples/docs.csv-spec/sort.md ::: -Explicitly sorting in ascending order with `ASC`: +### Sort in descending order with DESC :::{include} ../examples/docs.csv-spec/sortDesc.md ::: -Providing additional sort expressions to act as tie breakers: +### Break ties with additional sort expressions :::{include} ../examples/docs.csv-spec/sortTie.md ::: -Sorting `null` values first using `NULLS FIRST`: +### Control null placement with NULLS FIRST or NULLS LAST :::{include} ../examples/docs.csv-spec/sortNullsFirst.md ::: diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/stats-by.md b/docs/reference/query-languages/esql/_snippets/commands/layout/stats-by.md index 102f56d8e886d..d33db58af2c91 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/stats-by.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/stats-by.md @@ -186,7 +186,7 @@ employee, then pass the result to `AVG`: :::{include} ../examples/stats.csv-spec/docsStatsAvgNestedExpression.md ::: -Grouping expressions aren't limited to column references — any expression +Grouping expressions aren't limited to column references: any expression works. For example, group by a derived value using `LEFT`: :::{include} ../examples/stats.csv-spec/docsStatsByExpression.md diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/ts.md b/docs/reference/query-languages/esql/_snippets/commands/layout/ts.md index 20f35b755ede2..127e1a6bc0e8e 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/ts.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/ts.md @@ -3,8 +3,6 @@ serverless: preview stack: preview 9.2.0 ``` -**Brief description** - The `TS` source command is similar to the [`FROM`](/reference/query-languages/esql/commands/from.md) source command, with the following key differences: @@ -12,13 +10,13 @@ source command, with the following key differences: - Enables the use of [time series aggregation functions](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md) inside the [STATS](/reference/query-languages/esql/commands/stats-by.md) command -**Syntax** +## Syntax ```esql TS index_pattern [METADATA fields] ``` -**Parameters** +## Parameters `index_pattern` : A list of indices, data streams or aliases. Supports wildcards and date math. @@ -26,16 +24,16 @@ TS index_pattern [METADATA fields] `fields` : A comma-separated list of [metadata fields](/reference/query-languages/esql/esql-metadata-fields.md) to retrieve. -**Description** +## Description The `TS` source command enables time series semantics and adds support for [time series aggregation functions](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md) to the `STATS` command, such as [`AVG_OVER_TIME()`](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions/avg_over_time.md), or [`RATE`](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions/rate.md). These functions are implicitly evaluated per time series, then aggregated by group using a secondary aggregation -function. See the Examples section for a query that calculates the total rate of search requests per host and hour. +function. For an example, refer to [Calculate the rate of search requests per host](#calculate-the-rate-of-search-requests-per-host). -This paradigm—a pair of aggregation functions—is standard for time series +This paradigm (a pair of aggregation functions) is standard for time series querying. For supported inner (time series) functions per [metric type](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md#time-series-metric), refer to [](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md). These functions also @@ -44,17 +42,17 @@ apply to downsampled data, with the same semantics as for raw data. ::::{note} If a query is missing an inner (time series) aggregation function, [`LAST_OVER_TIME()`](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions/last_over_time.md) -is assumed and used implicitly. For instance, two equivalent queries are shown in the Examples section that return the average of the last memory usage values per time series. To calculate the average memory usage across per-time-series averages, see the Examples section. +is assumed and used implicitly. For example, two equivalent queries that return the average of the last memory usage values per time series are shown in [Aggregate with implicit LAST_OVER_TIME](#aggregate-with-implicit-last_over_time). To calculate the average memory usage across per-time-series averages, refer to [Calculate the average of per-time-series averages](#calculate-the-average-of-per-time-series-averages). :::: You can use [time series aggregation functions](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md) -directly in the `STATS` command ({applies_to}`stack: preview 9.3`). The output will contain one aggregate value per time series and time bucket (if specified). See the Examples section for examples. +directly in the `STATS` command ({applies_to}`stack: preview 9.3`). The output will contain one aggregate value per time series and time bucket (if specified). For an example, refer to [Use time series aggregation functions directly](#use-time-series-aggregation-functions-directly). -You can also combine time series aggregation functions with regular [aggregation functions](/reference/query-languages/esql/functions-operators/aggregation-functions.md) such as `SUM()`, as outer aggregation functions. See the Examples section for examples. +You can also combine time series aggregation functions with regular [aggregation functions](/reference/query-languages/esql/functions-operators/aggregation-functions.md) such as `SUM()`, as outer aggregation functions. For examples, refer to [Combine SUM and RATE](#combine-sum-and-rate) and [Combine SUM and AVG_OVER_TIME](#combine-sum-and-avg_over_time). -However, using a time series aggregation function in combination with an inner time series function causes an error. See the Examples section for an invalid query example. +However, using a time series aggregation function in combination with an inner time series function causes an error. For an example, refer to [Invalid query: nested time series functions](#invalid-query-nested-time-series-functions). -**Best practices** +## Best practices - Avoid aggregating multiple metrics in the same query when those metrics have different dimensional cardinalities. For example, in `STATS max(rate(foo)) + rate(bar))`, if `foo` and `bar` don't share the same dimension values, the rate @@ -69,7 +67,11 @@ However, using a time series aggregation function in combination with an inner t applied, you can process the tabular output with any applicable ES|QL operations. - Add a time range filter on `@timestamp` to limit the data volume scanned and improve query performance. -**Examples** +## Examples + +The following examples demonstrate common time series query patterns using `TS`. + +### Calculate the rate of search requests per host Calculate the total rate of search requests (tracked by the `search_requests` counter) per host and hour. The `RATE()` function is applied per time series in hourly buckets. These rates are summed for each @@ -81,6 +83,8 @@ TS metrics | STATS SUM(RATE(search_requests)) BY TBUCKET(1 hour), host ``` +### Aggregate with implicit LAST_OVER_TIME + The following two queries are equivalent, returning the average of the last memory usage values per time series. If a query is missing an inner (time series) aggregation function, `LAST_OVER_TIME()` is assumed and used implicitly: ```esql @@ -89,13 +93,17 @@ TS metrics | STATS AVG(memory_usage) TS metrics | STATS AVG(LAST_OVER_TIME(memory_usage)) ``` -Calculate the average memory usage across per-time-series averages: +### Calculate the average of per-time-series averages + +This query calculates the average memory usage across per-time-series averages, rather than the average of all raw values: ```esql TS metrics | STATS AVG(AVG_OVER_TIME(memory_usage)) ``` -Using a time series aggregation function directly ({applies_to}`stack: preview 9.3`): +### Use time series aggregation functions directly + +You can use a [time series aggregation function](/reference/query-languages/esql/functions-operators/time-series-aggregation-functions.md) directly in `STATS` ({applies_to}`stack: preview 9.3`): ```esql TS metrics @@ -103,13 +111,17 @@ TS metrics | STATS RATE(search_requests) BY TBUCKET(1 hour) ``` -Combining a time series aggregation function with a regular aggregation function: +### Combine SUM and RATE + +Use `SUM` as the outer aggregation to sum counter rates across groups: ```esql TS metrics | STATS SUM(RATE(search_requests)) BY host ``` -Combining a time series aggregation function with a regular aggregation function: +### Combine SUM and AVG_OVER_TIME + +Use `AVG_OVER_TIME` to compute per-time-series averages, then group the results by host and time bucket: ```esql TS metrics @@ -117,7 +129,9 @@ TS metrics | STATS SUM(AVG_OVER_TIME(memory_usage)) BY host, TBUCKET(1 hour) ``` -The following query is invalid because using a time series aggregation function in combination with an inner time series function causes an error: +### Invalid query: nested time series functions + +Using a time series aggregation function in combination with an inner time series function causes an error: ```esql TS metrics | STATS AVG_OVER_TIME(RATE(memory_usage)) diff --git a/docs/reference/query-languages/esql/_snippets/commands/layout/where.md b/docs/reference/query-languages/esql/_snippets/commands/layout/where.md index 2c97bf79bf212..5718e42659ae2 100644 --- a/docs/reference/query-languages/esql/_snippets/commands/layout/where.md +++ b/docs/reference/query-languages/esql/_snippets/commands/layout/where.md @@ -14,30 +14,41 @@ there is no value at all. For example: `WHERE field != "value"` will be interpre :::: -**Syntax** +## Syntax ```esql WHERE expression ``` -**Parameters** +## Parameters `expression` : A boolean expression. -**Examples** +## Examples + +The following examples show common ways to filter rows with `WHERE`. + +### Filter rows by field value :::{include} ../examples/docs.csv-spec/where.md ::: + Which, if `still_hired` is a boolean field, can be simplified to: :::{include} ../examples/docs.csv-spec/whereBoolean.md ::: + +### Filter by time range + Use date math to retrieve data from a specific time range. For example, to retrieve the last hour of logs: :::{include} ../examples/date.csv-spec/docsNowWhere.md ::: + +### Use WHERE with a function + `WHERE` supports various [functions](/reference/query-languages/esql/esql-functions-operators.md#esql-functions). For example the [`LENGTH`](/reference/query-languages/esql/functions-operators/string-functions/length.md) function: @@ -58,12 +69,9 @@ For NULL comparison, use the `IS NULL` and `IS NOT NULL` predicates. ### Matching text -For matching text, you can use [full text search functions](/reference/query-languages/esql/functions-operators/search-functions.md) like `MATCH`. - Use [`MATCH`](/reference/query-languages/esql/functions-operators/search-functions/match.md) to perform a [match query](/reference/query-languages/query-dsl/query-dsl-match-query.md) on a specified field. - -Match can be used on text fields, as well as other field types like boolean, dates, and numeric types. +`MATCH` works on text fields as well as boolean, date, and numeric types. :::{include} ../../functions/examples/match.md ::: diff --git a/docs/reference/query-languages/esql/esql-commands.md b/docs/reference/query-languages/esql/esql-commands.md index d50de09122b72..a5f8223c7f840 100644 --- a/docs/reference/query-languages/esql/esql-commands.md +++ b/docs/reference/query-languages/esql/esql-commands.md @@ -9,7 +9,8 @@ mapped_pages: # {{esql}} commands [esql-commands] -{{esql}} commands come in two flavors: source commands and processing commands: +{{esql}} queries are built from the following building blocks: -- An {{esql}} query must start with a [source command](./commands/source-commands.md). +- Optionally, start with [query directives](./commands/directives.md) to define query settings and general behavior. +- Every query must include a [source command](./commands/source-commands.md). - Use [processing commands](./commands/processing-commands.md) to modify an input table by adding, removing, or transforming rows and columns. \ No newline at end of file diff --git a/docs/reference/query-languages/esql/esql-multi-index.md b/docs/reference/query-languages/esql/esql-multi-index.md index 3d09932aab543..587f8b69eb2cd 100644 --- a/docs/reference/query-languages/esql/esql-multi-index.md +++ b/docs/reference/query-languages/esql/esql-multi-index.md @@ -112,7 +112,7 @@ This functionality is in technical preview and may be changed or removed in a fu :::: -{{esql}} has a way to handle [field type mismatches](#esql-multi-index-invalid-mapping). When the same field is mapped to multiple types in multiple indices, the type of the field is understood to be a *union* of the various types in the index mappings. As seen in the preceding examples, this *union type* cannot be used in the results, and cannot be referred to by the query — except in `KEEP`, `DROP` or when it’s passed to a type conversion function that accepts all the types in the *union* and converts the field to a single type. {{esql}} offers a suite of [type conversion functions](functions-operators/type-conversion-functions.md) to achieve this. +{{esql}} has a way to handle [field type mismatches](#esql-multi-index-invalid-mapping). When the same field is mapped to multiple types in multiple indices, the type of the field is understood to be a *union* of the various types in the index mappings. As seen in the preceding examples, this *union type* cannot be used in the results, and cannot be referred to by the query, except in `KEEP`, `DROP`, or when it’s passed to a type conversion function that accepts all the types in the *union* and converts the field to a single type. {{esql}} offers a suite of [type conversion functions](functions-operators/type-conversion-functions.md) to achieve this. In the above examples, the query can use a command like `EVAL client_ip = TO_IP(client_ip)` to resolve the union of `ip` and `keyword` to just `ip`. You can also use the type-conversion syntax `EVAL client_ip = client_ip::IP`. Alternatively, the query could use [`TO_STRING`](functions-operators/type-conversion-functions/to_string.md) to convert all supported types into `KEYWORD`.