From 6559e27485bc9ac9db78f4d9e0c2d0dd8ea943b2 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 21 Dec 2023 12:41:54 -0700 Subject: [PATCH 01/11] Copy dev authored content to new branch and perform doc review Signed-off-by: Melissa Vagi --- _search-plugins/sql/sql/nested-function.md | 243 +++++++++++++++++++++ 1 file changed, 243 insertions(+) create mode 100644 _search-plugins/sql/sql/nested-function.md diff --git a/_search-plugins/sql/sql/nested-function.md b/_search-plugins/sql/sql/nested-function.md new file mode 100644 index 00000000000..ad1367d5d30 --- /dev/null +++ b/_search-plugins/sql/sql/nested-function.md @@ -0,0 +1,243 @@ +--- +layout: default +title: Nested function +parent: SQL +grand_parent: SQL and PPL +nav_order: 13 +has_toc: true +redirect_from: +- /search-plugins/sql/nested/ +--- + +# Nested function + +The `nested` function is used to created a query within a query. See the following sections for how and why to use the function in OpenSearch. + +## Using the function in a SELECT clause + +To extract individual elements from nested object type collections within the `SELECT` clause, use the `nested` function. This process flattens nested structures, generating a Cartesian product when querying against nested collections. + +### Syntax + +The `field_expression` parameter is required. The `path_expression` parameter is optional. Dot notation is used to show the nesting level for both the `field_expression` and `path_expression` parameters. For example, `nestedObj.innerFieldName` denotes a field nested one level. If the user does not provide the `path_expression` parameter, the path value is generated dynamically. For example, the field `user.office.cubicle` would dynamically generate the path `user.office`. The syntax is shown in the following example: + +```sql +nested(field_expression | field_expression, path_expression) +``` +{% include copy.html %} + +### Selecting all inner fields of an object + +In the `SELECT` clause, the `*` character in a nested function's `field_expression` parameter selects all fields under a nested object. For example, `nestedObj.*` retrieves all fields within `nestedObj`. + +### Flattening + +Flattening is a process in OpenSearch that transforms the response format by converting nested objects into key/value pairs. The full path of each object becomes the key, while the object itself becomes the value. The following are examples of an input, an output, a query, and a dataset. + +#### Example input + +```json +{ + "comment": { + "data": "abc" + } +} +``` +{% include copy-curl.html %} + +#### Example output + +```json +[ + { "comment.data": "abc" } +] +``` + +#### Example query + +The following example uses a nested query in the `SELECT` clause: + +```sql +SELECT nested(comment.data), nested(message.info) FROM nested_objects; +``` +{% include copy.html %} + + +#### Example dataset + +```json +{ + "comment": { + "data": "abc" + }, + "message": [ + { "info": "letter1" }, + { "info": "letter2" } + ] +} +``` +{% include copy-curl.html %} + +The results contain the following documents that match the nested query: + +```json +| nested(comment.data) | nested(message.info) +:----------------------|---- +abc | letter1 +abc | letter2 +``` + +## Using the function in a WHERE clause + +Nested object documents can be filtered in SQL using the `nested` function in the `WHERE` clause. + +### Syntax + +The `nested` function in a SQL `WHERE` clause offers two syntax options for filtering a nested field with a literal value, both achieving the same outcome. The two options are described in the following sections. + +### Boolean condition inside the nested function + +This option specifies the Boolean condition inside the nested function using the `condition_expression` parameter, as shown in the following example: + +```sql +nested(path_expression, condition_expression) +``` +{% include copy.html %} + +### Nested function with predicate expression and literal expression + +This option, which is supported by the V2 engine, places the `nested` function on the left of the predicate expression and on the right of the `literal_expression`. The `path_expression` is optional and automatically determined by the SQL plugin if not provided, as shown in the following example: + +```sql +nested(field_expression | field_expression, path_expression) Operator Literal_expression +``` +{% include copy.html %} + +#### Example query + +The following are examples of a query using nested queries in the `WHERE` clause, a sample dataset, and the query results: + +```sql +SELECT nested(message.info) FROM nested_objects WHERE nested(message.info) = 'letter2'; +SELECT nested(message.info) FROM nested_objects WHERE nested(message, message.info = 'letter2'); +``` +{% include copy.html %} + +#### Example dataset + +```json +{ + "message": { + "message": [ + { "info": "letter1" }, + { "info": "letter2" } + ] + } +} +``` +{% include copy-curl.html %} + +The results contain documents that match the nested query: + +```json +| nested(message.info) | +:----------------------| +letter2 | +``` + +## Using the function in an ORDER BY clause + +In the `ORDER BY` clause, the `nested` function enables sorting across documents based on their nested fields. + +### Syntax + +The `ORDER BY` clause with the `nested` function defaults to ascending order (`ASC`). Specifying `DESC` (desending) after the function allows overriding this behavior. The syntax is shown in the following example: + +```sql +nested(field_expression | field_expression, path_expression) +``` +{% include copy.html %} + +#### Example query + +The following are examples of a query using nested queries in the `ORDER BY` clause, a sample dataset, and the query results: + +```sql +SELECT nested(message.info) FROM nested_objects ORDER BY nested(message.info) DESC; +``` + +#### Example dataset + +```json +{ + "message": { + "message": [ + { "info": "letter1" }, + { "info": "letter2" } + ] + } +} +``` +{% include copy-curl.html %} + +The results contain the following documents that match the nested query: + +```json +| nested(message.info) | +:----------------------| +letter2 | +letter1 | +``` + +## Using the function in aggregation queries + +Nested fields can be aggregated in the `GROUP BY` clause and filtered in the `HAVING` clause using the `nested` function within SQL queries. + +### Syntax + +The syntax is shown in the following example: + +```sql +nested(field_expression | field_expression, path_expression) +``` +{% include copy.html %} + +#### Example query + +The following are examples of a query using nested queries in the `GROUP BY` and `HAVING` clauses: + +```sql +SELECT count(*) FROM nested_objects GROUP BY nested(message.info) HAVING count(*) > 1; +``` + +#### Example dataset + +```json +{ + { + "message": [ + {"info": "letter1"}, + {"info": "letter2"} + ] + }, + { + "message": [ + {"info": "letter1"}, + {"info": "letter3"} + ] + } +} +``` +{% include copy-curl.html %} + +The results contain the following documents that match the nested query: + +```json +| count(*) | +:----------| +| 2 | +``` + +## Limitations + +The `nested` function is supported in the V2 engine in the `SELECT` and `WHERE` clauses. See [Query processing engines](https://opensearch.org/docs/latest/search-plugins/sql/limitation/#query-processing-engines). \ No newline at end of file From b64d68d77d2097835a7b9a4e6aa3bb9d9fd3ec7a Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 10 Jan 2024 15:54:25 -0700 Subject: [PATCH 02/11] Update _search-plugins/sql/sql/nested-function.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _search-plugins/sql/sql/nested-function.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_search-plugins/sql/sql/nested-function.md b/_search-plugins/sql/sql/nested-function.md index ad1367d5d30..a54d465b5a4 100644 --- a/_search-plugins/sql/sql/nested-function.md +++ b/_search-plugins/sql/sql/nested-function.md @@ -11,7 +11,7 @@ redirect_from: # Nested function -The `nested` function is used to created a query within a query. See the following sections for how and why to use the function in OpenSearch. +The `nested` function is used to create a query within a query. The following sections describe how and why to use this function in OpenSearch. ## Using the function in a SELECT clause From bb1d5fcce38d65c361569705660521015a1c2a7b Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 10 Jan 2024 15:57:29 -0700 Subject: [PATCH 03/11] Update _search-plugins/sql/sql/nested-function.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _search-plugins/sql/sql/nested-function.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_search-plugins/sql/sql/nested-function.md b/_search-plugins/sql/sql/nested-function.md index a54d465b5a4..0ce17ee5daa 100644 --- a/_search-plugins/sql/sql/nested-function.md +++ b/_search-plugins/sql/sql/nested-function.md @@ -32,7 +32,7 @@ In the `SELECT` clause, the `*` character in a nested function's `field_expressi ### Flattening -Flattening is a process in OpenSearch that transforms the response format by converting nested objects into key/value pairs. The full path of each object becomes the key, while the object itself becomes the value. The following are examples of an input, an output, a query, and a dataset. +Flattening is a process in OpenSearch that transforms the response format by converting nested objects into key-value pairs. The full path of each object becomes the key, while the object itself becomes the value. The following are examples of an input, an output, a query, and a dataset. #### Example input From e70fe4ed218be90556a272f048067cf06b8b7e10 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 10 Jan 2024 15:57:50 -0700 Subject: [PATCH 04/11] Update _search-plugins/sql/sql/nested-function.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _search-plugins/sql/sql/nested-function.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_search-plugins/sql/sql/nested-function.md b/_search-plugins/sql/sql/nested-function.md index 0ce17ee5daa..094037407cb 100644 --- a/_search-plugins/sql/sql/nested-function.md +++ b/_search-plugins/sql/sql/nested-function.md @@ -151,7 +151,7 @@ In the `ORDER BY` clause, the `nested` function enables sorting across documents ### Syntax -The `ORDER BY` clause with the `nested` function defaults to ascending order (`ASC`). Specifying `DESC` (desending) after the function allows overriding this behavior. The syntax is shown in the following example: +The `ORDER BY` clause with the `nested` function defaults to ascending order (`ASC`). Specifying `DESC` (desending) after the function allows you to override this behavior. The syntax is shown in the following example: ```sql nested(field_expression | field_expression, path_expression) From f0bc43ef40cd2f626bd7d63b451da2c155a403f7 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 10 Jan 2024 15:58:37 -0700 Subject: [PATCH 05/11] Update _search-plugins/sql/sql/nested-function.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _search-plugins/sql/sql/nested-function.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_search-plugins/sql/sql/nested-function.md b/_search-plugins/sql/sql/nested-function.md index 094037407cb..dcac1b2a18d 100644 --- a/_search-plugins/sql/sql/nested-function.md +++ b/_search-plugins/sql/sql/nested-function.md @@ -115,7 +115,7 @@ nested(field_expression | field_expression, path_expression) Operator Literal_ex #### Example query -The following are examples of a query using nested queries in the `WHERE` clause, a sample dataset, and the query results: +The following are examples of a query using nested queries in the `WHERE` clause: ```sql SELECT nested(message.info) FROM nested_objects WHERE nested(message.info) = 'letter2'; From e6c8a8fbff7971df45b2e39771d128c15d138126 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 10 Jan 2024 15:58:49 -0700 Subject: [PATCH 06/11] Update _search-plugins/sql/sql/nested-function.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _search-plugins/sql/sql/nested-function.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_search-plugins/sql/sql/nested-function.md b/_search-plugins/sql/sql/nested-function.md index dcac1b2a18d..59ca48bb2f3 100644 --- a/_search-plugins/sql/sql/nested-function.md +++ b/_search-plugins/sql/sql/nested-function.md @@ -160,7 +160,7 @@ nested(field_expression | field_expression, path_expression) #### Example query -The following are examples of a query using nested queries in the `ORDER BY` clause, a sample dataset, and the query results: +The following are examples of a query using nested queries in the `ORDER BY` clause: ```sql SELECT nested(message.info) FROM nested_objects ORDER BY nested(message.info) DESC; From 6e2e9cdd69f052db246e242b07c67b320008b088 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 10 Jan 2024 18:04:17 -0700 Subject: [PATCH 07/11] Update nested-function.md Signed-off-by: Melissa Vagi --- _search-plugins/sql/sql/nested-function.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/_search-plugins/sql/sql/nested-function.md b/_search-plugins/sql/sql/nested-function.md index 59ca48bb2f3..2faac60e6ae 100644 --- a/_search-plugins/sql/sql/nested-function.md +++ b/_search-plugins/sql/sql/nested-function.md @@ -55,7 +55,7 @@ Flattening is a process in OpenSearch that transforms the response format by con #### Example query -The following example uses a nested query in the `SELECT` clause: +The following example query uses a nested query in the `SELECT` clause: ```sql SELECT nested(comment.data), nested(message.info) FROM nested_objects; @@ -115,7 +115,7 @@ nested(field_expression | field_expression, path_expression) Operator Literal_ex #### Example query -The following are examples of a query using nested queries in the `WHERE` clause: +The following example query uses nested queries in the `WHERE` clause: ```sql SELECT nested(message.info) FROM nested_objects WHERE nested(message.info) = 'letter2'; @@ -160,7 +160,7 @@ nested(field_expression | field_expression, path_expression) #### Example query -The following are examples of a query using nested queries in the `ORDER BY` clause: +The following example query uses nested queries in the `ORDER BY` clause: ```sql SELECT nested(message.info) FROM nested_objects ORDER BY nested(message.info) DESC; @@ -204,7 +204,7 @@ nested(field_expression | field_expression, path_expression) #### Example query -The following are examples of a query using nested queries in the `GROUP BY` and `HAVING` clauses: +The following example query uses nested queries in the `GROUP BY` and `HAVING` clauses: ```sql SELECT count(*) FROM nested_objects GROUP BY nested(message.info) HAVING count(*) > 1; @@ -240,4 +240,4 @@ The results contain the following documents that match the nested query: ## Limitations -The `nested` function is supported in the V2 engine in the `SELECT` and `WHERE` clauses. See [Query processing engines](https://opensearch.org/docs/latest/search-plugins/sql/limitation/#query-processing-engines). \ No newline at end of file +The `nested` function is supported in the V2 engine in the `SELECT` and `WHERE` clauses. See [Query processing engines](https://opensearch.org/docs/latest/search-plugins/sql/limitation/#query-processing-engines). From 4f5935686fe6b21285b1b9247a0f22fb7d497625 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 10 Jan 2024 18:05:04 -0700 Subject: [PATCH 08/11] Update nested-function.md Signed-off-by: Melissa Vagi --- _search-plugins/sql/sql/nested-function.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_search-plugins/sql/sql/nested-function.md b/_search-plugins/sql/sql/nested-function.md index 2faac60e6ae..3ba0ad9e49e 100644 --- a/_search-plugins/sql/sql/nested-function.md +++ b/_search-plugins/sql/sql/nested-function.md @@ -19,7 +19,7 @@ To extract individual elements from nested object type collections within the `S ### Syntax -The `field_expression` parameter is required. The `path_expression` parameter is optional. Dot notation is used to show the nesting level for both the `field_expression` and `path_expression` parameters. For example, `nestedObj.innerFieldName` denotes a field nested one level. If the user does not provide the `path_expression` parameter, the path value is generated dynamically. For example, the field `user.office.cubicle` would dynamically generate the path `user.office`. The syntax is shown in the following example: +The `field_expression` parameter is required. The `path_expression` parameter is optional. Dot notation is used to show the nesting level for both the `field_expression` and `path_expression` parameters. For example, `nestedObj.innerFieldName` denotes a field nested one level deep. If the user does not provide the `path_expression` parameter, the path value is generated dynamically. For example, the field `user.office.cubicle` would dynamically generate the path `user.office`. The syntax is shown in the following example: ```sql nested(field_expression | field_expression, path_expression) From f2557fdd95007a5e302cccd32c6797ed876556e5 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 10 Jan 2024 18:18:38 -0700 Subject: [PATCH 09/11] Update nested-function.md Signed-off-by: Melissa Vagi --- _search-plugins/sql/sql/nested-function.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_search-plugins/sql/sql/nested-function.md b/_search-plugins/sql/sql/nested-function.md index 3ba0ad9e49e..2086f09b664 100644 --- a/_search-plugins/sql/sql/nested-function.md +++ b/_search-plugins/sql/sql/nested-function.md @@ -13,7 +13,7 @@ redirect_from: The `nested` function is used to create a query within a query. The following sections describe how and why to use this function in OpenSearch. -## Using the function in a SELECT clause +## the function in a SELECT clause To extract individual elements from nested object type collections within the `SELECT` clause, use the `nested` function. This process flattens nested structures, generating a Cartesian product when querying against nested collections. @@ -89,7 +89,7 @@ abc | letter2 ## Using the function in a WHERE clause -Nested object documents can be filtered in SQL using the `nested` function in the `WHERE` clause. +Nested object documents can be filtered in SQL by using the `nested` function in the `WHERE` clause. ### Syntax @@ -191,7 +191,7 @@ letter1 | ## Using the function in aggregation queries -Nested fields can be aggregated in the `GROUP BY` clause and filtered in the `HAVING` clause using the `nested` function within SQL queries. +Nested fields can be aggregated in the `GROUP BY` clause and filtered in the `HAVING` clause by using the `nested` function within SQL queries. ### Syntax From 21cd27344d0556154d523a3566de3c1a51736df1 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 10 Jan 2024 18:20:25 -0700 Subject: [PATCH 10/11] Update nested-function.md Signed-off-by: Melissa Vagi --- _search-plugins/sql/sql/nested-function.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/_search-plugins/sql/sql/nested-function.md b/_search-plugins/sql/sql/nested-function.md index 2086f09b664..f473616d17e 100644 --- a/_search-plugins/sql/sql/nested-function.md +++ b/_search-plugins/sql/sql/nested-function.md @@ -62,7 +62,6 @@ SELECT nested(comment.data), nested(message.info) FROM nested_objects; ``` {% include copy.html %} - #### Example dataset ```json @@ -95,7 +94,7 @@ Nested object documents can be filtered in SQL by using the `nested` function in The `nested` function in a SQL `WHERE` clause offers two syntax options for filtering a nested field with a literal value, both achieving the same outcome. The two options are described in the following sections. -### Boolean condition inside the nested function +#### Boolean condition inside the nested function This option specifies the Boolean condition inside the nested function using the `condition_expression` parameter, as shown in the following example: @@ -104,7 +103,7 @@ nested(path_expression, condition_expression) ``` {% include copy.html %} -### Nested function with predicate expression and literal expression +#### Nested function with predicate expression and literal expression This option, which is supported by the V2 engine, places the `nested` function on the left of the predicate expression and on the right of the `literal_expression`. The `path_expression` is optional and automatically determined by the SQL plugin if not provided, as shown in the following example: From e35fd7674fba15fc180271b04e572ceab6846b3d Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 10 Jan 2024 18:37:02 -0700 Subject: [PATCH 11/11] Update nested-function.md Signed-off-by: Melissa Vagi --- _search-plugins/sql/sql/nested-function.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_search-plugins/sql/sql/nested-function.md b/_search-plugins/sql/sql/nested-function.md index f473616d17e..b8332bd8532 100644 --- a/_search-plugins/sql/sql/nested-function.md +++ b/_search-plugins/sql/sql/nested-function.md @@ -13,7 +13,7 @@ redirect_from: The `nested` function is used to create a query within a query. The following sections describe how and why to use this function in OpenSearch. -## the function in a SELECT clause +## Using the function in a SELECT clause To extract individual elements from nested object type collections within the `SELECT` clause, use the `nested` function. This process flattens nested structures, generating a Cartesian product when querying against nested collections.