You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _search-plugins/star-tree-index.md
+89-46Lines changed: 89 additions & 46 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,71 +7,117 @@ nav_order: 54
7
7
8
8
# Star-tree index
9
9
10
-
A star-tree index is a multi-field index that improves the performance of aggregations.
10
+
A _star-tree index_ is a multi-field index that improves the performance of aggregations by precomputing metric values for combinations of dimension fields.
11
11
12
-
OpenSearch will automatically use a star-tree index to optimize aggregations if the queried fields are part of dimension fields and the aggregations are on star-tree metric fields. No changes are required in the query syntax or the request parameters.
12
+
Once you enable star-tree indexes, OpenSearch automatically builds and uses star-tree indexes to optimize aggregations if the filter fields match the defined dimensions and the aggregation fields match the defined metrics in the star-tree configuration. No changes to your query syntax or request parameters are required.
13
13
14
-
## When to use a star-tree index
15
-
16
-
A star-tree index can be used to perform faster aggregations. Consider the following criteria and features when deciding to use a star-tree index:
14
+
Use a star-tree index when you want to speed up aggregations:
17
15
18
16
- Star-tree indexes natively support multi-field aggregations.
19
-
- Star-tree indexes are created in real time as part of the indexing process, so the data in a star-tree will always be up to date.
20
-
- A star-tree index consolidates data, increasing index paging efficiency and using less IO for search queries.
17
+
- Star-tree indexes are created in real time as part of the indexing process, so the data in a star-tree is always current.
18
+
- A star-tree index consolidates data to improve paging efficiency and reduce disk I/O during search queries.
21
19
22
-
## Limitations
20
+
## Star-tree index structure
23
21
24
-
Star-tree indexes have the following limitations:
22
+
A star-tree index organizes and aggregates data across combinations of dimension fields and precomputes metric values for those combinations during ingestion. This structure enables OpenSearch to process aggregation queries quickly without scanning every document.
25
23
26
-
- A star-tree index should only be enabled on indexes whose data is not updated or deleted because updates and deletions are not accounted for in a star-tree index. To enforce this policy and use star-tree indexes, set the `index.append_only.enabled` setting to `true`.
27
-
- A star-tree index can be used for aggregation queries only if the queried fields are a subset of the star-tree's dimensions and the aggregated fields are a subset of the star-tree's metrics.
28
-
- After a star-tree index is enabled, it cannot be disabled. In order to disable a star-tree index, the data in the index must be reindexed without the star-tree mapping. Furthermore, changing a star-tree configuration will also require a reindex operation.
29
-
-[Multi-values/array values]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/index/#arrays) are not supported.
30
-
- Only [limited queries and aggregations](#supported-queries-and-aggregations) are supported. Support for more features will be added in future versions.
31
-
- The cardinality of the dimensions should not be very high (as with `_id` fields). Higher cardinality leads to increased storage usage and query latency.
24
+
The following is an example star-tree configuration:
32
25
33
-
## Star-tree index structure
34
-
35
-
The following image illustrates a standard star-tree index structure.
26
+
```json
27
+
"ordered_dimensions": [
28
+
{
29
+
"name": "status"
30
+
},
31
+
{
32
+
"name": "port"
33
+
}
34
+
],
35
+
"metrics": [
36
+
{
37
+
"name": "size",
38
+
"stats": [
39
+
"sum"
40
+
]
41
+
},
42
+
{
43
+
"name": "latency",
44
+
"stats": [
45
+
"avg"
46
+
]
47
+
}
48
+
]
49
+
```
36
50
37
-
<imgsrc="{{site.url}}{{site.baseurl}}/images/star-tree-index.png"alt="A star-tree index containing two dimensions and two metrics"width="700">
51
+
This configuration defines the following:
38
52
39
-
Sorted and aggregated star-tree documents are backed by `doc_values` in an index. The columnar data found in `doc_values` is stored using the following properties:
53
+
* Two dimension fields: `status` and `port`. The `ordered_dimension` field specifies how data is sorted (first, by `status`, then by `port`).
54
+
* Two metric fields: `size` and `latency` with their corresponding aggregations (`sum` and `avg`). For each unique dimension combination, metric values (`sum(size)` and `avg(latency)`) are pre-aggregated and stored in the star-tree structure.
40
55
41
-
- The values are sorted based on the fields set in the `ordered_dimension` setting. In the preceding image, the dimensions are determined by the `status` setting and then by the `port` for each status.
42
-
- For each unique dimension/value combination, the aggregated values for all the metrics, such as `avg(size)` and `count(requests)`, are precomputed during ingestion.
56
+
Based on this configuration, OpenSearch creates a star-tree index structure. Each node in the tree corresponds to a value (or wildcard `*`) for a dimension. At query time, OpenSearch traverses the tree based on which dimension values are provided in the query.
43
57
44
58
### Leaf nodes
45
59
46
-
Each node in a star-tree index points to a range of star-tree documents. Nodes can be further split into child nodes based on the [max_leaf_docs configuration]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/star-tree/#star-tree-index-configuration-options). The number of documents that a leaf node points to is less than or equal to the value set in `max_leaf_docs`. This ensures that the maximum number of documents that need to traverse nodes to derive an aggregated value is at most the number of `max_leaf_docs`, which provides predictable latency.
60
+
Leaf nodes contain the precomputed metric aggregations for specific combinations of dimensions. These are stored as doc values and referenced by star-tree nodes.
61
+
62
+
The `max_leaf_docs` setting controls how many documents each leaf node can reference, which helps keep query latency predictable by limiting how many documents are scanned for any given node.
47
63
48
64
### Star nodes
49
65
50
-
A star node contains the aggregated data of all the other nodes for a particular dimension, acting as a "catch-all" node. When a star node is found in a dimension, that dimension is skipped during aggregation. This groups together all values of that dimension and allows a query to skip non-competitive nodes when fetching the aggregated value of a particular field.
66
+
A _star node_ (marked as `*` in the following diagram) aggregates all values for a particular dimension. It allows OpenSearch to process partial-dimension queries more efficiently by skipping over non-filtered dimensions.
51
67
52
-
The star-tree index structure diagram contains the following three examples demonstrating how a query behaves when retrieving aggregations from nodes in the star-tree:
68
+
For example, if a query filters on `port` but not `status`, OpenSearch can use a star node that aggregates data for all status values.
53
69
54
-
-**Blue**: In a `terms` query that searches for the average request size aggregation, the `port` equals `8443` and the status equals `200`. Because the query contains values in both the `status` and `port` dimensions, the query traverses status node `200` and returns the aggregations from child node `8443`.
55
-
-**Green**: In a `term` query that searches for the number of aggregation requests, the `status` equals `200`. Because the query only contains a value from the `status` dimension, the query traverses the `200` node's child star node, which contains the aggregated value of all the `port` child nodes.
56
-
-**Red**: In a `term` query that searches for the average request size aggregation, the port equals `5600`. Because the query does not contain a value from the `status` dimension, the query traverses a star node and returns the aggregated result from the `5600` child node.
70
+
### How queries use the star-tree
57
71
58
-
Support for the `Terms` query will be added in a future version. For more information, see [GitHub issue #15257](https://github.com/opensearch-project/OpenSearch/issues/15257).
59
-
{: .note}
72
+
The following diagram shows a star-tree index created for this example and three example query paths.
73
+
74
+
<imgsrc="{{site.url}}{{site.baseurl}}/images/star-tree-index.png"alt="A star-tree index containing two dimensions and two metrics">
75
+
76
+
The colored arrows show three query examples:
77
+
78
+
***Blue arrow**: Multi-term query with metric aggregation
79
+
The query filters on both `status = 200` and `port = 5600` and calculates the sum of request sizes.
80
+
81
+
* OpenSearch follows the path: `Root → 200 → 5600`
82
+
* It retrieves the metric from Doc ID 1, where `Sum(size) = 988`
83
+
84
+
***Green arrow**: Single-term query with metric aggregation
85
+
The query filters on `status = 200` only and computes the average request latency.
86
+
87
+
* OpenSearch follows the path: `Root → 200 → *`
88
+
* It retrieves the metric from Doc ID 5, where `Avg(latency) = 70`
89
+
90
+
***Red arrow**: Single-term query with metric aggregation
91
+
The query filters on `port = 8443` only and calculates the sum of request sizes.
92
+
93
+
* OpenSearch follows the path: `Root → * → 8443`
94
+
* It retrieves the metric from Doc ID 7, where `Sum(size) = 1111`
95
+
96
+
These examples show how OpenSearch selects the shortest path in the star-tree and uses pre-aggregated values to process queries efficiently.
97
+
98
+
## Limitations
99
+
100
+
Star-tree indexes have the following limitations:
101
+
102
+
- Enable a star-tree index only for indexes whose data is not updated or deleted: star-tree indexes do not support updates or deletions. To enforce this policy, set `index.append_only.enabled` to `true` for the index you want to enable star-tree index on.
103
+
- Use a star-tree index for aggregation queries only if the queried fields are a subset of the star-tree's dimensions and the aggregated fields are a subset of the star-tree's metrics.
104
+
- Once enabled, a star-tree index cannot be disabled. In order to disable a star-tree index, the data in the index must be reindexed without the star-tree mapping. Changing a star-tree configuration also requires reindexing data.
105
+
-[Array values]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/index/#arrays) are not supported.
106
+
- Only [specific queries and aggregations](#supported-queries-and-aggregations) are supported.
107
+
- Avoid using high-cardinality fields like `_id` as dimensions, as they can significantly increase storage usage and query latency.
60
108
61
109
## Enabling a star-tree index
62
110
63
111
To use a star-tree index, modify the following settings:
64
112
65
-
- Set the feature flag `opensearch.experimental.feature.composite_index.star_tree.enabled` to `true`. For more information about enabling and disabling feature flags, see [Enabling experimental features]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/experimental/).
66
-
- Set the `indices.composite_index.star_tree.enabled` setting to `true`. For instructions on how to configure OpenSearch, see [Configuring settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index/#static-settings).
113
+
- Set the `indices.composite_index.star_tree.enabled` setting to `true`. For more information, see [Dynamic settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index/#dynamic-settings).
67
114
- Set the `index.composite_index` index setting to `true` during index creation.
68
115
- Set the `index.append_only.enabled` index setting to `true` during index creation.
69
-
- Ensure that the `doc_values` parameter is enabled for the `dimensions` and `metrics` fields used in your star-tree mapping.
70
-
116
+
- Ensure that the [`doc_values`]({{site.url}}{{site.baseurl}}/field-types/mapping-parameters/doc-values/) parameter is enabled for the dimension and metric fields used in your star-tree mapping.
71
117
72
118
## Example mapping
73
119
74
-
In the following example, index mappings define the star-tree configuration. The star-tree index precomputes aggregations in the `logs` index. The aggregations are calculated on the `size` and `latency` fields for all the combinations of values indexed in the `port` and `status` fields:
120
+
The following example shows how to create a star-tree index that precomputes aggregations in the `logs` index. The `sum` and `average`aggregations are calculated on the `size` and `latency` fields , respectively, for all combinations of values in the dimension fields. The dimensions are ordered by `status`, then `port`, and finally `method`, which determines how the data is organized in the tree structure:
75
121
76
122
```json
77
123
PUT logs
@@ -145,23 +191,24 @@ PUT logs
145
191
```
146
192
{% include copy.html %}
147
193
148
-
For detailed information about star-tree index mappings and parameters, see [Star-tree field type]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/star-tree/).
194
+
For more information about star-tree index mappings and parameters, see [Star-tree field type]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/star-tree/).
149
195
150
196
## Supported queries and aggregations
151
197
152
-
Star-tree indexes can be used to optimize queries and aggregations.
198
+
Star-tree indexes optimize aggregations. Every query must include at least one supported aggregation to use the star-tree optimization.
153
199
154
200
### Supported queries
155
201
156
-
The following queries are supported as of OpenSearch 2.19:
202
+
Queries without aggregations cannot use star-tree optimization. The query's fields must be present in the `ordered_dimensions` section of the star-tree configuration. The following queries are supported:
To use a query with a star-tree index, the query's fields must be present in the `ordered_dimensions` section of the star-tree configuration. Queries must also be paired with a supported aggregation. Queries without aggregations cannot be used with a star-tree index. Currently, queries on `date` fields are not supported and will be added in later versions.
210
+
Queries on `date` fields are not supported.
211
+
{: .note}
165
212
166
213
#### Boolean query restrictions
167
214
@@ -237,7 +284,7 @@ The following aggregations are supported by star-tree indexes.
237
284
238
285
#### Metric aggregations
239
286
240
-
The following metric aggregations are supported as of OpenSearch 2.18:
Bucket aggregations supported with star-tree index search can be nested and resolved using star-tree. [See more]({{site.url}}{{site.baseurl}}/aggregations/#nested-aggregations).
431
-
432
-
## Using queries without a star-tree index
475
+
#### Nested aggregations
433
476
434
-
Set the `indices.composite_index.star_tree.enabled` setting to `false` to run queries without using a star-tree index.
477
+
You can combine multiple supported bucket aggregations (such as terms and range) in a nested structure, and the star-tree index will optimize these nested aggregations. For more information about nested aggregations, see [Nested aggregations]({{site.url}}{{site.baseurl}}/aggregations/#nested-aggregations).
0 commit comments