Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions _posts/2022-10-05-one-million-enitities-in-one-minute.markdown
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ William Thomson, co-formulator of Thermodynamics

We continually strive to improve the existing OpenSearch features through harnessing the capabilities of OpenSearch itself. One such feature is the [Anomaly Detection (AD) plugin](https://opensearch.org/docs/latest/monitoring-plugins/ad/index/), which automatically detects anomalies in your OpenSearch data.

Because OpenSearch is used to index high volumes of data in a distributed fashion, we knew it was essential to design the AD feature to have minimal impact on application workloads. OpenSearch 1.0.1 did not scale beyond 360K entities. Since OpenSearch 1.2.4, it has been possible to track 1 million entities with a data arrival rate of 10 minutes using [36 data nodes](https://aws.amazon.com/blogs/big-data/detect-anomalies-on-one-million-unique-entities-with-amazon-opensearch-service/).
Because OpenSearch is used to index high volumes of data in a distributed fashion, we knew it was essential to design the AD feature to have minimal impact on application workloads. OpenSearch 1.0.1 did not scale beyond 360K entities. Since OpenSearch 1.2.4, it has been possible to track one million entities with a data arrival rate of 10 minutes using [36 data nodes](https://aws.amazon.com/blogs/big-data/detect-anomalies-on-one-million-unique-entities-with-amazon-opensearch-service/).

While the increase to one million entities was great, most monitoring solutions generate data at a far higher rate. If you want to react quickly to emergent scenarios within your cluster, that 10-minute interval is insufficient. In order for AD to be truly useful, our goal was simple: **Shorten the interval to one minute for one million entities**, without changing the model output or increasing the number of nodes.

Expand Down Expand Up @@ -89,7 +89,7 @@ request_body = {

When set up with the same category and sort order, the CPU spikes when using OpenSearch 2.0 were below 25%.

![CPU spikes after category adjustment]({{ site.baseurl }}/assets/media/blog-images/2022-09-30-one-in-one/cpu-spikes.png){: .img-fluid}
![CPU spikes after category adjustment]({{ site.baseurl }}/assets/media/blog-images/2022-09-30-one-in-one/cpu-spikes-below-25.png){: .img-fluid}

Finally, we achieved continuous anomaly detection of one million entities at a one-minute interval.

Expand All @@ -101,9 +101,9 @@ It was serendipitous that the measurements taken from improving the plugin helpe

Interestingly, our explorations of the 9 r6g.8xlarge Gravitron instances, each with a 128 GB (50 percent) heap, resulted in the measurements seen in the following readings. Notice the lower CPU spikes when compared with our measurements of the same instances from OpenSearch 2.0.

![Comparison of CPU spikes in Gravitron nodes vs OpenSearch 2.0]({{ site.baseurl }}/assets/media/blog-images/2022-09-30-one-in-one/jvm-measurements.png){: .img-fluid}
![Comparison of CPU spikes in Gravitron nodes vs OpenSearch 2.0]({{ site.baseurl }}/assets/media/blog-images/2022-09-30-one-in-one/cpu-compare.png){: .img-fluid}

![Memory pressure in Gravitron nodes]({{ site.baseurl }}/assets/media/blog-images/2022-09-30-one-in-one/jvm-measurements.png){: .img-fluid}
![Memory pressure in Gravitron nodes]({{ site.baseurl }}/assets/media/blog-images/2022-09-30-one-in-one/cpu-memory-pressure.png){: .img-fluid}

## See it for yourself

Expand All @@ -125,7 +125,7 @@ PUT /_cluster/settings
}
```

## But what if I don't one million entities
## But what if I don't use one million entities

From OpenSearch 1.2.4 to OpenSearch 2.2 or greater, many incremental improvements were made to AD, in particular to [historical analysis](https://opensearch.org/blog/technical-post/2021/11/real-time-and-historical-ad/) and other downstream [log analytics tasks](https://aws.amazon.com/blogs/security/analyze-aws-waf-logs-using-amazon-opensearch-service-anomaly-detection-built-on-random-cut-forests/). However, “cold start,” the gap between loading data and seeing results, has been a known challenge in AD since the beginning. Despite this challenge, the cold start gap has decreased from release to release as the AD model has improved.

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 4 additions & 2 deletions releases.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ title: Release Schedule and Maintenance Policy
---


_Updated August 30, 2022_
_Updated October 6, 2022_

[Release Schedule](#release-schedule) · [Maintenance Policy](#maintenance-policy)

Expand All @@ -27,7 +27,7 @@ For more information on the changes planned for each release, please see the [Pr
| 2.2.1 | August 19, 2022 | September 1, 2022 |
| 2.3 | September 7, 2022 | September 14, 2022 |
| 1.3.6 | September 30, 2022 | October 6, 2022 |
| 2.4 | November 4, 2022 | November 10, 2022 |
| 2.4 | November 3, 2022 | November 10, 2022 |
| 1.3.7 | December 1, 2022 | December 8, 2022 |
| 2.5 | January 10, 2023 | January 12, 2023 |

Expand All @@ -48,6 +48,8 @@ Sometimes an incompatible change is unavoidable. When this happens, the software

The duration of the maintenance window will vary from product to product and release to release. **By default, versions will remain under maintenance until the next major version enters maintenance, or 1 year passes, whichever is longer.** Therefore, at any given time, the current major version and previous major version will both be supported, as well as older major versions that have been in maintenance for less than 12 months. Please note that, maintenance windows are influenced by the support schedules for dependencies the software includes, community input, the scope of the changes introduced by the new version, and estimates for the effort required to continue maintenance of the previous version.

The software maintainers will not back-port fixes or features to versions outside of the maintenance window. That said, PRs with said back-ports are welcome and will follow the project’s [review process](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#review-process). No new releases will result from these changes, but interested parties can [create their own distribution](https://github.com/opensearch-project/opensearch-build#building-and-testing-an-opensearch-distribution) from the updated source after the PRs are merged.

<div class="table-styler"></div>

| Major Version | Latest Minor Version | Status | Initial Release | Maintenance Window Start | Maintenance Window End |
Expand Down