From 23cdaad13dd2aaefe5c13592db5cbc99a1b2059d Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Mon, 6 Feb 2023 11:48:25 -0600 Subject: [PATCH] Once more... Managing, Monitoring, Tuning. (#2653) Signed-off-by: Naarcha-AWS (cherry picked from commit 45834d6f78697fe291189c583f7d4d689faeb075) --- _api-reference/cluster-api/cluster-health.md | 1 + _api-reference/cluster-api/cluster-stats.md | 2 + _api-reference/index.md | 1 + _config.yml | 74 +++- _data-prepper/peer-forwarder.md | 181 ++++++++ .../job-scheduler/index.md | 132 ++++++ .../pa/api.md | 2 + .../pa/dashboards.md | 2 + .../pa/index.md | 1 + .../pa/rca/api.md | 2 + .../pa/rca/index.md | 2 + .../pa/rca/reference.md | 2 + .../pa/reference.md | 2 + .../ad/api.md | 2 + .../ad/index.md | 3 +- .../ad/result-mapping.md | 2 + .../ad/security.md | 2 + .../ad/settings.md | 2 + .../alerting/api.md | 2 + .../alerting/cron.md | 2 + .../alerting/index.md | 3 +- .../alerting/monitors.md | 2 + .../alerting/security.md | 2 + .../alerting/settings.md | 2 + .../app-analytics.md | 6 +- .../event-analytics.md | 6 +- .../index.md | 16 +- _observing-your-data/log-analytics.md | 0 .../log-ingestion.md | 8 +- .../notebooks.md | 4 +- .../notifications}/api.md | 4 +- .../notifications}/index.md | 3 +- .../observability-security.md | 2 + .../operational-panels.md | 6 +- .../trace/getting-started.md | 10 +- .../trace/index.md | 4 +- .../trace/ta-dashboards.md | 2 + _opensearch/snapshots/sm-dashboards.md | 97 ----- _search-plugins/sql/ppl/functions.md | 2 +- _search-plugins/sql/ppl/syntax.md | 2 +- _security/access-control/permissions.md | 2 + _security/configuration/index.md | 3 +- _security/index.md | 2 + .../availability-and-recovery/index.md | 9 + .../segment-replication/configuration.md | 84 ++++ .../segment-replication/index.md | 27 ++ .../shard-indexing-backpressure.md | 3 + .../shard-indexing-settings.md | 6 +- .../snapshots/index.md | 9 +- .../snapshots/sm-api.md | 7 +- .../snapshots/snapshot-management.md | 13 +- .../snapshots/snapshot-restore.md | 3 + .../availability-and-recovery}/stats-api.md | 3 + .../cluster-manager-task-throttling.md | 107 +++++ .../cluster.md | 8 +- .../replication-plugin/api.md | 394 ++++++++++++++++++ .../replication-plugin/auto-follow.md | 106 +++++ .../replication-plugin/getting-started.md | 295 +++++++++++++ .../replication-plugin/index.md | 24 ++ .../replication-plugin/permissions.md | 81 ++++ .../replication-plugin/settings.md | 39 ++ about.md | 2 +- 62 files changed, 1666 insertions(+), 161 deletions(-) create mode 100644 _data-prepper/peer-forwarder.md create mode 100644 _monitoring-your-cluster/job-scheduler/index.md rename {_monitoring-plugins => _monitoring-your-cluster}/pa/api.md (99%) rename {_monitoring-plugins => _monitoring-your-cluster}/pa/dashboards.md (99%) rename {_monitoring-plugins => _monitoring-your-cluster}/pa/index.md (99%) rename {_monitoring-plugins => _monitoring-your-cluster}/pa/rca/api.md (96%) rename {_monitoring-plugins => _monitoring-your-cluster}/pa/rca/index.md (95%) rename {_monitoring-plugins => _monitoring-your-cluster}/pa/rca/reference.md (83%) rename {_monitoring-plugins => _monitoring-your-cluster}/pa/reference.md (99%) rename {_monitoring-plugins => _observing-your-data}/ad/api.md (99%) rename {_monitoring-plugins => _observing-your-data}/ad/index.md (99%) rename {_monitoring-plugins => _observing-your-data}/ad/result-mapping.md (99%) rename {_monitoring-plugins => _observing-your-data}/ad/security.md (98%) rename {_monitoring-plugins => _observing-your-data}/ad/settings.md (99%) rename {_monitoring-plugins => _observing-your-data}/alerting/api.md (99%) rename {_monitoring-plugins => _observing-your-data}/alerting/cron.md (97%) rename {_monitoring-plugins => _observing-your-data}/alerting/index.md (95%) rename {_monitoring-plugins => _observing-your-data}/alerting/monitors.md (99%) rename {_monitoring-plugins => _observing-your-data}/alerting/security.md (99%) rename {_monitoring-plugins => _observing-your-data}/alerting/settings.md (98%) rename {_observability-plugin => _observing-your-data}/app-analytics.md (95%) rename {_observability-plugin => _observing-your-data}/event-analytics.md (95%) rename {_observability-plugin => _observing-your-data}/index.md (58%) create mode 100644 _observing-your-data/log-analytics.md rename _observability-plugin/log-analytics.md => _observing-your-data/log-ingestion.md (96%) rename {_observability-plugin => _observing-your-data}/notebooks.md (98%) rename {_notifications-plugin => _observing-your-data/notifications}/api.md (99%) rename {_notifications-plugin => _observing-your-data/notifications}/index.md (99%) rename {_observability-plugin => _observing-your-data}/observability-security.md (97%) rename {_observability-plugin => _observing-your-data}/operational-panels.md (87%) rename _observability-plugin/trace/get-started.md => _observing-your-data/trace/getting-started.md (93%) rename {_observability-plugin => _observing-your-data}/trace/index.md (91%) rename {_observability-plugin => _observing-your-data}/trace/ta-dashboards.md (95%) delete mode 100644 _opensearch/snapshots/sm-dashboards.md create mode 100644 _tuning-your-cluster/availability-and-recovery/index.md create mode 100644 _tuning-your-cluster/availability-and-recovery/segment-replication/configuration.md create mode 100644 _tuning-your-cluster/availability-and-recovery/segment-replication/index.md rename {_opensearch => _tuning-your-cluster/availability-and-recovery}/shard-indexing-backpressure.md (94%) rename {_opensearch => _tuning-your-cluster/availability-and-recovery}/shard-indexing-settings.md (97%) rename {_opensearch => _tuning-your-cluster/availability-and-recovery}/snapshots/index.md (75%) rename {_opensearch => _tuning-your-cluster/availability-and-recovery}/snapshots/sm-api.md (98%) rename {_opensearch => _tuning-your-cluster/availability-and-recovery}/snapshots/snapshot-management.md (94%) rename {_opensearch => _tuning-your-cluster/availability-and-recovery}/snapshots/snapshot-restore.md (99%) rename {_opensearch => _tuning-your-cluster/availability-and-recovery}/stats-api.md (99%) create mode 100644 _tuning-your-cluster/cluster-manager-task-throttling.md rename {_opensearch => _tuning-your-cluster}/cluster.md (99%) create mode 100644 _tuning-your-cluster/replication-plugin/api.md create mode 100644 _tuning-your-cluster/replication-plugin/auto-follow.md create mode 100644 _tuning-your-cluster/replication-plugin/getting-started.md create mode 100644 _tuning-your-cluster/replication-plugin/index.md create mode 100644 _tuning-your-cluster/replication-plugin/permissions.md create mode 100644 _tuning-your-cluster/replication-plugin/settings.md diff --git a/_api-reference/cluster-api/cluster-health.md b/_api-reference/cluster-api/cluster-health.md index 31a90bf4819..405194c70db 100644 --- a/_api-reference/cluster-api/cluster-health.md +++ b/_api-reference/cluster-api/cluster-health.md @@ -4,6 +4,7 @@ title: Cluster health nav_order: 40 parent: Cluster APIs has_children: false +redirect_from: /api-reference/cluster-health/ --- # Cluster health diff --git a/_api-reference/cluster-api/cluster-stats.md b/_api-reference/cluster-api/cluster-stats.md index 8097c27cd6a..9e56b178078 100644 --- a/_api-reference/cluster-api/cluster-stats.md +++ b/_api-reference/cluster-api/cluster-stats.md @@ -4,6 +4,8 @@ title: Cluster stats nav_order: 60 parent: Cluster APIs has_children: false +redirect_from: + - /api-reference/cluster-stats/ --- # Cluster stats diff --git a/_api-reference/index.md b/_api-reference/index.md index de6f13ae0d9..8b6038778f8 100644 --- a/_api-reference/index.md +++ b/_api-reference/index.md @@ -2,6 +2,7 @@ layout: default title: REST API reference nav_order: 1 +has_toc: true redirect_from: - /opensearch/rest-api/ --- diff --git a/_config.yml b/_config.yml index 55b5bd690aa..d7043fd75e5 100644 --- a/_config.yml +++ b/_config.yml @@ -5,10 +5,17 @@ baseurl: "/docs/latest" # the subpath of your site, e.g. /blog url: "https://opensearch.org" # the base hostname & protocol for your site, e.g. http://example.com permalink: /:path/ +<<<<<<< HEAD opensearch_version: 2.2.1 opensearch_dashboards_version: 2.2.1 opensearch_major_minor_version: 2.2 lucene_version: 9_3_0 +======= +opensearch_version: 2.5.0 +opensearch_dashboards_version: 2.5.0 +opensearch_major_minor_version: 2.5 +lucene_version: 9_4_1 +>>>>>>> 45834d6f (Once more... Managing, Monitoring, Tuning. (#2653)) # Build settings markdown: kramdown @@ -37,27 +44,46 @@ collections: opensearch: permalink: /:collection/:path/ output: true + im-plugin: + permalink: /:collection/:path/ + output: true dashboards: permalink: /:collection/:path/ output: true + tuning-your-cluster: + permalink: /:collection/:path/ + output: true security: permalink: /:collection/:path/ +<<<<<<< HEAD +======= + output: true + security-analytics: + permalink: /:collection/:path/ +>>>>>>> 45834d6f (Once more... Managing, Monitoring, Tuning. (#2653)) output: true search-plugins: + permalink: /:collection/:path/ + output: true + ml-commons-plugin: permalink: /:collection/:path/ output: true - im-plugin: + neural-search-plugin: + permalink: /:collection/:path/ + output: true + tuning-your-cluster: permalink: /:collection/:path/ output: true - replication-plugin: + monitoring-your-cluster: permalink: /:collection/:path/ output: true - observability-plugin: + observing-your-data: permalink: /:collection/:path/ output: true ml-commons-plugin: permalink: /:collection/:path/ output: true +<<<<<<< HEAD monitoring-plugins: permalink: /:collection/:path/ output: true @@ -65,6 +91,9 @@ collections: permalink: /:collection/:path/ output: true job-scheduler-plugin: +======= + neural-search-plugin: +>>>>>>> 45834d6f (Once more... Managing, Monitoring, Tuning. (#2653)) permalink: /:collection/:path/ output: true clients: @@ -94,35 +123,40 @@ just_the_docs: opensearch: name: OpenSearch nav_fold: true + im-plugin: + name: Managing Indexes + nav_fold: true dashboards: name: OpenSearch Dashboards nav_fold: true + tuning-your-cluster: + name: Tuning your cluster + nav_fold: true security: name: Security in OpenSearch nav_fold: true search-plugins: - name: Search plugins - nav_fold: true - im-plugin: - name: Index management plugin - nav_fold: true - replication-plugin: - name: Replication plugin - nav_fold: true - observability-plugin: - name: Observability plugin + name: Search nav_fold: true ml-commons-plugin: - name: ML Commons plugin + name: Machine learning nav_fold: true +<<<<<<< HEAD monitoring-plugins: name: Monitoring plugins +======= + neural-search-plugin: + name: Neural Search nav_fold: true - notifications-plugin: - name: Notifications plugin + tuning-your-cluster: + name: Tuning your cluster + nav_fold: true + monitoring-your-cluster: + name: Monitoring your cluster +>>>>>>> 45834d6f (Once more... Managing, Monitoring, Tuning. (#2653)) nav_fold: true - job-scheduler-plugin: - name: Job Scheduler plugin + observing-your-data: + name: Observing your data nav_fold: true clients: name: Clients and tools @@ -202,3 +236,7 @@ exclude: - vendor/gems/ - vendor/ruby/ - README.md +<<<<<<< HEAD +======= + - .idea +>>>>>>> 45834d6f (Once more... Managing, Monitoring, Tuning. (#2653)) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md new file mode 100644 index 00000000000..7f4ba14a3b5 --- /dev/null +++ b/_data-prepper/peer-forwarder.md @@ -0,0 +1,181 @@ +--- +layout: default +title: Peer Forwarder +nav_order: 12 +--- + +# Peer Forwarder + +Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. This HTTP service uses a hash-ring approach to aggregate events and determine which Data Prepper node it should handle on a given trace before rerouting it to that node. Currently, Peer Forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_trace_raw` [processors]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/processors/). + +Peer Forwarder groups events based on the identification keys provided by the supported processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys. + +Peer discovery allows Data Prepper to find other nodes that it will communicate with. Currently, peer discovery is provided by a static list, a DNS record lookup, or AWS Cloud Map. + +## Discovery modes + +The following sections provide information about discovery modes. + +### Static + +Static discovery mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. See the following YAML file for an example of static discovery mode: + +```yaml +peer_forwarder:4 + discovery_mode: static + static_endpoints: ["data-prepper1", "data-prepper2"] +``` + +### DNS lookup + +DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. DNS discovery configures a DNS provider to return a list of Data Prepper hosts when given a single domain name. This list consists of a [DNS A record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), and a list of IP addresses of a given domain. See the following YAML file for an example of DNS lookup: + +```yaml +peer_forwarder: + discovery_mode: dns + domain_name: "data-prepper-cluster.my-domain.net" +``` + +### AWS Cloud Map + +[AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. + +Peer Forwarder can use the API-based service discovery in AWS Cloud Map. To support this, you must have an existing namespace configured for API instance discovery. You can create a new one by following the instructions provided by the [AWS Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). + +Your Data Prepper configuration needs to include the following: +* `aws_cloud_map_namespace_name` – Set to your AWS Cloud Map namespace name. +* `aws_cloud_map_service_name` – Set to the service name within your specified namespace. +* `aws_region` – Set to the AWS Region in which your namespace exists. +* `discovery_mode` – Set to `aws_cloud_map`. + +Your Data Prepper configuration can optionally include the following: +* `aws_cloud_map_query_parameters` – Key-value pairs are used to filter the results based on the custom attributes attached to an instance. Results include only those instances that match all of the specified key-value pairs. + +#### Example configuration + +See the following YAML file example of AWS Cloud Map configuration: + +```yaml +peer_forwarder: + discovery_mode: aws_cloud_map + aws_cloud_map_namespace_name: "my-namespace" + aws_cloud_map_service_name: "data-prepper-cluster" + aws_cloud_map_query_parameters: + instance_type: "r5.xlarge" + aws_region: "us-east-1" +``` + +### IAM policy with necessary permissions + +Data Prepper must also be running with the necessary permissions. The following AWS Identity and Access Management (IAM) policy shows the necessary permissions: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "CloudMapPeerForwarder", + "Effect": "Allow", + "Action": "servicediscovery:DiscoverInstances", + "Resource": "*" + } + ] +} +``` + + +## Configuration + +The following table provides optional configuration values. + + +| Value | Type | Description | +| ---- | --- | ----------- | +| `port` | Integer | A value between 0 and 65535 that represents the port that the Peer Forwarder server is running on. Default value is `4994`. | +| `request_timeout` | Integer | Represents the request timeout duration in milliseconds for the Peer Forwarder HTTP server. Default value is `10000`. | +| `server_thread_count` | Integer | Represents the number of threads used by the Peer Forwarder server. Default value is `200`.| +| `client_thread_count` | Integer | Represents the number of threads used by the Peer Forwarder client. Default value is `200`.| +| `maxConnectionCount` | Integer | Represents the maximum number of open connections for the Peer Forwarder server. Default value is `500`. | +| `discovery_mode` | String | Represents the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node`, which processes events locally. | +| `static_endpoints` | List | Contains the endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. | +| `domain_name` | String | Represents the single domain name to query DNS against. Typically used by creating multiple [DNS A records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. | +| `aws_cloud_map_namespace_name` | String | Represents the AWS Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_cloud_map_service_name` | String | Represents the AWS Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_cloud_map_query_parameters` | Map | Key-value pairs used to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | +| `buffer_size` | Integer | Represents the maximum number of unchecked records the buffer accepts (the number of unchecked records equals the number of records written into the buffer plus the number of records that are still processing and not yet checked by the Checkpointing API). Default is `512`. | +| `batch_size` | Integer | Represents the maximum number of records that the buffer returns on read. Default is `48`. | +| `aws_region` | String | Represents the AWS Region that uses `ACM`, `Amazon S3`, or `AWS Cloud Map` and is required when any of the following conditions are met:
- The `use_acm_certificate_for_ssl` setting is set to `true`.
- Either `ssl_certificate_file` or `ssl_key_file` specifies an Amazon Simple Storage Service (Amazon S3) URI (for example, s3://mybucket/path/to/public.cert).
- The `discovery_mode` is set to `aws_cloud_map`. | +| `drain_timeout` | Duration | Represents the amount of time that Peer Forwarder will wait to complete data processing before shutdown. | + +## SSL configuration + +The following table provides optional SSL configuration values that allow you to set up a trust manager for the Peer Forwarder client in order to connect to other Data Prepper instances. + +| Value | Type | Description | +| ----- | ---- | ----------- | +| `ssl` | Boolean | Enables TLS/SSL. Default value is `true`. | +| `ssl_certificate_file`| String | Represents the SSL certificate chain file path or Amazon S3 path. The following is an example of an Amazon S3 path: `s3:///`. Defaults to the default certificate file,`config/default_certificate.pem`. See [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) for more information about how the certificate is generated. | +| `ssl_key_file`| String | Represents the SSL key file path or Amazon S3 path. Amazon S3 path example: `s3:///`. Defaults to `config/default_private_key.pem` which is the default private key file. See [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) for more information about how the private key file is generated. | +| `ssl_insecure_disable_verification` | Boolean | Disables the verification of the server's TLS certificate chain. Default value is `false`. | +| `ssl_fingerprint_verification_only` | Boolean | Disables the verification of the server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. | +| `use_acm_certificate_for_ssl` | Boolean | Enables TLS/SSL using the certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | +| `acm_certificate_arn`| String | Represents the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over Amazon S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | +| `acm_private_key_password` | String | Represents the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | +| `acm_certificate_timeout_millis` | Integer | Represents the timeout in milliseconds required for ACM to get certificates. Default value is `120000`. | +| `aws_region` | String | Represents the AWS Region that uses ACM, Amazon S3, or AWS Cloud Map. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to use the Amazon S3 path or if `discovery_mode` is set to `aws_cloud_map`. | + +#### Example configuration + +The following YAML file provides an example configuration: + +```yaml +peer_forwarder: + ssl: true + ssl_certificate_file: "" + ssl_key_file: "" +``` + +## Authentication + +`Authentication` is optional and is a `Map` that enables mutual TLS (mTLS). It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. The following YAML file provides an example of authentication: + +```yaml +peer_forwarder: + authentication: + mutual_tls: +``` + +## Metrics + +Core Peer Forwarder introduces the following custom metrics. All the metrics are prefixed by `core.peerForwarder`. + +### Timer + +Peer Forwarder's timer capability provides the following information: + +- `requestForwardingLatency`: Measures latency of requests forwarded by the Peer Forwarder client. +- `requestProcessingLatency`: Measures latency of requests processed by the Peer Forwarder server. + +### Counter + +The following table provides counter metric options. + +| Value | Description | +| ----- | ----------- | +| `requests`| Measures the total number of forwarded requests. | +| `requestsFailed`| Measures the total number of failed requests. Applies to requests with an HTTP response code other than `200`. | +| `requestsSuccessful`| Measures the total number of successful requests. Applies to requests with HTTP response code `200`. | +| `requestsTooLarge`| Measures the total number of requests that are too large to be written to the Peer Forwarder buffer. Applies to requests with HTTP response code `413`. | +| `requestTimeouts`| Measures the total number of requests that time out while writing content to the Peer Forwarder buffer. Applies to requests with HTTP response code `408`. | +| `requestsUnprocessable`| Measures the total number of requests that fail due to an unprocessable entity. Applies to requests with HTTP response code `422`. | +| `badRequests`| Measures the total number of requests with a bad request format. Applies to requests with HTTP response code `400`. | +| `recordsSuccessfullyForwarded`| Measures the total number of successfully forwarded records. | +| `recordsFailedForwarding`| Measures the total number of records that fail to be forwarded. | +| `recordsToBeForwarded` | Measures the total number of records to be forwarded. | +| `recordsToBeProcessedLocally` | Measures the total number of records to be processed locally. | +| `recordsActuallyProcessedLocally`| Measures the total number of records actually processed locally. This value is the sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. | +| `recordsReceivedFromPeers`| Measures the total number of records received from remote peers. | + +### Gauge + +`peerEndpoints` Measures the number of dynamically discovered peer Data Prepper endpoints. For `static` mode, the size is fixed. diff --git a/_monitoring-your-cluster/job-scheduler/index.md b/_monitoring-your-cluster/job-scheduler/index.md new file mode 100644 index 00000000000..938d79803ce --- /dev/null +++ b/_monitoring-your-cluster/job-scheduler/index.md @@ -0,0 +1,132 @@ +--- +layout: default +title: Job Scheduler +nav_order: 1 +has_children: false +has_toc: false +redirect_from: + - /job-scheduler-plugin/index/ +--- + +# Job Scheduler + +The OpenSearch Job Scheduler plugin provides a framework that can be used to build schedules for common tasks performed on your cluster. You can use Job Scheduler’s Service Provider Interface (SPI) to define schedules for cluster management tasks such as taking snapshots, managing your data’s lifecycle, and running periodic jobs. Job Scheduler has a sweeper that listens for updated events on the OpenSearch cluster and a scheduler that manages when jobs run. + +You can install the Job Scheduler plugin by following the standard [OpenSearch plugin installation]({{site.url}}{{site.baseurl}}/install-and-configure/install-opensearch/plugins/) process. The sample-extension-plugin example provided in the [Job Scheduler GitHub repository](https://github.com/opensearch-project/job-scheduler) provides a complete example of utilizing Job Scheduler when building a plugin. To define schedules, you build a plugin that implements the interfaces provided in the Job Scheduler library. You can schedule jobs by specifying an interval, or you can use a Unix cron expression such as `0 12 * * ?`, which runs at noon every day, to define a more flexible schedule. + +## Building a plugin for Job Scheduler + +OpenSearch plugin developers can extend the Job Scheduler plugin to schedule jobs to perform on the cluster. Jobs you can schedule include running aggregation queries against raw data, saving the aggregated data to a new index every hour, or continuing to monitor the shard allocation by calling the OpenSearch API and then posting the output to a webhook. + +For examples of building a plugin that uses the Job Scheduler plugin, see the Job Scheduler [README](https://github.com/opensearch-project/job-scheduler/blob/main/README.md). + +## Defining an endpoint + +You can configure your plugin's API endpoint by referencing the [example](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleExtensionRestHandler.java) `SampleExtensionRestHandler.java` file. Set the endpoint URL that your plugin will expose with `WATCH_INDEX_URI`: + +```java +public class SampleExtensionRestHandler extends BaseRestHandler { + public static final String WATCH_INDEX_URI = "/_plugins/scheduler_sample/watch"; +``` + +You can define the job configuration by [extending](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleJobParameter.java) `ScheduledJobParameter`. You can also define the fields used by your plugin, like `indexToWatch`, as shown in the [example](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleJobParameter.java) `SampleJobParameter` file. This job configuration will be saved as a document in an index you define, as shown in [this example](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleExtensionPlugin.java#L54). + +## Configuring parameters + +You can configure your plugin's parameters by referencing the [example](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleJobParameter.java) `SampleJobParameter.java` file and modifying it to fit your needs: + +```java +/** + * A sample job parameter. + *

+ * It adds an additional "indexToWatch" field to {@link ScheduledJobParameter}, which stores the index + * the job runner will watch. + */ +public class SampleJobParameter implements ScheduledJobParameter { + public static final String NAME_FIELD = "name"; + public static final String ENABLED_FILED = "enabled"; + public static final String LAST_UPDATE_TIME_FIELD = "last_update_time"; + public static final String LAST_UPDATE_TIME_FIELD_READABLE = "last_update_time_field"; + public static final String SCHEDULE_FIELD = "schedule"; + public static final String ENABLED_TIME_FILED = "enabled_time"; + public static final String ENABLED_TIME_FILED_READABLE = "enabled_time_field"; + public static final String INDEX_NAME_FIELD = "index_name_to_watch"; + public static final String LOCK_DURATION_SECONDS = "lock_duration_seconds"; + public static final String JITTER = "jitter"; + + private String jobName; + private Instant lastUpdateTime; + private Instant enabledTime; + private boolean isEnabled; + private Schedule schedule; + private String indexToWatch; + private Long lockDurationSeconds; + private Double jitter; +``` + +Next, configure the request parameters you would like your plugin to use with Job Scheduler. These will be based on the variables you declare when configuring your plugin. The following example shows the request parameters you set when building your plugin: + +```java +public SampleJobParameter(String id, String name, String indexToWatch, Schedule schedule, Long lockDurationSeconds, Double jitter) { + this.jobName = name; + this.indexToWatch = indexToWatch; + this.schedule = schedule; + + Instant now = Instant.now(); + this.isEnabled = true; + this.enabledTime = now; + this.lastUpdateTime = now; + this.lockDurationSeconds = lockDurationSeconds; + this.jitter = jitter; + } + + @Override + public String getName() { + return this.jobName; + } + + @Override + public Instant getLastUpdateTime() { + return this.lastUpdateTime; + } + + @Override + public Instant getEnabledTime() { + return this.enabledTime; + } + + @Override + public Schedule getSchedule() { + return this.schedule; + } + + @Override + public boolean isEnabled() { + return this.isEnabled; + } + + @Override + public Long getLockDurationSeconds() { + return this.lockDurationSeconds; + } + + @Override public Double getJitter() { + return jitter; + } +``` + +The following table describes the request parameters configured in the previous example. All the request parameters shown are required. + +| Field | Data type | Description | +:--- | :--- | :--- +| getName | String | Returns the name of the job. | +| getLastUpdateTime | Time unit | Returns the time that the job was last run. | +| getEnabledTime | Time unit | Returns the time that the job was enabled. | +| getSchedule | Unix cron | Returns the job schedule formatted in Unix cron syntax. | +| isEnabled | Boolean | Indicates whether or not the job is enabled. | +| getLockDurationSeconds | Integer | Returns the duration of time for which the job is locked. | +| getJitter | Integer | Returns the defined jitter value. | + +The logic used by your job should be defined by a class extended from `ScheduledJobRunner` in the `SampleJobParameter.java` sample file, such as `SampleJobRunner`. While the job is running, there is a locking mechanism you can use to prevent other nodes from running the same job. First, [acquire](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleJobRunner.java#L96) the lock. Then make sure to release the lock before the [job finishes](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleJobRunner.java#L116). + +For more information, see the Job Scheduler [sample extension](https://github.com/opensearch-project/job-scheduler/blob/main/sample-extension-plugin/src/main/java/org/opensearch/jobscheduler/sampleextension/SampleJobParameter.java) directory in the [Job Scheduler GitHub repo](https://github.com/opensearch-project/job-scheduler). diff --git a/_monitoring-plugins/pa/api.md b/_monitoring-your-cluster/pa/api.md similarity index 99% rename from _monitoring-plugins/pa/api.md rename to _monitoring-your-cluster/pa/api.md index f70481da244..8ead2b47a4a 100644 --- a/_monitoring-plugins/pa/api.md +++ b/_monitoring-your-cluster/pa/api.md @@ -3,6 +3,8 @@ layout: default title: API parent: Performance Analyzer nav_order: 1 +redirect_from: + - /monitoring-plugins/pa/api/ --- # Performance Analyzer API diff --git a/_monitoring-plugins/pa/dashboards.md b/_monitoring-your-cluster/pa/dashboards.md similarity index 99% rename from _monitoring-plugins/pa/dashboards.md rename to _monitoring-your-cluster/pa/dashboards.md index 561f6fa3ce0..300095d3af6 100644 --- a/_monitoring-plugins/pa/dashboards.md +++ b/_monitoring-your-cluster/pa/dashboards.md @@ -3,6 +3,8 @@ layout: default title: Create PerfTop Dashboards parent: Performance Analyzer nav_order: 2 +redirect_from: + - /monitoring-plugins/pa/dashboards/ --- # PerfTop dashboards diff --git a/_monitoring-plugins/pa/index.md b/_monitoring-your-cluster/pa/index.md similarity index 99% rename from _monitoring-plugins/pa/index.md rename to _monitoring-your-cluster/pa/index.md index 45acb55c966..d63b1795f8a 100644 --- a/_monitoring-plugins/pa/index.md +++ b/_monitoring-your-cluster/pa/index.md @@ -5,6 +5,7 @@ nav_order: 58 has_children: true redirect_from: - /monitoring-plugins/pa/ + - /monitoring-plugins/pa/index/ --- # Performance analyzer diff --git a/_monitoring-plugins/pa/rca/api.md b/_monitoring-your-cluster/pa/rca/api.md similarity index 96% rename from _monitoring-plugins/pa/rca/api.md rename to _monitoring-your-cluster/pa/rca/api.md index 2d3aeb3e80e..cb8762cd5fd 100644 --- a/_monitoring-plugins/pa/rca/api.md +++ b/_monitoring-your-cluster/pa/rca/api.md @@ -4,6 +4,8 @@ title: API parent: Root Cause Analysis grand_parent: Performance Analyzer nav_order: 1 +redirect_from: + - /monitoring-plugins/pa/rca/api/ --- # RCA API diff --git a/_monitoring-plugins/pa/rca/index.md b/_monitoring-your-cluster/pa/rca/index.md similarity index 95% rename from _monitoring-plugins/pa/rca/index.md rename to _monitoring-your-cluster/pa/rca/index.md index 765b9e23e00..cd63659529f 100644 --- a/_monitoring-plugins/pa/rca/index.md +++ b/_monitoring-your-cluster/pa/rca/index.md @@ -4,6 +4,8 @@ title: Root Cause Analysis nav_order: 50 parent: Performance Analyzer has_children: true +redirect_from: + - /monitoring-plugins/pa/rca/index/ --- # Root Cause Analysis diff --git a/_monitoring-plugins/pa/rca/reference.md b/_monitoring-your-cluster/pa/rca/reference.md similarity index 83% rename from _monitoring-plugins/pa/rca/reference.md rename to _monitoring-your-cluster/pa/rca/reference.md index 765942d0708..2805f894a69 100644 --- a/_monitoring-plugins/pa/rca/reference.md +++ b/_monitoring-your-cluster/pa/rca/reference.md @@ -4,6 +4,8 @@ title: RCA Reference parent: Root Cause Analysis grand_parent: Performance Analyzer nav_order: 3 +redirect_from: + - /monitoring-plugins/pa/rca/reference/ --- # RCA reference diff --git a/_monitoring-plugins/pa/reference.md b/_monitoring-your-cluster/pa/reference.md similarity index 99% rename from _monitoring-plugins/pa/reference.md rename to _monitoring-your-cluster/pa/reference.md index caa87a8f797..9fed7646b1e 100644 --- a/_monitoring-plugins/pa/reference.md +++ b/_monitoring-your-cluster/pa/reference.md @@ -3,6 +3,8 @@ layout: default title: Metrics Reference parent: Performance Analyzer nav_order: 3 +redirect_from: + - /monitoring-plugins/pa/reference/ --- # Metrics reference diff --git a/_monitoring-plugins/ad/api.md b/_observing-your-data/ad/api.md similarity index 99% rename from _monitoring-plugins/ad/api.md rename to _observing-your-data/ad/api.md index c8149424f7b..635914b99ae 100644 --- a/_monitoring-plugins/ad/api.md +++ b/_observing-your-data/ad/api.md @@ -3,6 +3,8 @@ layout: default title: Anomaly detection API parent: Anomaly detection nav_order: 1 +redirect_from: + - /monitoring-plugins/ad/api/ --- # Anomaly detection API diff --git a/_monitoring-plugins/ad/index.md b/_observing-your-data/ad/index.md similarity index 99% rename from _monitoring-plugins/ad/index.md rename to _observing-your-data/ad/index.md index a1997560ca8..454d649204b 100644 --- a/_monitoring-plugins/ad/index.md +++ b/_observing-your-data/ad/index.md @@ -1,10 +1,11 @@ --- layout: default title: Anomaly detection -nav_order: 46 +nav_order: 90 has_children: true redirect_from: - /monitoring-plugins/ad/ + - /monitoring-plugins/ad/index/ --- # Anomaly detection diff --git a/_monitoring-plugins/ad/result-mapping.md b/_observing-your-data/ad/result-mapping.md similarity index 99% rename from _monitoring-plugins/ad/result-mapping.md rename to _observing-your-data/ad/result-mapping.md index 54d8670540f..7e1482a0134 100644 --- a/_monitoring-plugins/ad/result-mapping.md +++ b/_observing-your-data/ad/result-mapping.md @@ -3,6 +3,8 @@ layout: default title: Anomaly result mapping parent: Anomaly detection nav_order: 6 +redirect_from: + - /monitoring-plugins/ad/result-mapping/ --- # Anomaly result mapping diff --git a/_monitoring-plugins/ad/security.md b/_observing-your-data/ad/security.md similarity index 98% rename from _monitoring-plugins/ad/security.md rename to _observing-your-data/ad/security.md index 777bb416791..e309a410299 100644 --- a/_monitoring-plugins/ad/security.md +++ b/_observing-your-data/ad/security.md @@ -4,6 +4,8 @@ title: Anomaly detection security nav_order: 10 parent: Anomaly detection has_children: false +redirect_from: + - /monitoring-plugins/ad/security/ --- # Anomaly detection security diff --git a/_monitoring-plugins/ad/settings.md b/_observing-your-data/ad/settings.md similarity index 99% rename from _monitoring-plugins/ad/settings.md rename to _observing-your-data/ad/settings.md index 9a736edfb91..cc21407077b 100644 --- a/_monitoring-plugins/ad/settings.md +++ b/_observing-your-data/ad/settings.md @@ -3,6 +3,8 @@ layout: default title: Settings parent: Anomaly detection nav_order: 4 +redirect_from: + - /monitoring-plugins/ad/settings/ --- # Settings diff --git a/_monitoring-plugins/alerting/api.md b/_observing-your-data/alerting/api.md similarity index 99% rename from _monitoring-plugins/alerting/api.md rename to _observing-your-data/alerting/api.md index 3650bba9025..bd98fee51f0 100644 --- a/_monitoring-plugins/alerting/api.md +++ b/_observing-your-data/alerting/api.md @@ -3,6 +3,8 @@ layout: default title: API parent: Alerting nav_order: 15 +redirect_from: + - /monitoring-plugins/alerting/api/ --- # Alerting API diff --git a/_monitoring-plugins/alerting/cron.md b/_observing-your-data/alerting/cron.md similarity index 97% rename from _monitoring-plugins/alerting/cron.md rename to _observing-your-data/alerting/cron.md index bba64d067bb..b37d13e576d 100644 --- a/_monitoring-plugins/alerting/cron.md +++ b/_observing-your-data/alerting/cron.md @@ -4,6 +4,8 @@ title: Cron nav_order: 20 parent: Alerting has_children: false +redirect_from: + - /monitoring-plugins/alerting/cron/ --- # Cron expression reference diff --git a/_monitoring-plugins/alerting/index.md b/_observing-your-data/alerting/index.md similarity index 95% rename from _monitoring-plugins/alerting/index.md rename to _observing-your-data/alerting/index.md index 3495e4251b4..fa033ce1110 100644 --- a/_monitoring-plugins/alerting/index.md +++ b/_observing-your-data/alerting/index.md @@ -1,10 +1,11 @@ --- layout: default title: Alerting -nav_order: 34 +nav_order: 70 has_children: true redirect_from: - /monitoring-plugins/alerting/ + - /monitoring-plugins/alerting/index/ --- # Alerting diff --git a/_monitoring-plugins/alerting/monitors.md b/_observing-your-data/alerting/monitors.md similarity index 99% rename from _monitoring-plugins/alerting/monitors.md rename to _observing-your-data/alerting/monitors.md index c4579255fb7..4a51c64e82c 100644 --- a/_monitoring-plugins/alerting/monitors.md +++ b/_observing-your-data/alerting/monitors.md @@ -4,6 +4,8 @@ title: Monitors nav_order: 1 parent: Alerting has_children: false +redirect_from: + - /monitoring-plugins/alerting/monitors/ --- # Monitors diff --git a/_monitoring-plugins/alerting/security.md b/_observing-your-data/alerting/security.md similarity index 99% rename from _monitoring-plugins/alerting/security.md rename to _observing-your-data/alerting/security.md index c71910e066d..ab3055719a9 100644 --- a/_monitoring-plugins/alerting/security.md +++ b/_observing-your-data/alerting/security.md @@ -4,6 +4,8 @@ title: Alerting security nav_order: 10 parent: Alerting has_children: false +redirect_from: + - /monitoring-plugins/alerting/security/ --- # Alerting security diff --git a/_monitoring-plugins/alerting/settings.md b/_observing-your-data/alerting/settings.md similarity index 98% rename from _monitoring-plugins/alerting/settings.md rename to _observing-your-data/alerting/settings.md index 6e44d7ae5b0..20ef388b63a 100644 --- a/_monitoring-plugins/alerting/settings.md +++ b/_observing-your-data/alerting/settings.md @@ -3,6 +3,8 @@ layout: default title: Management parent: Alerting nav_order: 5 +redirect_from: + - /monitoring-plugins/alerting/settings/ --- # Management diff --git a/_observability-plugin/app-analytics.md b/_observing-your-data/app-analytics.md similarity index 95% rename from _observability-plugin/app-analytics.md rename to _observing-your-data/app-analytics.md index 03a3dfa07a8..44e1939aaa4 100644 --- a/_observability-plugin/app-analytics.md +++ b/_observing-your-data/app-analytics.md @@ -1,7 +1,9 @@ --- layout: default title: Application analytics -nav_order: 80 +nav_order: 10 +redirect_from: + - /observing-your-data/app-analytics/ --- # Application analytics @@ -39,7 +41,7 @@ To see your visualizations, choose the **Panel** tab. ### Configure availability -Availability is the status of your application determined by availability levels set on a [time series metric]({{site.url}}{{site.baseurl}}/observability-plugin/app-analytics/#time-series-metric). +Availability is the status of your application determined by availability levels set on a [time series metric]({{site.url}}{{site.baseurl}}/observing-your-data/app-analytics/#time-series-metric). To create an availability level, you must configure the following: - color: The color of the availability badge on the home page. diff --git a/_observability-plugin/event-analytics.md b/_observing-your-data/event-analytics.md similarity index 95% rename from _observability-plugin/event-analytics.md rename to _observing-your-data/event-analytics.md index 030315eb28b..780a1f0039e 100644 --- a/_observability-plugin/event-analytics.md +++ b/_observing-your-data/event-analytics.md @@ -1,7 +1,9 @@ --- layout: default title: Event analytics -nav_order: 10 +nav_order: 20 +redirect_from: + - /observing-your-data/event-analytics/ --- # Event analytics @@ -28,7 +30,7 @@ For more information about building PPL queries, see [Piped Processing Language] ## Save a visualization -After Dashboards generates a visualization, you must save it if you want to return to it at a later time or if you want to add it to an [operational panel]({{site.url}}{{site.baseurl}}/observability-plugin/operational-panels). +After Dashboards generates a visualization, you must save it if you want to return to it at a later time or if you want to add it to an [operational panel]({{site.url}}{{site.baseurl}}/observing-your-data/operational-panels). To save a visualization, expand the save dropdown menu next to **Refresh**, enter a name for your visualization, then choose **Save**. You can reopen any saved visualizations on the event analytics page. diff --git a/_observability-plugin/index.md b/_observing-your-data/index.md similarity index 58% rename from _observability-plugin/index.md rename to _observing-your-data/index.md index 304cf2dbbf9..4c516855bb5 100644 --- a/_observability-plugin/index.md +++ b/_observing-your-data/index.md @@ -1,13 +1,13 @@ --- layout: default -title: About Observability +title: Observing your data nav_order: 1 has_children: false redirect_from: - - /observability-plugin/ + - /observability-plugin/index/ --- -# About Observability +# Observing your data OpenSearch Dashboards {: .label .label-yellow :} @@ -16,12 +16,12 @@ Observability is collection of plugins and applications that let you visualize d Your experience of exploring data might differ, but if you're new to exploring data to create visualizations, we recommend trying a workflow like the following: 1. Explore data within a certain timeframe using [Piped Processing Language]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index). -2. Use [event analytics]({{site.url}}{{site.baseurl}}/observability-plugin/event-analytics) to turn data-driven events into visualizations. +2. Use [event analytics]({{site.url}}{{site.baseurl}}/observing-your-data/event-analytics) to turn data-driven events into visualizations. ![Sample Event Analytics View]({{site.url}}{{site.baseurl}}/images/event-analytics.png) -3. Create [operational panels]({{site.url}}{{site.baseurl}}/observability-plugin/operational-panels) and add visualizations to compare data the way you like. +3. Create [operational panels]({{site.url}}{{site.baseurl}}/observing-your-data/operational-panels) and add visualizations to compare data the way you like. ![Sample Operational Panel View]({{site.url}}{{site.baseurl}}/images/operational-panel.png) -4. Use [log analytics]({{site.url}}{{site.baseurl}}/observability-plugin/log-analytics) to transform unstructured log data. -5. Use [trace analytics]({{site.url}}{{site.baseurl}}/observability-plugin/trace/index) to create traces and dive deep into your data. +4. Use [log analytics]({{site.url}}{{site.baseurl}}/observing-your-data/log-ingestion/) to transform unstructured log data. +5. Use [trace analytics]({{site.url}}{{site.baseurl}}/observing-your-data/trace/index) to create traces and dive deep into your data. ![Sample Trace Analytics View]({{site.url}}{{site.baseurl}}/images/observability-trace.png) -6. Leverage [notebooks]({{site.url}}{{site.baseurl}}/observability-plugin/notebooks) to combine different visualizations and code blocks that you can share with team members. +6. Leverage [notebooks]({{site.url}}{{site.baseurl}}/observing-your-data/notebooks) to combine different visualizations and code blocks that you can share with team members. ![Sample Notebooks View]({{site.url}}{{site.baseurl}}/images/notebooks.png) diff --git a/_observing-your-data/log-analytics.md b/_observing-your-data/log-analytics.md new file mode 100644 index 00000000000..e69de29bb2d diff --git a/_observability-plugin/log-analytics.md b/_observing-your-data/log-ingestion.md similarity index 96% rename from _observability-plugin/log-analytics.md rename to _observing-your-data/log-ingestion.md index 8128d6d5610..43267f84ffb 100644 --- a/_observability-plugin/log-analytics.md +++ b/_observing-your-data/log-ingestion.md @@ -1,7 +1,9 @@ --- layout: default -title: Log analytics -nav_order: 70 +title: Log ingestion +nav_order: 30 +redirect_from: + - /observability-plugin/log-analytics/ --- # Log Ingestion @@ -90,4 +92,4 @@ The response should show the parsed log data: ] ``` -The same data can be viewed in OpenSearch Dashboards by visiting the **Discover** page and searching the `apache_logs` index. Remember, you must create the index in OpenSearch Dashboards if this is your first time searching for the index. +The same data can be viewed in OpenSearch Dashboards by visiting the **Discover** page and searching the `apache_logs` index. Remember, you must create the index in OpenSearch Dashboards if this is your first time searching for the index. \ No newline at end of file diff --git a/_observability-plugin/notebooks.md b/_observing-your-data/notebooks.md similarity index 98% rename from _observability-plugin/notebooks.md rename to _observing-your-data/notebooks.md index c4392696cbc..e85750fff5e 100644 --- a/_observability-plugin/notebooks.md +++ b/_observing-your-data/notebooks.md @@ -2,7 +2,9 @@ layout: default title: Notebooks nav_order: 50 -redirect_from: /notebooks/ +redirect_from: + - /notebooks/ + - /observability-plugin/notebooks/ has_children: false --- diff --git a/_notifications-plugin/api.md b/_observing-your-data/notifications/api.md similarity index 99% rename from _notifications-plugin/api.md rename to _observing-your-data/notifications/api.md index e9263ad020e..761d0855fbd 100644 --- a/_notifications-plugin/api.md +++ b/_observing-your-data/notifications/api.md @@ -2,8 +2,10 @@ layout: default title: API nav_order: 50 -has_children: false +has_children: true +parent: Notifications redirect_from: + - /notifications-plugin/api/ --- # Notifications API diff --git a/_notifications-plugin/index.md b/_observing-your-data/notifications/index.md similarity index 99% rename from _notifications-plugin/index.md rename to _observing-your-data/notifications/index.md index 521d4384419..c04d9787639 100644 --- a/_notifications-plugin/index.md +++ b/_observing-your-data/notifications/index.md @@ -1,10 +1,11 @@ --- layout: default title: Notifications -nav_order: 1 +nav_order: 80 has_children: false redirect_from: - /notifications-plugin/ + - /notifications-plugin/index/ --- # Notifications diff --git a/_observability-plugin/observability-security.md b/_observing-your-data/observability-security.md similarity index 97% rename from _observability-plugin/observability-security.md rename to _observing-your-data/observability-security.md index aacf40c30ba..da3ad613085 100644 --- a/_observability-plugin/observability-security.md +++ b/_observing-your-data/observability-security.md @@ -3,6 +3,8 @@ layout: default title: Observability security nav_order: 5 has_children: false +redirect_from: + - /observing-your-data/security/ --- # Observability security diff --git a/_observability-plugin/operational-panels.md b/_observing-your-data/operational-panels.md similarity index 87% rename from _observability-plugin/operational-panels.md rename to _observing-your-data/operational-panels.md index 8b8db539a49..1b05b9ded8b 100644 --- a/_observability-plugin/operational-panels.md +++ b/_observing-your-data/operational-panels.md @@ -1,7 +1,9 @@ --- layout: default title: Operational panels -nav_order: 30 +nav_order: 60 +redirect_from: + - /observing-your-data/operational-panels/ --- # Operational panels @@ -16,7 +18,7 @@ If you want to start using operational panels without adding any data, expand th To create an operational panel and add visualizations: -1. From the **Add Visualization** dropdown menu, choose **Select Existing Visualization** or **Create New Visualization**, which takes you to the [event analytics]({{site.url}}{{site.baseurl}}/observability-plugin/event-analytics) explorer, where you can use PPL to create visualizations. +1. From the **Add Visualization** dropdown menu, choose **Select Existing Visualization** or **Create New Visualization**, which takes you to the [event analytics]({{site.url}}{{site.baseurl}}/observing-your-data/event-analytics) explorer, where you can use PPL to create visualizations. 1. If you're adding already existing visualizations, choose a visualization from the dropdown menu. 1. Choose **Add**. diff --git a/_observability-plugin/trace/get-started.md b/_observing-your-data/trace/getting-started.md similarity index 93% rename from _observability-plugin/trace/get-started.md rename to _observing-your-data/trace/getting-started.md index 852626a9e83..1e48b739a18 100644 --- a/_observability-plugin/trace/get-started.md +++ b/_observing-your-data/trace/getting-started.md @@ -1,11 +1,13 @@ --- layout: default -title: Get Started +title: Getting Started parent: Trace analytics nav_order: 1 +redirect_from: + - /observability-plugin/trace/get-started/ --- -# Get started with Trace Analytics +# Getting started with Trace Analytics OpenSearch Trace Analytics consists of two components---Data Prepper and the Trace Analytics OpenSearch Dashboards plugin---that fit into the OpenTelemetry and OpenSearch ecosystems. The Data Prepper repository has several [sample applications](https://github.com/opensearch-project/data-prepper/tree/main/examples) to help you get started. @@ -21,7 +23,7 @@ OpenSearch Trace Analytics consists of two components---Data Prepper and the Tra 1. [Data Prepper]({{site.url}}{{site.baseurl}}/clients/data-prepper/index/) processes the OpenTelemetry data, transforms it for use in OpenSearch, and indexes it on an OpenSearch cluster. -1. The [Trace Analytics OpenSearch Dashboards plugin]({{site.url}}{{site.baseurl}}/observability-plugin/trace/ta-dashboards/) displays the data in near real-time as a series of charts and tables, with an emphasis on service architecture, latency, error rate, and throughput. +1. The [Trace Analytics OpenSearch Dashboards plugin]({{site.url}}{{site.baseurl}}/observing-your-data/trace/ta-dashboards/) displays the data in near real-time as a series of charts and tables, with an emphasis on service architecture, latency, error rate, and throughput. ## Jaeger HotROD @@ -78,4 +80,4 @@ curl -X GET -u 'admin:admin' -k 'https://localhost:9200/otel-v1-apm-span-000001/ Navigate to `http://localhost:5601` in a web browser and choose **Trace Analytics**. You can see the results of your single click in the Jaeger HotROD web interface: the number of traces per API and HTTP method, latency trends, a color-coded map of the service architecture, and a list of trace IDs that you can use to drill down on individual operations. -If you don't see your trace, adjust the timeframe in OpenSearch Dashboards. For more information on using the plugin, see [OpenSearch Dashboards plugin]({{site.url}}{{site.baseurl}}/observability-plugin/trace/ta-dashboards/). +If you don't see your trace, adjust the timeframe in OpenSearch Dashboards. For more information on using the plugin, see [OpenSearch Dashboards plugin]({{site.url}}{{site.baseurl}}/observing-your-data/trace/ta-dashboards/). diff --git a/_observability-plugin/trace/index.md b/_observing-your-data/trace/index.md similarity index 91% rename from _observability-plugin/trace/index.md rename to _observing-your-data/trace/index.md index ca8f72e7c69..08bfa58a850 100644 --- a/_observability-plugin/trace/index.md +++ b/_observing-your-data/trace/index.md @@ -1,9 +1,11 @@ --- layout: default title: Trace analytics -nav_order: 60 +nav_order: 40 has_children: true has_toc: false +redirect_from: + - /observability-plugin/trace/index/ --- # Trace Analytics diff --git a/_observability-plugin/trace/ta-dashboards.md b/_observing-your-data/trace/ta-dashboards.md similarity index 95% rename from _observability-plugin/trace/ta-dashboards.md rename to _observing-your-data/trace/ta-dashboards.md index 94e78c44ece..979fd07f8cb 100644 --- a/_observability-plugin/trace/ta-dashboards.md +++ b/_observing-your-data/trace/ta-dashboards.md @@ -3,6 +3,8 @@ layout: default title: OpenSearch Dashboards plugin parent: Trace analytics nav_order: 50 +redirect_from: + - /observability-plugin/trace/ta-dashboards/ --- # Trace Analytics OpenSearch Dashboards plugin diff --git a/_opensearch/snapshots/sm-dashboards.md b/_opensearch/snapshots/sm-dashboards.md deleted file mode 100644 index 5844e708604..00000000000 --- a/_opensearch/snapshots/sm-dashboards.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -layout: default -title: Snapshot Management in OpenSearch Dashboards -parent: Snapshots -nav_order: 35 -has_children: false ---- - -# Using Snapshot Management in OpenSearch Dashboards - -You can set up Snapshot Management (SM) in OpenSearch Dashboards. - -## Create a repository - -Before you create an SM policy, you need to set up a repository for snapshots. - -1. On the top menu bar, go to **OpenSearch Plugins > Snapshot Management**. -1. In the left panel, under **Snapshot Management**, select **Repositories**. -1. Select the **Create Repository** button. -1. Enter the repository name, type, and location. -1. (Optional) Select **Advanced Settings** and enter additional settings for this repository as a JSON object. Example: -```json -{ - "chunk_size": null, - "compress": false, - "max_restore_bytes_per_sec": "40m", - "max_snapshot_bytes_per_sec": "40m", - "readonly": false -} -``` -1. Select the **Add** button. - -## Create an SM policy - -Create an SM policy to set up automatic snapshots. An SM policy defines an automated snapshot creation schedule and an optional automated deletion schedule. - -1. On the top menu bar, go to **OpenSearch Plugins > Snapshot Management**. -1. In the left panel, under **Snapshot Management**, select **Snapshot Policies**. -1. Select the **Create Policy** button. -1. In the **Policy settings** section: - 1. Enter the policy name. - 1. (Optional) Enter the policy description. -1. In the **Source and destination** section: - 1. Select or enter source indexes either as a list or as an index pattern. - 1. Select a repository for snapshots. To [create a new repository](#create-a-repository), select the **Create** button. -1. In the **Snapshot schedule** section: - 1. Select the desired snapshot frequency or enter a custom cron expression for snapshot frequency. - 1. Select the start time and time zone. -1. In the **Retention period** section: - 1. Choose to retain all snapshots or specify retention conditions (the maximum age of retained snapshots). - 1. (Optional) In **Additional settings**, select the minimum and maximum number of retained snapshots, deletion frequency, and deletion start time. -1. In the **Notifications** section, select the snapshot activities you want to be notified about. -1. (Optional) In the **Advanced settings** section, select the desired options: - - **Include cluster state in snapshots** - - **Ignore unavailable indices** - - **Allow partial snapshots** -1. Select the **Create** button. - -## View, edit, or delete an SM policy - -You can view, edit, or delete an SM policy on the policy details page. - -1. On the top menu bar, go to **OpenSearch Plugins > Snapshot Management**. -1. In the left panel, under **Snapshot Management**, select **Snapshot Policies**. -1. Click on the **Policy name** of the policy you want to view, edit, or delete.
-The policy settings, snapshot schedule, snapshot retention period, notifications, and last creation and deletion are displayed in the policy details page.
If a snapshot creation or deletion fails, you can view information about the failure in the **Last Creation/Deletion** section. To view the failure message, click on the **cause** in the **Info** column. -1. To edit or delete the SM policy, select the **Edit** or **Delete** button. - -## Enable, disable, or delete SM policies - -1. On the top menu bar, go to **OpenSearch Plugins > Snapshot Management**. -1. In the left panel, under **Snapshot Management**, select **Snapshot Policies**. -1. Select one or more policies in the list. -1. To enable or disable selected SM policies, select the **Enable** or **Disable** button. To delete selected SM policies, in the **Actions** list, select the **Delete** option. - -## View snapshots - -1. On the top menu bar, go to **OpenSearch Plugins > Snapshot Management**. -1. In the left panel, under **Snapshot Management**, select **Snapshots**. -All automatically or manually taken snapshots appear in the list. -1. To view a snapshot, click on its **Name**. - -## Take a snapshot - -Use the steps below to take a snapshot manually. If you need to restore a snapshot, use the [restore snapshot API operation]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#restore-snapshots). - -1. On the top menu bar, go to **OpenSearch Plugins > Snapshot Management**. -1. In the left panel, under **Snapshot Management**, select **Snapshots**. -1. Select the **Take snapshot** button. -1. Enter the snapshot name. -1. Select or enter source indexes either as a list or as an index pattern. -1. Select a repository for the snapshot. -1. (Optional) In the **Advanced options** section, select the desired options: - - **Include cluster state in snapshots** - - **Ignore unavailable indices** - - **Allow partial snapshots** -1. Select the **Add** button. diff --git a/_search-plugins/sql/ppl/functions.md b/_search-plugins/sql/ppl/functions.md index 9ae2e6cf6e3..49a546e50ab 100644 --- a/_search-plugins/sql/ppl/functions.md +++ b/_search-plugins/sql/ppl/functions.md @@ -1,7 +1,7 @@ --- layout: default title: Commands -parent: PPL - Piped Processing Language +parent: PPL – Piped Processing Language grand_parent: SQL and PPL nav_order: 2 --- diff --git a/_search-plugins/sql/ppl/syntax.md b/_search-plugins/sql/ppl/syntax.md index 9ccd701fb50..08d6882fb40 100644 --- a/_search-plugins/sql/ppl/syntax.md +++ b/_search-plugins/sql/ppl/syntax.md @@ -1,7 +1,7 @@ --- layout: default title: Syntax -parent: PPL - Piped Processing Language +parent: PPL – Piped Processing Language grand_parent: SQL and PPL nav_order: 1 --- diff --git a/_security/access-control/permissions.md b/_security/access-control/permissions.md index bdfeaf39189..47f2becacc6 100644 --- a/_security/access-control/permissions.md +++ b/_security/access-control/permissions.md @@ -3,6 +3,8 @@ layout: default title: Permissions parent: Access control nav_order: 110 +redirect_from: + - /security-plugin/access-control/permissions/ --- # Permissions diff --git a/_security/configuration/index.md b/_security/configuration/index.md index 7bcdb256102..d0acd8b620f 100644 --- a/_security/configuration/index.md +++ b/_security/configuration/index.md @@ -5,7 +5,8 @@ nav_order: 2 has_children: true has_toc: false redirect_from: - - /security/configuration/ + - /security-plugin/configuration/ + - /security-plugin/configuration/index/ --- # Security configuration diff --git a/_security/index.md b/_security/index.md index 900f7a26cae..1e8abcdbda6 100755 --- a/_security/index.md +++ b/_security/index.md @@ -5,6 +5,8 @@ nav_order: 1 has_children: false has_toc: false redirect_from: + - /security-plugin/ + - /security-plugin/index/ - /security/ --- diff --git a/_tuning-your-cluster/availability-and-recovery/index.md b/_tuning-your-cluster/availability-and-recovery/index.md new file mode 100644 index 00000000000..3a68fd4c22e --- /dev/null +++ b/_tuning-your-cluster/availability-and-recovery/index.md @@ -0,0 +1,9 @@ +--- +layout: default +title: Availability and Recovery +nav_order: 20 +has_children: true +has_toc: true +--- + +The following OpenSearch features help ensure consistent uptime so that your cluster can complete and scale based on your use case, as well as creating snapshots. \ No newline at end of file diff --git a/_tuning-your-cluster/availability-and-recovery/segment-replication/configuration.md b/_tuning-your-cluster/availability-and-recovery/segment-replication/configuration.md new file mode 100644 index 00000000000..b336df6985a --- /dev/null +++ b/_tuning-your-cluster/availability-and-recovery/segment-replication/configuration.md @@ -0,0 +1,84 @@ +--- +layout: default +title: Segment replication configuration +nav_order: 12 +parent: Segment replication +grand_parent: Availability and Recovery +--- + +# Segment replication configuration + +Segment replication is an experimental feature. Therefore, we do not recommend the use of segment replication in a production environment. For updates on the progress of segment replication or if you want to leave feedback that could help improve the feature, see the [Segment replication issue](https://github.com/opensearch-project/OpenSearch/issues/2194). +{: .warning } + +To enable the segment replication type, reference the steps below. + +## Enabling the feature flag + +There are several methods for enabling segment replication, depending on the install type. You will also need to set the replication strategy to `SEGMENT` when creating the index. + +### Enable on a node using a tarball install + +The flag is toggled using a new jvm parameter that is set either in `OPENSEARCH_JAVA_OPTS` or in config/jvm.options. + +1. Option 1: Update config/jvm.options by adding the following line: + + ````json + -Dopensearch.experimental.feature.replication_type.enabled=true + ```` + +1. Option 2: Use the `OPENSEARCH_JAVA_OPTS` environment variable: + + ````json + export OPENSEARCH_JAVA_OPTS="-Dopensearch.experimental.feature.replication_type.enabled=true" + ```` +1. Option 3: For developers using Gradle, update run.gradle by adding the following lines: + + ````json + testClusters { + runTask { + testDistribution = 'archive' + if (numZones > 1) numberOfZones = numZones + if (numNodes > 1) numberOfNodes = numNodes + systemProperty 'opensearch.experimental.feature.replication_type.enabled', 'true' + } + } + ```` + +### Enable with Docker containers + +If you're running Docker, add the following line to docker-compose.yml underneath the `opensearch-node` and `environment` section: + +````json +OPENSEARCH_JAVA_OPTS="-Dopensearch.experimental.feature.replication_type.enabled=true" # Enables segment replication +```` + +### Setting the replication strategy on the index + +To set the replication strategy to segment replication, create an index with replication.type set to `SEGMENT`: + +````json +PUT /my-index1 +{ + "settings": { + "index": { + "replication.type": "SEGMENT" + } + } +} +```` + +## Known limitations + +1. Enabling segment replication for an existing index requires [reindexing](https://github.com/opensearch-project/OpenSearch/issues/3685). +1. Rolling upgrades are currently not supported. Full cluster restarts are required when upgrading indexes using segment replication. [Issue 3881](https://github.com/opensearch-project/OpenSearch/issues/3881). +1. [Cross-cluster replication](https://github.com/opensearch-project/OpenSearch/issues/4090) does not currently use segment replication to copy between clusters. +1. Increased network congestion on primary shards. [Issue - Optimize network bandwidth on primary shards](https://github.com/opensearch-project/OpenSearch/issues/4245). +1. Shard allocation algorithms have not been updated to evenly spread primary shards across nodes. +1. Integration with remote-backed storage as the source of replication is [currently unsupported](https://github.com/opensearch-project/OpenSearch/issues/4448). + +### Further resources regarding segment replication + +1. [Known issues](https://github.com/opensearch-project/OpenSearch/issues/2194). +1. Steps for testing (link coming soon). +1. Segment replication blog post (link coming soon). \ No newline at end of file diff --git a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md new file mode 100644 index 00000000000..b7641f81920 --- /dev/null +++ b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md @@ -0,0 +1,27 @@ +--- +layout: default +title: Segment replication +nav_order: 70 +has_children: true +parent: Availability and Recovery +redirect_from: + - /opensearch/segment-replication/ + - /opensearch/segment-replication/index/ +--- + +# Segment replication + +Segment replication is an experimental feature with OpenSearch 2.3. Therefore, we do not recommend the use of segment replication in a production environment. For updates on the progress of segment replication or if you want leave feedback that could help improve the feature, see the [Segment replication git issue](https://github.com/opensearch-project/OpenSearch/issues/2194). +{: .warning} + +With segment replication, segment files are copied across shards instead of documents being indexed on each shard copy. This improves indexing throughput and lowers resource utilization at the expense of increased network utilization. + +As an experimental feature, segment replication will be behind a feature flag and must be enabled on **each node** of a cluster and pass a new setting during index creation. +{: .note } + +### Potential use cases + +- Users who have high write loads but do not have high search requirements and are comfortable with longer refresh times. +- Users with very high loads who want to add new nodes, as you do not need to index all nodes when adding a new node to the cluster. + +This is the first step in a series of features designed to decouple reads and writes in order to lower compute costs. \ No newline at end of file diff --git a/_opensearch/shard-indexing-backpressure.md b/_tuning-your-cluster/availability-and-recovery/shard-indexing-backpressure.md similarity index 94% rename from _opensearch/shard-indexing-backpressure.md rename to _tuning-your-cluster/availability-and-recovery/shard-indexing-backpressure.md index ac58c7d358a..cde2f125cb6 100644 --- a/_opensearch/shard-indexing-backpressure.md +++ b/_tuning-your-cluster/availability-and-recovery/shard-indexing-backpressure.md @@ -3,6 +3,9 @@ layout: default title: Shard indexing backpressure nav_order: 62 has_children: true +parent: Availability and Recovery +redirect_from: + - /opensearch/shard-indexing-backpressure/ --- # Shard indexing backpressure diff --git a/_opensearch/shard-indexing-settings.md b/_tuning-your-cluster/availability-and-recovery/shard-indexing-settings.md similarity index 97% rename from _opensearch/shard-indexing-settings.md rename to _tuning-your-cluster/availability-and-recovery/shard-indexing-settings.md index 8726906352b..88b0ea70b4c 100644 --- a/_opensearch/shard-indexing-settings.md +++ b/_tuning-your-cluster/availability-and-recovery/shard-indexing-settings.md @@ -2,8 +2,10 @@ layout: default title: Settings parent: Shard indexing backpressure -nav_order: 1 -has_children: false +nav_order: 50 +grand_parent: Availability and Recovery +redirect_from: + - /opensearch/shard-indexing-settings/ --- # Settings diff --git a/_opensearch/snapshots/index.md b/_tuning-your-cluster/availability-and-recovery/snapshots/index.md similarity index 75% rename from _opensearch/snapshots/index.md rename to _tuning-your-cluster/availability-and-recovery/snapshots/index.md index 192f6f02ec8..43ceec1d7b9 100644 --- a/_opensearch/snapshots/index.md +++ b/_tuning-your-cluster/availability-and-recovery/snapshots/index.md @@ -1,9 +1,12 @@ --- layout: default title: Snapshots -nav_order: 65 +nav_order: 30 has_children: true -redirect_from: /opensearch/snapshots/ +parent: Availability and Recovery +redirect_from: + - /opensearch/snapshots/ + - /opensearch/snapshots/index/ has_toc: false --- @@ -24,4 +27,4 @@ Snapshots have two main uses: You can take and restore snapshots using the [snapshot API]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore). -If you need to automate taking snapshots, you can use the [Snapshot Management]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-management) feature. +If you need to automate taking snapshots, you can use the [snapshot management]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-management) feature. diff --git a/_opensearch/snapshots/sm-api.md b/_tuning-your-cluster/availability-and-recovery/snapshots/sm-api.md similarity index 98% rename from _opensearch/snapshots/sm-api.md rename to _tuning-your-cluster/availability-and-recovery/snapshots/sm-api.md index 563e7bbfe26..e3cc316482d 100644 --- a/_opensearch/snapshots/sm-api.md +++ b/_tuning-your-cluster/availability-and-recovery/snapshots/sm-api.md @@ -1,14 +1,17 @@ --- layout: default -title: Snapshot Management API +title: Snapshot management API parent: Snapshots nav_order: 30 has_children: false +grand_parent: Availability and Recovery +redirect_from: + - /opensearch/snapshots/sm-api/ --- # Snapshot Management API -Use the [Snapshot Management (SM)]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#take-snapshots) API to automate [taking snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#take-snapshots). +Use the snapshot management (SM) API to automate [taking snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#take-snapshots). --- diff --git a/_opensearch/snapshots/snapshot-management.md b/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-management.md similarity index 94% rename from _opensearch/snapshots/snapshot-management.md rename to _tuning-your-cluster/availability-and-recovery/snapshots/snapshot-management.md index 8cedd0b4d51..9a25b28683a 100644 --- a/_opensearch/snapshots/snapshot-management.md +++ b/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-management.md @@ -1,14 +1,17 @@ --- layout: default -title: Snapshot Management +title: Snapshot management parent: Snapshots nav_order: 20 has_children: false +grand_parent: Availability and Recovery +redirect_from: + - /opensearch/snapshots/snapshot-management/ --- -# Snapshot Management +# Snapshot management -Snapshot Management (SM) lets you automate [taking snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#take-snapshots). To use this feature, you need to install the [Index Management (IM) Plugin]({{site.url}}{{site.baseurl}}/im-plugin). Snapshots store only incremental changes since the last snapshot. Thus, while taking an initial snapshot may be a heavy operation, subsequent snapshots have minimal overhead. To set up automatic snapshots, you have to create an SM policy with a desired SM schedule and configuration. +Snapshot management (SM) lets you automate [taking snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#take-snapshots). To use this feature, you need to install the [Index Management (IM) Plugin]({{site.url}}{{site.baseurl}}/im-plugin). Snapshots store only incremental changes since the last snapshot. Thus, while taking an initial snapshot may be a heavy operation, subsequent snapshots have minimal overhead. To set up automatic snapshots, you have to create an SM policy with a desired SM schedule and configuration. When you create an SM policy, its document ID is given the name `-sm-policy`. Because of this, SM policies have to obey the following rules: @@ -18,7 +21,7 @@ When you create an SM policy, its document ID is given the name `-s SM-created snapshots have names in the format `--`. Two snapshots created by different policies at the same time always have different names because of the `` prefix. To avoid name collisions within the same policy, each snapshot's name contains a random string suffix. -Each policy has associated metadata that stores the policy status. Snapshot Management saves SM policies and metadata in the system index and reads them from the system index. Thus, Snapshot Management depends on the OpenSearch cluster's indexing and searching functions. The policy's metadata keeps information about the latest creation and deletion only. The metadata is read before running every scheduled job so that SM can continue execution from the previous job's state. You can view the metadata using the [explain API]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-api#explain). +Each policy has associated metadata that stores the policy status. Snapshot management saves SM policies and metadata in the system index and reads them from the system index. Thus, Snapshot Management depends on the OpenSearch cluster's indexing and searching functions. The policy's metadata keeps information about the latest creation and deletion only. The metadata is read before running every scheduled job so that SM can continue execution from the previous job's state. You can view the metadata using the [explain API]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-api#explain). An SM schedule is a custom [cron]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/cron) expression. It consists of two parts: a creation schedule and a deletion schedule. You must set up a creation schedule that specifies the frequency and timing of snapshot creation. Optionally, you can set up a separate schedule for deleting snapshots. @@ -44,7 +47,7 @@ We don't recommend setting up the same repository for multiple SM policies with ## Failure management -If a snapshot operation fails, it is retried a maximum of three times. The failure message is saved in `metadata.latest_execution` and is overwritten when a subsequent snapshot operation starts. You can view the failure message using the [explain API]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-api#explain). When using OpenSearch Dashboards, you can view the failure message on the [policy details page]({{site.url}}{{site.baseurl}}/dashboards/admin-ui-index/sm-dashboards/#enable-disable-or-delete-sm-policies). Possible reasons for failure include red index status and shard reallocation. +If a snapshot operation fails, it is retried a maximum of three times. The failure message is saved in `metadata.latest_execution` and is overwritten when a subsequent snapshot operation starts. You can view the failure message using the [explain API]({{site.url}}{{site.baseurl}}/opensearch/snapshots/sm-api#explain). When using OpenSearch Dashboards, you can view the failure message on the [policy details page]({{site.url}}{{site.baseurl}}/dashboards/admin-ui-index/sm-dashboards#view-edit-or-delete-an-sm-policy). Possible reasons for failure include red index status and shard reallocation. ## Security diff --git a/_opensearch/snapshots/snapshot-restore.md b/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore.md similarity index 99% rename from _opensearch/snapshots/snapshot-restore.md rename to _tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore.md index 8a6d06cf757..f01ff1c96b1 100644 --- a/_opensearch/snapshots/snapshot-restore.md +++ b/_tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore.md @@ -4,6 +4,9 @@ title: Take and restore snapshots parent: Snapshots nav_order: 10 has_children: false +grand_parent: Availability and Recovery +redirect_from: + - /opensearch/snapshots/snapshot-restore/ --- # Take and restore snapshots diff --git a/_opensearch/stats-api.md b/_tuning-your-cluster/availability-and-recovery/stats-api.md similarity index 99% rename from _opensearch/stats-api.md rename to _tuning-your-cluster/availability-and-recovery/stats-api.md index 050d0bb6625..4ac5e4e6ad5 100644 --- a/_opensearch/stats-api.md +++ b/_tuning-your-cluster/availability-and-recovery/stats-api.md @@ -3,7 +3,10 @@ layout: default title: Stats API parent: Shard indexing backpressure nav_order: 2 +grand_parent: Availability and Recovery has_children: false +redirect_from: + - /opensearch/stats-api/ --- # Stats API diff --git a/_tuning-your-cluster/cluster-manager-task-throttling.md b/_tuning-your-cluster/cluster-manager-task-throttling.md new file mode 100644 index 00000000000..0ccb51496ed --- /dev/null +++ b/_tuning-your-cluster/cluster-manager-task-throttling.md @@ -0,0 +1,107 @@ +--- +layout: default +title: Cluster manager task throttling +nav_order: 10 +has_children: false +--- + +# Cluster manager task throttling + +For many cluster state updates, such as defining a mapping or creating an index, nodes submit tasks to the cluster manager. The cluster manager maintains a pending task queue for these tasks and runs them in a single-threaded environment. When nodes send tens of thousands of resource-intensive tasks, like `put-mapping` or snapshot tasks, these tasks can pile up in the queue and flood the cluster manager. This affects the cluster manager's performance and may in turn affect the availability of the whole cluster. + +The first line of defense is to implement mechanisms in the caller nodes to avoid task overload on the cluster manager. However, even with those mechanisms in place, the cluster manager needs a built-in way to protect itself: cluster manager task throttling. + +To turn on cluster manager task throttling, you need to set throttling limits. The cluster manager uses the throttling limits to determine whether to reject a task. + +The cluster manager rejects a task based on its type. For any incoming task, the cluster manager evaluates the total number of tasks of the same type in the pending task queue. If this number exceeds the threshold for this task type, the cluster manager rejects the incoming task. Rejecting a task does not affect tasks of a different type. For example, if the cluster manager rejects a `put-mapping` task, it can still accept a subsequent `create-index` task. + +When the cluster manager rejects a task, the node performs retries with exponential backoff to resubmit the task to the cluster manager. If retries are unsuccessful within the timeout period, OpenSearch returns a cluster timeout error. + +## Setting throttling limits + +You can set throttling limits by specifying them in the `cluster_manager.throttling.thresholds` object and updating the [OpenSearch cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-settings). The setting is dynamic, so you can change the behavior of this feature without restarting your cluster. + +By default, throttling is disabled for all task types. +{: .note} + +The request has the following format: + +```json +PUT _cluster/settings +{ + "persistent": { + "cluster_manager.throttling.thresholds" : { + "" : { + "value" : + } + } + } +} +``` + +The following table describes the `cluster_manager.throttling.thresholds` object. + +Field Name | Description +:--- | :--- +task-type | The task type. See [supported task types](#supported-task-types) for a list of valid values. +value | The maximum number of tasks of the `task-type` type in the cluster manager's pending task queue. Default is `-1` (no task throttling). + +## Supported task types + +The following task types are supported: + +- `create-index` +- `update-settings` +- `cluster-update-settings` +- `auto-create` +- `delete-index` +- `delete-dangling-index` +- `create-data-stream` +- `remove-data-stream` +- `rollover-index` +- `index-aliases` +- `put-mapping` +- `create-index-template` +- `remove-index-template` +- `create-component-template` +- `remove-component-template` +- `create-index-template-v2` +- `remove-index-template-v2` +- `put-pipeline` +- `delete-pipeline` +- `create-persistent-task` +- `finish-persistent-task` +- `remove-persistent-task` +- `update-task-state` +- `put-script` +- `delete-script` +- `put-repository` +- `delete-repository` +- `create-snapshot` +- `delete-snapshot` +- `update-snapshot-state` +- `restore-snapshot` +- `cluster-reroute-api` + +#### Sample request + +The following request sets the throttling threshold for the `put-mapping` task type to 100: + +```json +PUT _cluster/settings +{ + "persistent": { + "cluster_manager.throttling.thresholds": { + "put-mapping": { + "value": 100 + } + } + } +} +``` + +Set the threshold to `-1` to disable throttling for a task type. +{: .note} + + + diff --git a/_opensearch/cluster.md b/_tuning-your-cluster/cluster.md similarity index 99% rename from _opensearch/cluster.md rename to _tuning-your-cluster/cluster.md index dc4ecfa1126..0bd08c1fbf9 100644 --- a/_opensearch/cluster.md +++ b/_tuning-your-cluster/cluster.md @@ -1,10 +1,12 @@ --- layout: default -title: Cluster formation -nav_order: 7 +title: Creating a cluster +nav_order: 8 +redirect_from: + - /opensearch/cluster/ --- -# Cluster formation +# Creating a cluster Before diving into OpenSearch and searching and aggregating data, you first need to create an OpenSearch cluster. diff --git a/_tuning-your-cluster/replication-plugin/api.md b/_tuning-your-cluster/replication-plugin/api.md new file mode 100644 index 00000000000..60a0a37590a --- /dev/null +++ b/_tuning-your-cluster/replication-plugin/api.md @@ -0,0 +1,394 @@ +--- +layout: default +title: API +nav_order: 50 +parent: Cross-cluster replication +redirect_from: + - /replication-plugin/api/ +--- + +# Cross-cluster replication API + +Use these replication operations to programmatically manage cross-cluster replication. + +#### Table of contents +- TOC +{:toc} + +## Start replication +Introduced 1.1 +{: .label .label-purple } + +Initiate replication of an index from the leader cluster to the follower cluster. Send this request to the follower cluster. + + +#### Request + +```json +PUT /_plugins/_replication//_start +{ + "leader_alias":"", + "leader_index":"", + "use_roles":{ + "leader_cluster_role":"", + "follower_cluster_role":"" + } +} +``` + +Specify the following options: + +Options | Description | Type | Required +:--- | :--- |:--- |:--- | +`leader_alias` | The name of the cross-cluster connection. You define this alias when you [set up a cross-cluster connection]({{site.url}}{{site.baseurl}}/replication-plugin/get-started/#set-up-a-cross-cluster-connection). | `string` | Yes +`leader_index` | The index on the leader cluster that you want to replicate. | `string` | Yes +`use_roles` | The roles to use for all subsequent backend replication tasks between the indexes. Specify a `leader_cluster_role` and `follower_cluster_role`. See [Map the leader and follower cluster roles]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#map-the-leader-and-follower-cluster-roles). | `string` | If security plugin is enabled + +#### Sample response + +```json +{ + "acknowledged": true +} +``` + +## Stop replication +Introduced 1.1 +{: .label .label-purple } + +Terminates replication and converts the follower index to a standard index. Send this request to the follower cluster. + +#### Request + +```json +POST /_plugins/_replication//_stop +{} +``` + +#### Sample response + +```json +{ + "acknowledged": true +} +``` + +## Pause replication +Introduced 1.1 +{: .label .label-purple } + +Pauses replication of the leader index. Send this request to the follower cluster. + +#### Request + +```json +POST /_plugins/_replication//_pause +{} +``` + +You can't resume replication after it's been paused for more than 12 hours. You must [stop replication]({{site.url}}{{site.baseurl}}/replication-plugin/api/#stop-replication), delete the follower index, and restart replication of the leader. + +#### Sample response + +```json +{ + "acknowledged": true +} +``` + +## Resume replication +Introduced 1.1 +{: .label .label-purple } + +Resumes replication of the leader index. Send this request to the follower cluster. + +#### Request + +```json +POST /_plugins/_replication//_resume +{} +``` + +#### Sample response + +```json +{ + "acknowledged": true +} +``` + +## Get replication status +Introduced 1.1 +{: .label .label-purple } + +Gets the status of index replication. Possible statuses are `SYNCING`, `BOOTSTRAPING`, `PAUSED`, and `REPLICATION NOT IN PROGRESS`. Use the syncing details to measure replication lag. Send this request to the follower cluster. + +#### Request + +```json +GET /_plugins/_replication//_status +``` + +#### Sample response + +```json +{ + "status" : "SYNCING", + "reason" : "User initiated", + "leader_alias" : "my-connection-name", + "leader_index" : "leader-01", + "follower_index" : "follower-01", + "syncing_details" : { + "leader_checkpoint" : 19, + "follower_checkpoint" : 19, + "seq_no" : 0 + } +} +``` +To include shard replication details in the response, add the `&verbose=true` parameter. + +The leader and follower checkpoint values begin as negative integers and reflect the shard count (-1 for one shard, -5 for five shards, and so on). The values increment toward positive integers with each change that you make. For example, when you make a change on the leader index, the `leader_checkpoint` becomes `0`. The `follower_checkpoint` is initially still `-1` until the follower index pulls the change from the leader, at which point it increments to `0`. If the values are the same, it means the indexes are fully synced. + +## Get leader cluster stats +Introduced 1.1 +{: .label .label-purple } + +Gets information about replicated leader indexes on a specified cluster. + +#### Request + +```json +GET /_plugins/_replication/leader_stats +``` + +#### Sample response + +```json +{ + "num_replicated_indices": 2, + "operations_read": 15, + "translog_size_bytes": 1355, + "operations_read_lucene": 0, + "operations_read_translog": 15, + "total_read_time_lucene_millis": 0, + "total_read_time_translog_millis": 659, + "bytes_read": 1000, + "index_stats":{ + "leader-index-1":{ + "operations_read": 7, + "translog_size_bytes": 639, + "operations_read_lucene": 0, + "operations_read_translog": 7, + "total_read_time_lucene_millis": 0, + "total_read_time_translog_millis": 353, + "bytes_read":466 + }, + "leader-index-2":{ + "operations_read": 8, + "translog_size_bytes": 716, + "operations_read_lucene": 0, + "operations_read_translog": 8, + "total_read_time_lucene_millis": 0, + "total_read_time_translog_millis": 306, + "bytes_read": 534 + } + } +} +``` + +## Get follower cluster stats +Introduced 1.1 +{: .label .label-purple } + +Gets information about follower (syncing) indexes on a specified cluster. + +#### Request + +```json +GET /_plugins/_replication/follower_stats +``` + +#### Sample response + +```json +{ + "num_syncing_indices": 2, + "num_bootstrapping_indices": 0, + "num_paused_indices": 0, + "num_failed_indices": 0, + "num_shard_tasks": 2, + "num_index_tasks": 2, + "operations_written": 3, + "operations_read": 3, + "failed_read_requests": 0, + "throttled_read_requests": 0, + "failed_write_requests": 0, + "throttled_write_requests": 0, + "follower_checkpoint": 1, + "leader_checkpoint": 1, + "total_write_time_millis": 2290, + "index_stats":{ + "follower-index-1":{ + "operations_written": 2, + "operations_read": 2, + "failed_read_requests": 0, + "throttled_read_requests": 0, + "failed_write_requests": 0, + "throttled_write_requests": 0, + "follower_checkpoint": 1, + "leader_checkpoint": 1, + "total_write_time_millis": 1355 + }, + "follower-index-2":{ + "operations_written": 1, + "operations_read": 1, + "failed_read_requests": 0, + "throttled_read_requests": 0, + "failed_write_requests": 0, + "throttled_write_requests": 0, + "follower_checkpoint": 0, + "leader_checkpoint": 0, + "total_write_time_millis": 935 + } + } +} +``` + +## Get auto-follow stats +Introduced 1.1 +{: .label .label-purple } + +Gets information about auto-follow activity and any replication rules configured on the specified cluster. + +#### Request + +```json +GET /_plugins/_replication/autofollow_stats +``` + +#### Sample response + +```json +{ + "num_success_start_replication": 2, + "num_failed_start_replication": 0, + "num_failed_leader_calls": 0, + "failed_indices":[ + + ], + "autofollow_stats":[ + { + "name":"my-replication-rule", + "pattern":"movies*", + "num_success_start_replication": 2, + "num_failed_start_replication": 0, + "num_failed_leader_calls": 0, + "failed_indices":[ + + ] + } + ] +} +``` + +## Update settings +Introduced 1.1 +{: .label .label-purple } + +Updates settings on the follower index. + +#### Request + +```json +PUT /_plugins/_replication//_update +{ + "settings":{ + "index.number_of_shards": 4, + "index.number_of_replicas": 2 + } +} +``` + +#### Sample response + +```json +{ + "acknowledged": true +} +``` + +## Create replication rule +Introduced 1.1 +{: .label .label-purple } + +Automatically starts replication on indexes matching a specified pattern. If a new index on the leader cluster matches the pattern, OpenSearch automatically creates a follower index and begins replication. You can also use this API to update existing replication rules. + +Send this request to the follower cluster. + +Make sure to note the names of all auto-follow patterns after you create them. The replication plugin currently does not include an API operation to retrieve a list of existing patterns. +{: .tip } + +#### Request + +```json +POST /_plugins/_replication/_autofollow +{ + "leader_alias" : "", + "name": "", + "pattern": "", + "use_roles":{ + "leader_cluster_role": "", + "follower_cluster_role": "" + } +} +``` + +Specify the following options: + +Options | Description | Type | Required +:--- | :--- |:--- |:--- | +`leader_alias` | The name of the cross-cluster connection. You define this alias when you [set up a cross-cluster connection]({{site.url}}{{site.baseurl}}/replication-plugin/get-started/#set-up-a-cross-cluster-connection). | `string` | Yes +`name` | A name for the auto-follow pattern. | `string` | Yes +`pattern` | An array of index patterns to match against indexes in the specified leader cluster. Supports wildcard characters. For example, `leader-*`. | `string` | Yes +`use_roles` | The roles to use for all subsequent backend replication tasks between the indexes. Specify a `leader_cluster_role` and `follower_cluster_role`. See [Map the leader and follower cluster roles]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#map-the-leader-and-follower-cluster-roles). | `string` | If security plugin is enabled + +#### Sample response + +```json +{ + "acknowledged": true +} +``` + +## Delete replication rule +Introduced 1.1 +{: .label .label-purple } + +Deletes the specified replication rule. This operation prevents any new indexes from being replicated but does not stop existing replication that the rule has already initiated. Replicated indexes remain read-only until you stop replication. + +Send this request to the follower cluster. + +#### Request + +```json +DELETE /_plugins/_replication/_autofollow +{ + "leader_alias" : "", + "name": "", +} +``` + +Specify the following options: + +Options | Description | Type | Required +:--- | :--- |:--- |:--- | +`leader_alias` | The name of the cross-cluster connection. You define this alias when you [set up a cross-cluster connection]({{site.url}}{{site.baseurl}}/replication-plugin/get-started/#set-up-a-cross-cluster-connection). | `string` | Yes +`name` | The name of the pattern. | `string` | Yes + +#### Sample response + +```json +{ + "acknowledged": true +} +``` diff --git a/_tuning-your-cluster/replication-plugin/auto-follow.md b/_tuning-your-cluster/replication-plugin/auto-follow.md new file mode 100644 index 00000000000..d3103df91ea --- /dev/null +++ b/_tuning-your-cluster/replication-plugin/auto-follow.md @@ -0,0 +1,106 @@ +--- +layout: default +title: Auto-follow +nav_order: 20 +parent: Cross-cluster replication +redirect_from: + - /replication-plugin/auto-follow/ +--- + +# Auto-follow for cross-cluster replication + +Auto-follow lets you automatically replicate indexes created on the leader cluster based on matching patterns. When you create an index on the leader cluster with a name that matches a specified pattern (for example, `index-01*`), a corresponding follower index is automatically created on the follower cluster. + +You can configure multiple replication rules for a single cluster. The patterns currently only support wildcard matching. + +## Prerequisites + +You need to [set up a cross-cluster connection]({{site.url}}{{site.baseurl}}/replication-plugin/get-started/#set-up-a-cross-cluster-connection) between two clusters before you can enable auto-follow. + +## Permissions + +If the security plugin is enabled, make sure that non-admin users are mapped to the appropriate permissions so they can perform replication actions. For index and cluster-level permissions requirements, see [Cross-cluster replication permissions]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/). + +## Get started with auto-follow + +Replication rules are a collection of patterns that you create against a single follower cluster. When you create a replication rule, it starts by automatically replicating any *existing* indexes that match the pattern. It will then continue to replicate any *new* indexes that you create that match the pattern. + +Create a replication rule on the follower cluster: + +```bash +curl -XPOST -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/_autofollow?pretty' -d ' +{ + "leader_alias" : "my-connection-alias", + "name": "my-replication-rule", + "pattern": "movies*", + "use_roles":{ + "leader_cluster_role": "all_access", + "follower_cluster_role": "all_access" + } +}' +``` + +If the security plugin is disabled, you can leave out the `use_roles` parameter. If it's enabled, however, you need to specify the leader and follower cluster roles that OpenSearch uses to authenticate requests. This example uses `all_access` for simplicity, but we recommend creating a replication user on each cluster and [mapping it accordingly]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#map-the-leader-and-follower-cluster-roles). +{: .tip } + +To test the rule, create a matching index on the leader cluster: + +```bash +curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9201/movies-0001?pretty' +``` + +And confirm its replica shows up on the follower cluster: + +```bash +curl -XGET -u 'admin:admin' -k 'https://localhost:9200/_cat/indices?v' +``` + +It might take several seconds for the index to appear. + +```bash +health status index uuid pri rep docs.count docs.deleted store.size pri.store.size +yellow open movies-0001 kHOxYYHxRMeszLjTD9rvSQ 1 1 0 0 208b 208b +``` + +## Retrieve replication rules + +To retrieve a list of existing replication rules that are configured on a cluster, send the following request: + +```bash +curl -XGET -u 'admin:admin' -k 'https://localhost:9200/_plugins/_replication/autofollow_stats' + +{ + "num_success_start_replication": 1, + "num_failed_start_replication": 0, + "num_failed_leader_calls": 0, + "failed_indices":[ + + ], + "autofollow_stats":[ + { + "name":"my-replication-rule", + "pattern":"movies*", + "num_success_start_replication": 1, + "num_failed_start_replication": 0, + "num_failed_leader_calls": 0, + "failed_indices":[ + + ] + } + ] +} +``` + +## Delete a replication rule + +To delete a replication rule, send the following request to the follower cluster: + +```bash +curl -XDELETE -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/_autofollow?pretty' -d ' +{ + "leader_alias" : "my-conection-alias", + "name": "my-replication-rule" +}' +``` + +When you delete a replication rule, OpenSearch stops replicating *new* indexes that match the pattern, but existing indexes that the rule previously created remain read-only and continue to replicate. If you need to stop existing replication activity and open the indexes up for writes, use the [stop replication API operation]({{site.url}}{{site.baseurl}}/replication-plugin/api/#stop-replication). \ No newline at end of file diff --git a/_tuning-your-cluster/replication-plugin/getting-started.md b/_tuning-your-cluster/replication-plugin/getting-started.md new file mode 100644 index 00000000000..05f515f3c72 --- /dev/null +++ b/_tuning-your-cluster/replication-plugin/getting-started.md @@ -0,0 +1,295 @@ +--- +layout: default +title: Getting started +nav_order: 15 +parent: Cross-cluster replication +redirect_from: + - /replication-plugin/get-started/ +--- + +# Getting started with cross-cluster replication + +With cross-cluster replication, you index data to a leader index, and OpenSearch replicates that data to one or more read-only follower indexes. All subsequent operations on the leader are replicated on the follower, such as creating, updating, or deleting documents. + +## Prerequisites + +Cross-cluster replication has the following prerequisites: +- Both the leader and follower cluster must have the replication plugin installed. +- If you've overridden `node.roles` in `opensearch.yml` on the follower cluster, make sure it also includes the `remote_cluster_client` role: + + ```yaml + node.roles: [, remote_cluster_client] + ``` + +## Permissions + +Make sure the security plugin is either enabled on both clusters or disabled on both clusters. If you disabled the security plugin, you can skip this section. However, we strongly recommend enabling the security plugin in production scenarios. + +If the security plugin is enabled, make sure that non-admin users are mapped to the appropriate permissions so they can perform replication actions. For index and cluster-level permissions requirements, see [Cross-cluster replication permissions]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/). + +In addition, verify and add the distinguished names (DNs) of each follower cluster node on the leader cluster to allow connections from the followers to the leader. + +First, get the node's DN from each follower cluster: + + ```bash +curl -XGET -k -u 'admin:admin' 'https://localhost:9200/_opendistro/_security/api/ssl/certs?pretty' + +{ + "transport_certificates_list": [ + { + "issuer_dn" : "CN=Test,OU=Server CA 1B,O=Test,C=US", + "subject_dn" : "CN=follower.test.com", # To be added under leader's nodes_dn configuration + "not_before" : "2021-11-12T00:00:00Z", + "not_after" : "2022-12-11T23:59:59Z" + } + ] +} + ``` + +Then verify that it's part of the leader cluster configuration in `opensearch.yml`. Otherwise, add it under the following setting: + + ```yaml +plugins.security.nodes_dn: + - "CN=*.leader.com, OU=SSL, O=Test, L=Test, C=DE" # Already part of the configuration + - "CN=follower.test.com" # From the above response from follower + ``` +## Example setup + +To start two single-node clusters on the same network, save this sample file as `docker-compose.yml` and run `docker-compose up`: + +```yml +version: '3' +services: + replication-node1: + image: opensearchproject/opensearch:{{site.opensearch_version}} + container_name: replication-node1 + environment: + - cluster.name=leader-cluster + - discovery.type=single-node + - bootstrap.memory_lock=true + - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" + ulimits: + memlock: + soft: -1 + hard: -1 + volumes: + - opensearch-data2:/usr/share/opensearch/data + ports: + - 9201:9200 + - 9700:9600 # required for Performance Analyzer + networks: + - opensearch-net + replication-node2: + image: opensearchproject/opensearch:{{site.opensearch_version}} + container_name: replication-node2 + environment: + - cluster.name=follower-cluster + - discovery.type=single-node + - bootstrap.memory_lock=true + - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" + ulimits: + memlock: + soft: -1 + hard: -1 + volumes: + - opensearch-data1:/usr/share/opensearch/data + ports: + - 9200:9200 + - 9600:9600 # required for Performance Analyzer + networks: + - opensearch-net + +volumes: + opensearch-data1: + opensearch-data2: + +networks: + opensearch-net: +``` + +After the clusters start, verify the names of each: + +```bash +curl -XGET -u 'admin:admin' -k 'https://localhost:9201' +{ + "cluster_name" : "leader-cluster", + ... +} + +curl -XGET -u 'admin:admin' -k 'https://localhost:9200' +{ + "cluster_name" : "follower-cluster", + ... +} +``` + +For this example, use port 9201 (`replication-node1`) as the leader and port 9200 (`replication-node2`) as the follower cluster. + +To get the IP address for the leader cluster, first identify its container ID: + +```bash +docker ps +CONTAINER ID IMAGE PORTS NAMES +3b8cdc698be5 opensearchproject/opensearch:{{site.opensearch_version}} 0.0.0.0:9200->9200/tcp, 0.0.0.0:9600->9600/tcp, 9300/tcp replication-node2 +731f5e8b0f4b opensearchproject/opensearch:{{site.opensearch_version}} 9300/tcp, 0.0.0.0:9201->9200/tcp, 0.0.0.0:9700->9600/tcp replication-node1 +``` + +Then get that container's IP address: + +```bash +docker inspect --format='{% raw %}{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}{% endraw %}' 731f5e8b0f4b +172.22.0.3 +``` + +## Set up a cross-cluster connection + +Cross-cluster replication follows a "pull" model, so most changes occur on the follower cluster, not the leader cluster. + +On the follower cluster, add the IP address (with port 9300) for each seed node. Because this is a single-node cluster, you only have one seed node. Provide a descriptive name for the connection, which you'll use in the request to start replication: + +```bash +curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_cluster/settings?pretty' -d ' +{ + "persistent": { + "cluster": { + "remote": { + "my-connection-alias": { + "seeds": ["172.22.0.3:9300"] + } + } + } + } +}' +``` + +## Start replication + +To get started, create an index called `leader-01` on the leader cluster: + +```bash +curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9201/leader-01?pretty' +``` + +Then start replication from the follower cluster. In the request body, provide the connection name and leader index that you want to replicate, along with the security roles you want to use: + +```bash +curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_start?pretty' -d ' +{ + "leader_alias": "my-connection-alias", + "leader_index": "leader-01", + "use_roles":{ + "leader_cluster_role": "all_access", + "follower_cluster_role": "all_access" + } +}' +``` + +If the security plugin is disabled, omit the `use_roles` parameter. If it's enabled, however, you must specify the leader and follower cluster roles that OpenSearch will use to authenticate the request. This example uses `all_access` for simplicity, but we recommend creating a replication user on each cluster and [mapping it accordingly]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#map-the-leader-and-follower-cluster-roles). +{: .tip } + +This command creates an identical read-only index named `follower-01` on the follower cluster that continuously stays updated with changes to the `leader-01` index on the leader cluster. Starting replication creates a follower index from scratch -- you can't convert an existing index to a follower index. + +## Confirm replication + +After replication starts, get the status: + +```bash +curl -XGET -k -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_status?pretty' + +{ + "status" : "SYNCING", + "reason" : "User initiated", + "leader_alias" : "my-connection-alias", + "leader_index" : "leader-01", + "follower_index" : "follower-01", + "syncing_details" : { + "leader_checkpoint" : -1, + "follower_checkpoint" : -1, + "seq_no" : 0 + } +} +``` + +Possible statuses are `SYNCING`, `BOOTSTRAPPING`, `PAUSED`, and `REPLICATION NOT IN PROGRESS`. + +The leader and follower checkpoint values begin as negative numbers and reflect the shard count (-1 for one shard, -5 for five shards, and so on). The values increment with each change and illustrate how many updates the follower is behind the leader. If the indexes are fully synced, the values are the same. + +To confirm that replication is actually happening, add a document to the leader index: + +```bash +curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9201/leader-01/_doc/1?pretty' -d '{"The Shining": "Stephen King"}' +``` + +Then validate the replicated content on the follower index: + +```bash +curl -XGET -k -u 'admin:admin' 'https://localhost:9200/follower-01/_search?pretty' + +{ + ... + "hits": [{ + "_index": "follower-01", + "_id": "1", + "_score": 1.0, + "_source": { + "The Shining": "Stephen King" + } + }] +} +``` + +## Pause and resume replication + +You can temporarily pause replication of an index if you need to remediate issues or reduce load on the leader cluster: + +```bash +curl -XPOST -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_pause?pretty' -d '{}' +``` + +To confirm that replication is paused, get the status: + +```bash +curl -XGET -k -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_status?pretty' + +{ + "status" : "PAUSED", + "reason" : "User initiated", + "leader_alias" : "my-connection-alias", + "leader_index" : "leader-01", + "follower_index" : "follower-01" +} +``` + +When you're done making changes, resume replication: + +```bash +curl -XPOST -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_resume?pretty' -d '{}' +``` + +When replication resumes, the follower index picks up any changes that were made to the leader index while replication was paused. + +Note that you can't resume replication after it's been paused for more than 12 hours. You must [stop replication]({{site.url}}{{site.baseurl}}/replication-plugin/api/#stop-replication), delete the follower index, and restart replication of the leader. + +## Stop replication + +When you no longer need to replicate an index, terminate replication from the follower cluster: + +```bash +curl -XPOST -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_stop?pretty' -d '{}' +``` + +When you stop replication, the follower index un-follows the leader and becomes a standard index that you can write to. You can't restart replication after stopping it. + +Get the status to confirm that the index is no longer being replicated: + +```bash +curl -XGET -k -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_status?pretty' + +{ + "status" : "REPLICATION NOT IN PROGRESS" +} +``` + +You can further confirm that replication is stopped by making modifications to the leader index and confirming they don't show up on the follower index. + + diff --git a/_tuning-your-cluster/replication-plugin/index.md b/_tuning-your-cluster/replication-plugin/index.md new file mode 100644 index 00000000000..46eede78e91 --- /dev/null +++ b/_tuning-your-cluster/replication-plugin/index.md @@ -0,0 +1,24 @@ +--- +layout: default +title: Cross-cluster replication +nav_order: 12 +has_children: true +redirect_from: + - /replication-plugin/ + - /replication-plugin/index/ +--- + +# Cross-cluster replication + +The cross-cluster replication plugin lets you replicate indexes, mappings, and metadata from one OpenSearch cluster to another. Cross-cluster replication has the following benefits: +- By replicating your indexes, you ensure that you can continue to handle search requests if there's an outage. +- Replicating data across geographically distant data centers minimizes the distance between the data and the application server. This reduces expensive latencies. +- You can replicate data from multiple smaller clusters to a centralized reporting cluster, which is useful when it's inefficient to query across a large network. + +Replication follows an active-passive model where the follower index (where the data is replicated) pulls data from the leader (remote) index. + +The replication plugin supports replication of indexes using wildcard pattern matching and provides commands to pause, resume, and stop replication. Once replication starts on an index, it initiates persistent background tasks on all primary shards on the follower cluster, which continuously poll corresponding shards from the leader cluster for updates. + +You can use the replication plugin with the security plugin to encrypt cross-cluster traffic with node-to-node encryption and control access to replication activities. + +To start, see [Get started with cross-cluster replication]({{site.url}}{{site.baseurl}}/replication-plugin/get-started/). diff --git a/_tuning-your-cluster/replication-plugin/permissions.md b/_tuning-your-cluster/replication-plugin/permissions.md new file mode 100644 index 00000000000..9c8a8b10b64 --- /dev/null +++ b/_tuning-your-cluster/replication-plugin/permissions.md @@ -0,0 +1,81 @@ +--- +layout: default +title: Replication security +nav_order: 30 +parent: Cross-cluster replication +redirect_from: + - /replication-plugin/permissions/ +--- + +# Cross-cluster replication security + +You can use the [security plugin]({{site.url}}{{site.baseurl}}/security/index/) with cross-cluster replication to limit users to certain actions. For example, you might want certain users to only perform replication activity on the leader or follower cluster. + +Because cross-cluster replication involves multiple clusters, it's possible that clusters might have different security configurations. The following configurations are supported: + +- Security plugin fully enabled on both clusters +- Security plugin enabled only for TLS on both clusters (`plugins.security.ssl_only`) +- Security plugin absent or disabled on both clusters (not recommended) + +Enable node-to-node encryption on both the leader and the follower cluster to ensure that replication traffic between the clusters is encrypted. + +## Basic permissions + +In order for non-admin users to perform replication activities, they must be mapped to the appropriate permissions. + +The security plugin has two built-in roles that cover most replication use cases: `cross_cluster_replication_leader_full_access`, which provides replication permissions on the leader cluster, and `cross_cluster_replication_follower_full_access`, which provides replication permissions on the follower cluster. For descriptions of each, see [Predefined roles]({{site.url}}{{site.baseurl}}/security/access-control/users-roles#predefined-roles). + +If you don't want to use the default roles, you can combine individual replication [permissions]({{site.url}}{{site.baseurl}}/replication-plugin/permissions/#replication-permissions) to meet your needs. Most permissions correspond to specific REST API operations. For example, the `indices:admin/plugins/replication/index/pause` permission lets you pause replication. + +## Map the leader and follower cluster roles + +The [start replication]({{site.url}}{{site.baseurl}}/replication-plugin/api/#start-replication) and [create replication rule]({{site.url}}{{site.baseurl}}/replication-plugin/api/#create-replication-rule) operations are special cases. They involve background processes on the leader and follower clusters that must be associated with roles. When you perform one of these actions, you must explicitly pass the `leader_cluster_role` and +`follower_cluster_role` in the request, which OpenSearch then uses in all backend replication tasks. + +To enable non-admins to start replication and create replication rules, create an identical user on each cluster (for example, `replication_user`) and map them to the `cross_cluster_replication_leader_full_access` role on the remote cluster and `cross_cluster_replication_follower_full_access` on the follower cluster. For instructions, see [Map users to roles]({{site.url}}{{site.baseurl}}/security/access-control/users-roles/#map-users-to-roles). + +Then add those roles to the request, and sign it with the appropriate credentials: + +```bash +curl -XPUT -k -H 'Content-Type: application/json' -u 'replication_user:password' 'https://localhost:9200/_plugins/_replication/follower-01/_start?pretty' -d ' +{ + "leader_alias": "leader-cluster", + "leader_index": "leader-01", + "use_roles":{ + "leader_cluster_role": "cross_cluster_replication_leader_full_access", + "follower_cluster_role": "cross_cluster_replication_follower_full_access" + } +}' +``` + +You can create your own, custom leader and follower cluster roles using individual permissions, but we recommend using the default roles, which are a good fit for most use cases. + +## Replication permissions + +The following sections list the available index and cluster-level permissions for cross-cluster replication. + +### Follower cluster + +The security plugin supports these permissions for the follower cluster: + +``` +indices:admin/plugins/replication/index/setup/validate +indices:admin/plugins/replication/index/start +indices:admin/plugins/replication/index/pause +indices:admin/plugins/replication/index/resume +indices:admin/plugins/replication/index/stop +indices:admin/plugins/replication/index/update +indices:admin/plugins/replication/index/status_check +indices:data/write/plugins/replication/changes +cluster:admin/plugins/replication/autofollow/update +``` + +### Leader cluster + +The security plugin supports these permissions for the leader cluster: + +``` +indices:admin/plugins/replication/validate +indices:data/read/plugins/replication/file_chunk +indices:data/read/plugins/replication/changes +``` diff --git a/_tuning-your-cluster/replication-plugin/settings.md b/_tuning-your-cluster/replication-plugin/settings.md new file mode 100644 index 00000000000..4a5d266d087 --- /dev/null +++ b/_tuning-your-cluster/replication-plugin/settings.md @@ -0,0 +1,39 @@ +--- +layout: default +title: Replication settings +nav_order: 40 +parent: Cross-cluster replication +redirect_from: + - /replication-plugin/settings/ +--- + +# Replication settings + +The replication plugin adds several settings to the standard OpenSearch cluster settings. +The settings are dynamic, so you can change the default behavior of the plugin without restarting your cluster. +You can mark settings as `persistent` or `transient`. + +For example, to update how often the follower cluster polls the leader cluster for updates: + +```json +PUT _cluster/settings +{ + "persistent": { + "plugins.replication.follower.metadata_sync_interval": "30s" + } +} +``` + +These settings manage the resources consumed by remote recoveries. We don’t recommend changing these settings; the defaults should work well for most use cases. + +Setting | Default | Description +:--- | :--- | :--- +`plugins.replication.follower.index.recovery.chunk_size` | 10 MB | The chunk size requested by the follower cluster during file transfer. Specify the chunk size as a value and unit, for example, 10 MB, 5 KB. See [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/). +`plugins.replication.follower.index.recovery.max_concurrent_file_chunks` | 4 | The number of file chunk requests that can be sent in parallel for each recovery. +`plugins.replication.follower.index.ops_batch_size` | 5000 | The number of operations that can be fetched at a time during the syncing phase of replication. +`plugins.replication.follower.concurrent_readers_per_shard` | 2 | The number of concurrent requests from the follower cluster per shard during the syncing phase of replication. +`plugins.replication.autofollow.fetch_poll_interval` | 30s | How often auto-follow tasks poll the leader cluster for new matching indexes. +`plugins.replication.follower.metadata_sync_interval` | 60s | How often the follower cluster polls the leader cluster for updated index metadata. +`plugins.replication.translog.retention_lease.pruning.enabled` | true | If enabled, prunes the translog based on retention leases on the leader index. +`plugins.replication.translog.retention_size` | 512 MB | Controls the size of the translog on the leader index. + diff --git a/about.md b/about.md index 1b6a45fdb0e..13842071b23 100644 --- a/about.md +++ b/about.md @@ -17,7 +17,7 @@ OpenSearch is a distributed search and analytics engine based on [Apache Lucene] Unsurprisingly, people often use search engines like OpenSearch as the backend for a search application---think [Wikipedia](https://en.wikipedia.org/wiki/Wikipedia:FAQ/Technical#What_software_is_used_to_run_Wikipedia?) or an online store. It offers excellent performance and can scale up and down as the needs of the application grow or shrink. -An equally popular, but less obvious use case is log analytics, in which you take the logs from an application, feed them into OpenSearch, and use the rich search and visualization functionality to identify issues. For example, a malfunctioning web server might throw a 500 error 0.5% of the time, which can be hard to notice unless you have a real-time graph of all HTTP status codes that the server has thrown in the past four hours. You can use [OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/) to build these sorts of visualizations from data in OpenSearch. +An equally popular, but less obvious use case is log analytics, in which you take the logs from an application, feed them into OpenSearch, and use the rich search and visualization functionality to identify issues. For example, a malfunctioning web server might throw a 500 error 0.5% of the time, which can be hard to notice unless you have a real-time graph of all HTTP status codes that the server has thrown in the past four hours. You can use [OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/dashboards/index/) to build these sorts of visualizations from data in OpenSearch. ## Clusters and nodes