From 30586d3a9d8bb79b48675af2bec5d973387e25d0 Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 16 Nov 2022 15:38:21 -0500 Subject: [PATCH 01/32] Add Peer forwarder to doc site repo. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 156 ++++++++++++++++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 _data-prepper/peer-forwarder.md diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md new file mode 100644 index 00000000000..13e3421e0e8 --- /dev/null +++ b/_data-prepper/peer-forwarder.md @@ -0,0 +1,156 @@ +--- +layout: default +title: Configuration reference +nav_order: 30 +--- + +## Peer Forwarder +An HTTP service which performs peer forwarding of `Event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. + +Peer Forwarder groups events based on the identification keys provided the processors. +For `service_map_stateful` and `otel_trace_raw` it's `traceId` by default and can not be configured. +It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). + +--- + +Presently peer discovery is provided by either a static list or by a DNS record lookup or AWS Cloudmap. + +### Static discovery mode +Static discover mode allows Data Prepper node to discover nodes using a list of IP addresses or domain names. +```yaml +peer_forwarder: + discovery_mode: static + static_endpoints: ["data-prepper1", "data-prepper2"] +``` + +### DNS lookup discovery mode +We recommend using DNS discovery over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. +This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) which indicates a list of IP addresses of a given domain. +```yaml +peer_forwarder: + discovery_mode: dns + domain_name: "data-prepper-cluster.my-domain.net" +``` + +### AWS Cloud Map discovery mode + +[AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. + +Peer forwarder can use the API-based service discovery. To support this you must have an existing +namespace configured for API instance discovery. You can create a new one following the instructions +provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). + +Your Data Prepper configuration needs to include: +* `aws_cloud_map_namespace_name` - Set to your Cloud Map Namespace name +* `aws_cloud_map_service_name` - Set to the service name within your specified Namespace +* `aws_region` - The AWS region where your namespace exists. +* `discovery_mode` - Set to `aws_cloud_map` + +Your Data Prepper configuration can optionally include: +* `aws_cloud_map_query_parameters` - Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. + +Example configuration: + +```yaml +peer_forwarder: + discovery_mode: aws_cloud_map + aws_cloud_map_namespace_name: "my-namespace" + aws_cloud_map_service_name: "data-prepper-cluster" + aws_cloud_map_query_parameters: + instance_type: "r5.xlarge" + aws_region: "us-east-1" +``` + +The Data Prepper must also be running with the necessary permissions. The following +IAM policy shows the necessary permissions. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "CloudMapPeerForwarder", + "Effect": "Allow", + "Action": "servicediscovery:DiscoverInstances", + "Resource": "*" + } + ] +} +``` +--- +## Configuration + +* `port`(Optional): An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`. +* `request_timeout`(Optional): Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. +* `server_thread_count`(Optional): An `int` representing number of threads used by Peer Forwarder server. Defaults to `200`. +* `client_thread_count`(Optional): An `int` representing number of threads used by Peer Forwarder client. Defaults to `200`. +* `maxConnectionCount`(Optional): An `int` representing maximum number of open connections for Peer Forwarder server. Default value is `500`. +* `discovery_mode`(Optional): A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node` which processes events locally. +* `static_endpoints`(Optional): A `list` containing endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. +* `domain_name`(Optional): A `String` representing single domain name to query DNS against. Typically, used by creating multiple [DNS A Records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. +* `aws_cloud_map_namespace_name`(Optional) - A `String` representing the Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. +* `aws_cloud_map_service_name`(Optional) - A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. +* `aws_cloud_map_query_parameters`(Optional): A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. +* `buffer_size`(Optional): An `int` representing max number of unchecked records the buffer accepts (num of unchecked records = num of records written into the buffer + num of in-flight records not yet checked by the Checkpointing API). Default is `512`. +* `batch_size`(Optional): An `int` representing max number of records the buffer returns on read. Default is `48`. +* `aws_region`(Optional) : A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. +* `drain_timeout`(Optional) : A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. + +### SSL +The SSL configuration for setting up trust manager for peer forwarding client to connect to other Data Prepper instances. + +* `ssl`(Optional) : A `boolean` that enables TLS/SSL. Default value is `true`. +* `ssl_certificate_file`(Optional) : A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). +* `ssl_key_file`(Optional) : A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). +* `ssl_insecure_disable_verification`(Optional) : A `boolean` that disables the verification of server's TLS certificate chain. Default value is `false`. +* `ssl_fingerprint_verification_only`(Optional) : A `boolean` that disables the verification of server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. +* `use_acm_certificate_for_ssl`(Optional) : A `boolean` that enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. +* `acm_certificate_arn`(Optional) : A `String` represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. +* `acm_private_key_password`(Optional) : A `String` that represents the ACM private key password which that will be used to decrypt the private key. If it's not provided, a random password will be generated. +* `acm_certificate_timeout_millis`(Optional) : An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. +* `aws_region`(Optional) : A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. + +```yaml +peer_forwarder: + ssl: true + ssl_certificate_file: "" + ssl_key_file: "" +``` + +### Authentication + +* `authentication`(Optional) : A `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. Default value is `unauthenticated`. +```yaml +peer_forwarder: + authentication: + mutual_tls: +``` + +## Metrics + +Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder` + +### Timer + +- `requestForwardingLatency`: measures latency of forwarding requests by peer forwarder client. +- `requestProcessingLatency`: measures latency of processing requests by peer forwarder server. + +### Counter + +- `requests`: measures total number of forwarded requests. +- `requestsFailed`: measures total number of failed requests. Requests with HTTP response code other than `200`. +- `requestsSuccessful`: measures total number of successful requests. Requests with HTTP response code `200`. +- `requestsTooLarge`: measures total number of requests which are too large to be written to peer forwarder buffer. Requests with HTTP response code `413`. +- `requestTimeouts`: measures total number of requests which timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. +- `requestsUnprocessable`: measures total number of requests which failed due to unprocessable entity. Requests with HTTP response code `422`. +- `badRequests`: measures total number of requests with bad request format. Requests with HTTP response code `400`. +- `recordsSuccessfullyForwarded`: measures total number of forwarded records successfully. +- `recordsFailedForwarding`: measures total number of records failed to be forwarded. +- `recordsToBeForwarded`: measures total number of records to be forwarded. +- `recordsToBeProcessedLocally`: measures total number of records to be processed locally. +- `recordsActuallyProcessedLocally`: measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. +- `recordsReceivedFromPeers`: measures total number of records received from remote peers. + +### Gauge + +- `peerEndpoints`: measures number of dynamically discovered peer data-prepper endpoints. For `static` mode, the size is fixed. From 73fab9fe7c3391de21e008cf1d062004d1ef48b9 Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 16 Nov 2022 15:46:16 -0500 Subject: [PATCH 02/32] Minor updates. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 13e3421e0e8..1d6c0a7d8fa 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -1,7 +1,7 @@ --- layout: default title: Configuration reference -nav_order: 30 +nav_order: 12 --- ## Peer Forwarder From f8feef6743a6edca4a0c68f03a63739a050dee1a Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 16 Nov 2022 15:47:37 -0500 Subject: [PATCH 03/32] Minor updates to header section. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 1d6c0a7d8fa..17501b976bd 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -1,6 +1,6 @@ --- layout: default -title: Configuration reference +title: Peer forwarder nav_order: 12 --- From 24d4a5e3980e37251c149f81fecf794e4d65aa68 Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 21 Nov 2022 15:16:46 -0500 Subject: [PATCH 04/32] Minor copyedits and heading adjustements. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 41 +++++++++++++++------------------ 1 file changed, 19 insertions(+), 22 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 17501b976bd..c182ab93297 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -4,18 +4,17 @@ title: Peer forwarder nav_order: 12 --- -## Peer Forwarder -An HTTP service which performs peer forwarding of `Event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. +Peer Forwarder is an HTTP service which performs peer forwarding of `Event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided the processors. -For `service_map_stateful` and `otel_trace_raw` it's `traceId` by default and can not be configured. -It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw` it's `traceId` by default and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). --- -Presently peer discovery is provided by either a static list or by a DNS record lookup or AWS Cloudmap. +Right now, peer discovery is provided by either a static list or by a DNS record lookup or AWS Cloudmap. -### Static discovery mode +## Discovery modes + +### Static discovery Static discover mode allows Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml peer_forwarder: @@ -23,22 +22,20 @@ peer_forwarder: static_endpoints: ["data-prepper1", "data-prepper2"] ``` -### DNS lookup discovery mode -We recommend using DNS discovery over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. -This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) which indicates a list of IP addresses of a given domain. +### DNS lookup discovery +We recommend using DNS discovery over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) which indicates a list of IP addresses of a given domain. + ```yaml peer_forwarder: discovery_mode: dns domain_name: "data-prepper-cluster.my-domain.net" ``` -### AWS Cloud Map discovery mode +### AWS Cloud Map discovery [AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. -Peer forwarder can use the API-based service discovery. To support this you must have an existing -namespace configured for API instance discovery. You can create a new one following the instructions -provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). +Peer forwarder can use the API-based service discovery. To support this you must have an existing namespace configured for API instance discovery. You can create a new one following the instructions provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). Your Data Prepper configuration needs to include: * `aws_cloud_map_namespace_name` - Set to your Cloud Map Namespace name @@ -49,7 +46,7 @@ Your Data Prepper configuration needs to include: Your Data Prepper configuration can optionally include: * `aws_cloud_map_query_parameters` - Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. -Example configuration: +#### Example configuration ```yaml peer_forwarder: @@ -78,7 +75,7 @@ IAM policy shows the necessary permissions. } ``` --- -## Configuration +# Configuration * `port`(Optional): An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`. * `request_timeout`(Optional): Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. @@ -96,7 +93,7 @@ IAM policy shows the necessary permissions. * `aws_region`(Optional) : A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. * `drain_timeout`(Optional) : A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. -### SSL +## SSL The SSL configuration for setting up trust manager for peer forwarding client to connect to other Data Prepper instances. * `ssl`(Optional) : A `boolean` that enables TLS/SSL. Default value is `true`. @@ -117,7 +114,7 @@ peer_forwarder: ssl_key_file: "" ``` -### Authentication +## Authentication * `authentication`(Optional) : A `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. Default value is `unauthenticated`. ```yaml @@ -126,16 +123,16 @@ peer_forwarder: mutual_tls: ``` -## Metrics +# Metrics Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder` -### Timer +## Timer - `requestForwardingLatency`: measures latency of forwarding requests by peer forwarder client. - `requestProcessingLatency`: measures latency of processing requests by peer forwarder server. -### Counter +## Counter - `requests`: measures total number of forwarded requests. - `requestsFailed`: measures total number of failed requests. Requests with HTTP response code other than `200`. @@ -151,6 +148,6 @@ Core Peer Forwarder introduces the following custom metrics and all the metrics - `recordsActuallyProcessedLocally`: measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. - `recordsReceivedFromPeers`: measures total number of records received from remote peers. -### Gauge +## Gauge - `peerEndpoints`: measures number of dynamically discovered peer data-prepper endpoints. For `static` mode, the size is fixed. From 7fb9b33985d87a1a1e2256f9ad64b2fe29f25183 Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 21 Nov 2022 15:27:30 -0500 Subject: [PATCH 05/32] Minor copyedits and heading updates. Additional of Optional section to simplify wording. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 54 ++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 25 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index c182ab93297..8d78d48b7d0 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -77,35 +77,39 @@ IAM policy shows the necessary permissions. --- # Configuration -* `port`(Optional): An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`. -* `request_timeout`(Optional): Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. -* `server_thread_count`(Optional): An `int` representing number of threads used by Peer Forwarder server. Defaults to `200`. -* `client_thread_count`(Optional): An `int` representing number of threads used by Peer Forwarder client. Defaults to `200`. -* `maxConnectionCount`(Optional): An `int` representing maximum number of open connections for Peer Forwarder server. Default value is `500`. -* `discovery_mode`(Optional): A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node` which processes events locally. -* `static_endpoints`(Optional): A `list` containing endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. -* `domain_name`(Optional): A `String` representing single domain name to query DNS against. Typically, used by creating multiple [DNS A Records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. -* `aws_cloud_map_namespace_name`(Optional) - A `String` representing the Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. -* `aws_cloud_map_service_name`(Optional) - A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. -* `aws_cloud_map_query_parameters`(Optional): A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. -* `buffer_size`(Optional): An `int` representing max number of unchecked records the buffer accepts (num of unchecked records = num of records written into the buffer + num of in-flight records not yet checked by the Checkpointing API). Default is `512`. -* `batch_size`(Optional): An `int` representing max number of records the buffer returns on read. Default is `48`. -* `aws_region`(Optional) : A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. -* `drain_timeout`(Optional) : A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. +## Optional + +* `port`: An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`. +* `request_timeout`: Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. +* `server_thread_count`: An `int` representing number of threads used by Peer Forwarder server. Defaults to `200`. +* `client_thread_count`: An `int` representing number of threads used by Peer Forwarder client. Defaults to `200`. +* `maxConnectionCount`: An `int` representing maximum number of open connections for Peer Forwarder server. Default value is `500`. +* `discovery_mode`: A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node` which processes events locally. +* `static_endpoints`: A `list` containing endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. +* `domain_name`: A `String` representing single domain name to query DNS against. Typically, used by creating multiple [DNS A Records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. +* `aws_cloud_map_namespace_name`: A `String` representing the Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. +* `aws_cloud_map_service_name`: A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. +* `aws_cloud_map_query_parameters`: A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. +* `buffer_size`: An `int` representing max number of unchecked records the buffer accepts (num of unchecked records = num of records written into the buffer + num of in-flight records not yet checked by the Checkpointing API). Default is `512`. +* `batch_size`: An `int` representing max number of records the buffer returns on read. Default is `48`. +* `aws_region`: A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. +* `drain_timeout`: A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. ## SSL The SSL configuration for setting up trust manager for peer forwarding client to connect to other Data Prepper instances. -* `ssl`(Optional) : A `boolean` that enables TLS/SSL. Default value is `true`. -* `ssl_certificate_file`(Optional) : A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). -* `ssl_key_file`(Optional) : A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). -* `ssl_insecure_disable_verification`(Optional) : A `boolean` that disables the verification of server's TLS certificate chain. Default value is `false`. -* `ssl_fingerprint_verification_only`(Optional) : A `boolean` that disables the verification of server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. -* `use_acm_certificate_for_ssl`(Optional) : A `boolean` that enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. -* `acm_certificate_arn`(Optional) : A `String` represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. -* `acm_private_key_password`(Optional) : A `String` that represents the ACM private key password which that will be used to decrypt the private key. If it's not provided, a random password will be generated. -* `acm_certificate_timeout_millis`(Optional) : An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. -* `aws_region`(Optional) : A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. +### Optional + +* `ssl`: A `boolean` that enables TLS/SSL. Default value is `true`. +* `ssl_certificate_file`: A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). +* `ssl_key_file`: A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). +* `ssl_insecure_disable_verification`: A `boolean` that disables the verification of server's TLS certificate chain. Default value is `false`. +* `ssl_fingerprint_verification_only`: A `boolean` that disables the verification of server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. +* `use_acm_certificate_for_ssl`: A `boolean` that enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. +* `acm_certificate_arn`: A `String` represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. +* `acm_private_key_password`: A `String` that represents the ACM private key password which that will be used to decrypt the private key. If it's not provided, a random password will be generated. +* `acm_certificate_timeout_millis`: An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. +* `aws_region`: A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. ```yaml peer_forwarder: From e709ce91c7747e4f3ef9118cd9aadb18e51b362d Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 22 Nov 2022 13:12:03 -0500 Subject: [PATCH 06/32] Minor copyedits and heading adjustments. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 8d78d48b7d0..27dde1e0485 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -12,9 +12,9 @@ Peer Forwarder groups events based on the identification keys provided the proce Right now, peer discovery is provided by either a static list or by a DNS record lookup or AWS Cloudmap. -## Discovery modes +# Discovery modes -### Static discovery +## Static Static discover mode allows Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml peer_forwarder: @@ -22,7 +22,7 @@ peer_forwarder: static_endpoints: ["data-prepper1", "data-prepper2"] ``` -### DNS lookup discovery +## DNS lookup We recommend using DNS discovery over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) which indicates a list of IP addresses of a given domain. ```yaml @@ -31,7 +31,7 @@ peer_forwarder: domain_name: "data-prepper-cluster.my-domain.net" ``` -### AWS Cloud Map discovery +## AWS Cloud Map [AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. @@ -46,7 +46,7 @@ Your Data Prepper configuration needs to include: Your Data Prepper configuration can optionally include: * `aws_cloud_map_query_parameters` - Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. -#### Example configuration +### Example configuration ```yaml peer_forwarder: @@ -58,8 +58,9 @@ peer_forwarder: aws_region: "us-east-1" ``` -The Data Prepper must also be running with the necessary permissions. The following -IAM policy shows the necessary permissions. +### IAM policy with necessary permissions + +The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions. ```json { @@ -77,7 +78,7 @@ IAM policy shows the necessary permissions. --- # Configuration -## Optional +## Optional configuration * `port`: An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`. * `request_timeout`: Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. @@ -95,10 +96,10 @@ IAM policy shows the necessary permissions. * `aws_region`: A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. * `drain_timeout`: A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. -## SSL +# SSL configuration The SSL configuration for setting up trust manager for peer forwarding client to connect to other Data Prepper instances. -### Optional +## Optional SSL configuration * `ssl`: A `boolean` that enables TLS/SSL. Default value is `true`. * `ssl_certificate_file`: A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). @@ -118,7 +119,7 @@ peer_forwarder: ssl_key_file: "" ``` -## Authentication +# Authentication * `authentication`(Optional) : A `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. Default value is `unauthenticated`. ```yaml @@ -129,14 +130,14 @@ peer_forwarder: # Metrics -Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder` +Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder`. -## Timer +# Timer - `requestForwardingLatency`: measures latency of forwarding requests by peer forwarder client. - `requestProcessingLatency`: measures latency of processing requests by peer forwarder server. -## Counter +# Counter - `requests`: measures total number of forwarded requests. - `requestsFailed`: measures total number of failed requests. Requests with HTTP response code other than `200`. From 74d1ed3495e2b0ba315aef11341e7efb95f6ae66 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 09:19:54 -0700 Subject: [PATCH 07/32] Changed capitalization in title. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 27dde1e0485..8b3f2214b24 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -1,6 +1,6 @@ --- layout: default -title: Peer forwarder +title: Peer Forwarder nav_order: 12 --- From 5f3fb82c74fc4d890cac2c6ac5863e5a517428a9 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 14:46:53 -0700 Subject: [PATCH 08/32] Minor changes from doc review feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 8b3f2214b24..2af16ffabae 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -4,7 +4,7 @@ title: Peer Forwarder nav_order: 12 --- -Peer Forwarder is an HTTP service which performs peer forwarding of `Event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. +Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw` it's `traceId` by default and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). From f32c31b9f1b3288542ba74e55a9618259e4bceba Mon Sep 17 00:00:00 2001 From: Caroline <113052567+carolxob@users.noreply.github.com> Date: Fri, 9 Dec 2022 14:50:14 -0700 Subject: [PATCH 09/32] Update _data-prepper/peer-forwarder.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 2af16ffabae..2f066f7b58a 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,7 +6,7 @@ nav_order: 12 Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw` it's `traceId` by default and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw`, it's `traceId` by default, and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). --- From 01c4fb05bd26b39aa043f4f6118550534a4f1be7 Mon Sep 17 00:00:00 2001 From: Caroline <113052567+carolxob@users.noreply.github.com> Date: Fri, 9 Dec 2022 14:50:26 -0700 Subject: [PATCH 10/32] Update _data-prepper/peer-forwarder.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 2f066f7b58a..22b9cbc99a8 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -10,7 +10,7 @@ Peer Forwarder groups events based on the identification keys provided the proce --- -Right now, peer discovery is provided by either a static list or by a DNS record lookup or AWS Cloudmap. +Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloudmap. # Discovery modes From d4b1faef0f0d32820328b19de8e95b58a4e53645 Mon Sep 17 00:00:00 2001 From: Caroline <113052567+carolxob@users.noreply.github.com> Date: Fri, 9 Dec 2022 14:50:35 -0700 Subject: [PATCH 11/32] Update _data-prepper/peer-forwarder.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 22b9cbc99a8..68491bdb6a4 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -15,7 +15,7 @@ Peer discovery is currently provided by either a static list or by a DNS record # Discovery modes ## Static -Static discover mode allows Data Prepper node to discover nodes using a list of IP addresses or domain names. +Static discover mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml peer_forwarder: discovery_mode: static From 73cc07e63d982f4f5bf0a95ee959c97cd992ca30 Mon Sep 17 00:00:00 2001 From: Caroline <113052567+carolxob@users.noreply.github.com> Date: Fri, 9 Dec 2022 14:50:45 -0700 Subject: [PATCH 12/32] Update _data-prepper/peer-forwarder.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 68491bdb6a4..f7026e5b824 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -60,7 +60,7 @@ peer_forwarder: ### IAM policy with necessary permissions -The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions. +The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions: ```json { From c9c3727ac63a29fb4af03be1454184e59dab7ea5 Mon Sep 17 00:00:00 2001 From: Caroline <113052567+carolxob@users.noreply.github.com> Date: Fri, 9 Dec 2022 14:50:54 -0700 Subject: [PATCH 13/32] Update _data-prepper/peer-forwarder.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index f7026e5b824..6370daba5bc 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -97,7 +97,7 @@ The Data Prepper must also be running with the necessary permissions. The follow * `drain_timeout`: A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. # SSL configuration -The SSL configuration for setting up trust manager for peer forwarding client to connect to other Data Prepper instances. +The SSL configuration for setting up trust manager for the peer forwarding client to connect to other Data Prepper instances. ## Optional SSL configuration From db822eeb1d9c9afdc1cc3b50ff6d17d59cc79d8b Mon Sep 17 00:00:00 2001 From: Caroline <113052567+carolxob@users.noreply.github.com> Date: Fri, 9 Dec 2022 14:51:50 -0700 Subject: [PATCH 14/32] Update _data-prepper/peer-forwarder.md Co-authored-by: Aria Marble <111301581+ariamarble@users.noreply.github.com> --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 6370daba5bc..3adf7bfa4ed 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -143,7 +143,7 @@ Core Peer Forwarder introduces the following custom metrics and all the metrics - `requestsFailed`: measures total number of failed requests. Requests with HTTP response code other than `200`. - `requestsSuccessful`: measures total number of successful requests. Requests with HTTP response code `200`. - `requestsTooLarge`: measures total number of requests which are too large to be written to peer forwarder buffer. Requests with HTTP response code `413`. -- `requestTimeouts`: measures total number of requests which timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. +- `requestTimeouts`: measures the total number of requests which timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. - `requestsUnprocessable`: measures total number of requests which failed due to unprocessable entity. Requests with HTTP response code `422`. - `badRequests`: measures total number of requests with bad request format. Requests with HTTP response code `400`. - `recordsSuccessfullyForwarded`: measures total number of forwarded records successfully. From 491db107cf2046b376e4a1a02c62da4f1da937f6 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 15:07:59 -0700 Subject: [PATCH 15/32] Trying to commit file. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 3adf7bfa4ed..45968a8c389 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,7 +6,11 @@ nav_order: 12 Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. +<<<<<<< HEAD Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw`, it's `traceId` by default, and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +======= +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The identification key is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) +>>>>>>> c1b85e08 (Minor changes.) --- From a87761079160a818d779786a9d979cbc77f77ba4 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 15:17:08 -0700 Subject: [PATCH 16/32] Trying to push file. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 45968a8c389..7b7a0139b4a 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,11 +6,7 @@ nav_order: 12 Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -<<<<<<< HEAD -Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw`, it's `traceId` by default, and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). -======= -Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The identification key is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) ->>>>>>> c1b85e08 (Minor changes.) +Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw`, the value is `traceId` by default, and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). --- From ab8edae279d585d2a6cc738f90695f38970ddc11 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 15:21:25 -0700 Subject: [PATCH 17/32] Trying to push file again. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 7b7a0139b4a..bc43871282e 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,7 +6,11 @@ nav_order: 12 Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. +<<<<<<< HEAD Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw`, the value is `traceId` by default, and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +======= +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The identification key is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +>>>>>>> 0d83461f (Trying to push file.) --- From f83dffd7fa951ec117441f42c8a37817bbf0d377 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 15:23:09 -0700 Subject: [PATCH 18/32] Trying to push file one more time after rebasing. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index bc43871282e..83480e4a728 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,15 +6,11 @@ nav_order: 12 Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -<<<<<<< HEAD -Peer Forwarder groups events based on the identification keys provided the processors. For `service_map_stateful` and `otel_trace_raw`, the value is `traceId` by default, and can not be configured. It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). -======= Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The identification key is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). ->>>>>>> 0d83461f (Trying to push file.) --- -Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloudmap. +Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloudmap. # Discovery modes From 0ee78500cf5a5ac40b86979d313770e440dede2c Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 15:32:01 -0700 Subject: [PATCH 19/32] Minor change. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 83480e4a728..6cc3a906b9e 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,7 +6,7 @@ nav_order: 12 Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The identification key is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The value is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). --- @@ -17,7 +17,7 @@ Peer discovery is currently provided by either a static list or by a DNS record ## Static Static discover mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml -peer_forwarder: +peer_forwarder:4 discovery_mode: static static_endpoints: ["data-prepper1", "data-prepper2"] ``` From 2cb064d6b94fd0e69e1f9ac71e8abe125ad70487 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 15:45:17 -0700 Subject: [PATCH 20/32] Minor edits. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 6cc3a906b9e..a75831ea2d9 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -14,6 +14,8 @@ Peer discovery is currently provided by either a static list or by a DNS record # Discovery modes +See the following information about discovery modes below. + ## Static Static discover mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml From 0609ee4ad9c203d9fdb1c695f8064e1c7ae9cceb Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 16:00:55 -0700 Subject: [PATCH 21/32] Converted optional configuration section to table. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 43 ++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 19 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index a75831ea2d9..1f029541188 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -17,6 +17,7 @@ Peer discovery is currently provided by either a static list or by a DNS record See the following information about discovery modes below. ## Static + Static discover mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml peer_forwarder:4 @@ -78,25 +79,29 @@ The Data Prepper must also be running with the necessary permissions. The follow } ``` --- -# Configuration - -## Optional configuration - -* `port`: An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`. -* `request_timeout`: Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. -* `server_thread_count`: An `int` representing number of threads used by Peer Forwarder server. Defaults to `200`. -* `client_thread_count`: An `int` representing number of threads used by Peer Forwarder client. Defaults to `200`. -* `maxConnectionCount`: An `int` representing maximum number of open connections for Peer Forwarder server. Default value is `500`. -* `discovery_mode`: A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node` which processes events locally. -* `static_endpoints`: A `list` containing endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. -* `domain_name`: A `String` representing single domain name to query DNS against. Typically, used by creating multiple [DNS A Records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. -* `aws_cloud_map_namespace_name`: A `String` representing the Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. -* `aws_cloud_map_service_name`: A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. -* `aws_cloud_map_query_parameters`: A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. -* `buffer_size`: An `int` representing max number of unchecked records the buffer accepts (num of unchecked records = num of records written into the buffer + num of in-flight records not yet checked by the Checkpointing API). Default is `512`. -* `batch_size`: An `int` representing max number of records the buffer returns on read. Default is `48`. -* `aws_region`: A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. -* `drain_timeout`: A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. +## Configuration + +See the following for optional configuration values. + +### Optional configuration + +| Value | Description | +| ---- | ----------- | +| `port` | An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`.| +| `request_timeout` | Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. | +| `server_thread_count` | An `int` representing number of threads used by Peer Forwarder server. Defaults to `200`.| +| `client_thread_count` | An `int` representing number of threads used by Peer Forwarder client. Defaults to `200`.| +| `maxConnectionCount` | An `int` representing maximum number of open connections for Peer Forwarder server. Default value is `500`. | +| `discovery_mode` | A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node` which processes events locally. | +| `static_endpoints` | A `list` containing endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. | +| `domain_name` | A `String` representing single domain name to query DNS against. Typically, used by creating multiple [DNS A Records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. | +| `aws_cloud_map_namespace_name` | A `String` representing the Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_cloud_map_service_name` | A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_cloud_map_query_parameters` | A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | +| `buffer_size` | An `int` representing max number of unchecked records the buffer accepts (num of unchecked records = num of records written into the buffer + num of in-flight records not yet checked by the Checkpointing API). Default is `512`. | +| `batch_size` | An `int` representing max number of records the buffer returns on read. Default is `48`. | +| `aws_region` | A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | +| `drain_timeout` | A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. | # SSL configuration The SSL configuration for setting up trust manager for the peer forwarding client to connect to other Data Prepper instances. From 2d1cf02a5b521f1e3f58d7aff1e9e5c620c456ca Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 16:03:44 -0700 Subject: [PATCH 22/32] Minor adjustmenets. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 1f029541188..95de4816206 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -4,11 +4,12 @@ title: Peer Forwarder nav_order: 12 --- +#Peer Forwarder + Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The value is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The value is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information here about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). ---- Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloudmap. @@ -103,10 +104,10 @@ See the following for optional configuration values. | `aws_region` | A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | | `drain_timeout` | A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. | -# SSL configuration +## SSL configuration The SSL configuration for setting up trust manager for the peer forwarding client to connect to other Data Prepper instances. -## Optional SSL configuration +### Optional SSL configuration * `ssl`: A `boolean` that enables TLS/SSL. Default value is `true`. * `ssl_certificate_file`: A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). From cb5134af34a4fb64d3690c478a426012ecb75749 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 9 Dec 2022 16:05:19 -0700 Subject: [PATCH 23/32] Minor adjustments again. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 95de4816206..fe5155f14aa 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -4,7 +4,7 @@ title: Peer Forwarder nav_order: 12 --- -#Peer Forwarder +# Peer Forwarder Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. @@ -13,11 +13,11 @@ Peer Forwarder groups events based on the identification keys provided by the pr Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloudmap. -# Discovery modes +## Discovery modes See the following information about discovery modes below. -## Static +### Static Static discover mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml @@ -26,7 +26,7 @@ peer_forwarder:4 static_endpoints: ["data-prepper1", "data-prepper2"] ``` -## DNS lookup +### DNS lookup We recommend using DNS discovery over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) which indicates a list of IP addresses of a given domain. ```yaml @@ -35,7 +35,7 @@ peer_forwarder: domain_name: "data-prepper-cluster.my-domain.net" ``` -## AWS Cloud Map +### AWS Cloud Map [AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. @@ -127,7 +127,7 @@ peer_forwarder: ssl_key_file: "" ``` -# Authentication +## Authentication * `authentication`(Optional) : A `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. Default value is `unauthenticated`. ```yaml @@ -136,16 +136,16 @@ peer_forwarder: mutual_tls: ``` -# Metrics +## Metrics Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder`. -# Timer +### Timer - `requestForwardingLatency`: measures latency of forwarding requests by peer forwarder client. - `requestProcessingLatency`: measures latency of processing requests by peer forwarder server. -# Counter +### Counter - `requests`: measures total number of forwarded requests. - `requestsFailed`: measures total number of failed requests. Requests with HTTP response code other than `200`. @@ -161,6 +161,6 @@ Core Peer Forwarder introduces the following custom metrics and all the metrics - `recordsActuallyProcessedLocally`: measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. - `recordsReceivedFromPeers`: measures total number of records received from remote peers. -## Gauge +### Gauge - `peerEndpoints`: measures number of dynamically discovered peer data-prepper endpoints. For `static` mode, the size is fixed. From 0b4a055e0a7894726726818b3c1504254bbcc39d Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 12 Dec 2022 11:32:14 -0700 Subject: [PATCH 24/32] Updates from doc review feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 73 ++++++++++++++++++++------------- 1 file changed, 45 insertions(+), 28 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index fe5155f14aa..ff5a090e23a 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,9 +6,9 @@ nav_order: 12 # Peer Forwarder -Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. +Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, Peer Forwarder is supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The value is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information here about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The value is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) on the aggregate processor page. Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloudmap. @@ -64,7 +64,7 @@ peer_forwarder: ### IAM policy with necessary permissions -The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions: +The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions. ```json { @@ -109,16 +109,20 @@ The SSL configuration for setting up trust manager for the peer forwarding clien ### Optional SSL configuration -* `ssl`: A `boolean` that enables TLS/SSL. Default value is `true`. -* `ssl_certificate_file`: A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). -* `ssl_key_file`: A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). -* `ssl_insecure_disable_verification`: A `boolean` that disables the verification of server's TLS certificate chain. Default value is `false`. -* `ssl_fingerprint_verification_only`: A `boolean` that disables the verification of server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. -* `use_acm_certificate_for_ssl`: A `boolean` that enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. -* `acm_certificate_arn`: A `String` represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. -* `acm_private_key_password`: A `String` that represents the ACM private key password which that will be used to decrypt the private key. If it's not provided, a random password will be generated. -* `acm_certificate_timeout_millis`: An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. -* `aws_region`: A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. +See the table below for optional SSL configuration descriptions. + +| Value | Description | +| ----- | ----------- | +| `ssl` | A `boolean` that enables TLS/SSL. Default value is `true`. | +| `ssl_certificate_file`| A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). | +| `ssl_key_file`| A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). | +| `ssl_insecure_disable_verification` | A `boolean` that disables the verification of server's TLS certificate chain. Default value is `false`. | +| `ssl_fingerprint_verification_only` | A `boolean` that disables the verification of server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. | +| `use_acm_certificate_for_ssl` | A `boolean` that enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. | +| `acm_certificate_arn`| A `String` represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | +| `acm_private_key_password` | A `String` that represents the ACM private key password which that will be used to decrypt the private key. If it's not provided, a random password will be generated. | +| `acm_certificate_timeout_millis` | An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. | +| `aws_region` | A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | ```yaml peer_forwarder: @@ -129,7 +133,10 @@ peer_forwarder: ## Authentication -* `authentication`(Optional) : A `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. Default value is `unauthenticated`. +This section describes optional authentication. + +`authentication`(optional) : A `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. + ```yaml peer_forwarder: authentication: @@ -138,29 +145,39 @@ peer_forwarder: ## Metrics +This section describes custom metrics capabilities, including timer, counter, and gauge. + Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder`. ### Timer +This section describes the metrics feature, timer. + - `requestForwardingLatency`: measures latency of forwarding requests by peer forwarder client. - `requestProcessingLatency`: measures latency of processing requests by peer forwarder server. ### Counter -- `requests`: measures total number of forwarded requests. -- `requestsFailed`: measures total number of failed requests. Requests with HTTP response code other than `200`. -- `requestsSuccessful`: measures total number of successful requests. Requests with HTTP response code `200`. -- `requestsTooLarge`: measures total number of requests which are too large to be written to peer forwarder buffer. Requests with HTTP response code `413`. -- `requestTimeouts`: measures the total number of requests which timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. -- `requestsUnprocessable`: measures total number of requests which failed due to unprocessable entity. Requests with HTTP response code `422`. -- `badRequests`: measures total number of requests with bad request format. Requests with HTTP response code `400`. -- `recordsSuccessfullyForwarded`: measures total number of forwarded records successfully. -- `recordsFailedForwarding`: measures total number of records failed to be forwarded. -- `recordsToBeForwarded`: measures total number of records to be forwarded. -- `recordsToBeProcessedLocally`: measures total number of records to be processed locally. -- `recordsActuallyProcessedLocally`: measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. -- `recordsReceivedFromPeers`: measures total number of records received from remote peers. +See the table below for counter metric options. + +| Value | Description | +| ----- | ----------- | +| `requests`| measures total number of forwarded requests. | +| `requestsFailed`| measures total number of failed requests. Requests with HTTP response code other than `200`. | +| `requestsSuccessful`| measures total number of successful requests. Requests with HTTP response code `200`. | +| `requestsTooLarge`| measures total number of requests which are too large to be written to peer forwarder buffer. Requests with HTTP response code `413`. | +| `requestTimeouts`| measures the total number of requests which timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. | +| `requestsUnprocessable`| measures total number of requests which failed due to unprocessable entity. Requests with HTTP response code `422`. | +| `badRequests`| measures total number of requests with bad request format. Requests with HTTP response code `400`. | +| `recordsSuccessfullyForwarded`| measures total number of forwarded records successfully. | +| `recordsFailedForwarding`| measures total number of records failed to be forwarded. | +| `recordsToBeForwarded` | measures total number of records to be forwarded. | +| `recordsToBeProcessedLocally` | measures total number of records to be processed locally. | +| `recordsActuallyProcessedLocally`| measures total number of records actually processed locally. Sum of recordsToBeProcessedLocally` and `recordsFailedForwarding`. | +| `recordsReceivedFromPeers`| measures total number of records received from remote peers. | ### Gauge -- `peerEndpoints`: measures number of dynamically discovered peer data-prepper endpoints. For `static` mode, the size is fixed. +This section describes gauge metric options. + +`peerEndpoints` measures number of dynamically discovered peer data-prepper endpoints. For `static` mode, the size is fixed. From 850625f01f50d87399da174267f29c3719d2c3fb Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 12 Dec 2022 15:04:17 -0700 Subject: [PATCH 25/32] Made changes based on doc review feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index ff5a090e23a..2a5f4b6ecac 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,12 +6,12 @@ nav_order: 12 # Peer Forwarder -Peer Forwarder is an HTTP service which performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, Peer Forwarder is supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. +Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, Peer Forwarder is supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The value is configurable for the `aggregate` processor using the `identification_keys` configuration option. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) on the aggregate processor page. +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) on the aggregate processor page. -Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloudmap. +Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloud Map. ## Discovery modes @@ -19,7 +19,7 @@ See the following information about discovery modes below. ### Static -Static discover mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. +Static discovery mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. ```yaml peer_forwarder:4 discovery_mode: static @@ -27,7 +27,7 @@ peer_forwarder:4 ``` ### DNS lookup -We recommend using DNS discovery over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) which indicates a list of IP addresses of a given domain. +DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. ```yaml peer_forwarder: @@ -39,7 +39,7 @@ peer_forwarder: [AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. -Peer forwarder can use the API-based service discovery. To support this you must have an existing namespace configured for API instance discovery. You can create a new one following the instructions provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). +Peer Forwarder can use the API-based service discovery. To support this you must have an existing namespace configured for API instance discovery. You can create a new one following the instructions provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). Your Data Prepper configuration needs to include: * `aws_cloud_map_namespace_name` - Set to your Cloud Map Namespace name @@ -145,7 +145,7 @@ peer_forwarder: ## Metrics -This section describes custom metrics capabilities, including timer, counter, and gauge. +This section describes Peer Forwarder's metrics capabilities, including timer, counter, and gauge. Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder`. @@ -165,19 +165,19 @@ See the table below for counter metric options. | `requests`| measures total number of forwarded requests. | | `requestsFailed`| measures total number of failed requests. Requests with HTTP response code other than `200`. | | `requestsSuccessful`| measures total number of successful requests. Requests with HTTP response code `200`. | -| `requestsTooLarge`| measures total number of requests which are too large to be written to peer forwarder buffer. Requests with HTTP response code `413`. | -| `requestTimeouts`| measures the total number of requests which timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. | -| `requestsUnprocessable`| measures total number of requests which failed due to unprocessable entity. Requests with HTTP response code `422`. | +| `requestsTooLarge`| measures total number of requests that are too large to be written to Peer Forwarder buffer. Requests with HTTP response code `413`. | +| `requestTimeouts`| measures the total number of requests that timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. | +| `requestsUnprocessable`| measures total number of requests that failed due to unprocessable entity. Requests with HTTP response code `422`. | | `badRequests`| measures total number of requests with bad request format. Requests with HTTP response code `400`. | | `recordsSuccessfullyForwarded`| measures total number of forwarded records successfully. | | `recordsFailedForwarding`| measures total number of records failed to be forwarded. | | `recordsToBeForwarded` | measures total number of records to be forwarded. | | `recordsToBeProcessedLocally` | measures total number of records to be processed locally. | -| `recordsActuallyProcessedLocally`| measures total number of records actually processed locally. Sum of recordsToBeProcessedLocally` and `recordsFailedForwarding`. | +| `recordsActuallyProcessedLocally`| measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. | | `recordsReceivedFromPeers`| measures total number of records received from remote peers. | ### Gauge This section describes gauge metric options. -`peerEndpoints` measures number of dynamically discovered peer data-prepper endpoints. For `static` mode, the size is fixed. +`peerEndpoints` measures number of dynamically discovered peer Data Prepper endpoints. For `static` mode, the size is fixed. From 588da3f4ac65a229471de9fdf5968dcb6cdeabe9 Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 13 Dec 2022 10:49:24 -0700 Subject: [PATCH 26/32] Made minor heading adjustements. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 2a5f4b6ecac..fb4d3493c78 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -20,6 +20,7 @@ See the following information about discovery modes below. ### Static Static discovery mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. + ```yaml peer_forwarder:4 discovery_mode: static @@ -50,7 +51,7 @@ Your Data Prepper configuration needs to include: Your Data Prepper configuration can optionally include: * `aws_cloud_map_query_parameters` - Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. -### Example configuration +#### Example configuration ```yaml peer_forwarder: @@ -124,6 +125,8 @@ See the table below for optional SSL configuration descriptions. | `acm_certificate_timeout_millis` | An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. | | `aws_region` | A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | +#### Example config + ```yaml peer_forwarder: ssl: true From 81e08ed7eea52b4339a6e08f2b7fe807382c98bf Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 13 Dec 2022 11:54:34 -0700 Subject: [PATCH 27/32] Made edits based on doc review feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 64 ++++++++++++++++----------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index fb4d3493c78..e9be7c2f0ee 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -40,16 +40,16 @@ peer_forwarder: [AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. -Peer Forwarder can use the API-based service discovery. To support this you must have an existing namespace configured for API instance discovery. You can create a new one following the instructions provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). +Peer Forwarder can use the API-based service discovery. To support this, you must have an existing Namespace configured for API instance discovery. You can create a new one following the instructions provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). Your Data Prepper configuration needs to include: * `aws_cloud_map_namespace_name` - Set to your Cloud Map Namespace name * `aws_cloud_map_service_name` - Set to the service name within your specified Namespace -* `aws_region` - The AWS region where your namespace exists. +* `aws_region` - The AWS Region where your Namespace exists. * `discovery_mode` - Set to `aws_cloud_map` Your Data Prepper configuration can optionally include: -* `aws_cloud_map_query_parameters` - Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. +* `aws_cloud_map_query_parameters` - Key-value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. #### Example configuration @@ -65,7 +65,7 @@ peer_forwarder: ### IAM policy with necessary permissions -The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions. +The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions: ```json { @@ -89,21 +89,21 @@ See the following for optional configuration values. | Value | Description | | ---- | ----------- | -| `port` | An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`.| +| `port` | An `int` between 0 and 65535 represents the port Peer Forwarder server is running on. Default value is `4994`.| | `request_timeout` | Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. | -| `server_thread_count` | An `int` representing number of threads used by Peer Forwarder server. Defaults to `200`.| -| `client_thread_count` | An `int` representing number of threads used by Peer Forwarder client. Defaults to `200`.| +| `server_thread_count` | An `int` representing number of threads used by the Peer Forwarder server. Defaults to `200`.| +| `client_thread_count` | An `int` representing number of threads used by the Peer Forwarder client. Defaults to `200`.| | `maxConnectionCount` | An `int` representing maximum number of open connections for Peer Forwarder server. Default value is `500`. | | `discovery_mode` | A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node` which processes events locally. | | `static_endpoints` | A `list` containing endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. | | `domain_name` | A `String` representing single domain name to query DNS against. Typically, used by creating multiple [DNS A Records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. | -| `aws_cloud_map_namespace_name` | A `String` representing the Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_cloud_map_namespace_name` | A `String` representing the Cloud Map Namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | | `aws_cloud_map_service_name` | A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | | `aws_cloud_map_query_parameters` | A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | -| `buffer_size` | An `int` representing max number of unchecked records the buffer accepts (num of unchecked records = num of records written into the buffer + num of in-flight records not yet checked by the Checkpointing API). Default is `512`. | +| `buffer_size` | An `int` representing max number of unchecked records the buffer accepts (num of unchecked records equals the number of records written into the buffer + the number of in-flight records not yet checked by the Checkpointing API). Default is `512`. | | `batch_size` | An `int` representing max number of records the buffer returns on read. Default is `48`. | -| `aws_region` | A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | -| `drain_timeout` | A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. | +| `aws_region` | A `String` represents the AWS region to use `ACM`, `S3`, or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | +| `drain_timeout` | A `Duration` representing the wait time for the Peer Forwarder to complete processing data before shutdown. | ## SSL configuration The SSL configuration for setting up trust manager for the peer forwarding client to connect to other Data Prepper instances. @@ -115,15 +115,15 @@ See the table below for optional SSL configuration descriptions. | Value | Description | | ----- | ----------- | | `ssl` | A `boolean` that enables TLS/SSL. Default value is `true`. | -| `ssl_certificate_file`| A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). | -| `ssl_key_file`| A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). | +| `ssl_certificate_file`| A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to the default certificate file,`config/default_certificate.pem`. Read more about how the certificate file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | +| `ssl_key_file`| A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | | `ssl_insecure_disable_verification` | A `boolean` that disables the verification of server's TLS certificate chain. Default value is `false`. | | `ssl_fingerprint_verification_only` | A `boolean` that disables the verification of server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. | | `use_acm_certificate_for_ssl` | A `boolean` that enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. | | `acm_certificate_arn`| A `String` represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | -| `acm_private_key_password` | A `String` that represents the ACM private key password which that will be used to decrypt the private key. If it's not provided, a random password will be generated. | +| `acm_private_key_password` | A `String` that represents the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | | `acm_certificate_timeout_millis` | An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. | -| `aws_region` | A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_region` | A `String` represents the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | #### Example config @@ -154,10 +154,10 @@ Core Peer Forwarder introduces the following custom metrics and all the metrics ### Timer -This section describes the metrics feature, timer. +This section describes the timer metrics feature. -- `requestForwardingLatency`: measures latency of forwarding requests by peer forwarder client. -- `requestProcessingLatency`: measures latency of processing requests by peer forwarder server. +- `requestForwardingLatency`: measures latency of forwarding requests by the Peer Forwarder client. +- `requestProcessingLatency`: measures latency of processing requests by Peer Forwarder server. ### Counter @@ -165,22 +165,22 @@ See the table below for counter metric options. | Value | Description | | ----- | ----------- | -| `requests`| measures total number of forwarded requests. | -| `requestsFailed`| measures total number of failed requests. Requests with HTTP response code other than `200`. | -| `requestsSuccessful`| measures total number of successful requests. Requests with HTTP response code `200`. | -| `requestsTooLarge`| measures total number of requests that are too large to be written to Peer Forwarder buffer. Requests with HTTP response code `413`. | -| `requestTimeouts`| measures the total number of requests that timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. | -| `requestsUnprocessable`| measures total number of requests that failed due to unprocessable entity. Requests with HTTP response code `422`. | -| `badRequests`| measures total number of requests with bad request format. Requests with HTTP response code `400`. | -| `recordsSuccessfullyForwarded`| measures total number of forwarded records successfully. | -| `recordsFailedForwarding`| measures total number of records failed to be forwarded. | -| `recordsToBeForwarded` | measures total number of records to be forwarded. | -| `recordsToBeProcessedLocally` | measures total number of records to be processed locally. | -| `recordsActuallyProcessedLocally`| measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. | -| `recordsReceivedFromPeers`| measures total number of records received from remote peers. | +| `requests`| Measures total number of forwarded requests. | +| `requestsFailed`| Measures total number of failed requests. Requests with HTTP response code other than `200`. | +| `requestsSuccessful`| Measures total number of successful requests. Requests with HTTP response code `200`. | +| `requestsTooLarge`| Measures total number of requests that are too large to be written to Peer Forwarder buffer. Requests with HTTP response code `413`. | +| `requestTimeouts`| Measures the total number of requests that timed out while writing content to Peer Forwarder buffer. Requests with HTTP response code `408`. | +| `requestsUnprocessable`| Measures total number of requests that failed due to unprocessable entity. Requests with HTTP response code `422`. | +| `badRequests`| Measures total number of requests with bad request format. Requests with HTTP response code `400`. | +| `recordsSuccessfullyForwarded`| Measures total number of forwarded records successfully. | +| `recordsFailedForwarding`| Measures total number of records failed to be forwarded. | +| `recordsToBeForwarded` | Measures total number of records to be forwarded. | +| `recordsToBeProcessedLocally` | Measures total number of records to be processed locally. | +| `recordsActuallyProcessedLocally`| Measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. | +| `recordsReceivedFromPeers`| Measures total number of records received from remote peers. | ### Gauge This section describes gauge metric options. -`peerEndpoints` measures number of dynamically discovered peer Data Prepper endpoints. For `static` mode, the size is fixed. +`peerEndpoints` Measures number of dynamically discovered peer Data Prepper endpoints. For `static` mode, the size is fixed. From 9f5313eb1d2ef54e70b0927bab66c599a5d71142 Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 14 Dec 2022 19:04:14 -0700 Subject: [PATCH 28/32] Made updates based on editorial feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 40 +++++++++++++++------------------ 1 file changed, 18 insertions(+), 22 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index e9be7c2f0ee..b38aa6112f3 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -6,20 +6,19 @@ nav_order: 12 # Peer Forwarder -Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, Peer Forwarder is supported by `aggregate`, `service_map_stateful`, `otel_trace_raw` processors. +Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, Peer Forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_trace_raw` processors. Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) on the aggregate processor page. - -Peer discovery is currently provided by either a static list or by a DNS record lookup or AWS Cloud Map. +Peer discovery is currently provided by a static list, a DNS record lookup, or AWS Cloud Map. ## Discovery modes -See the following information about discovery modes below. +The following sections provide information about discovery modes. ### Static -Static discovery mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. +Static discovery mode allows a Data Prepper node to discover nodes using a list of IP addresses or domain names. See the following yaml file for an example of static discovery mode: ```yaml peer_forwarder:4 @@ -28,7 +27,8 @@ peer_forwarder:4 ``` ### DNS lookup -DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. + +DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. See the following yaml file for an example of DNS lookup: ```yaml peer_forwarder: @@ -154,8 +154,6 @@ Core Peer Forwarder introduces the following custom metrics and all the metrics ### Timer -This section describes the timer metrics feature. - - `requestForwardingLatency`: measures latency of forwarding requests by the Peer Forwarder client. - `requestProcessingLatency`: measures latency of processing requests by Peer Forwarder server. @@ -165,22 +163,20 @@ See the table below for counter metric options. | Value | Description | | ----- | ----------- | -| `requests`| Measures total number of forwarded requests. | -| `requestsFailed`| Measures total number of failed requests. Requests with HTTP response code other than `200`. | -| `requestsSuccessful`| Measures total number of successful requests. Requests with HTTP response code `200`. | -| `requestsTooLarge`| Measures total number of requests that are too large to be written to Peer Forwarder buffer. Requests with HTTP response code `413`. | +| `requests`| Measures the total number of forwarded requests. | +| `requestsFailed`| Measures the total number of failed requests. Requests with HTTP response code other than `200`. | +| `requestsSuccessful`| Measures the total number of successful requests. Requests with HTTP response code `200`. | +| `requestsTooLarge`| Measures the total number of requests that are too large to be written to Peer Forwarder buffer. Requests with HTTP response code `413`. | | `requestTimeouts`| Measures the total number of requests that timed out while writing content to Peer Forwarder buffer. Requests with HTTP response code `408`. | -| `requestsUnprocessable`| Measures total number of requests that failed due to unprocessable entity. Requests with HTTP response code `422`. | -| `badRequests`| Measures total number of requests with bad request format. Requests with HTTP response code `400`. | -| `recordsSuccessfullyForwarded`| Measures total number of forwarded records successfully. | -| `recordsFailedForwarding`| Measures total number of records failed to be forwarded. | -| `recordsToBeForwarded` | Measures total number of records to be forwarded. | -| `recordsToBeProcessedLocally` | Measures total number of records to be processed locally. | -| `recordsActuallyProcessedLocally`| Measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. | -| `recordsReceivedFromPeers`| Measures total number of records received from remote peers. | +| `requestsUnprocessable`| Measures the total number of requests that failed due to unprocessable entity. Requests with HTTP response code `422`. | +| `badRequests`| Measures the total number of requests with bad request format. Requests with HTTP response code `400`. | +| `recordsSuccessfullyForwarded`| Measures the total number of forwarded records successfully. | +| `recordsFailedForwarding`| Measures the total number of records failed to be forwarded. | +| `recordsToBeForwarded` | Measures the total number of records to be forwarded. | +| `recordsToBeProcessedLocally` | Measures the total number of records to be processed locally. | +| `recordsActuallyProcessedLocally`| Measures the total number of records actually processed locally. This value is the sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. | +| `recordsReceivedFromPeers`| Measures the total number of records received from remote peers. | ### Gauge -This section describes gauge metric options. - `peerEndpoints` Measures number of dynamically discovered peer Data Prepper endpoints. For `static` mode, the size is fixed. From 156ddc7d15db9f2975c996a05185ce37799dfe65 Mon Sep 17 00:00:00 2001 From: carolxob Date: Fri, 16 Dec 2022 16:01:26 -0700 Subject: [PATCH 29/32] Made extensive changes based on editorial feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 90 ++++++++++++++++----------------- 1 file changed, 45 insertions(+), 45 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index b38aa6112f3..769929c9951 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -8,7 +8,7 @@ nav_order: 12 Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, Peer Forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. You can find more information about [identification keys](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) on the aggregate processor page. +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor Page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys. Peer discovery is currently provided by a static list, a DNS record lookup, or AWS Cloud Map. @@ -28,7 +28,7 @@ peer_forwarder:4 ### DNS lookup -DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. See the following yaml file for an example of DNS lookup: +DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. See the following yaml file example of DNS lookup: ```yaml peer_forwarder: @@ -40,19 +40,21 @@ peer_forwarder: [AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. -Peer Forwarder can use the API-based service discovery. To support this, you must have an existing Namespace configured for API instance discovery. You can create a new one following the instructions provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). +Peer Forwarder can use the API-based service discovery. To support this, you must have an existing namespace configured for API instance discovery. You can create a new one by following the instructions provided by the [AWS Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). -Your Data Prepper configuration needs to include: -* `aws_cloud_map_namespace_name` - Set to your Cloud Map Namespace name -* `aws_cloud_map_service_name` - Set to the service name within your specified Namespace -* `aws_region` - The AWS Region where your Namespace exists. -* `discovery_mode` - Set to `aws_cloud_map` +Your Data Prepper configuration needs to include the following: +* `aws_cloud_map_namespace_name` – Set to your AWS Cloud Map namespace name. +* `aws_cloud_map_service_name` – Set to the service name within your specified namespace. +* `aws_region` – Set to the AWS Region where your namespace exists. +* `discovery_mode` – Set to `aws_cloud_map`. -Your Data Prepper configuration can optionally include: -* `aws_cloud_map_query_parameters` - Key-value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. +Your Data Prepper configuration can optionally include the following: +* `aws_cloud_map_query_parameters` – Key-value pairs are used to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. #### Example configuration +See the following yaml file example of AWS Cloud Map configuration: + ```yaml peer_forwarder: discovery_mode: aws_cloud_map @@ -65,7 +67,7 @@ peer_forwarder: ### IAM policy with necessary permissions -The Data Prepper must also be running with the necessary permissions. The following IAM policy shows the necessary permissions: +Data Prepper must also be running with the necessary permissions. The following AWS Identity and Access Management (IAM) policy shows the necessary permissions: ```json { @@ -83,49 +85,47 @@ The Data Prepper must also be running with the necessary permissions. The follow --- ## Configuration -See the following for optional configuration values. +The following table provides optional configuration values. -### Optional configuration | Value | Description | | ---- | ----------- | -| `port` | An `int` between 0 and 65535 represents the port Peer Forwarder server is running on. Default value is `4994`.| -| `request_timeout` | Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. | -| `server_thread_count` | An `int` representing number of threads used by the Peer Forwarder server. Defaults to `200`.| -| `client_thread_count` | An `int` representing number of threads used by the Peer Forwarder client. Defaults to `200`.| -| `maxConnectionCount` | An `int` representing maximum number of open connections for Peer Forwarder server. Default value is `500`. | -| `discovery_mode` | A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node` which processes events locally. | -| `static_endpoints` | A `list` containing endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. | -| `domain_name` | A `String` representing single domain name to query DNS against. Typically, used by creating multiple [DNS A Records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. | -| `aws_cloud_map_namespace_name` | A `String` representing the Cloud Map Namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | +| `port` | An `int` between 0 and 65535 represents the port that the Peer Forwarder server is running on. Default value is `4994`. | +| `request_timeout` | An `int` representing the request timeout duration in milliseconds for the Peer Forwarder HTTP server. Default value is `10000`. | +| `server_thread_count` | An `int` representing the number of threads used by the Peer Forwarder server. Default value is `200`.| +| `client_thread_count` | An `int` representing the number of threads used by the Peer Forwarder client. Default value is `200`.| +| `maxConnectionCount` | An `int` representing the maximum number of open connections for Peer Forwarder server. Default value is `500`. | +| `discovery_mode` | A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node`, which processes events locally. | +| `static_endpoints` | A `list` containing the endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. | +| `domain_name` | A `String` representing the single domain name to query DNS against. Typically, used by creating multiple [DNS A records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. | +| `aws_cloud_map_namespace_name` | A `String` representing the AWS Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | | `aws_cloud_map_service_name` | A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | | `aws_cloud_map_query_parameters` | A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | -| `buffer_size` | An `int` representing max number of unchecked records the buffer accepts (num of unchecked records equals the number of records written into the buffer + the number of in-flight records not yet checked by the Checkpointing API). Default is `512`. | +| `buffer_size` | An `int` representing the maximum number of unchecked records the buffer accepts (the number of unchecked records equals the number of records written into the buffer plus the number of in-flight records not yet checked by the Checkpointing API). Default is `512`. | | `batch_size` | An `int` representing max number of records the buffer returns on read. Default is `48`. | -| `aws_region` | A `String` represents the AWS region to use `ACM`, `S3`, or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_region` | A `String` represents the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`, and `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | | `drain_timeout` | A `Duration` representing the wait time for the Peer Forwarder to complete processing data before shutdown. | ## SSL configuration -The SSL configuration for setting up trust manager for the peer forwarding client to connect to other Data Prepper instances. - -### Optional SSL configuration -See the table below for optional SSL configuration descriptions. +The following SSL configuration table provides optional SSL configuration values and allows you to set up trust manager for the peer forwarding client to connect to other Data Prepper instances. | Value | Description | | ----- | ----------- | -| `ssl` | A `boolean` that enables TLS/SSL. Default value is `true`. | -| `ssl_certificate_file`| A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to the default certificate file,`config/default_certificate.pem`. Read more about how the certificate file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | +| `ssl` | A `Boolean` that enables TLS/SSL. Default value is `true`. | +| `ssl_certificate_file`| A `String` representing the SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. The following is an example of an Amazon S3 path: `s3:///`. Defaults to the default certificate file,`config/default_certificate.pem`. See [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) for more information about how the certificate is generated. | | `ssl_key_file`| A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated at the [Default Certificates](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates) page. | -| `ssl_insecure_disable_verification` | A `boolean` that disables the verification of server's TLS certificate chain. Default value is `false`. | -| `ssl_fingerprint_verification_only` | A `boolean` that disables the verification of server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. | -| `use_acm_certificate_for_ssl` | A `boolean` that enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. | -| `acm_certificate_arn`| A `String` represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | -| `acm_private_key_password` | A `String` that represents the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | -| `acm_certificate_timeout_millis` | An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. | -| `aws_region` | A `String` represents the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. | +| `ssl_insecure_disable_verification` | A `Boolean` that disables the verification of the server's TLS certificate chain. Default value is `false`. | +| `ssl_fingerprint_verification_only` | A `Boolean` that disables the verification of the server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. | +| `use_acm_certificate_for_ssl` | A `Boolean` that enables TLS/SSL using the certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | +| `acm_certificate_arn`| A `String` representing the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | +| `acm_private_key_password` | A `String` representing the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | +| `acm_certificate_timeout_millis` | An `Int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. | +| `aws_region` | A `String` representing the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`, and `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | + +#### Example configuration -#### Example config +The following yaml file provides an example configuration: ```yaml peer_forwarder: @@ -138,7 +138,7 @@ peer_forwarder: This section describes optional authentication. -`authentication`(optional) : A `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. +`authentication` is optional and is a `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. The following yaml file provides an exmaple for authentication: ```yaml peer_forwarder: @@ -150,24 +150,24 @@ peer_forwarder: This section describes Peer Forwarder's metrics capabilities, including timer, counter, and gauge. -Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder`. +Core Peer Forwarder introduces the following custom metrics. All the metrics are prefixed by `core.peerForwarder`. ### Timer -- `requestForwardingLatency`: measures latency of forwarding requests by the Peer Forwarder client. -- `requestProcessingLatency`: measures latency of processing requests by Peer Forwarder server. +- `requestForwardingLatency`: Measures latency of requestes forwarded by the Peer Forwarder client. +- `requestProcessingLatency`: Measures latency of requests processed by the Peer Forwarder server. ### Counter -See the table below for counter metric options. +The following table provides counter metric options. | Value | Description | | ----- | ----------- | | `requests`| Measures the total number of forwarded requests. | | `requestsFailed`| Measures the total number of failed requests. Requests with HTTP response code other than `200`. | | `requestsSuccessful`| Measures the total number of successful requests. Requests with HTTP response code `200`. | -| `requestsTooLarge`| Measures the total number of requests that are too large to be written to Peer Forwarder buffer. Requests with HTTP response code `413`. | -| `requestTimeouts`| Measures the total number of requests that timed out while writing content to Peer Forwarder buffer. Requests with HTTP response code `408`. | +| `requestsTooLarge`| Measures the total number of requests that are too large to be written to the Peer Forwarder buffer. Requests with HTTP response code `413`. | +| `requestTimeouts`| Measures the total number of requests that time out while writing content to the Peer Forwarder buffer. Requests with HTTP response code `408`. | | `requestsUnprocessable`| Measures the total number of requests that failed due to unprocessable entity. Requests with HTTP response code `422`. | | `badRequests`| Measures the total number of requests with bad request format. Requests with HTTP response code `400`. | | `recordsSuccessfullyForwarded`| Measures the total number of forwarded records successfully. | From 5c640ca8bdad0aba52b3e3f4a3e6068bd6433a80 Mon Sep 17 00:00:00 2001 From: carolxob Date: Mon, 19 Dec 2022 16:16:55 -0800 Subject: [PATCH 30/32] Incorporated minor editorial feedback changes. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 769929c9951..979597b1e3e 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -8,7 +8,7 @@ nav_order: 12 Peer Forwarder is an HTTP service that performs peer forwarding of an `event` between Data Prepper nodes for aggregation. Currently, Peer Forwarder is supported by the `aggregate`, `service_map_stateful`, and `otel_trace_raw` processors. -Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor Page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys. +Peer Forwarder groups events based on the identification keys provided by the processors. For `service_map_stateful` and `otel_trace_raw`, the identification key is `traceId` by default and cannot be configured. The `aggregate` processor is configured using the `identification_keys` configuration option. From here, you can specify which keys to use for Peer Forwarder. See [Aggregate Processor page](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys) for more information about identification keys. Peer discovery is currently provided by a static list, a DNS record lookup, or AWS Cloud Map. @@ -28,7 +28,7 @@ peer_forwarder:4 ### DNS lookup -DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. See the following yaml file example of DNS lookup: +DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. DNS discovery configures a DNS provider to return a list of Data Prepper hosts when given a single domain name. This list that is returned is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. See the following yaml file example of DNS lookup: ```yaml peer_forwarder: From a0eb2ecf234dade8050257a9553e0fccd04e82c4 Mon Sep 17 00:00:00 2001 From: carolxob Date: Tue, 3 Jan 2023 15:26:38 -0700 Subject: [PATCH 31/32] Incorporated more editorial feedback. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 979597b1e3e..71b16fe957c 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -28,7 +28,7 @@ peer_forwarder:4 ### DNS lookup -DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. DNS discovery configures a DNS provider to return a list of Data Prepper hosts when given a single domain name. This list that is returned is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. See the following yaml file example of DNS lookup: +DNS discovery is preferred over static discovery when scaling out a Data Prepper cluster. DNS discovery configures a DNS provider to return a list of Data Prepper hosts when given a single domain name. This list that is returned is a [DNS A record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/), which indicates a list of IP addresses of a given domain. See the following yaml file example of DNS lookup: ```yaml peer_forwarder: @@ -94,13 +94,13 @@ The following table provides optional configuration values. | `request_timeout` | An `int` representing the request timeout duration in milliseconds for the Peer Forwarder HTTP server. Default value is `10000`. | | `server_thread_count` | An `int` representing the number of threads used by the Peer Forwarder server. Default value is `200`.| | `client_thread_count` | An `int` representing the number of threads used by the Peer Forwarder client. Default value is `200`.| -| `maxConnectionCount` | An `int` representing the maximum number of open connections for Peer Forwarder server. Default value is `500`. | +| `maxConnectionCount` | An `int` representing the maximum number of open connections for the Peer Forwarder server. Default value is `500`. | | `discovery_mode` | A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node`, which processes events locally. | | `static_endpoints` | A `list` containing the endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. | | `domain_name` | A `String` representing the single domain name to query DNS against. Typically, used by creating multiple [DNS A records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. | | `aws_cloud_map_namespace_name` | A `String` representing the AWS Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | -| `aws_cloud_map_service_name` | A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | -| `aws_cloud_map_query_parameters` | A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | +| `aws_cloud_map_service_name` | A `String` representing the AWS Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. | +| `aws_cloud_map_query_parameters` | A `Map` of key-value pairs used to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. | | `buffer_size` | An `int` representing the maximum number of unchecked records the buffer accepts (the number of unchecked records equals the number of records written into the buffer plus the number of in-flight records not yet checked by the Checkpointing API). Default is `512`. | | `batch_size` | An `int` representing max number of records the buffer returns on read. Default is `48`. | | `aws_region` | A `String` represents the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`, and `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | @@ -120,8 +120,8 @@ The following SSL configuration table provides optional SSL configuration values | `use_acm_certificate_for_ssl` | A `Boolean` that enables TLS/SSL using the certificate and private key from AWS Certificate Manager (ACM). Default value is `false`. | | `acm_certificate_arn`| A `String` representing the ACM certificate Amazon Resource Name (ARN). The ACM certificate takes precedence over S3 or the local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. | | `acm_private_key_password` | A `String` representing the ACM private key password that will be used to decrypt the private key. If it's not provided, a random password will be generated. | -| `acm_certificate_timeout_millis` | An `Int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. | -| `aws_region` | A `String` representing the AWS Region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`, and `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | +| `acm_certificate_timeout_millis` | An `Int` representing the timeout in milliseconds required for ACM to get certificates. Default value is `120000`. | +| `aws_region` | A `String` representing the AWS Region that uses `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file`. Also required when the `ssl_key_file` is set to the `AWS S3` path, or if `discovery_mode` is set to `aws_cloud_map`. | #### Example configuration @@ -138,7 +138,7 @@ peer_forwarder: This section describes optional authentication. -`authentication` is optional and is a `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. The following yaml file provides an exmaple for authentication: +`Authentication` is optional and is a `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. The default value is `unauthenticated`. The following yaml file provides an exmaple for authentication: ```yaml peer_forwarder: @@ -154,7 +154,9 @@ Core Peer Forwarder introduces the following custom metrics. All the metrics are ### Timer -- `requestForwardingLatency`: Measures latency of requestes forwarded by the Peer Forwarder client. +Peer Forwarder's timer capability provides the following information: + +- `requestForwardingLatency`: Measures latency of requests forwarded by the Peer Forwarder client. - `requestProcessingLatency`: Measures latency of requests processed by the Peer Forwarder server. ### Counter @@ -168,10 +170,10 @@ The following table provides counter metric options. | `requestsSuccessful`| Measures the total number of successful requests. Requests with HTTP response code `200`. | | `requestsTooLarge`| Measures the total number of requests that are too large to be written to the Peer Forwarder buffer. Requests with HTTP response code `413`. | | `requestTimeouts`| Measures the total number of requests that time out while writing content to the Peer Forwarder buffer. Requests with HTTP response code `408`. | -| `requestsUnprocessable`| Measures the total number of requests that failed due to unprocessable entity. Requests with HTTP response code `422`. | +| `requestsUnprocessable`| Measures the total number of requests that fail due to an unprocessable entity. Requests with HTTP response code `422`. | | `badRequests`| Measures the total number of requests with bad request format. Requests with HTTP response code `400`. | -| `recordsSuccessfullyForwarded`| Measures the total number of forwarded records successfully. | -| `recordsFailedForwarding`| Measures the total number of records failed to be forwarded. | +| `recordsSuccessfullyForwarded`| Measures the total number of successfully forwarded records. | +| `recordsFailedForwarding`| Measures the total number of records fail to be forwarded. | | `recordsToBeForwarded` | Measures the total number of records to be forwarded. | | `recordsToBeProcessedLocally` | Measures the total number of records to be processed locally. | | `recordsActuallyProcessedLocally`| Measures the total number of records actually processed locally. This value is the sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. | From 069ef713a9f6a1a91c515bfbeb53a562405ca92a Mon Sep 17 00:00:00 2001 From: carolxob Date: Wed, 4 Jan 2023 10:00:55 -0700 Subject: [PATCH 32/32] Minor changes. Signed-off-by: carolxob --- _data-prepper/peer-forwarder.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/peer-forwarder.md b/_data-prepper/peer-forwarder.md index 71b16fe957c..f2c5358350a 100644 --- a/_data-prepper/peer-forwarder.md +++ b/_data-prepper/peer-forwarder.md @@ -181,4 +181,4 @@ The following table provides counter metric options. ### Gauge -`peerEndpoints` Measures number of dynamically discovered peer Data Prepper endpoints. For `static` mode, the size is fixed. +`peerEndpoints` Measures the number of dynamically discovered peer Data Prepper endpoints. For `static` mode, the size is fixed.