Skip to content

[exporter/awss3exporter] Add the retry mode, max attempts and max backoff settings #39509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
atoulme merged 1 commit into from
May 20, 2025
Merged

[exporter/awss3exporter] Add the retry mode, max attempts and max backoff settings #39509

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .chloggen/main.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: enhancement

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: awss3exporter

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Add the retry mode, max attempts and max backoff to the settings

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [36264]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: [user]
60 changes: 40 additions & 20 deletions exporter/awss3exporter/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,26 +20,29 @@ This exporter targets to support proto/json format.

The following exporter configuration parameters are supported.

| Name | Description | Default |
|:--------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------|
| `region` | AWS region. | "us-east-1" |
| `s3_bucket` | S3 bucket | |
| `s3_prefix` | prefix for the S3 key (root directory inside bucket). | |
| `s3_partition_format` | filepath formatting for the partition; See [strftime](https://www.man7.org/linux/man-pages/man3/strftime.3.html) for format specification. | "year=%Y/month=%m/day=%d/hour=%H/minute=%M" |
| `role_arn` | the Role ARN to be assumed | |
| `file_prefix` | file prefix defined by user | |
| `marshaler` | marshaler used to produce output data | `otlp_json` |
| `encoding` | Encoding extension to use to marshal data. Overrides the `marshaler` configuration option if set. | |
| `encoding_file_extension` | file format extension suffix when using the `encoding` configuration option. May be left empty for no suffix to be appended. | |
| `endpoint` | (REST API endpoint) overrides the endpoint used by the exporter instead of constructing it from `region` and `s3_bucket` | |
| `storage_class` | [S3 storageclass](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) | STANDARD |
| `acl` | [S3 Object Canned ACL](https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl) | none (does not set by default) |
| `s3_force_path_style` | [set this to `true` to force the request to use path-style addressing](http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html) | false |
| `disable_ssl` | set this to `true` to disable SSL when sending requests | false |
| `compression` | should the file be compressed | none |
| `sending_queue` | [exporters common queuing](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md) | disabled |
| `timeout` | [exporters common timeout](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md) | 5s |
| `resource_attrs_to_s3` | determines the mapping of S3 configuration values to resource attribute values for uploading operations. | |
| Name | Description | Default |
|:--------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------|
| `region` | AWS region. | "us-east-1" |
| `s3_bucket` | S3 bucket | |
| `s3_prefix` | prefix for the S3 key (root directory inside bucket). | |
| `s3_partition_format` | filepath formatting for the partition; See [strftime](https://www.man7.org/linux/man-pages/man3/strftime.3.html) for format specification. | "year=%Y/month=%m/day=%d/hour=%H/minute=%M" |
| `role_arn` | the Role ARN to be assumed | |
| `file_prefix` | file prefix defined by user | |
| `marshaler` | marshaler used to produce output data | `otlp_json` |
| `encoding` | Encoding extension to use to marshal data. Overrides the `marshaler` configuration option if set. | |
| `encoding_file_extension` | file format extension suffix when using the `encoding` configuration option. May be left empty for no suffix to be appended. | |
| `endpoint` | (REST API endpoint) overrides the endpoint used by the exporter instead of constructing it from `region` and `s3_bucket` | |
| `storage_class` | [S3 storageclass](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) | STANDARD |
| `acl` | [S3 Object Canned ACL](https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl) | none (does not set by default) |
| `s3_force_path_style` | [set this to `true` to force the request to use path-style addressing](http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html) | false |
| `disable_ssl` | set this to `true` to disable SSL when sending requests | false |
| `compression` | should the file be compressed | none |
| `sending_queue` | [exporters common queuing](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md) | disabled |
| `timeout` | [exporters common timeout](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md) | 5s |
| `resource_attrs_to_s3` | determines the mapping of S3 configuration values to resource attribute values for uploading operations. | |
| `retry_mode` | The retryer implementation, the supported values are "standard", "adaptive" and "nop". "nop" will set the retryer as `aws.NopRetryer`, which effectively disable the retry. | standard |
| `retry_max_attempts` | The max number of attempts for retrying a request if the `retry_mode` is set. Setting max attempts to 0 will allow the SDK to retry all retryable errors until the request succeeds, or a non-retryable error is returned. | 3 |
| `retry_max_backoff` | the max backoff delay that can occur before retrying a request if `retry_mode` is set | 20s |

### Marshaler

Expand Down Expand Up @@ -141,6 +144,23 @@ metric/YYYY/MM/DD/HH/mm
...
```

## Retry

Standard is the default retryer implementation used by service clients. See the [retry](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/aws/retry) package documentation for details on what errors are considered as retryable by the standard retryer implementation.

See also the [aws-sdk-go reference](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-retries-timeouts.html) for more information.

```yaml
exporters:
awss3:
s3uploader:
region: 'eu-central-1'
s3_bucket: 'databucket'
s3_prefix: 'metric'
retry_mode: "standard"
retry_max_attempts: 5
retry_max_backoff: "30s"
```

## AWS Credential Configuration

Expand Down
22 changes: 22 additions & 0 deletions exporter/awss3exporter/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,20 @@ package awss3exporter // import "github.com/open-telemetry/opentelemetry-collect

import (
"errors"
"time"

"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/config/configcompression"
"go.opentelemetry.io/collector/exporter/exporterhelper"
"go.uber.org/multierr"
)

const (
DefaultRetryMode = "standard"
DefaultRetryMaxAttempts = 3
DefaultRetryMaxBackoff = 20 * time.Second
)

// S3UploaderConfig contains aws s3 uploader related config to controls things
// like bucket, prefix, batching, connections, retries, etc.
type S3UploaderConfig struct {
Expand Down Expand Up @@ -40,6 +47,17 @@ type S3UploaderConfig struct {
// before uploading to S3.
// Valid values are: `gzip` or no value set.
Compression configcompression.Type `mapstructure:"compression"`

// RetryMode specifies the retry mode for S3 client, default is "standard".
// Valid values are: "standard", "adaptive", or "nop".
// "nop" will disable retry by setting the retryer to aws.NopRetryer.
RetryMode string `mapstructure:"retry_mode"`
// RetryMaxAttempts specifies the maximum number of attempts for S3 client.
// Default is 3 (SDK default).
RetryMaxAttempts int `mapstructure:"retry_max_attempts"`
// RetryMaxBackoff specifies the maximum backoff delay for S3 client.
// Default is 20 seconds (SDK default).
RetryMaxBackoff time.Duration `mapstructure:"retry_max_backoff"`
}

type MarshalerType string
Expand Down Expand Up @@ -116,5 +134,9 @@ func (c *Config) Validate() error {
errs = multierr.Append(errs, errors.New("marshaler does not support compression"))
}
}

if c.S3Uploader.RetryMode != "nop" && c.S3Uploader.RetryMode != "standard" && c.S3Uploader.RetryMode != "adaptive" {
errs = multierr.Append(errs, errors.New("invalid retry mode, must be either 'standard', 'adaptive' or 'nop'"))
}
return errs
}
Loading
Loading