403 errors when delivering logs to AWS OpenSearch Serverless after a successful period #228
Description
Describe the bug
I'm attempting to deliver logs to AWS Opensearch Serverless. I'm running logstash as a deployment on AWS EKS. I'm attempting to using the IAM role to attached to the EKS EC2 node that's running the associated pod to authenticate with Opensearch serverless.
When I start the deployment/pod up it successfully delivers messages into Opensearch serverless, however after a short period, (20 seconds - 5 minutes) logs fail to be delivered to Opensearch serverless with 403 errors e.g.
[2023-08-31T11:18:47,585][ERROR][logstash.outputs.opensearch][main][43ac7955e25a1efb882bfe67309ff3cf447bfc3b85dc94a4119f84872473b07b] Encountered a retryable error (will retry with exponential backoff) {:code=>403, :url=>"[https://REDACTED.eu-west-1.aoss.amazonaws.com:443/_bulk ](https://REDACTED.eu-west-1.aoss.amazonaws.com/_bulk) ", :content_length=>52619}
If I stop the deployment/pod and start it again the process repeats itself. Logs can be delivered for a short period after which they are rejected with 403 errors.
My output config is as follows
output {
opensearch {
hosts => ["@Model.OpenSearchHost"]
index => "audit-%{[@@metadata][index]}"
action => "create"
auth_type => {
type => 'aws_iam'
aws_access_key_id => ''
aws_secret_access_key => ''
service_name => 'aoss'
region => 'eu-west-1'
}
default_server_major_version => 2
legacy_template => false
}
}
To Reproduce
See Above
Expected behavior
Logs should be able to be delivered to Opensearch serverless consistently
Plugins
none
Screenshots
none
Host/Environment (please complete the following information):
- Docker Hub container - opensearchproject/logstash-oss-with-opensearch-output-plugin:8.9.0
- EKS version 1.25 running Amazon Linux AMD nodes
Additional context
none