Could not fetch objects from "requester pays" enabled bucket

[dkayahan]

Hello,

I need to fetch objects from "requester pays" enabled S3 bucket in order to do so I have added "request_payer" => "requester" parameter to additional_settings field. However logstash does not recognize this option and throws the error below.

If I remove "request_payer" => "requester" option, logstash returns access denied error as expected since AWS explicitly denotes that request payer field should be added to request header when requester pays option is enabled on the bucket. Reference: https://docs.aws.amazon.com/AmazonS3/latest/userguide/ObjectsinRequesterPaysBuckets.html

I can fetch the objects by using aws cli with "--request-payer requester" successfully. So there is nothing wrong with the bucket or access key configurations.

Could you please let me know whether S3 input plugin supports requester pays option?

Thanks

Logstash Version: 7.10.2
S3 Input plugin version: 3.5.0

Config:

input {
s3 {
"access_key_id" => "xx"
"secret_access_key" => "xx"
"bucket" => "xx"
"sincedb_path" => "xx"
"prefix" => "xx"
"region" => "xx"
"additional_settings" => {
"request_payer" => "requester"
}
}
}

Error:
[ERROR] 2021-03-09 16:55:57.519 [[main]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"main", :exception=>#<ArgumentError: invalid configuration option :request_payer'>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/configuration.rb:166:in block in apply_options'", "org/jruby/RubyHash.java:1415:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/configuration.rb:161:in apply_options'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/configuration.rb:149:in build!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/base.rb:68:in build_config'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/base.rb:19:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/base.rb:105:in new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/resource.rb:169:in extract_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/resource.rb:15:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.5.0/lib/logstash/inputs/s3.rb:417:in get_s3object'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.5.0/lib/logstash/inputs/s3.rb:96:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in block in register_plugins'", "org/jruby/RubyArray.java:1809:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:386:in start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:311:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/home/xxx"], :thread=>"#<Thread:0x794e17e0 run>"}
[INFO ] 2021-03-09 16:55:57.520 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[ERROR] 2021-03-09 16:55:57.524 [Converge PipelineAction::Create

] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}

[dkayahan]

I have resolved it by manually adding required header field to aws-sdk-core signer api but a high level solution would be appreciated if there is any.