generated from amazon-archives/__template_Apache-2.0
-
Notifications
You must be signed in to change notification settings - Fork 84
Closed
Labels
Milestone
Description
Describe the bug
When batch size is large, data will be sent out in multiple partial bulk requests. If http compression is enabled in this case, we get 500 error when sending data to OpenSearch.
To Reproduce
Steps to reproduce the behavior:
- Create a sample log file with 1000 characters per line and 100 lines
- Set
pipeline.batch.size: 50inlogstash.yml; settarget_bulk_bytes => 10240andhttp_compression => truefor logstash-output-opensearch plugin in logstash.conf. This will make batch byte size larger thantarget_bulk_bytesso that the plugin will send multiple partial bulk requests. - Start logstash to read the sample log and send logs to an Amazon OpenSearch domain
- See errors like this and keep retry:
[2022-12-22T12:21:32,103][ERROR][logstash.outputs.opensearch][main][67a1dea2c259b674532fcabe49e3d0dd7401d7cab38266ee76e8920ddf9faca3] Encountered a retryable error (will retry with exponential backoff) {:code=>500, :url=>"https://xxx-domain-blnft6zvvl64fq2v43jn4b3hky.us-east-1.es.amazonaws.com:443/_bulk", :content_length=>3412, :body=>""} - See server side error:
java.io.EOFException: Unexpected end of ZLIB input stream
Expected behavior
Be able to send data to OpenSearch even when batch size is large and http compression is enabled
Plugins
Please list all plugins currently enabled.
Screenshots
If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):
- OS: [e.g. iOS]
- Version [e.g. 22]
Additional context
Add any other context about the problem here.