The aws_kinesis_firehose sink is logging "connection closed before message completed" errors

[yalinglee]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

We are trying out Vector's aws_kinesis_firehose sink and noticed that every few minutes we'd get an error like this:

{"error":"Error during dispatch: Failed to make HTTP(S) request: connection closed before message completed","host":"10.x.x.x","message":"Retrying after error.","metadata":{"kind":"event","level":"WARN","module_path":"vector::sinks::util::retries","target":"vector::sinks::util::retries"},"pid":123,"source_type":"internal_logs","timestamp":"2022-04-28T20:50:12.508104113Z"}

Could this be related to the issue reported in #9671? If those are retries, does that mean we are not actually losing messages?

Configuration

sinks:
  firehose_kinesis_sink:
    type: aws_kinesis_firehose
    inputs:
      - log_file
    stream_name: my_stream
    region: us-west-1
    batch:
      max_bytes: 1000000
      max_events: 100
      timeout_secs: 5
    encoding:
      codec: ndjson

Version

0.20.11.58.1

Example Data

- - 10.131.75.132 [24/Apr/2022:23:59:59 +0000] "GET / HTTP/1.1" 200 9187 "-" "0.17.0" 0.000

References

#9671

[jszwedko]

Hi @yalinglee ! Yeah, that does look similar to #9671. Vector will retry those requests so you won't be dropping data.

[marcus-lim-lrn]

It seems although the issue looks similar to #12850, which was closed due to PR #12930, and #9671 which was closed due to PR #9703, neither actually addresses this issue which is still ongoing :(

Although it is clear this is not impacting the data being sent, it would still be nice to have the logging lowered from WARN to INFO