out_http: ReadTimeout resulting in an invalid json in following chunks

[tyarimi]

Check CONTRIBUTING guideline first and here is the list to help us investigate the problem.

Describe the bug
I use out_http to output json arrays over HTTP. When the server is slow to respond, and ReadTimeout occurs, fluentd retries sending the chunk, but the JSON format is invalid (looks like the end of the buffer is trimmed).

To Reproduce
Send any log over http and make the server not respond for at least read_timeout seconds.

Expected behavior
Fluentd should retry the exact same chunk, or at least not slice the chunk in an arbitrary position.

Your Environment

• Fluentd version 1.11.2
• Operating system: Debian container running on MacOS

Your Configuration

<match **>
        @type http

        endpoint "http://localhost:8080" # local test server
        open_timeout 2
        read_timeout 10

        content_type "application/json"
        retryable_response_codes [503]

        json_array true
        <format>
            @type json
            add_newline false
        </format>

        <buffer>
            # Interval for flushing event chunks
            flush_interval 5s

            # The number of threads to flush/write chunks in parallel
            flush_thread_count 2

            # The max size of each chunks: events will be written into chunks until
            # the size of chunks become this size
            chunk_limit_size 16mb
        </buffer>
    </match>

Your Error Log

Example ReadTimeout error:

[warn]: #0 failed to flush the buffer. retry_time=275 next_retry_seconds=2020-09-29 10:01:02 +0000 chunk="5b070cb025df79d5cd538adb86617115" error_class=Net::ReadTimeout error="Net::ReadTimeout"

Additional context

Example for an invalid JSON output (note the json array closing AFTER the payload is trimmed):

[{"worker=0","service":"fluentd","tag":"fluent.info"},"sourcetype":"fluentd","agent_timestamp":"2020-09-29T09:44:27Z","hostname":]

(the actual payload is larger, ±14kb)

[repeatedly]

Patch: #3144