[Feature Request] Partial retry of output

[nokute78]

Is your feature request related to a problem? Please describe.

#2935 (comment)
To fix the issue, we need to send multiple http requests at flush callback.
In this situation, if some requests are failed to send, we should retry for these failed requests.

e.g.

• http request 1 : 200
• http request 2 : 503
• http request 3 : 200

We should retry only request 2.

Current flush design

In my understanding,

1. Flush callback passes const pointer of messagepack. source
2. If it needs to retry, engine passes same pointer to callback again.

Problem

Because of 1, we can't modify the events in callback.
e.g. we can't remove the record which is sent successfully.
It means we can't notify to engine what events are failed.

Because of 2, we needs to cache the failed requests and events and compare between incoming events and cached events at plugin side.

Describe the solution you'd like

For example, I want to modify incoming record at cb_flush of output plugin.
Or engine passes only failed events to cb_flush at retry.

Describe alternatives you've considered

Additional context

Similar issue is opened at fluentd repo.It is still opened.
fluent/fluentd#1911

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.