Nakadi clients resilience to partial outage and partial success

[adyach]

Nakadi publishing API accepts events in batches. It can fail to publish some events from the batch to underlying storage (Apache Kafka). In that case Nakadi publishing API will return error that batch was partially successful.
It can create problems the following problems, depending on how the Nakadi client and the publishing application deals with this partial success response:

• increase in traffic on Nakadi publishing API due to Nakadi clients retrying the whole batch over and over
• the application retries identical batches which prevents application from progressing

The following should be done to decrease the possibility of mentioned problems:

• Nakadi client should contain a note to developers that publishing can experience partial success. This should be in the client documentation and ideally also within the self contained code documentation, raising awareness for the users, e.g. via docstrings.

• An optional retry method on batch level can be provided for the whole batch, but the default strategy must contain a backoff solution in case of continued errors to publish to Nakadi.

• An optional retry method can be provided that only re-publishes unsuccessful events to Nakadi. This retry must also support a backoff strategy by default.

• Clients must expose the result of a publishing request in a way that developers can understand that there is the possibility of a partial success for batch publishing.