transient_error_retry - Retry transient errors when attempting to insert_tasks, and re-raise if the reinsert fails.

[markshaule-wf]

Instead of silently failing on a TransientError on task inserts, we check the options to see if we should retry the failure (after a delay). If the retry fails, we re-raise.

Currently, the default option is to retry. I can't think of a scenario that the called would not want to have the errors retry, or at least trigger an error.
As a side effect, this may break custom implementations of _insert_tasks since we have added a parameter.

@beaulyddon-wf @robertkluin-wf @johnlockwood-wf @tannermiller-wf @rosshendrickson-wf @macleodbroad-wf @tylertreat-wf

FYI: @jasonaguilon-wf @erikpetersen-wf

[erikpetersen-wf]

Agree, why even have a kwarg to indicate raising an error, that should be a default?

[erikpetersen-wf]

Why is there a backoff on performing the retry?

[tylertreat-wf]

@erikpetersen-wf TransientError is typically a result of the queue being unavailable or contended for some reason. Google recommends a small backoff when you're seeing these types of errors.

[tylertreat-wf]

To clarify, the current behavior for how TransientError is handled is not to fail silently but to split the batch into two and re-insert. Is this mainly to handle the case where len(tasks) == 1? Also I suppose TransientErrors tend to be mitigated with a backoff, so this probably makes sense.

[markshaule-wf]

@erikpetersen-wf - Yes we do have kwarg argument for the retry, and the default is 'True' at this point.

[markshaule-wf]

@tylertreat - Yes the current behaviour is to split and retry on TransientError. The issue is when we get down to 1 task and hit a Transient error, we fail silently. The other exception types (BadTaskStateError, TaskAlreadyExistsError, TombstonedTaskError), as I understand, indicate that the task has already been inserted, and we don't need to re-raise the exception.

But yes - that was the main fix - for len(tasks)==1 and a TransientError

[tylertreat-wf]

+1

[tannermiller-wf]

+1

[jasonaguilon-wf]

+1

[beaulyddon-wf]

+1