feat(writer): add retry logic #2459
Conversation
This change introduces a Fibonacci retry policy (with jitter) to the agent writer to mitigate networking issues (e.g. timeouts, broken pipes, ...), similar to what the profiler does already. Resolves DataDog#1413.
ImpactWhen the agent is available, the impact of the retry logic is negligible
However, when the agent is not available, or unreachable over the majority of the connection attempts, there is a slight impact from attempting to connect multiple times in a row. |
| self._metrics_dist("http.errors", tags=["type:err"]) | ||
| self._metrics_dist("http.dropped.bytes", len(encoded)) | ||
| self._metrics_dist("http.dropped.traces", len(enc_traces)) | ||
| if raise_exc: | ||
| raise | ||
| e.reraise() |
There was a problem hiding this comment.
There's also a reraise=True in tenacity if that fits you better.
There was a problem hiding this comment.
Ah thanks, I missed that from the docs! I think I'll keep it as it is so to avoid having the same set of exception classes in two different places.
|
@Kyle-Verhoog what release did this land in? I couldn't see it in the release notes? |
|
@inverse It will be available in v0.50.0 for which we have pre-release versions. You can try them out from pypi: https://pypi.org/project/ddtrace/#history. |
Description
This change introduces a Fibonacci retry policy (with jitter) to the agent writer to mitigate networking issues (e.g. timeouts, broken pipes, ...), similar to what the profiler does already.
Memory usage might increase up to an extra payload buffer size, meaning that occasionally the tracer would require about 8MB more memory for processing. That is because the buffer will start filling up again while the writer retries flushing the previous payload.
Resolves #1413.
Checklist