Exponential backoff is not calculated right

[LukasJerabek]

Describe the bug

In documentation is written:

With exponential_backoff, retry_wait interval will be calculated as below:
c: constant factor, @retry_wait
b: base factor, @retry_exponential_backoff_base
k: number of retry times
total retry time: c + c * b^1 + (...) + cb^k = cb^(k+1) - 1

I was not sure, if alone c element counts as first retry time, or if first retry is issued after (c + c * b^1) time, so I dug inside source code and found out that the first retry is counted for iteration = -1, which results for the first element to be counted as c * b^(-1) - not as in the equation above. Also even if this would not go into negative numbers, proper left part of equation is probably c + c * b^1 + (...) + c * b^(k-1). Since according to source code I would assume that first retry time is c element itself (c*b^0)

inside retry_state.rb there is function calc_max_retry_timeout which gets "i" iteration variable starting from zero, but inside calc_interval function call is raw_interval function call which substracts 1 from num variable, always resulting that the first exponent is -1.

This happened probably when calc_max_retry_timeout was created in commit 92dbf54

It is important to know the correct time for exponential backoff, to know when buffer files get deleted, if target instance is not responding (for example when using forward plugin). If the time is too long, there might be too many buffer files resulting in "to many open files" error. This has happend to us once since we didnt count buffer files created before they are thrown away and there was way too many.

To Reproduce

Use buffer with exponential backoff.

Expected behavior

Retries are issued as in documentation and the documentation has correct equation. Also documentation states if the first c element is the time for the first retry, or the c + the element with k = 1 is, current state is unclear.

Your Environment

- Fluentd version: master branch with top commit a60603346483da721799ccb3faebcbdd757c5f16
- TD Agent version: -
- Operating system: -
- Kernel version: -

Your Configuration

Not important here

Your Error Log

None

Additional context

No response

[ashie]

Confirmed. Thanks for your report!
I'll fix it by #3640.

[LukasJerabek]

Thank you for the fix.

Could you please change documentation as well? Right now there is this equation in the very bottom of here:
c + c * b^1 + (...) + c*b^k = c*b^(k+1) - 1

c: constant factor, @retry_wait
b: base factor, @retry_exponential_backoff_base
k: number of retry times

the left part can be rewritten as:
c*b^0 + c * b^1 + (...) + c*b^k

where it seems that values of exponents are <0,k>, but in reality according to your change it is <0,k). so it should be rewritten like this:
c*b^0 + c * b^1 + (...) + c*b^(k-1)
since last exponent is k-1, not k. Also I dont know how about the right part of equation, it might also not fit.

[ashie]

Ah, thanks for pointing out it.
I think the documentation is correct, the implementation is still wrong.

c + c * b^1 + (...) + c*b^k = c*b^(k+1) - 1

Here, k is retry times, not try times.
First c is for the timeout of the first try, not retry.
Since k means max_retry_times in buffer config, it shouldn't k - 1.
I'll recheck it.

[LukasJerabek]

So what I originally wanted to know and why I started digging into this is the time what it takes from the first try to first retry:
What you are saying is that it is counted as c? And that the first retry time, time between first retry and second retry is counted as c + c * b^1?

[LukasJerabek]

@ashie could you please confirm this for me?

[ashie]

The current implementation might be still wrong, so we are checking it in #3649, please wait for a while.

[LukasJerabek]

ok, can we reopen this?

[daipom]

@LukasJerabek

I have sorted out the behavior here: #3649 (comment)

This is so complicated.

The correct total time must be c + c*b^1 + (...) + c*b^(k-1) = c*(b^k - 1) / (b - 1).
This fix will be done.

The existing behavior was strange as follows.

• actual time: c + c + c*b^1 + (...) + c*b^(k-1)
  • note that there are two c terms, and totally k+1 terms (One more retry was made by mistake...)
• calculated total time: c + c*b^1 + (...) + c*b^(k-2)
  • this is completely wrong. almost half the actual time.

So actual behavior, total time calculation, and documents were all wrong!