lease: rate limit revoke runLoop #8099
Conversation
gyuho
force-pushed
the
rate-limit-lease-expiration
branch
from
1a5f816
to
cc9f075
Compare
etcdserver/server.go
Outdated
| } | ||
| revokerLimiter.Wait(context.Background()) |
There was a problem hiding this comment.
Should we use c.ctx instead of the context.Background()? and if Wait returns an error, how should we handle it?
There was a problem hiding this comment.
Wait won't never return error if we pass Background, unless we mis-configure in the beginning
etcdserver/server.go
Outdated
| } | ||
| revokerLimiter.Wait(context.Background()) | ||
| lid := lease.ID | ||
| s.goAttach(func() { |
There was a problem hiding this comment.
maxPendingRevokes is designed to take at most maxPendingRevokes(16 in this case) lease revoke at same time. By removing that, it means that there can be 1000 go routines that revoking the lease at the same time. Is that fine? I think the original design was to limit number of simultaneous revokes call to etcd.
etcdserver/server.go
Outdated
| maxPendingRevokes = 16 | ||
| // TODO: make this configurable? | ||
| // maximum number of leases to revoke per second | ||
| leaseRevokeRate = 1000 |
There was a problem hiding this comment.
leaseRevokesPerSecond?
etcdserver/server.go
Outdated
| } | ||
| revokerLimiter.Wait(context.Background()) |
There was a problem hiding this comment.
this rate limiting probably belongs in lessor.runLoop instead of the etcd server
gyuho
force-pushed
the
rate-limit-lease-expiration
branch
from
cc9f075
to
1e5a8ac
Compare
gyuho
changed the title
lease/lessor.go
Outdated
| @@ -412,7 +419,8 @@ func (le *lessor) Stop() { | |||
| func (le *lessor) runLoop() { | |||
| defer close(le.doneC) | |||
|
|
|||
| for { | |||
| revokerLimiter := rate.NewLimiter(rate.Limit(leaseRevokesPerSecond), leaseRevokesPerSecond) | |||
| for revokerLimiter.Wait(context.Background()) == nil { | |||
There was a problem hiding this comment.
I don't think this will work since findExpiredLeases can give len(ls) > leaseRevokesPerSecond. Maybe just have something like if len(ls) > leaseRevokesPerSecond/2 { ls = ls[:leaseRevokesPerSecond/2]} before posting to expiredC?
gyuho
force-pushed
the
rate-limit-lease-expiration
branch
2 times, most recently
from
515ee34
to
6b9338c
Compare
lease/lessor.go
Outdated
| @@ -422,6 +426,10 @@ func (le *lessor) runLoop() { | |||
| le.mu.Unlock() | |||
|
|
|||
| if len(ls) != 0 { | |||
| // keep half for next iteration | |||
Fix etcd-io#8097. Signed-off-by: Gyu-Ho Lee <[email protected]>
gyuho
force-pushed
the
rate-limit-lease-expiration
branch
from
6b9338c
to
0011b78
Compare
|
lgtm |
Fix #8097.