-
Notifications
You must be signed in to change notification settings - Fork 399
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: processor rate limit #1321
Conversation
3346726
to
9d1bac1
Compare
return "rate limit processor for logtail" | ||
} | ||
|
||
func (p *ProcessorRateLimit) ProcessLogs(logArray []*protocol.Log) []*protocol.Log { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
新的插件,v1、v2都实现下吧
|
||
Algorithm algorithm | ||
limitMetric pipeline.CounterMetric | ||
processedMetric pipeline.CounterMetric |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
time.Sleep(time.Second) | ||
outLogs = s.processor.ProcessLogs(logArray) | ||
c.Assert(len(outLogs), check.Equals, 3) | ||
c.Assert(len(outLogs[0].Contents), check.Equals, 3) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
指标也要增加观测项
9d1bac1
to
e199f1a
Compare
047d052
to
cb0c1aa
Compare
cb0c1aa
to
4694ff4
Compare
* feat: processor rate limit * fix racing
* feat: processor rate limit * fix racing
算法设计
限速插件基于令牌桶算法的设计:
数据结构
令牌桶
key的计算方法为:
垃圾回收
令牌桶中一旦出现了key所对应的一条日志,那么无论其后续还是否出现,都会在内存中占用一个bucket。因此,引入了一个垃圾回收机制: