Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: processor rate limit #1321

Merged
merged 8 commits into from
Feb 27, 2024
Merged

Conversation

Abingcbc
Copy link
Collaborator

算法设计

限速插件基于令牌桶算法的设计:

  1. 在获取令牌桶时,会根据距离上次填充的时间长度,向令牌桶中补充相应数量的令牌。
  2. 如果无法获取到令牌(令牌桶中的数量小于等于0),那么该日志就会被丢弃掉。

数据结构

令牌桶

type tokenBucket struct {
	mu sync.Mutex

	limit   rate
	buckets sync.Map

	// GC thresholds and metrics
	gc struct {
		thresholds tokenBucketGCConfig
		metrics    struct {
			numCalls atomic.Int32
		}
	}
}
  1. mu:GC操作的并发锁
  2. limit:限速的速率
  3. buckets:map,存储了key到对应的令牌桶之间的关系。
    key的计算方法为:
    1. 对field名进行排序
    2. 按照排序后的顺序,取出field的值,并拼接成一个字符串
    3. 计算该字符串的hash值(uint64)作为key,节约内存空间
  4. gc:GC配置参数
type bucket struct {
	tokens        float64
	lastReplenish time.Time
}
  1. tokens:该桶内剩余的令牌数量。由于距离上一次补充的时间间隔不一定是完整的秒,所以可能会补充非整数的令牌。
  2. lastReplenish:上次补充令牌的时间。

垃圾回收

令牌桶中一旦出现了key所对应的一条日志,那么无论其后续还是否出现,都会在内存中占用一个bucket。因此,引入了一个垃圾回收机制:

  1. 在处理了一定数量的日志后(默认10000条),触发垃圾回收
  2. 垃圾回收会重新补充每个桶的令牌
  3. 如果该桶的令牌大于等于了limit的令牌数,那么说明该桶所对应key,在一段时间内都没有出现
  4. 回收这些桶

return "rate limit processor for logtail"
}

func (p *ProcessorRateLimit) ProcessLogs(logArray []*protocol.Log) []*protocol.Log {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

新的插件,v1、v2都实现下吧


Algorithm algorithm
limitMetric pipeline.CounterMetric
processedMetric pipeline.CounterMetric
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

time.Sleep(time.Second)
outLogs = s.processor.ProcessLogs(logArray)
c.Assert(len(outLogs), check.Equals, 3)
c.Assert(len(outLogs[0].Contents), check.Equals, 3)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

指标也要增加观测项

docs/cn/plugins/processor/extended/processor-rate-limit.md Outdated Show resolved Hide resolved
plugins/processor/ratelimit/token_bucket.go Outdated Show resolved Hide resolved
plugins/processor/ratelimit/processor_rate_limit.go Outdated Show resolved Hide resolved
plugins/processor/ratelimit/token_bucket.go Outdated Show resolved Hide resolved
plugins/processor/ratelimit/token_bucket.go Outdated Show resolved Hide resolved
@Abingcbc Abingcbc force-pushed the processor_rate_limit branch from 9d1bac1 to e199f1a Compare February 25, 2024 15:28
@Abingcbc Abingcbc force-pushed the processor_rate_limit branch 2 times, most recently from 047d052 to cb0c1aa Compare February 26, 2024 03:36
@Abingcbc Abingcbc force-pushed the processor_rate_limit branch from cb0c1aa to 4694ff4 Compare February 26, 2024 04:31
@yyuuttaaoo yyuuttaaoo merged commit ddf8f6f into alibaba:main Feb 27, 2024
15 checks passed
linrunqi08 pushed a commit that referenced this pull request Feb 29, 2024
* feat: processor rate limit

* fix racing
yyuuttaaoo pushed a commit that referenced this pull request Feb 29, 2024
* feat: processor rate limit

* fix racing
@yyuuttaaoo yyuuttaaoo added this to the v2.0 milestone Mar 27, 2024
@yyuuttaaoo yyuuttaaoo added the feature request New feature request label Mar 27, 2024
@Abingcbc Abingcbc deleted the processor_rate_limit branch November 27, 2024 03:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants