log missing rate is about 60% when setting buffer chunk size to 32k #8024
Description
Bug Report
Describe the bug
fluentbit is used to upload logs into external servers.
when using fluentbit 2.0.8
The log missing rate is about 60% with below settings
buffer_chunk_size 32k
buffer_max_size 32k
Solution #1: The log missing rate is about 0% with below settings.
buffer_chunk_size 1M
buffer_max_size 1M
Solution #2: using fluentbit 1.9.5/1.9.6, the log missing rate is about 0% without changing parameters.
buffer_chunk_size 32k
buffer_max_size 32k
https://docs.fluentbit.io/manual/pipeline/inputs/tail.
The document is not clear to set the parameters, like Buffer_Chunk_Size and Buffer_Max_Size.
Is it correct to apply such fix(Solution #1 changing buffer_chunk_size and buffer_max_size)? If yes, could you explain more technical details based on this?
To Reproduce
- Example configuration of fluentbit when the issue happened:
[INPUT]
name tail
tag event.kafka.ingress
alias kafka.ingress
**buffer_chunk_size 32k
buffer_max_size 32k**
read_from_head true
refresh_interval 5
rotate_wait 10
skip_empty_lines off
skip_long_lines true
key message
db /var/log/logshipper/kafka.ingress.db
db.sync normal
db.locking true
db.journal_mode off
path /var/log/aaa/ingress/*/*/*/*,/var/log/aaa/ingress/*/*/*/*/*,/var/log/aaa/ingress/*/*/*/*/*/*
exclude_path /var/log/logshipper/logshipper.log,/var/log/aaa/ingress/*.gz,/var/log/aaa/ingress/*.tgz
mem_buf_limit 20MB
parser json
ignore_older 11m
- Steps to reproduce the problem:
- generate lots of logs
- fluentbit to upload logs to external log server
- check the logs missing by counting the log records
Expected behavior
No logs missings or close to 0 missing rate.
Your Environment
- Version used: 2.0.8, 1.9.7, 1.9.9, etc
- Configuration: above
- Environment name and version (e.g. Kubernetes? What version?): Kubernetes