could not enqueue records into the ring buffer #7071
Description
Bug Report
Describe the bug
Error Log:
[2023/03/26 07:32:13] [error] [input:tail:tail.0] could not enqueue records into the ring buffer
[tail.0] failed buffer write, retries=0
[tail.0] failed buffer write, retries=1
[tail.0] failed buffer write, retries=2
[tail.0] failed buffer write, retries=3
[tail.0] failed buffer write, retries=4
[tail.0] failed buffer write, retries=5
[tail.0] failed buffer write, retries=0
[tail.0] failed buffer write, retries=1
[tail.0] failed buffer write, retries=2
[tail.0] failed buffer write, retries=3
[tail.0] failed buffer write, retries=4
[tail.0] failed buffer write, retries=5
[tail.0] failed buffer write, retries=6
[tail.0] failed buffer write, retries=7
[tail.0] failed buffer write, retries=8
[tail.0] failed buffer write, retries=0
[tail.0] failed buffer write, retries=0
[tail.0] failed buffer write, retries=0
[tail.0] failed buffer write, retries=0
[tail.0] failed buffer write, retries=1
[tail.0] failed buffer write, retries=2
[tail.0] failed buffer write, retries=0
fluent-bit config
apiVersion: v1
data:
custom_parsers.conf: |
[PARSER]
Name docker_no_time
Format json
Time_Keep Off
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
[PARSER]
Name custom-nginx
Format regex
Regex ^(?<domain>[^ ]*) (?<remote_ip>[^\, ]*)(?:,? ?[^\[]*) \[(?<time>[^\]]*)\]\[(?<msec>[^ ]*)\] "(?<method>[^ ]*) (?<url>[^\?]*)(?:\??)(?<url_params>.*) (HTTP\/)?(?<http_version>[^ ]*)" (?<status>[^ ]*) (?<body_bytes>[^ ]*) "(?<referer>[^\"]*)" "(?<http_user_agent>[^\"]*)" "(?<body>[^ ]*)" (?<request_time>[^ ]*) (?<response_time>[^ ]*) (?<app_id>[^ ]*) (?<digest>[^ ]*) "(?<firmware_version>[^ ]*)" "(?<device_model>[^ ]*)" "(?<app_version>[^ ]*)" "(?<token>[^\?]*)" "(?<resp_body>[^"]*)" "(?<request_id>[^ "]*)"
Time_Format %d/%b/%Y:%H:%M:%S %z
Time_Key time
[PARSER]
Name logfmt
Format logfmt
fluent-bit.conf: |
[SERVICE]
Daemon Off
Flush 5
Log_Level warn
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
storage.path /var/log/flb-storage/
storage.sync normal
storage.checksum off
storage.backlog.mem_limit 300M
storage.max_chunks_up 256
storage.metrics on
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser docker, cri
Tag kube.*
DB /var/log/flb_kube.db
Mem_Buf_Limit 256MB
Rotate_Wait 30
Buffer_Chunk_Size 5MB
Buffer_Max_Size 512MB
threaded on
[INPUT]
Name tail
Tag kube_audit.*
Path /var/log/kube-apiserver-audit.log
Parser docker_no_time
DB /var/log/flb_kube_audit.db
Mem_Buf_Limit 32MB
Rotate_Wait 30
Buffer_Chunk_Size 5MB
Buffer_Max_Size 32MB
threaded on
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude Off
[FILTER]
Name grep
Match *
Exclude http_user_agent ELB-HealthChecker
[FILTER]
Name geoip2
Match *
Database /fluent-bit/geo/GeoLite2-City.mmdb
Lookup_key remote_ip
Record country_name remote_ip %{country.names.en}
Record country_code2 remote_ip %{country.iso_code}
[FILTER]
Name nest
Match *
Operation nest
Wildcard country*
Nest_under geoip
[FILTER]
Name lua
Match *
script /fluent-bit/scripts/qnap_filter.lua
call ip_filter
[OUTPUT]
Name loki
Match *
Host loki-loki-distributed-distributor.monitoring.svc
Port 3100
labels job=fluent-bit
label_map_path /fluent-bit/etc/labelmap.json
remove_keys kubernetes.container_hash, kubernetes.docker_id, kubernetes.annotations, stream, _p
auto_kubernetes_labels off
line_format json
Retry_Limit False
labelmap.json: |
{
"kubernetes": {
"container_name": "container",
"host": "node",
"labels": {
"app": "app",
"k8s_app": "app",
"release": "release"
},
"namespace_name": "namespace",
"pod_name": "instance"
},
"stream": "stream"
}
edit deployment:
**To Reproduce**
When I add "fluent-bit/parser" in deployment
template:
metadata:
annotations:
fluentbit.io/parser: custom-nginx
Fluentibit can't send log to loki. Log miss.
**Your Environment**
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used: kubernetes 1.25.6. fluent-bit 2.0.8
Could everyone have any suggestion?