Tail input plugin doesn't respect SIGTERM signal #3323
Comments
|
It may be a same issue with #3185 (comment) |
v1.12.2 which fixes for #3324 is already released: https://rubygems.org/gems/fluentd/versions/1.12.2 |
@shenmuxiaosen @ashie For what it is worth, I could not reproduce this <source>
@type tail
path test.log
pos_file fluentd-containers.log.pos
time_key @timestamp
tag kubernetes.*
read_from_head true
format json
</source>and set up a moderately-busy writer: $ while true; do cat log >> test.log; sleep 0.01; done;I can confirm Fluentd always shutdown cleanly on SIGTERM. I tested Fluentd v1.12.1 and v1.12.2. Both worked fine. Neither I saw I'm a bit suspecting that the real issue may not in Fluentd per se, but I'd like to hear from @shenmuxiaosen if upgrading to v1.12.2 resolves |
ashie
added
waiting-for-user
In our Kubernetes cluster, we configured the shut down grace period for fluentd container to be 5 minutes. So K8s will send SIGTERM signal right away after pod shutdown requested and waited 5 minutes to send the kill signal. Then our observation is that tail input plugin kept reading new logs during this 5 minutes. Ideally we expect tail to stop right away and the grace period is just used to clear buffers and preparing for shutdown. FYI, we are still using v1.12.1 as ashie mentioned v1.12.2 introduces another bug. |
|
Maybe I've reproduced this issue. fluentd/lib/fluent/plugin/in_tail.rb Lines 912 to 940 in 4455d34 The log throttling feature #3185 may resolve this issue. |
ashie
added
bug
Why it blocks shoudown is that it is occurred on the main thread during startup process. It can avoid by But when I use this option and killing the process while reading a large file, following error is occurred. It doesn't seem graceful: |
|
@ashie If that is the case, wouldn't this resolve this issue? do
....
read_more = false if shutdown?
while read_more |
Thanks. Yes, I think something like that is required. |
|
#3380 has been merged. |
Check CONTRIBUTING guideline first and here is the list to help us investigate the problem.
Describe the bug
We use tail input plugin to read logs. And Fluentd is deployed as daemonset in Kubernetes. During deployment, Kubernetes will first send a SIGTERM signal to Fluentd container to ask it to gracefully shut down. But tail input plugin keeps reading data until the container eventually killed.
To Reproduce
Run Fluentd in Kubernetes and trigger a deployment.
Expected behavior
When receive SIGTERM, tail should stop reading, keep record of current position and flush out memory buffers.
Your Environment
td-agent 4.1.0 fluentd 1.12.1 (e3effa337593618cbd7f0f4ef071766df1ec69a0)Ubuntu 18.04.5 LTS (Bionic Beaver)5.4.0-1039-azureIf you hit the problem with older fluentd version, try latest version first.
Your Configuration
Your Error Log
Additional context
N/A
The text was updated successfully, but these errors were encountered: