Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fluentd not releasing log file on log rotation #2426

Closed
kevinkirkup opened this issue May 29, 2019 · 4 comments · Fixed by #3782
Closed

fluentd not releasing log file on log rotation #2426

kevinkirkup opened this issue May 29, 2019 · 4 comments · Fixed by #3782
Labels
bug Something isn't working

Comments

@kevinkirkup
Copy link

kevinkirkup commented May 29, 2019 •
edited
Loading

  • fluentd or td-agent version.
  • td-agent 1.4.2
td-agent:
  Installed: 3.4.1-0
  • Environment information:
    • Operating system:
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
  • Kernel version:
4.15.0-48-generic

  • Your configuration
    Tailing multiple log files

  • Your problem explanation. If you have an error logs, write it together.
    Looks like fluentd is holding on to it's own log file during rotation and that is causing the disks to fill up event though the file is being deleted:

root@xxxx:/var/log/td-agent# lsof +L1
COMMAND     PID USER   FD   TYPE DEVICE   SIZE/OFF NLINK   NODE NAME
ovsdb-ser  2893 root    7u   REG    8,1        159     0  63256 /tmp/#63256 (deleted)
fluentd   10897 root    7w   REG    8,1 6624796672     0 257220 /var/log/td-agent/td-agent.log (deleted)
fluentd   10897 root    9w   REG    8,1 6624796672     0 257220 /var/log/td-agent/td-agent.log (deleted)
ruby      10902 root    7w   REG    8,1 6624796672     0 257220 /var/log/td-agent/td-agent.log (deleted)
@ganmacs ganmacs added the bug Something isn't working label Dec 23, 2019
@kenhys
Copy link
Contributor

kenhys commented Oct 28, 2020 •
edited
Loading

I can reproduce it:

$ dpkg -l | grep td-agent
ii  td-agent                               3.4.1-0                                         amd64        Treasure Agent: A data collector for Treasure Data

Summary:

Result:

  • sudo /usr/sbin/logrotate -f /etc/logrotate.d/td-agent
$ ls -la
total 16
drwxr-xr-x  3 td-agent td-agent 4096 Oct 28 10:54 .
drwxrwxr-x 12 root     syslog   4096 Oct 28 10:21 ..
drwxr-xr-x  3 td-agent td-agent 4096 Oct 28 10:21 buffer
-rw-r--r--  1 td-agent td-agent 3111 Oct 28 10:54 td-agent.log

$ sudo lsof | grep td-agent.log
fluentd   12476               td-agent    7w      REG              252,3     3111    2228581 /var/log/td-agent/td-agent.log
fluentd   12476               td-agent   10w      REG              252,3     3111    2228581 /var/log/td-agent/td-agent.log
ruby-time 12476 12479         td-agent    7w      REG              252,3     3111    2228581 /var/log/td-agent/td-agent.log
ruby-time 12476 12479         td-agent   10w      REG              252,3     3111    2228581 /var/log/td-agent/td-agent.log
signal_th 12476 12480         td-agent    7w      REG              252,3     3111    2228581 /var/log/td-agent/td-agent.log
signal_th 12476 12480         td-agent   10w      REG              252,3     3111    2228581 /var/log/td-agent/td-agent.log
socket_ma 12476 12481         td-agent    7w      REG              252,3     3111    2228581 /var/log/td-agent/td-agent.log
socket_ma 12476 12481         td-agent   10w      REG              252,3     3111    2228581 /var/log/td-agent/td-agent.log
ruby      12482               td-agent    7w      REG              252,3     3111    2228581 /var/log/td-agent/td-agent.log
ruby-time 12482 12485         td-agent    7w      REG              252,3     3111    2228581 /var/log/td-agent/td-agent.log
thread.rb 12482 12486         td-agent    7w      REG              252,3     3111    2228581 /var/log/td-agent/td-agent.log
thread.rb 12482 12487         td-agent    7w      REG              252,3     3111    2228581 /var/log/td-agent/td-agent.log
drb.rb:14 12482 12488         td-agent    7w      REG              252,3     3111    2228581 /var/log/td-agent/td-agent.log
thread.rb 12482 12489         td-agent    7w      REG              252,3     3111    2228581 /var/log/td-agent/td-agent.log
thread.rb 12482 12493         td-agent    7w      REG              252,3     3111    2228581 /var/log/td-agent/td-agent.log

sudo /usr/sbin/logrotate -f /etc/logrotate.d/td-agent 

$ sudo lsof | grep td-agent.log
fluentd   12476               td-agent    7w      REG              252,3     3179    2228581 /var/log/td-agent/td-agent.log.1
fluentd   12476               td-agent   10w      REG              252,3       66    2229665 /var/log/td-agent/td-agent.log
ruby-time 12476 12479         td-agent    7w      REG              252,3     3179    2228581 /var/log/td-agent/td-agent.log.1
ruby-time 12476 12479         td-agent   10w      REG              252,3       66    2229665 /var/log/td-agent/td-agent.log
signal_th 12476 12480         td-agent    7w      REG              252,3     3179    2228581 /var/log/td-agent/td-agent.log.1
signal_th 12476 12480         td-agent   10w      REG              252,3       66    2229665 /var/log/td-agent/td-agent.log
socket_ma 12476 12481         td-agent    7w      REG              252,3     3179    2228581 /var/log/td-agent/td-agent.log.1
socket_ma 12476 12481         td-agent   10w      REG              252,3       66    2229665 /var/log/td-agent/td-agent.log
ruby      12482               td-agent    7w      REG              252,3       66    2229665 /var/log/td-agent/td-agent.log
ruby-time 12482 12485         td-agent    7w      REG              252,3       66    2229665 /var/log/td-agent/td-agent.log
thread.rb 12482 12486         td-agent    7w      REG              252,3       66    2229665 /var/log/td-agent/td-agent.log
thread.rb 12482 12487         td-agent    7w      REG              252,3       66    2229665 /var/log/td-agent/td-agent.log
drb.rb:14 12482 12488         td-agent    7w      REG              252,3       66    2229665 /var/log/td-agent/td-agent.log
thread.rb 12482 12489         td-agent    7w      REG              252,3       66    2229665 /var/log/td-agent/td-agent.log
thread.rb 12482 12493         td-agent    7w      REG              252,3       66    2229665 /var/log/td-agent/td-agent.log

$ sudo /usr/sbin/logrotate -vf /etc/logrotate.d/td-agent 
reading config file /etc/logrotate.d/td-agent
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state

Handling 1 logs

rotating pattern: /var/log/td-agent/td-agent.log  forced from command line (30 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/td-agent/td-agent.log
  Now: 2020-10-28 10:56
  Last rotated at 2020-10-28 10:55
  log needs rotating
rotating log /var/log/td-agent/td-agent.log, log->rotateCount is 30
dateext suffix '-20201028'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
renaming /var/log/td-agent/td-agent.log.30.gz to /var/log/td-agent/td-agent.log.31.gz (rotatecount 30, logstart 1, i 30), 
old log /var/log/td-agent/td-agent.log.30.gz does not exist
renaming /var/log/td-agent/td-agent.log.29.gz to /var/log/td-agent/td-agent.log.30.gz (rotatecount 30, logstart 1, i 29), 
old log /var/log/td-agent/td-agent.log.29.gz does not exist
renaming /var/log/td-agent/td-agent.log.28.gz to /var/log/td-agent/td-agent.log.29.gz (rotatecount 30, logstart 1, i 28), 
old log /var/log/td-agent/td-agent.log.28.gz does not exist
renaming /var/log/td-agent/td-agent.log.27.gz to /var/log/td-agent/td-agent.log.28.gz (rotatecount 30, logstart 1, i 27), 
old log /var/log/td-agent/td-agent.log.27.gz does not exist
renaming /var/log/td-agent/td-agent.log.26.gz to /var/log/td-agent/td-agent.log.27.gz (rotatecount 30, logstart 1, i 26), 
old log /var/log/td-agent/td-agent.log.26.gz does not exist
renaming /var/log/td-agent/td-agent.log.25.gz to /var/log/td-agent/td-agent.log.26.gz (rotatecount 30, logstart 1, i 25), 
old log /var/log/td-agent/td-agent.log.25.gz does not exist
renaming /var/log/td-agent/td-agent.log.24.gz to /var/log/td-agent/td-agent.log.25.gz (rotatecount 30, logstart 1, i 24), 
old log /var/log/td-agent/td-agent.log.24.gz does not exist
renaming /var/log/td-agent/td-agent.log.23.gz to /var/log/td-agent/td-agent.log.24.gz (rotatecount 30, logstart 1, i 23), 
old log /var/log/td-agent/td-agent.log.23.gz does not exist
renaming /var/log/td-agent/td-agent.log.22.gz to /var/log/td-agent/td-agent.log.23.gz (rotatecount 30, logstart 1, i 22), 
old log /var/log/td-agent/td-agent.log.22.gz does not exist
renaming /var/log/td-agent/td-agent.log.21.gz to /var/log/td-agent/td-agent.log.22.gz (rotatecount 30, logstart 1, i 21), 
old log /var/log/td-agent/td-agent.log.21.gz does not exist
renaming /var/log/td-agent/td-agent.log.20.gz to /var/log/td-agent/td-agent.log.21.gz (rotatecount 30, logstart 1, i 20), 
old log /var/log/td-agent/td-agent.log.20.gz does not exist
renaming /var/log/td-agent/td-agent.log.19.gz to /var/log/td-agent/td-agent.log.20.gz (rotatecount 30, logstart 1, i 19), 
old log /var/log/td-agent/td-agent.log.19.gz does not exist
renaming /var/log/td-agent/td-agent.log.18.gz to /var/log/td-agent/td-agent.log.19.gz (rotatecount 30, logstart 1, i 18), 
old log /var/log/td-agent/td-agent.log.18.gz does not exist
renaming /var/log/td-agent/td-agent.log.17.gz to /var/log/td-agent/td-agent.log.18.gz (rotatecount 30, logstart 1, i 17), 
old log /var/log/td-agent/td-agent.log.17.gz does not exist
renaming /var/log/td-agent/td-agent.log.16.gz to /var/log/td-agent/td-agent.log.17.gz (rotatecount 30, logstart 1, i 16), 
old log /var/log/td-agent/td-agent.log.16.gz does not exist
renaming /var/log/td-agent/td-agent.log.15.gz to /var/log/td-agent/td-agent.log.16.gz (rotatecount 30, logstart 1, i 15), 
old log /var/log/td-agent/td-agent.log.15.gz does not exist
renaming /var/log/td-agent/td-agent.log.14.gz to /var/log/td-agent/td-agent.log.15.gz (rotatecount 30, logstart 1, i 14), 
old log /var/log/td-agent/td-agent.log.14.gz does not exist
renaming /var/log/td-agent/td-agent.log.13.gz to /var/log/td-agent/td-agent.log.14.gz (rotatecount 30, logstart 1, i 13), 
old log /var/log/td-agent/td-agent.log.13.gz does not exist
renaming /var/log/td-agent/td-agent.log.12.gz to /var/log/td-agent/td-agent.log.13.gz (rotatecount 30, logstart 1, i 12), 
old log /var/log/td-agent/td-agent.log.12.gz does not exist
renaming /var/log/td-agent/td-agent.log.11.gz to /var/log/td-agent/td-agent.log.12.gz (rotatecount 30, logstart 1, i 11), 
old log /var/log/td-agent/td-agent.log.11.gz does not exist
renaming /var/log/td-agent/td-agent.log.10.gz to /var/log/td-agent/td-agent.log.11.gz (rotatecount 30, logstart 1, i 10), 
old log /var/log/td-agent/td-agent.log.10.gz does not exist
renaming /var/log/td-agent/td-agent.log.9.gz to /var/log/td-agent/td-agent.log.10.gz (rotatecount 30, logstart 1, i 9), 
old log /var/log/td-agent/td-agent.log.9.gz does not exist
renaming /var/log/td-agent/td-agent.log.8.gz to /var/log/td-agent/td-agent.log.9.gz (rotatecount 30, logstart 1, i 8), 
old log /var/log/td-agent/td-agent.log.8.gz does not exist
renaming /var/log/td-agent/td-agent.log.7.gz to /var/log/td-agent/td-agent.log.8.gz (rotatecount 30, logstart 1, i 7), 
old log /var/log/td-agent/td-agent.log.7.gz does not exist
renaming /var/log/td-agent/td-agent.log.6.gz to /var/log/td-agent/td-agent.log.7.gz (rotatecount 30, logstart 1, i 6), 
old log /var/log/td-agent/td-agent.log.6.gz does not exist
renaming /var/log/td-agent/td-agent.log.5.gz to /var/log/td-agent/td-agent.log.6.gz (rotatecount 30, logstart 1, i 5), 
old log /var/log/td-agent/td-agent.log.5.gz does not exist
renaming /var/log/td-agent/td-agent.log.4.gz to /var/log/td-agent/td-agent.log.5.gz (rotatecount 30, logstart 1, i 4), 
old log /var/log/td-agent/td-agent.log.4.gz does not exist
renaming /var/log/td-agent/td-agent.log.3.gz to /var/log/td-agent/td-agent.log.4.gz (rotatecount 30, logstart 1, i 3), 
old log /var/log/td-agent/td-agent.log.3.gz does not exist
renaming /var/log/td-agent/td-agent.log.2.gz to /var/log/td-agent/td-agent.log.3.gz (rotatecount 30, logstart 1, i 2), 
old log /var/log/td-agent/td-agent.log.2.gz does not exist
renaming /var/log/td-agent/td-agent.log.1.gz to /var/log/td-agent/td-agent.log.2.gz (rotatecount 30, logstart 1, i 1), 
renaming /var/log/td-agent/td-agent.log.0.gz to /var/log/td-agent/td-agent.log.1.gz (rotatecount 30, logstart 1, i 0), 
old log /var/log/td-agent/td-agent.log.0.gz does not exist
log /var/log/td-agent/td-agent.log.31.gz doesn't exist -- won't try to dispose of it
renaming /var/log/td-agent/td-agent.log to /var/log/td-agent/td-agent.log.1
creating new /var/log/td-agent/td-agent.log mode = 0640 uid = 111 gid = 116
running postrotate script

$ sudo lsof | grep td-agent.log
fluentd   12476               td-agent    7w      REG              252,3     3179    2228581 /var/log/td-agent/td-agent.log.1 (deleted)
fluentd   12476               td-agent   10w      REG              252,3       66    2229759 /var/log/td-agent/td-agent.log
ruby-time 12476 12479         td-agent    7w      REG              252,3     3179    2228581 /var/log/td-agent/td-agent.log.1 (deleted)
ruby-time 12476 12479         td-agent   10w      REG              252,3       66    2229759 /var/log/td-agent/td-agent.log
signal_th 12476 12480         td-agent    7w      REG              252,3     3179    2228581 /var/log/td-agent/td-agent.log.1 (deleted)
signal_th 12476 12480         td-agent   10w      REG              252,3       66    2229759 /var/log/td-agent/td-agent.log
socket_ma 12476 12481         td-agent    7w      REG              252,3     3179    2228581 /var/log/td-agent/td-agent.log.1 (deleted)
socket_ma 12476 12481         td-agent   10w      REG              252,3       66    2229759 /var/log/td-agent/td-agent.log
ruby      12482               td-agent    7w      REG              252,3       66    2229759 /var/log/td-agent/td-agent.log
ruby-time 12482 12485         td-agent    7w      REG              252,3       66    2229759 /var/log/td-agent/td-agent.log
thread.rb 12482 12486         td-agent    7w      REG              252,3       66    2229759 /var/log/td-agent/td-agent.log
thread.rb 12482 12487         td-agent    7w      REG              252,3       66    2229759 /var/log/td-agent/td-agent.log
drb.rb:14 12482 12488         td-agent    7w      REG              252,3       66    2229759 /var/log/td-agent/td-agent.log
thread.rb 12482 12489         td-agent    7w      REG              252,3       66    2229759 /var/log/td-agent/td-agent.log
thread.rb 12482 12493         td-agent    7w      REG              252,3       66    2229759 /var/log/td-agent/td-agent.log

$ ls -la
total 24
drwxr-xr-x  3 td-agent td-agent 4096 Oct 28 11:04 .
drwxrwxr-x 12 root     syslog   4096 Oct 28 10:21 ..
drwxr-xr-x  3 td-agent td-agent 4096 Oct 28 11:03 buffer
-rw-r-----  1 td-agent td-agent   66 Oct 28 11:04 td-agent.log
-rw-r-----  1 td-agent td-agent  134 Oct 28 11:04 td-agent.log.1
-rw-r--r--  1 td-agent td-agent  934 Oct 28 11:04 td-agent.log.2.gz

@kenhys
Copy link
Contributor

kenhys commented Oct 28, 2020

/etc/logrotate.d/td-agent coniguration is not changed, so it still affects to td-agent 4.0.1.

@repeatedly
Copy link
Member

fluentd uses serverengine's logger feature.
So need to fix serverengine first.

@kenhys
Copy link
Contributor

kenhys commented Oct 29, 2020

Thanks, it seems that #2200 is also same issue.

@kenhys kenhys mentioned this issue Nov 26, 2020
Closed
@ashie ashie moved this to In-Progress in Fluentd Kanban Jan 6, 2022
@fujimotos fujimotos removed the status in Fluentd Kanban Feb 10, 2022
This was referenced Jun 13, 2022
Merged
Merged
ashie added a commit that referenced this issue Jun 13, 2022
@ashie
Bump up required serverengine to 2.3.0
a102d9e 
* Enable to set interval of restarting workers
  See #3749 and #3768
* Fix a bug that fluentd doesn't release its own log file even after
  rotated by external tools
  Fix #2200 and #2426

Signed-off-by: Takuro Ashie <[email protected]>
@ashie ashie moved this to Done in Fluentd Kanban Jun 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Fluentd Kanban
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Bump up required serverengine to 2.3.0 fluent/fluentd
4 participants
@repeatedly @kevinkirkup @kenhys @ganmacs