[0.14.20] [in_tail] prevents docker from removing containers

[zette]

• fluentd 0.14.20
• cenos 7.3
• running docker version 17.05.0-ce, build 89658be
• docker runinng with devicmapper storage driver, json file log and log rotation set to 100m: -s devicemapper --log-opt max-size=100m --log-opt
• fluentd config:

<system>
  # equal to -qq option
  log_level debug
</system>

<source>
  @type tail
  path /var/lib/docker/containers/**/*-json.log
  pos_file /var/log/fluentd.pos
  format json
  keep_time_key true
  time_format %Y-%m-%dT%H:%M:%S.%N%z
  tag dev.*
</source>

With this configuration local fluentd aggregates logs from all running containers json files, and sends them to external host for processing. This works well for running containers, but when the container is stopped and is being removed with docker rm [container_name], the docker demon reports:
Error response from daemon: Unable to remove filesystem for [container_id]: remove /var/lib/docker/containers/[container_id]/shm: device or resource busy

The above behavior is not happeing all the times, but often enough to cripple CI builds. Unfortunately we can not use docker fluentd forwarder, since the developers need to have realtime access to container logs.

Some work with docker log handling, and file rotation for fluent-bit has been already done in fluent/fluent-bit#287
After stopping/restarting the fluentd, container directory is unlocked and removed without problems. It seems that in_tail is holding lock on container log file, preventing docker from removing container's filesystem.

[ph-One]

This is far and away my #1 issue with fluentd; has been for years now. Please find a solution -- does anyone have a good workaround for having both docker logs work locally, as well as having fluentd forward on those same logs?

[zette]

I've end up with stopping fluentd each time I'm removing old containers.
Not elegant or efficient, but at least works... for my current container load and rotation.

[roffe]

@zette i have a cron job restarting fluentd every 30 minutes to clean up pos_file and to release log files

https://github.com/roffe/kube-gelf/blob/master/cron.yaml

[repeatedly]

It seems that in_tail is holding lock on container log file

Yes and in_tail opens target files to tail logs until shutdown by default.
I'm not sure but how about setting open_on_every_update true in in_tail setting?

[ayushmathur86]

Hi, I'm facing a similar issue as #287 eventhough the following configuration is present for docker log file rotation:
--log-driver=json-file --log-opt max-size=2G --log-opt max-file=10

I can see multiple files being generated, i.e. log files are being rotated once they hit 2G size mark, but fluentd is still reading the main file (*-json.log) and not the others (*log.1, .2, etc).
Is there any modification required on fluentd configuration which I'm missing ?

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions]

This issue was automatically closed because of stale in 30 days