Filebeat 1.1.2 does not record correct inode values of rotated logs on exit

[jstangroome]

From the discussion at https://discuss.elastic.co/t/filebeat-1-1-2-does-not-record-correct-inode-values-of-rotated-logs-on-exit/45749

I am seeing filebeat re-ship rotated log files when the filebeat service restarts.

We are experiencing the problem on a local ext4 file system on Ubuntu 14.04 running in AWS EC2.

While filebeat remains running, it is correctly processing the files. The problem is that when it persists its state to the registry file, it is persisting the older inode values of the files before they were rotated, not the inodes of the files at those paths at the time filebeat exited.

I have created a basic configuration and bash script which starts filebeat, writes to a log file, rotates it a couple of times (with pauses in between to allow filebeat to notice) and then stops filebeat.

Full repro script, conf, and output log available as a gist:
https://gist.github.com/jstangroome/a1a70306cbd7a9420be15ec2bbe109a0

After filebeat stops, the registry has persisted the wrong inodes for the log files present on disk. As such, when filebeat starts again, an existing log file will appear to be a new file without an existing offset and its contents will be shipped a second time.

My repro script was run on a local ext4 file system on Ubuntu 14.04 running in a Virtualbox VM.

This might be an edge case not handled in #1010

[tsg]

Thanks for the well documented bug report. We'll investigate.

[ruflin]

@jstangroome I can confirm this bug also on master and Mac OS X. Will not create a system test to reproduce it.

[ruflin]

Here is a PR that introduces a system check to test for the problem: #1375 Next step: Fixing it.

[ruflin]

@jstangroome I think I found a solution (see #1375). The problem occured when multiple files were rotated at the same time and it was written to the first file before the rotation was detected. Can you test if master works now for you?

@tsg I tried to find a fix that is minimal and backportable. Should we backport it?

[tsg]

@ruflin We're not yet sure if we'll have another release in the 1.X series, but if the cherry-pick is trivial, please do it to the 1.2 branch. Alternatively, you can tag this as backport and we do it when we're sure we will have a release.

[ruflin]

@jstangroome We really appreciate all the work you put into this one. Thanks a lot.

[jstangroome]

@ruflin No problems. I'm just happy you've been able to push a fix so quickly and I'm hoping I'll get the opportunity to try the latest build in our environment soon.