Prevent thread switching in the interval between seek and write operations to pos_file

[vitclone]

We're trying to migrate to Fluentd for transferring web-server logs to HDFS and I can't understand why we are the first who face this issue.

After deploying to one server with production load (read_from_head true) we restarted td-agent and after a while we found duplicated log records in HDFS storage. A quick examination led us to a strange inconsistency in pos_file like this line:
/data/htlogs/00000000b2ebab5f/data/htlogs/nginx/w11009.log 00000000029ba830 000000000000601a9/data00000000000332a15875.log 0000000000000000 0000000000421e96

I added debug log and found the mixing of seek and write operations from different threads. The first thread was updating log files current position (set file.pos, write) and the second thread was adding new records to pos_file (set file.pos, write, write, read file.pos, write, read file.pos).

Also, I added a warning for unparsable lines in pos_file.

[repeatedly]

Thanks for the patch.

I added debug log and found the mixing of seek and write operations from different threads.

Does this mean one in_tail causes broken pos_file?

[vitclone]

Yes, we have only one in_tail in config and this mixing of operation is the cause of inconsistencies. On our server, I found several different situations, for example:

• writing a new path while a file offset set by update_pos,
• remembering seek from an offset set by update_pos,
• remembering last_pos from an offset set by update_pos.

[repeatedly]

Sorry for the delay. Just merged!

[johanneswuerbach]

@repeatedly any chance to get a new version including this? Thanks :-)

[repeatedly]

@johanneswuerbach sorry for late reply. v1.2.6 is released in last week.