log rotation/compressing/purging

[kfox1111]

I've tried many log shippers and in general, have had a hard time configuring them with appropriate log rotation and log purging to ensure it doesn't end up with the possibility of either dropping log entries or sending duplicate entries. This is because the different tools handle the different jobs, but the workflow between log rotation, log shipping, log compressing, and log purging seems to be intimately tied together. Its usually left as an exercise to the operator to figure out how to setup this complex orchestration.

Should fluent-bit implement it all internally so the operator doesn't need to? It knows when it has sent a log entry, so knows when its safe to delete a log file. knows when its ok to compress a log file since it doesn't need to ever read it again, etc. It could be a killer feature.

[edsiper]

Actually log shippers needs to be generic for all application use cases, log rotation is always complicated by it nature. So if a feature like this exits, do you mean that it could do:

• monitored files, check file size, if exceed limit perform rotation and compressing
• purge old log files based on configuration

so it can behave as a logrotate on steroids ?

[kfox1111]

In some environments, its really bad to ever drop a message. If log rotation(deletion) isn't properly syncronized with log shipping, the log data could get deleted before shipped. Its been pretty hard to ensure this workflow is completely safe with the tools being separate/unaware of each other. Its the log shipper that has the knowledge of if data has been successfully sent or not, not the log rotation daemon/job that knows that.

[xujihui1985]

it is also better if fluentbit can handle purge, otherwise we will need another cronjob to purge the history log

[debu99]

any update ?

[tlefevre]

I don't think fluent-bit should perform rotation. The only thing I think fluent-bit could do is compression / purging.

If you take ex. logback, or similar frameworks, logback needs to handle the rotation otherwise I think it would most likely just blow up? Logback supports compression, but that would not work because fluent-bit would need to keep reading that file and if hasn't shipped it completely yet that would be almost as good as deleting the file?

But if fluent-bit is knowledgeable of the fact that log files do get rotated by the logging framework, it could then handle compression and purging of log files. If fluent-bit follows a file, has the file handle on it and witnesses the rotation by the logging framework it could make sure that all lines in the log file are transmitted (like the tail plugin does). After it is done with the file, it could compress them and keep them on disk for a set number of days and after those days purge the compressed log files?

It would need to be an entirely new class of plugin I guess, because it's outside the data pipeline of fluent-bit.

[macropin]

We're looking to implement fluent-bit as a log shipper for applications deployed on Kubernetes. In our use case we are deploying fluent-bit as a sidecar pod which picks up the logs and ships them to AWS CloudWatch Logs.

For our use case it would be preferable if fluent-bit could automatically truncate the log files as the logs are shipped.

Right now to truncate logs I need to implement a process which truncates the files (or moves & renames), or add another sidecar with crond and logrotate. It would be far better solution for fluent-bit to truncate the file when tailing logs.

[sudeep-ib]

Is this feature request going to be prioritised at some point?

[mrmacow]

So will we ever see this? The best to purge is the one whom knows if the data is shipped

[bbkfhq]

It seems like as of now it's not possible to safely rotate the output log file without losing logs (it's possible for logs to be lost while an external program such as logrotate is busy performing the log rotation).

If log rotation is not going to be performed by fluent-bit, is it possible to provide a mechanism for a safe way of rotating the logs?

[nuclearpidgeon]

I don't know why they were never linked but pull request #1890 has been open for implementing this feature since... Jan 2020. (although it has required several rounds of feedback/updates since then).

There seems to still be a lot of outstanding open comments on it, and the whole thing is quite out of date so it will likely need a rebase (and fighting whatever conflict demons may spawn from attempting that).

Edit: PR #7541 also addresses this but was also never linked here...

[nuclearpidgeon]

While I'm here. Anyone got any tips for a nice way to do an equivalent of logrotate on Windows when running Fluentbit as a Windows Service?