fix(loki.source.file): Keep positions for compressed files when reading is finished#5723
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a regression introduced in Alloy 1.8 where compressed files read by loki.source.file would have their position removed from the positions file after being fully consumed (at EOF). This caused the compressed files to be re-ingested on every Alloy restart. The fix adds a shouldKeepPosition() method that preserves the position when decompression is enabled, relying on the existing positions cleanup mechanism to remove entries for files that no longer exist.
Changes:
- Added
shouldKeepPositon()method to the tailer that preserves positions when the component is stopping or when decompression is enabled - Updated
TestTailer_Compressionsto validate that positions are kept after compressed file consumption and that files are not re-ingested on subsequent runs
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
internal/component/loki/source/file/tailer.go |
Adds shouldKeepPositon() method and replaces direct componentStopping() check with it in the stop() method |
internal/component/loki/source/file/tailer_test.go |
Updates TestTailer_Compressions to pass false for componentStopping to verify position retention with decompression |
Contributor
Author
@thampiotr TestTailer_Compressions was already trying to test this behavior but because it always returned true for componentIsStopping it passed. I changed that to false, this makes this test fail on main but pass in this pr |
kalleep
force-pushed
the
kalleep/loki-source-file-decompression-position
branch
from
c602558 to
e756377
Compare
thampiotr
approved these changes
Mar 10, 2026
grafana-alloybot Bot
pushed a commit
that referenced
this pull request
Mar 10, 2026
…ng is finished (#5723) ### Pull Request Details This is a [regression that have been in alloy since 1.8](#2428). When tailing compressed files we stop once we hit EOF. The issue is that tailer, and previously decompressor, will remove the stored position because component is not stopping. Then when alloy is re-started we read the files again. Instead we should never remove the position here when decompression is enabled, position structure will run [cleanup](https://github.com/grafana/alloy/blob/c47338fec6eb26fa9f2d040cc002f78721b93427/internal/component/loki/source/internal/positions/positions.go#L303) on files that no longer exists so it will be able to clean it up when files disapears. ### Issue(s) fixed by this Pull Request Fixes: #5365 ### Notes to the Reviewer <!-- Add any relevant notes for the reviewers and testers of this PR. --> ### PR Checklist <!-- Remove items that do not apply. For completed items, change [ ] to [x]. --> - [ ] Documentation added - [x] Tests updated - [ ] Config converters updated (cherry picked from commit fb41d0a)
kalleep
added a commit
that referenced
this pull request
Mar 10, 2026
…ng is finished [backport] (#5741) ## Backport of #5723 This PR backports #5723 to release/v1.14. ### Original PR Author @kalleep ### Description ### Pull Request Details This is a [regression that have been in alloy since 1.8](#2428). When tailing compressed files we stop once we hit EOF. The issue is that tailer, and previously decompressor, will remove the stored position because component is not stopping. Then when alloy is re-started we read the files again. Instead we should never remove the position here when decompression is enabled, position structure will run [cleanup](https://github.com/grafana/alloy/blob/c47338fec6eb26fa9f2d040cc002f78721b93427/internal/component/loki/source/internal/positions/positions.go#L303) on files that no longer exists so it will be able to clean it up when files disapears. ### Issue(s) fixed by this Pull Request Fixes: #5365 ### Notes to the Reviewer <!-- Add any relevant notes for the reviewers and testers of this PR. --> ### PR Checklist <!-- Remove items that do not apply. For completed items, change [ ] to [x]. --> - [ ] Documentation added - [x] Tests updated - [ ] Config converters updated --- *This backport was created automatically.* Co-authored-by: Karl Persson <23356117+kalleep@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Details
This is a regression that have been in alloy since 1.8. When tailing compressed files we stop once we hit EOF. The issue is that tailer, and previously decompressor, will remove the stored position because component is not stopping. Then when alloy is re-started we read the files again.
Instead we should never remove the position here when decompression is enabled, position structure will run cleanup on files that no longer exists so it will be able to clean it up when files disapears.
Issue(s) fixed by this Pull Request
Fixes: #5365
Notes to the Reviewer
PR Checklist