[8.17](backport #44751) Log that files are too small to ingest at warn level #44808
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed commit message
Filestream now logs one line at warn level per scan with the number of files that are too small to be ingested.
Checklist
I have commented my code, particularly in hard-to-understand areasI have made corresponding changes to the documentationI have made corresponding change to the default configuration filesI have added tests that prove my fix is effective or that my feature worksCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.## Disruptive User Impact## Author's ChecklistHow to test this PR locally
Create a few small files
Build & start Filebeat with the following configuration:
Look for the following logs
{ "log.level": "debug", "@timestamp": "2025-06-12T13:41:17.034-0400", "log.logger": "scanner", "log.origin": { "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles", "file.name": "filestream/fswatch.go", "file.line": 398 }, "message": "cannot start ingesting from file \"/tmp/small-1.log\": filesize of \"/tmp/small-1.log\" is 4 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion", "service.name": "filebeat", "filestream_id": "log-small-files-cannot-be-ingested", "ecs.version": "1.6.0" } { "log.level": "debug", "@timestamp": "2025-06-12T13:41:17.034-0400", "log.logger": "scanner", "log.origin": { "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles", "file.name": "filestream/fswatch.go", "file.line": 398 }, "message": "cannot start ingesting from file \"/tmp/small-2.log\": filesize of \"/tmp/small-2.log\" is 4 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion", "service.name": "filebeat", "filestream_id": "log-small-files-cannot-be-ingested", "ecs.version": "1.6.0" } { "log.level": "debug", "@timestamp": "2025-06-12T13:41:17.034-0400", "log.logger": "scanner", "log.origin": { "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles", "file.name": "filestream/fswatch.go", "file.line": 398 }, "message": "cannot start ingesting from file \"/tmp/small-3.log\": filesize of \"/tmp/small-3.log\" is 8 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion", "service.name": "filebeat", "filestream_id": "log-small-files-cannot-be-ingested", "ecs.version": "1.6.0" } { "log.level": "warn", "@timestamp": "2025-06-12T13:41:17.034-0400", "log.logger": "scanner", "log.origin": { "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles", "file.name": "filestream/fswatch.go", "file.line": 421 }, "message": "3 files are too small to be ingested, files need to be at least 1024 in size for ingestion to start. To change this behaviour set 'prospector.scanner.fingerprint.length' and 'prospector.scanner.fingerprint.offset'. Enable debug logging to see all file names.", "service.name": "filebeat", "filestream_id": "log-small-files-cannot-be-ingested", "ecs.version": "1.6.0" }They will repeat every scan of the file system (default is 10s)
Related issues
## Use cases## Screenshots## LogsThis is an automatic backport of pull request #44751 done by [Mergify](https://mergify.com).