Log that files are too small to ingest at warn level

[belimawr]

Proposed commit message

Filestream now logs one line at warn level per scan with the number of files that are too small to be ingested.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

## Disruptive User Impact
## Author's Checklist

How to test this PR locally

1. Create a few small files

   echo "foo" > /tmp/small-1.log
   echo "bar" > /tmp/small-2.log
   echo "foo bar" > /tmp/small-3.log

2. Build & start Filebeat with the following configuration:

   filebeat.inputs:
   - type: filestream
     id: log-small-files-cannot-be-ingested
     enabled: true
     paths:
       - /tmp/small*.log
   
   output.discard:
     enabled: true
   
   logging:
     level: debug
     to_stderr: true

3. Look for the following logs

   {
     "log.level": "debug",
     "@timestamp": "2025-06-12T13:41:17.034-0400",
     "log.logger": "scanner",
     "log.origin": {
       "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles",
       "file.name": "filestream/fswatch.go",
       "file.line": 398
     },
     "message": "cannot start ingesting from file \"/tmp/small-1.log\": filesize of \"/tmp/small-1.log\" is 4 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion",
     "service.name": "filebeat",
     "filestream_id": "log-small-files-cannot-be-ingested",
     "ecs.version": "1.6.0"
   }
   {
     "log.level": "debug",
     "@timestamp": "2025-06-12T13:41:17.034-0400",
     "log.logger": "scanner",
     "log.origin": {
       "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles",
       "file.name": "filestream/fswatch.go",
       "file.line": 398
     },
     "message": "cannot start ingesting from file \"/tmp/small-2.log\": filesize of \"/tmp/small-2.log\" is 4 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion",
     "service.name": "filebeat",
     "filestream_id": "log-small-files-cannot-be-ingested",
     "ecs.version": "1.6.0"
   }
   {
     "log.level": "debug",
     "@timestamp": "2025-06-12T13:41:17.034-0400",
     "log.logger": "scanner",
     "log.origin": {
       "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles",
       "file.name": "filestream/fswatch.go",
       "file.line": 398
     },
     "message": "cannot start ingesting from file \"/tmp/small-3.log\": filesize of \"/tmp/small-3.log\" is 8 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion",
     "service.name": "filebeat",
     "filestream_id": "log-small-files-cannot-be-ingested",
     "ecs.version": "1.6.0"
   }
   {
     "log.level": "warn",
     "@timestamp": "2025-06-12T13:41:17.034-0400",
     "log.logger": "scanner",
     "log.origin": {
       "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles",
       "file.name": "filestream/fswatch.go",
       "file.line": 421
     },
     "message": "3 files are too small to be ingested, files need to be at least 1024 in size for ingestion to start. To change this behaviour set 'prospector.scanner.fingerprint.length' and 'prospector.scanner.fingerprint.offset'. Enable debug logging to see all file names.",
     "service.name": "filebeat",
     "filestream_id": "log-small-files-cannot-be-ingested",
     "ecs.version": "1.6.0"
   }

   They will repeat every scan of the file system (default is 10s)

Related issues

• Relates Add warning about Filestream only ingesting files >= 1024 bytes. #44749
• Relates [filestream] add warning about only ingesting files >= 1024 bytes integrations#14209

## Use cases
## Screenshots
## Logs

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[leehinman]

Would it be possible to add the input id to the log message (maybe it is a field added to the logger?). If we had that, the user would know which input needs to be changed.

Right now, you have to switch to debug mode, and then you just get the file name, which means you have to work backwards from the globs to determine which input needs to be modified.

[belimawr]

Would it be possible to add the input id to the log message (maybe it is a field added to the logger?). If we had that, the user would know which input needs to be changed.

Right now, you have to switch to debug mode, and then you just get the file name, which means you have to work backwards from the globs to determine which input needs to be modified.

Thanks Lee, that was a very good point. I added it on 76ccd18, but I don't like having to use the global logger in so many tests 😞. All the "testing loggers" I found in elastic-agent-libs/logp build upon the development logger that ends up logging everything 😭 .

I'll put this PR in draft and write a noop logger for elastic-agent-libs/logp.

[leehinman]

LGTM

One optional suggestion.

[github-actions]

@Mergifyio backport 8.17 8.18 8.19 9.0

[mergify]

backport 8.17 8.18 8.19 9.0

✅ Backports have been created

Details

• #44808 [8.17](backport #44751) Log that files are too small to ingest at warn level has been created for branch 8.17 but encountered conflicts
• #44809 [8.18](backport #44751) Log that files are too small to ingest at warn level has been created for branch 8.18 but encountered conflicts
• #44810 [8.19](backport #44751) Log that files are too small to ingest at warn level has been created for branch 8.19
• #44811 [9.0](backport #44751) Log that files are too small to ingest at warn level has been created for branch 9.0 but encountered conflicts