Skip to content

Conversation

@mergify
Copy link
Contributor

@mergify mergify bot commented Jun 13, 2025

Proposed commit message

Filestream now logs one line at warn level per scan with the number of files that are too small to be ingested.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

## Disruptive User Impact
## Author's Checklist

How to test this PR locally

  1. Create a few small files

    echo "foo" > /tmp/small-1.log
    echo "bar" > /tmp/small-2.log
    echo "foo bar" > /tmp/small-3.log
  2. Build & start Filebeat with the following configuration:

    filebeat.inputs:
    - type: filestream
      id: log-small-files-cannot-be-ingested
      enabled: true
      paths:
        - /tmp/small*.log
    
    output.discard:
      enabled: true
    
    logging:
      level: debug
      to_stderr: true
  3. Look for the following logs

    {
      "log.level": "debug",
      "@timestamp": "2025-06-12T13:41:17.034-0400",
      "log.logger": "scanner",
      "log.origin": {
        "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles",
        "file.name": "filestream/fswatch.go",
        "file.line": 398
      },
      "message": "cannot start ingesting from file \"/tmp/small-1.log\": filesize of \"/tmp/small-1.log\" is 4 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion",
      "service.name": "filebeat",
      "filestream_id": "log-small-files-cannot-be-ingested",
      "ecs.version": "1.6.0"
    }
    {
      "log.level": "debug",
      "@timestamp": "2025-06-12T13:41:17.034-0400",
      "log.logger": "scanner",
      "log.origin": {
        "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles",
        "file.name": "filestream/fswatch.go",
        "file.line": 398
      },
      "message": "cannot start ingesting from file \"/tmp/small-2.log\": filesize of \"/tmp/small-2.log\" is 4 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion",
      "service.name": "filebeat",
      "filestream_id": "log-small-files-cannot-be-ingested",
      "ecs.version": "1.6.0"
    }
    {
      "log.level": "debug",
      "@timestamp": "2025-06-12T13:41:17.034-0400",
      "log.logger": "scanner",
      "log.origin": {
        "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles",
        "file.name": "filestream/fswatch.go",
        "file.line": 398
      },
      "message": "cannot start ingesting from file \"/tmp/small-3.log\": filesize of \"/tmp/small-3.log\" is 8 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion",
      "service.name": "filebeat",
      "filestream_id": "log-small-files-cannot-be-ingested",
      "ecs.version": "1.6.0"
    }
    {
      "log.level": "warn",
      "@timestamp": "2025-06-12T13:41:17.034-0400",
      "log.logger": "scanner",
      "log.origin": {
        "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles",
        "file.name": "filestream/fswatch.go",
        "file.line": 421
      },
      "message": "3 files are too small to be ingested, files need to be at least 1024 in size for ingestion to start. To change this behaviour set 'prospector.scanner.fingerprint.length' and 'prospector.scanner.fingerprint.offset'. Enable debug logging to see all file names.",
      "service.name": "filebeat",
      "filestream_id": "log-small-files-cannot-be-ingested",
      "ecs.version": "1.6.0"
    }

    They will repeat every scan of the file system (default is 10s)

Related issues

## Use cases
## Screenshots
## Logs


This is an automatic backport of pull request #44751 done by [Mergify](https://mergify.com).

(cherry picked from commit b91d891)

# Conflicts:
#	NOTICE.txt
#	filebeat/input/filestream/prospector_creator.go
#	go.mod
#	go.sum
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Jun 13, 2025
@mergify mergify bot requested review from a team as code owners June 13, 2025 15:31
@mergify mergify bot added the backport label Jun 13, 2025
@mergify mergify bot requested review from belimawr and removed request for a team June 13, 2025 15:31
@mergify mergify bot added the conflicts There is a conflict in the backported pull request label Jun 13, 2025
@mergify mergify bot requested a review from rdner June 13, 2025 15:31
@mergify
Copy link
Contributor Author

mergify bot commented Jun 13, 2025

Cherry-pick of b91d891 has failed:

On branch mergify/bp/8.17/pr-44751
Your branch is up to date with 'origin/8.17'.

You are currently cherry-picking commit b91d8913e.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   filebeat/input/filestream/fswatch.go
	modified:   filebeat/input/filestream/fswatch_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   NOTICE.txt
	both modified:   filebeat/input/filestream/prospector_creator.go
	both modified:   go.mod
	both modified:   go.sum

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 13, 2025
@pierrehilbert pierrehilbert added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Jun 15, 2025
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 15, 2025
@mergify
Copy link
Contributor Author

mergify bot commented Jun 16, 2025

This pull request has not been merged yet. Could you please review and merge it @belimawr? 🙏

@belimawr belimawr merged commit e2eeb4b into 8.17 Jun 16, 2025
170 of 173 checks passed
@belimawr belimawr deleted the mergify/bp/8.17/pr-44751 branch June 16, 2025 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport conflicts There is a conflict in the backported pull request Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants