Skip to content

Conversation

@mergify
Copy link
Contributor

@mergify mergify bot commented Oct 14, 2024

Proposed commit message

This commit adds Debian 12 support to our system module, to support
Debian 12 we need to use the journald input to collect the system
logs.

To support it, a new, internal, input system-logsis introduced, it is responsible
for deciding whether the log input or journald must be used. If var.paths is defined
in the module configuration, system-logs looks at the files, if any of the globs resolves
to one or more files the log input is used, otherwise the jouranld input is used.

This behaviour can be overridden by setting var.use_journald or var.use_files,
which will force the use of journald or files.

Other changes:

  • Journald input now support filtering by facilities
  • System tests for modules now support handling journal files
  • The TESTING_FILEBEAT_FILEPATTERN environment variable now is a
    comma separated list of globs, it defaults to .log,*.journal
  • Multiple lint warnings are fixed
  • The documentation has been updated where needed.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

  • Debian 12 is now supported by Filebeat's system module
  • On Debian 12 the system module will use the journald input
  • The ingest pipelines for the system module were modified from 1 per fileset to 3
  • The journald input currently only reads logs from the current boot

## Author's Checklist

How to test this PR locally

Run the tests

mage buildSystemTestBinary
mage docker:ComposeUp
source $(mage PythonVirtualEnv)/bin/activate
INTEGRATION_TESTS=1 BEAT_STRICT_PERMS=false ES_PASS=testing ES_USER=admin TESTING_FILEBEAT_MODULES=system pytest tests/system/test_modules.py

Run the system module

Package Filebeat from this PR.
Start the Debian 12 VM, run Filebeat

vagrant up debian12
vagrant ssh debian12
cp /vagrant/filebeat/build/distributions/filebeat-oss-9.0.0-SNAPSHOT-linux-x86_64.tar.gz ./
tar -xf filebeat-oss-9.0.0-SNAPSHOT-linux-x86_64.tar.gz 
cd filebeat-9.0.0-SNAPSHOT-linux-x86_64/
./filebeat modules enable system
./filebeat setup -e -v
# edit modules.d/system.yml and enable both filesets
# edit filebeat.yml and add the ES output and Kibana URL/credentials
./filebeat -e -v

Ensure data is ingested (datastream filebeat-9.0.0)

Related issues

## Use cases
## Screenshots
## Logs


This is an automatic backport of pull request #41061 done by [Mergify](https://mergify.com).

This commit adds Debian 12 support to our system module, to support
Debian 12 we need to use the journald input to collect the system
logs.

To support it, a new, internal, input  `system-logs`is introduced, it is responsible
for deciding whether the log input or journald must be used. If `var.paths` is defined
in the module configuration, `system-logs` looks at the files, if any of the globs resolves
to one or more files the `log` input is used, otherwise the `jouranld` input is used.

This behaviour can be overridden by setting `var.use_journald` or `var.use_files`,
which will force the use of journald or files.

Other changes:
 - Journald input now support filtering by facilities
 - System tests for modules now support handling journal files
 - The `TESTING_FILEBEAT_FILEPATTERN` environment variable now is a
 comma separated list of globs, it defaults to `.log,*.journal`
 - Multiple lint warnings are fixed
 - The documentation has been updated where needed.

(cherry picked from commit cfd1f1c)
@mergify mergify bot added the backport label Oct 14, 2024
@mergify mergify bot requested a review from a team as a code owner October 14, 2024 16:25
@mergify mergify bot removed the request for review from a team October 14, 2024 16:25
@mergify mergify bot requested review from faec and leehinman October 14, 2024 16:25
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 14, 2024
@botelastic
Copy link

botelastic bot commented Oct 14, 2024

This pull request doesn't have a Team:<team> label.

@pierrehilbert pierrehilbert merged commit cd23cc3 into 8.x Oct 15, 2024
@pierrehilbert pierrehilbert deleted the mergify/bp/8.x/pr-41061 branch October 15, 2024 06:43
@khushijain21 khushijain21 mentioned this pull request Jun 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport needs_team Indicates that the issue/PR needs a Team:* label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants