diff --git a/docs/reference/filebeat/filebeat-input-filestream.md b/docs/reference/filebeat/filebeat-input-filestream.md index f292571e2d5b..da73b8ecacb6 100644 --- a/docs/reference/filebeat/filebeat-input-filestream.md +++ b/docs/reference/filebeat/filebeat-input-filestream.md @@ -325,57 +325,83 @@ The setting relies on the modification time of the file to determine if a file i To remove the state of previously harvested files from the registry file, use the `clean_inactive` configuration option. -## Take over [filebeat-input-filestream-take-over] +#### `take_over` [filebeat-input-filestream-take-over] + +```{applies_to} +stack: beta +``` + When `take_over` is enabled, this `filestream` input will take over states from the [`log`](/reference/filebeat/filebeat-input-log.md) input or other `filestream` inputs. Only states of files being actively harvested by this input are taken over. -To take over files from a `log` input, simply set `take_over.enabled: true`. +The syntax for enabling take over mode varies by version: + +* {applies_to}`stack: beta 9.0.0` Use `take_over: true`. +* {applies_to}`stack: beta 9.1.0` Use `take_over.enabled: true`. -To take over states from other `filestream` inputs, set -`take_over.enabled: true` and set `take_over.from_ids` to a list of -existing `filestream` IDs you want to migrate files from. +:::{note} +While `take_over: true` is still supported to migrate state from the `log` input to +`filestream`, we plan to remove support in a future version so you should use +the new syntax if possible. +::: -On both cases make sure the files you want this input to take over -match the configured globs in `paths`. +##### `log` input -When `take_over.from_ids` is set, files are not taken over from `log` -inputs. The migration is limited to `filestream` inputs only. +To take over files from a `log` input, enable take over mode +and make sure the files you want this input to take over match the configured globs in `paths`. +::::{tab-set} +:::{tab-item} 9.1.0 ```yaml take_over: enabled: true - from_ids: ["foo", "bar"] # omit to take over from the log input ``` -:::{important} -The `take over` mode can work correctly only if the source (taken from) inputs are no longer active. If source inputs are still harvesting the files which are being migrated, it will lead to data duplication and in some cases might cause data loss. ::: - -::::{important} -`take_over.enabled: true` requires the `filestream` to have a unique ID. +:::{tab-item} 9.0.0 +```yaml +take_over: true +``` +::: :::: -This `take over` mode was created to enable smooth migration from + + +##### `filestream` input + +```{applies_to} +stack: beta 9.1.0 +``` + +To take over states from other `filestream` inputs, enable take over mode, +set `take_over.from_ids` to a list of existing `filestream` IDs you want to migrate files from, +and make sure the files you want this input to take over match the configured globs in `paths`. + +When `take_over.from_ids` is set, files are not taken over from `log` +inputs. The migration is limited to `filestream` inputs only. + +```yaml +take_over: + enabled: true + from_ids: ["foo", "bar"] +``` + +This take over mode was created to enable smooth migration from deprecated `log` inputs to the new `filestream` inputs and to allow changing `filestream` input IDs without data re-ingestion. -See [*Migrate `log` input configurations to `filestream`*](/reference/filebeat/migrate-to-filestream.md) for more details about the migration process. +Refer to [Migrate `log` input configurations to `filestream`](/reference/filebeat/migrate-to-filestream.md) for more details about the migration process. -The previous configuration format `take_over: true`, while -deprecated, is still supported to migrate state from the `log` input -to `filestream`. - -::::{warning} -The `take over` mode is still in beta, however, it should be generally safe to use. -:::: +##### Notes and limitations +When using take over mode, it is important to note: -### Limitations -Take over can only migrate states from existing files that are not -ignored during the `filestream` input start up. Once the input is -ingesting data, if a new file appears, `filestream` will not try to -migrate its state. +* Take over mode only works correctly if the source (taken from) inputs are no longer active. + If source inputs are still harvesting the files which are being migrated, it will lead to data duplication and in some cases might cause data loss. +* Using `take_over.enabled: true` requires the `filestream` to have a unique ID. +* Take over can only migrate states from existing files that are not ignored during the `filestream` input start up. + Once the input is ingesting data, if a new file appears, `filestream` will not try to migrate its state. #### `close.*` [filebeat-input-filestream-close-options] diff --git a/docs/reference/filebeat/migrate-to-filestream.md b/docs/reference/filebeat/migrate-to-filestream.md index 1208563df3d6..777708f368ee 100644 --- a/docs/reference/filebeat/migrate-to-filestream.md +++ b/docs/reference/filebeat/migrate-to-filestream.md @@ -145,7 +145,10 @@ filebeat.inputs: ### Step 2: The "take over" mode [step_2] -In order to indicate that the new `filestream` inputs are supposed to take over the files from the previously defined `log` or `container` inputs and continue where they left off, we need to activate the "take over" mode by adding `take_over.enabled: true` to each new `filestream`. +In order to indicate that the new `filestream` inputs are supposed to take over the files from the previously defined `log` or `container` inputs and continue where they left off, activate the [take over](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-take-over) mode in each new `filestream` using: + +* {applies_to}`stack: ga 9.1` `take_over.enabled: true` +* {applies_to}`stack: ga 9.0` `take_over: true` ::::{important} For the "take over" mode to work the `paths` list in each new `filestream` inputs must match the `paths` list in each old `log` or `container` inputs accordingly. @@ -153,6 +156,10 @@ For the "take over" mode to work the `paths` list in each new `filestream` input After enabling the "take over" mode the configuration should look like this: +::::{tab-set} +:group: versions +:::{tab-item} 9.1.0 +:sync: 9.1.0 ```yaml filebeat.inputs: - type: filestream @@ -187,6 +194,41 @@ filebeat.inputs: paths: - /var/log/my-old-files*.log ``` +::: +:::{tab-item} 9.0.0 +:sync: 9.0.0 +```yaml +filebeat.inputs: + - type: filestream + enabled: true + id: my-java-collector + take_over: true + paths: + - /var/log/java-exceptions*.log + + - type: filestream + enabled: true + id: my-container-input + take_over: true + paths: + - /var/lib/docker/containers/*.log + + - type: filestream + enabled: true + id: my-application-input + take_over: true + paths: + - /var/log/my-application*.json + + - type: filestream + enabled: true + id: my-old-files + take_over: true + paths: + - /var/log/my-old-files*.log +``` +::: +:::: ### Step 3: Migrate additional parameters if necessary [step_3] @@ -219,6 +261,10 @@ Furthermore, the `json` parser was renamed to [`ndjson`](/reference/filebeat/fil The example configuration shown earlier needs to be adjusted according to the changes described in the table above: +::::{tab-set} +:group: versions +:::{tab-item} 9.1.0 +:sync: 9.1.0 ```yaml filebeat.inputs: - type: filestream @@ -239,9 +285,9 @@ filebeat.inputs: - type: filestream enabled: true id: my-container-input - prospector.scanner.symlinks: true # container logs often use symlinks, they should be enabled + prospector.scanner.symlinks: true <1> parsers: - - container: ~ # the container parser replaces everything the container input did before + - container: ~ <2> take_over: enabled: true paths: @@ -268,6 +314,60 @@ filebeat.inputs: - /var/log/my-old-files*.log ignore_inactive: since_last_start ``` +1. Container logs often use symlinks, they should be enabled. +2. The container parser replaces everything the container input did before. +::: +:::{tab-item} 9.0.0 +:sync: 9.0.0 +```yaml +filebeat.inputs: + - type: filestream + enabled: true + id: my-java-collector + take_over: true + paths: + - /var/log/java-exceptions*.log + parsers: + - multiline: + pattern: '^\[' + negate: true + match: after + close.on_state_change.removed: true + close.on_state_change.renamed: true + + - type: filestream + enabled: true + id: my-container-input + prospector.scanner.symlinks: true <1> + parsers: + - container: ~ <2> + take_over: true + paths: + - /var/lib/docker/containers/*.log + + - type: filestream + enabled: true + id: my-application-input + take_over: true + paths: + - /var/log/my-application*.json + prospector.scanner.check_interval: 1m + parsers: + - ndjson: + keys_under_root: true + + - type: filestream + enabled: true + id: my-old-files + take_over: true + paths: + - /var/log/my-old-files*.log + ignore_inactive: since_last_start +``` +1. Container logs often use symlinks, they should be enabled. +2. The container parser replaces everything the container input did before. +::: +:::: Now you finally have your configuration fully migrated to using `filestream` inputs instead of `log` and `container` inputs. @@ -282,8 +382,8 @@ Double-check that: Start Filebeat with the new migrated configuration. -All the events produced by a `filestream` input with `take_over.enabled: true` contain the `take_over` tag. You can filter on this tag in Kibana Discover and see all the events which came from filestreams in the "take over" mode. +All the events produced by a `filestream` input with [take over](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-take-over) mode enabled contain the `take_over` tag. You can filter on this tag in Kibana Discover and see all the events which came from filestreams in the "take over" mode. -Once you start receiving events with this tag and validate that all new `filestream` inputs behave correctly, you can remove `take_over.enabled: true` and restart Filebeat again. +Once you start receiving events with this tag and validate that all new `filestream` inputs behave correctly, you can stop using take over mode and restart Filebeat again. Congratulations, you've completed the migration process.