Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 55 additions & 29 deletions docs/reference/filebeat/filebeat-input-filestream.md
Original file line number Diff line number Diff line change
Expand Up @@ -325,57 +325,83 @@ The setting relies on the modification time of the file to determine if a file i

To remove the state of previously harvested files from the registry file, use the `clean_inactive` configuration option.

## Take over [filebeat-input-filestream-take-over]
#### `take_over` [filebeat-input-filestream-take-over]

```{applies_to}
stack: beta
Comment thread
colleenmcginnis marked this conversation as resolved.
```

When `take_over` is enabled, this `filestream` input will take over
states from the [`log`](/reference/filebeat/filebeat-input-log.md) input
or other `filestream` inputs. Only states of files being actively
harvested by this input are taken over.

To take over files from a `log` input, simply set `take_over.enabled: true`.
The syntax for enabling take over mode varies by version:

* {applies_to}`stack: beta 9.0.0` Use `take_over: true`.
* {applies_to}`stack: beta 9.1.0` Use `take_over.enabled: true`.

To take over states from other `filestream` inputs, set
`take_over.enabled: true` and set `take_over.from_ids` to a list of
existing `filestream` IDs you want to migrate files from.
:::{note}
While `take_over: true` is still supported to migrate state from the `log` input to
`filestream`, we plan to remove support in a future version so you should use
the new syntax if possible.
:::

On both cases make sure the files you want this input to take over
match the configured globs in `paths`.
##### `log` input

When `take_over.from_ids` is set, files are not taken over from `log`
inputs. The migration is limited to `filestream` inputs only.
To take over files from a `log` input, enable take over mode
and make sure the files you want this input to take over match the configured globs in `paths`.

::::{tab-set}
:::{tab-item} 9.1.0
```yaml
take_over:
enabled: true
from_ids: ["foo", "bar"] # omit to take over from the log input
```
:::{important}
The `take over` mode can work correctly only if the source (taken from) inputs are no longer active. If source inputs are still harvesting the files which are being migrated, it will lead to data duplication and in some cases might cause data loss.
:::

::::{important}
`take_over.enabled: true` requires the `filestream` to have a unique ID.
:::{tab-item} 9.0.0
```yaml
take_over: true
```
:::
::::

This `take over` mode was created to enable smooth migration from


##### `filestream` input

```{applies_to}
stack: beta 9.1.0
```

To take over states from other `filestream` inputs, enable take over mode,
set `take_over.from_ids` to a list of existing `filestream` IDs you want to migrate files from,
and make sure the files you want this input to take over match the configured globs in `paths`.

When `take_over.from_ids` is set, files are not taken over from `log`
inputs. The migration is limited to `filestream` inputs only.

```yaml
take_over:
enabled: true
from_ids: ["foo", "bar"]
```

This take over mode was created to enable smooth migration from
deprecated `log` inputs to the new `filestream` inputs and to allow
changing `filestream` input IDs without data re-ingestion.

See [*Migrate `log` input configurations to `filestream`*](/reference/filebeat/migrate-to-filestream.md) for more details about the migration process.
Refer to [Migrate `log` input configurations to `filestream`](/reference/filebeat/migrate-to-filestream.md) for more details about the migration process.

The previous configuration format `take_over: true`, while
deprecated, is still supported to migrate state from the `log` input
to `filestream`.

::::{warning}
The `take over` mode is still in beta, however, it should be generally safe to use.
::::
##### Notes and limitations

When using take over mode, it is important to note:

### Limitations
Take over can only migrate states from existing files that are not
ignored during the `filestream` input start up. Once the input is
ingesting data, if a new file appears, `filestream` will not try to
migrate its state.
* Take over mode only works correctly if the source (taken from) inputs are no longer active.
If source inputs are still harvesting the files which are being migrated, it will lead to data duplication and in some cases might cause data loss.
* Using `take_over.enabled: true` requires the `filestream` to have a unique ID.
* Take over can only migrate states from existing files that are not ignored during the `filestream` input start up.
Once the input is ingesting data, if a new file appears, `filestream` will not try to migrate its state.

#### `close.*` [filebeat-input-filestream-close-options]

Expand Down
110 changes: 105 additions & 5 deletions docs/reference/filebeat/migrate-to-filestream.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,14 +145,21 @@ filebeat.inputs:

### Step 2: The "take over" mode [step_2]

In order to indicate that the new `filestream` inputs are supposed to take over the files from the previously defined `log` or `container` inputs and continue where they left off, we need to activate the "take over" mode by adding `take_over.enabled: true` to each new `filestream`.
In order to indicate that the new `filestream` inputs are supposed to take over the files from the previously defined `log` or `container` inputs and continue where they left off, activate the [take over](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-take-over) mode in each new `filestream` using:

* {applies_to}`stack: ga 9.1` `take_over.enabled: true`
* {applies_to}`stack: ga 9.0` `take_over: true`

::::{important}
For the "take over" mode to work the `paths` list in each new `filestream` inputs must match the `paths` list in each old `log` or `container` inputs accordingly.
::::

After enabling the "take over" mode the configuration should look like this:

::::{tab-set}
:group: versions
:::{tab-item} 9.1.0
:sync: 9.1.0
```yaml
filebeat.inputs:
- type: filestream
Expand Down Expand Up @@ -187,6 +194,41 @@ filebeat.inputs:
paths:
- /var/log/my-old-files*.log
```
:::
:::{tab-item} 9.0.0
:sync: 9.0.0
```yaml
filebeat.inputs:
- type: filestream
enabled: true
id: my-java-collector
take_over: true
paths:
- /var/log/java-exceptions*.log

- type: filestream
enabled: true
id: my-container-input
take_over: true
paths:
- /var/lib/docker/containers/*.log

- type: filestream
enabled: true
id: my-application-input
take_over: true
paths:
- /var/log/my-application*.json

- type: filestream
enabled: true
id: my-old-files
take_over: true
paths:
- /var/log/my-old-files*.log
```
:::
::::

### Step 3: Migrate additional parameters if necessary [step_3]

Expand Down Expand Up @@ -219,6 +261,10 @@ Furthermore, the `json` parser was renamed to [`ndjson`](/reference/filebeat/fil

The example configuration shown earlier needs to be adjusted according to the changes described in the table above:

::::{tab-set}
:group: versions
:::{tab-item} 9.1.0
:sync: 9.1.0
```yaml
filebeat.inputs:
- type: filestream
Expand All @@ -239,9 +285,9 @@ filebeat.inputs:
- type: filestream
enabled: true
id: my-container-input
prospector.scanner.symlinks: true # container logs often use symlinks, they should be enabled
prospector.scanner.symlinks: true <1>
parsers:
- container: ~ # the container parser replaces everything the container input did before
- container: ~ <2>
take_over:
enabled: true
paths:
Expand All @@ -268,6 +314,60 @@ filebeat.inputs:
- /var/log/my-old-files*.log
ignore_inactive: since_last_start
```
1. Container logs often use symlinks, they should be enabled.
2. The container parser replaces everything the container input did before.
:::
:::{tab-item} 9.0.0
:sync: 9.0.0
```yaml
filebeat.inputs:
- type: filestream
enabled: true
id: my-java-collector
take_over: true
paths:
- /var/log/java-exceptions*.log
parsers:
- multiline:
pattern: '^\['
negate: true
match: after
close.on_state_change.removed: true
close.on_state_change.renamed: true

- type: filestream
enabled: true
id: my-container-input
prospector.scanner.symlinks: true <1>
parsers:
- container: ~ <2>
take_over: true
paths:
- /var/lib/docker/containers/*.log

- type: filestream
enabled: true
id: my-application-input
take_over: true
paths:
- /var/log/my-application*.json
prospector.scanner.check_interval: 1m
parsers:
- ndjson:
keys_under_root: true

- type: filestream
enabled: true
id: my-old-files
take_over: true
paths:
- /var/log/my-old-files*.log
ignore_inactive: since_last_start
```
1. Container logs often use symlinks, they should be enabled.
2. The container parser replaces everything the container input did before.
:::
::::

Now you finally have your configuration fully migrated to using `filestream` inputs instead of `log` and `container` inputs.

Expand All @@ -282,8 +382,8 @@ Double-check that:

Start Filebeat with the new migrated configuration.

All the events produced by a `filestream` input with `take_over.enabled: true` contain the `take_over` tag. You can filter on this tag in Kibana Discover and see all the events which came from filestreams in the "take over" mode.
All the events produced by a `filestream` input with [take over](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-take-over) mode enabled contain the `take_over` tag. You can filter on this tag in Kibana Discover and see all the events which came from filestreams in the "take over" mode.

Once you start receiving events with this tag and validate that all new `filestream` inputs behave correctly, you can remove `take_over.enabled: true` and restart Filebeat again.
Once you start receiving events with this tag and validate that all new `filestream` inputs behave correctly, you can stop using take over mode and restart Filebeat again.

Congratulations, you've completed the migration process.
Loading