diff --git a/libbeat/docs/breaking.asciidoc b/libbeat/docs/breaking.asciidoc index 0b5f5962eaef..a5c323e80fc6 100644 --- a/libbeat/docs/breaking.asciidoc +++ b/libbeat/docs/breaking.asciidoc @@ -4,18 +4,14 @@ == Breaking changes As a general rule, we strive to keep backwards compatibility between minor -versions (e.g. 6.x to 6.y) so you can upgrade without any configuration file -changes, but there are breaking changes between major versions (e.g. 5.x to -6.y). Migrating directly between non consecutive major versions (e.g. 1.x to -6.x) is not recommended. +versions (e.g. 7.x to 7.y) so you can upgrade without any configuration file +changes, but there are breaking changes between major versions (e.g. 6.x to +7.y). Migrating directly between non consecutive major versions (e.g. 5.x to +7.x) is not recommended. See the following topics for a description of breaking changes: * <> -* <> -* <> -* <> -* <> [[breaking-changes-7.0]] @@ -80,375 +76,4 @@ with the data types used in the Elasticsearch index template. |`process.ppid` |string |number |====================== -[[breaking-changes-6.7]] -=== Breaking changes in 6.7 - -The data structure used to store Beats central management data in Kibana has changed. As a result of the -template changes, enrolled Beats and configuration tags created prior to 6.7 will not work until you migrate them to the new format. -To migrate to the new format, use the https://github.com/elastic/migrate-management-beats/releases[migration tool] we've created. -For more information, see the https://github.com/elastic/migrate-management-beats/blob/master/README.md[documentation] about the tool. {see-relnotes} - -[[breaking-changes-6.3]] -=== Breaking changes in 6.3 - -This section discusses the main changes that you should be aware of if you -upgrade the Beats to version 6.3. {see-relnotes} - -[[breaking-changes-monitoring]] -==== Beats monitoring schema changes - -Starting with version 6.3, the monitoring field `beat.cpu.*.time.metrics` is -renamed to `beat.cpu.*.time.ms`. As a result of this change, Beats shippers -released prior to version 6.3 are unable to send monitoring data to clusters -running on Elasticsearch 6.3 and later. {see-relnotes} - -[[breaking-changes-mapping-conflict]] -==== New `host` namespace may cause mapping conflicts for Logstash - -This breaking change applies only to users who send Beats events to Logstash. - -Starting with version 6.3, Beats provides an `add_host_metadata` processor for -adding fields, such as `host.name` and `host.id`, to Beats events. These fields -are grouped under a `host` prefix and conform to the -https://github.com/elastic/ecs[Elastic Common Schema (ECS)]. The `host` object -is defined in the Elasticsearch index template even if the processor is not -used. - -We've also added a `host.name` field to all events sent by Beats. This field -prevents the Beats input plugin in Logstash from adding a default `host` field. -(By default, the plugin adds a `host` field if the event doesn't already have -one. We don't want the plugin to add this field because it causes a mapping -conflict with the `host` object defined in the index template.) - -*What does this mean to you?* - -See the info for your particular use case: - -* <> -* <> -* <> - -[[beats-template-versioned-indices]] -===== Use case: You use the Beats index template and versioned indices - -In this use case, you load the versioned template manually and use the Beats -versioned index pattern, `%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}`, -as recommended in the {logstash-ref}/plugins-inputs-beats.html[Beats input -plugin] documentation. This results in a `host` field in 6.2 indices, and a -`host.name` field in 6.3 indices. There are no mapping conflicts, but -any visualizations or searches that use `host` will no longer show results for -6.3 indices. - -*What do you need to change?* - -If you searched for the `host` field previously, modify your searches to use the -`beat.hostname` field instead. The `beat.hostname` field existed prior to 6.3 -and contains the same information as `host.name`. Using this field ensures that -your queries and aggregations will work as expected in earlier releases and 6.3. - -To save time when you have a large number of objects to update, you can batch -this process. Use either the Kibana UI or API to export the objects to JSON, -make the JSON modification, and then re-import the objects. For more -information, see: - -* {kibana-ref}/managing-saved-objects.html[Managing Saved Objects] -* {kibana-ref}/saved-objects-api.html[Kibana Saved Objects API] - - -[[custom-template-non-versioned-indices]] -===== Use case: You use a custom template and your indices are not versioned - -Mapping conflicts are likely in this use case because two different Beats -versions (6.2 and 6.3) are sending data to the same index. For 6.2, Logstash -adds the default `host` field, and for 6.3, Beats adds the `host.name` field, -which results in a mapping conflict. - -*What do you need to change?* - -To resolve the mapping issue, do *one* of the following: - -* Use versioned indices to prevent the mapping conflict. In the Logstash -pipeline configuration, set `manage_template => false` and use an index naming -pattern that includes `[version]` metadata: -+ -[source,yaml] ----- -manage_template => false -index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" ----- -+ -For more information, see the documentation for the -{logstash-ref}/plugins-inputs-beats.html[Beats input plugin]. - -* Or, in the Beats config file, configure Beats to drop all `host.*` fields: -+ -[source,yaml] ----- - processors: - - drop_fields: - fields: ["host"] ----- -+ --- -IMPORTANT: If you drop the `host` fields, you cannot use the `add_host_metadata` -processor. - --- -+ -With this configuration, Beats drops the `host` fields before sending the -event to Logstash, and Logstash adds a default `host` field, as it did with -previous Beats versions. This approach resolves the mapping conflict, but you -should plan to migrate your Logstash configurations to use `host.name` in -future releases. - -[[beats-template-non-versioned-indices]] -===== Use case: You use the Beats index template and your indices are not versioned - -In this use case, you load the Beats index template manually into Elasticsearch, -and send your data through Logstash, but you don’t use the versioned index -pattern to create versioned indices. - -You cannot resolve the problem by dropping the `host.*` fields, because Logstash -will add a default `host` field, resulting in a mapping conflict with the -`host` field defined as an object in the index template. - -*What do you need to change?* - -To resolve the mapping issue, do *one* of the following: - -* Drop the `host.*` fields in a Logstash filter. For example: -+ -[source,yaml] ----- -filter { - mutate { - remove_field => [ "[host]" ] - } -} ----- -+ -With this configuration, there will be no `host` field in the final event at -ingestion time, and the mapping conflict is avoided. - -* Or: -** Modify the Beats index template by removing the `host.*` fields, and -** Configure Beats to drop all `host.*` fields, as described in -<>. -+ -This solution prevents a mapping conflict because the fields are no longer -defined in the Elasticsearch template. Elasticsearch can use the `host` -mapping created when Logstash added a default `host` field. - -The difference between these two approaches is that the first approach, using -a Logstash filter, drops the `host` fields completely. There will be no `host` -field in the final event. The second approach drops the `host` fields from the -Beats event, but because Logstash adds a default `host` field, there will be a -`host` field in the final event. - -[[breaking-changes-6.2]] -=== Breaking changes in 6.2 - -{see-relnotes} - -[[breaking-changes-6.1]] -=== Breaking changes in 6.1 - -{see-relnotes} - -[[breaking-changes-6.0]] -=== Breaking changes in 6.0 - -This section discusses the main changes that you should be aware of if you -upgrade the Beats from version 5.x to 6.x. {see-relnotes} - -// TODO: better link to the consolidated release notes for 6.0.0. - - -[[breaking-changes-spooler-removed]] -==== Filebeat spooler removed - -Version 6.0 comes with a new architecture for the internal pipeline of all -Beats. This architecture refactoring is mostly internal, but one of the more -visible effects is that the Spooler component of Filebeat is removed. The -functionality of the Spooler was similar to the one of the publisher queue from -libbeat (the code shared by all Beats), and the presence of multiple queues -made the performance tuning of Filebeat more complex than it needed to be. - -As a result, the following options are removed: - -- `filebeat.spool_size` -- `filebeat.publish_async` -- `filebeat.idle_timeout` -- `queue_size` -- `bulk_queue_size` - -The first three are specific to Filebeat, while `queue_size` and -`bulk_queue_size` exist in all Beats. If any of these options is set, Filebeat -6.0 will refuse to start. - -Instead of the settings above, the `queue.mem` settings are introduced. If you -had to tune the `spool_size` or the `queue_size` before, you might want to tune -the `queue.mem.events` when upgrading. However, it is best to leave the rest of -the `queue.mem` settings to their default values, as they are appropriate for -all loads. - -The `publish_async` option (which was deprecated since 5.3) is removed because -the new pipeline already works asynchronously by default. - -// TODO: for the above new settings, link to their configuration settings. - -[[breaking-changes-single-output]] -==== Only one enabled output - -In versions prior to 6.0, you could enabled multiple outputs at the same time, -but only of different types. For example, you were able to enable the -Elasticsearch and Logstash outputs, but not two Logstash outputs. The drawback -of enabling multiple outputs was that the Beats that wait for acknowledgments -(Filebeat and Winlogbeat) before proceeding slowed down to the slowest output. -This implication was not obvious and hindered the use cases where multiple -outputs would have been useful. - -As part of the pipeline re-architecture that we did for 6.0, we removed the -option to enable multiple outputs at the same time. This helps with keeping the -pipeline simple and with clarifying the scope of outputs in Beats. - -If you require multiple outputs, you have the following options: - -* use the Logstash output and then use Logstash to pipe the events to multiple - outputs -* run multiple instances of the same Beat - -If you used the `file` or `console` outputs for debugging purposes, in addition -to the main output, we recommend using the `-d "publish"` option which logs the -published events in the Filebeat logs. - -[[breaking-changes-ls-index]] -==== Logstash index setting now requires version - -If you use the Logstash output to send data from Beats to Logstash, you need to -update the `index` setting in your Logstash configuration to include the Beat -version: - -[source,json] ----- -output { - elasticsearch { - hosts => "localhost:9200" - manage_template => false - index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" - } -} ----- - - -Prior to 6.0, the recommended setting was: - -[source,yaml] ----- - index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" ----- - - -The index templates that ship with 6.0 are applied to new indices that match the -pattern `[beat]-[version]-*`. You must update your Logstash config, or the -templates will not be applied. - -[[breaking-changes-types]] -==== Filebeat prospector type and document type changes - -The `document_type` setting, from the prospector configuration, was removed -because the `_type` concept is being -{ref}/removal-of-types.html[removed from Elasticsearch]. Instead of -the `document_type` setting, you can use a custom field. - -This has led also to the rename of the `input_type` configuration setting to -`type`. This change is backwards compatible because the old setting still -works. However, the `input_type` output field was renamed to `prospector.type`. - -[[breaking-changes-default-config]] -==== Filebeat default prospector disabled in the configuration file - -The default startup behaviour (based on the included sample configuration) of -Filebeat was to read all the files matching the `/var/log/*.log` pattern. -Starting with version 6.0, Filebeat doesn't read any files in its default -configuration. However, you can easily enable the `system` module, for example -with a CLI flag: - -["source","sh",subs="attributes"] ----- -filebeat --modules=system ----- - -==== Other settings changed or moved - -The `outputs.elasticsearch.template.*` settings have been moved under -`setup.template.*`, but are otherwise unchanged. - -The `dashboards.*` settings have been moved under `setup.dashboards.*`. - -The Filebeat deprecated options `force_close_files` and `close_older` are -removed. - -[[breaking-changes-import-dashboards]] -==== Changes for importing the Kibana dashboards - -The `import_dashboards` program, used to load the Kibana dashboards in previous -versions of Beats, is replaced by the `setup` command. For example, the -following command: - -["source","sh",subs="attributes"] ----- -./scripts/import_dashboards -user elastic -pass {pwd} ----- - -Can be replaced with: - -["source","sh",subs="attributes"] ----- -./filebeat setup -E "output.elasticsearch.username=elastic" -E "output.elasticsearch.password={pwd}" ----- - -Note that the `-E` flags are only required if the Elasticsearch output is not -already configured in the configuration file. - -Besides the change in the commands, it's important to note that loading the -Kibana dashboards works differently in the 6.0 version of the stack. Prior to -6.0, the dashboards were inserted directly in the `.kibana` Elasticsearch -index. Starting with 6.0, the Beats use a Kibana server API. This means that -the Beat that loads the dashboards needs direct access to Kibana and that the -Kibana URL needs to be set. The option to set the Kibana URL is -`setup.kibana.host`, which you can set in the configuration file or via the -`-E` CLI flag: - - -["source","sh",subs="attributes"] ----- -./filebeat setup -E "setup.kibana.host=http://kibana-host:5601" ----- - -The default value for the Kibana host is `localhost:5601`. - -[[breaking-changes-filters]] -==== Metricbeat filters renamed to processors - -The "local" processors, which are configured at the module level, used to be -called `filters` in Metricbeat, but were offering similar functionality with -the global `processors`. A notable difference between the two was that the -filters accessed fields relatively to the metricset (for example, -`mount_point`), while the processors referred to fields by their fully -qualified name (for example, `system.filesystem.mount_point`). - -Starting with version 6.0, the `filters` are renamed to `processors` and they -can access the fields only by using the fully qualified names. - -[[breaking-changes-cgo]] -==== Binaries are dynamically compiled against libc - -Prior to 6.0, Metricbeat and Packetbeat were compiled using -https://golang.org/cmd/cgo/[Cgo], while Filebeat, Winlogbeat, and Heartbeat -were compiled using the pure Go compiler. One of the side-effects of compiling -with Cgo is that libc is dynamically compiled. Starting with 6.0, all the Beats -are compiled using Cgo and therefore dynamically compiled against libc. This -can reduce the portability of the binaries, but none of the supported platforms -is affected. diff --git a/libbeat/docs/release-notes/6.0.0.asciidoc b/libbeat/docs/release-notes/6.0.0.asciidoc index dd4691e90936..63412008f284 100644 --- a/libbeat/docs/release-notes/6.0.0.asciidoc +++ b/libbeat/docs/release-notes/6.0.0.asciidoc @@ -3,7 +3,7 @@ The list below covers the changes during the 6.0.0-alpha1, -alpha2, -beta1, -beta2, -rc1 and -rc2 releases. -Also read <> for more detail about changes that affect +Also read {beats-ref-60}/breaking-changes-6.0.html[Breaking changes] for more detail about changes that affect upgrade. ==== Breaking changes