Backport PR #12830 to 7.x: ecs-compatibility docs

config/logstash.yml

@@ -64,7 +64,7 @@
 # if there are multiple workers.
 # "false" will disable any extra processing necessary for preserving ordering.
 #
-pipeline.ordered: auto
+# pipeline.ordered: auto
 #
 # ------------ Pipeline Configuration Settings --------------
 #

docs/index.asciidoc

@@ -55,6 +55,10 @@ include::static/advanced-pipeline.asciidoc[]
 :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/life-of-an-event.asciidoc
 include::static/life-of-an-event.asciidoc[]
 
+// Elastic Common Schema (ECS)
+:editurl!:
+include::static/ecs-compatibility.asciidoc[]
+
 // Processing details
 
 :edit_url!:

docs/static/ecs-compatibility.asciidoc

@@ -0,0 +1,97 @@
+[[ecs-ls]]
+=== ECS in Logstash
+
+// LS8 will ship with ECS v8, but until ECS v8 is ready we rely on ECS v1 as an approximation.
+:ls8-ecs-major-version: v1
+
+The {ecs-ref}/index.html[Elastic Common Schema (ECS)] is an open source specification, developed with support from the Elastic user community.
+ECS defines a common set of fields to be used for storing event data, such as logs and metrics, in {es}.
+With ECS, users can normalize event data to better analyze, visualize, and correlate the data represented in their events.
+
+[[ecs-compatibility]]
+==== ECS compatibility
+
+Many plugins implement an ECS-compatibility mode, which causes them to produce and manipulate events in a manner that is compatible with the Elastic Common Schema (ECS).
+
+Any plugin that supports this mode will also have an `ecs_compatibility` option, which allows you to configure which mode the individual plugin instance should operate in.
+If left unspecified for an individual plugin, the pipeline's `pipeline.ecs_compatibility` setting will be observed.
+This allows you to configure plugins to use ECS -- or to lock in your existing non-ECS behavior -- in advance of your {ls} 8.0 upgrade where ECS will be enabled by default.
+
+ECS Compatibility modes do not prevent you from explicitly configuring a plugin in a manner that conflicts with ECS.
+Instead, they ensure that _implicit_ configuration avoids conflicts.
+
+NOTE: Until {ls} 8.0 and the final 7.x are released, any value for `pipeline.ecs_compatibility` other than `disabled` are considered BETA and unsupported.
+      As we continue to release plugins with ECS Compatibility modes, having this flag set will cause even patch-level upgrades to _automatically_ consume breaking changes in the upgraded plugins, changing the shape of data the plugin produces.
+
+ifeval::["{ls8-ecs-major-version}"!="v8"]
+NOTE: ECS `v8` will be the default in the GA release of {ls} 8.0.0, and will be available at or before the final minor release of {ls} 7.
+      We expect the scope of breaking changes in ECS 8 to be limited.
+      We are https://github.com/elastic/ecs/issues/839[tracking progress toward ECS v8] in a GitHub issue.
+endif::[]
+
+[[ecs-configuration]]
+===== Configuring ECS
+
+ECS will be on by default in {ls} 8, but you can begin using it now by configuring individual plugins with `ecs_compatibility`.
+You can also "lock in" the existing non-ECS behavior for an entire pipeline to ensure its behavior doesn't change when you perform your next major upgrade.
+
+====== Specific plugin instance
+
+Use a plugin's `ecs_compatibility` option to override the default value on the plugin instance.
+
+For example, if you want a specific instance of the GeoIP Filter to behave without ECS compatibility, you can adjust its definition in your pipeline without affecting any other plugin instances.
+
+[source,text]
+-----
+filter {
+  geoip {
+    source => "[host][ip]"
+    ecs_compatibility => disabled
+  }
+}
+-----
+
+Alternatively, if you had a UDP input with a CEF codec, and wanted both to use an ECS mode while still running {ls} 7, you can adjust their definitions to specify the major version of ECS to use.
+
+[source,text,subs="attributes"]
+-----
+input {
+  udp {
+    port => 1234
+    ecs_compatibility => {ls8-ecs-major-version}
+    codec => cef {
+      ecs_compatibility => {ls8-ecs-major-version}
+    }
+  }
+}
+-----
+
+[[ecs-configuration-pipeline]]
+====== All plugins in a given pipeline
+
+If you wish to provide a specific default value for `ecs_compatibility` to _all_ plugins in a pipeline, you can do so with the `pipeline.ecs_compatibility` setting in your pipeline definition in `config/pipelines.yml` or Central Management.
+This setting will be used unless overridden by a specific plugin instance.
+If unspecified for an individual pipeline, the global value will be used.
+
+[source,yaml,subs="attributes"]
+-----
+- pipeline.id: my-legacy-pipeline
+  path.config: "/etc/path/to/legacy-pipeline.config"
+  pipeline.ecs_compatibility: disabled
+- pipeline.id: my-ecs-pipeline
+  path.config: "/etc/path/to/ecs-pipeline.config"
+  pipeline.ecs_compatibility: {ls8-ecs-major-version}
+-----
+
+NOTE: Until the final minor release of {ls} 7 that coincides with the General Availability of {ls} 8.0.0, any value for `pipeline.ecs_compatibility` other than `disabled` is considered BETA and unsupported because it will produce undesireable consequences when performing upgrades.
+      As we continue to release updated plugins with ECS-Compatibility modes, opting into them at a pipeline or process level will cause the affected plugins to silently and automatically consume breaking changes with each upgrade, which may change the shape of data your pipeline produces.
+
+[[ecs-configuration-all]]
+====== All plugins in all pipelines
+
+Similarly, you can set the default value for the whole {ls} process by setting the `pipeline.ecs_compatibility` value in `config/logstash.yml`.
+
+[source,yaml]
+-----
+pipeline.ecs_compatibility: disabled
+-----