[DSL] Introduce data stream global retention - Part 3

docs/changelog/105682.yaml

@@ -0,0 +1,20 @@
+pr: 105682
+summary: Introduce global retention in data stream lifecycle.
+area: Data streams
+type: feature
+issues:
+  - 106169
+highlight:
+  title: Add global retention in data stream lifecycle
+  body: |-
+    Data stream lifecycle now supports configuring retention on a cluster level, namely global retention. Global retention 
+    allows us to configure two different retentions:
+
+    - `default_retention` is applied to all data streams managed by the data stream lifecycle that do not have retention
+    defined on the data stream level.
+    - `max_retention` is applied to all data streams managed by the data stream lifecycle and it allows any data stream 
+    data to be deleted after the `max_retention` has passed.
+
+    Furthermore, we introduce the term `effective_retention` which is the retention applied at a certain moment to a data
+    stream considering all the available retention configurations.
+  notable: true

docs/reference/data-streams/data-stream-apis.asciidoc

@@ -25,7 +25,13 @@ preview:[]
 preview:[]
 * <<data-streams-explain-lifecycle,Explain data stream lifecycle>>
 preview:[]
-* <<data-streams-get-lifecycle-stats, Get data stream lifecycle stats>>
+* <<data-streams-get-lifecycle-stats,Get data stream lifecycle stats>>
+preview:[]
+* <<data-streams-put-global-retention,Update the global retention for data stream lifecycle managed data streams>>
+preview:[]
+* <<data-streams-get-global-retention,Get the global retention for data stream lifecycle managed data streams>>
+preview:[]
+* <<data-streams-delete-global-retention,Delete the global retention for data stream lifecycle managed data streams>>
 preview:[]
 
 The following API is available for <<tsds,time series data streams>>:
@@ -59,4 +65,10 @@ include::{es-repo-dir}/data-streams/lifecycle/apis/explain-lifecycle.asciidoc[]
 
 include::{es-repo-dir}/data-streams/lifecycle/apis/get-lifecycle-stats.asciidoc[]
 
+include::{es-repo-dir}/data-streams/lifecycle/apis/put-global-retention.asciidoc[]
+
+include::{es-repo-dir}/data-streams/lifecycle/apis/get-global-retention.asciidoc[]
+
+include::{es-repo-dir}/data-streams/lifecycle/apis/delete-global-retention.asciidoc[]
+
 include::{es-repo-dir}/indices/downsample-data-stream.asciidoc[]

docs/reference/data-streams/lifecycle/apis/delete-global-retention.asciidoc

@@ -0,0 +1,121 @@
+[[data-streams-delete-global-retention]]
+=== Delete the global retention of data streams
+++++
+<titleabbrev>Delete Data Stream Global Retention</titleabbrev>
+++++
+
+preview::[]
+
+Deletes the global retention configuration that applies on every data stream managed by <<data-stream-lifecycle,data stream lifecycle>>.
+
+[[delete-global-retention-api-prereqs]]
+==== {api-prereq-title}
+
+** If the {es} {security-features} are enabled, you must have the `manage_data_stream_global_retention` <<privileges-list-cluster,cluster privilege>> to use this API.
+
+[[data-streams-delete-global-retention-request]]
+==== {api-request-title}
+
+`DELETE _data_stream/_global_retention`
+
+[[data-streams-delete-global-retention-desc]]
+==== {api-description-title}
+
+Deletes the global retention configuration that is applied on data streams managed by data stream lifecycle.
+
+[role="child_attributes"]
+[[delete-global-retention-api-query-parms]]
+==== {api-query-parms-title}
+
+`dry_run`::
+(Boolean) Signals that the request should determine the effect of the removal of the existing without updating
+the global retention. The default value is `false`, which means the removal will happen.
+
+[[delete-global-retention-api-response-body]]
+==== {api-response-body-title}
+
+`acknowledged`::
+(boolean)
+True, if the global retention has been removed. False, if it fails or if it was a dry run.
+
+`dry_run`::
+(boolean)
+True, if this was a dry run, false otherwise.
+
+`affected_data_streams`::
+(array of objects)
+Contains information about the data streams affected by the change.
++
+.Properties of objects in `affected_data_streams`
+[%collapsible%open]
+====
+`name`::
+(string)
+Name of the data stream.
+`previous_effective_retention`::
+(string)
+The retention that was effective before the change of this request. `infinite` if there was no retention applicable.
+`new_effective_retention`::
+(string)
+The retention that is or would be effective after this request. `infinite` if there is no retention applicable.
+====
+
+[[data-streams-delete-global-retention-example]]
+==== {api-examples-title}
+
+////
+
+[source,console]
+--------------------------------------------------
+PUT _data_stream/_global_retention
+{
+  "default_retention": "7d",
+  "max_retention": "90d"
+}
+
+PUT /_index_template/template
+{
+  "index_patterns": ["my-data-stream*"],
+  "template": {
+    "lifecycle": {}
+  },
+  "data_stream": { }
+}
+
+PUT /_data_stream/my-data-stream
+----
+// TESTSETUP
+////
+
+////
+[source,console]
+----
+DELETE /_data_stream/my-data-stream*
+DELETE /_index_template/template
+DELETE /_data_stream/_global_retention
+----
+// TEARDOWN
+////
+
+Let's update the global retention:
+[source,console]
+--------------------------------------------------
+DELETE _data_stream/_global_retention
+--------------------------------------------------
+
+The response will look like the following:
+
+[source,console-result]
+--------------------------------------------------
+{
+  "acknowledged": true,
+  "dry_run": false,
+  "affected_data_streams": [
+    {
+      "name": "my-data-stream",
+      "previous_effective_retention": "7d",
+      "new_effective_retention": "infinite"
+    }
+  ]
+}
+--------------------------------------------------

docs/reference/data-streams/lifecycle/apis/get-global-retention.asciidoc

@@ -0,0 +1,90 @@
+[[data-streams-get-global-retention]]
+=== Get the global retention of data streams
+++++
+<titleabbrev>Get Data Stream Global Retention</titleabbrev>
+++++
+
+preview::[]
+
+Gets the global retention that applies on every data stream managed by <<data-stream-lifecycle,data stream lifecycle>>.
+
+[[get-global-retention-api-prereqs]]
+==== {api-prereq-title}
+
+** If the {es} {security-features} are enabled, you must have the `monitor_data_stream_global_retention` or
+`manage_data_stream_global_retention` <<privileges-list-cluster,cluster privilege>> to use this API.
+
+[[data-streams-get-global-retention-request]]
+==== {api-request-title}
+
+`GET _data_stream/_global_retention`
+
+[[data-streams-get-global-retention-desc]]
+==== {api-description-title}
+
+Gets the global retention configuration that is applied on data streams managed by data stream lifecycle.
+
+[role="child_attributes"]
+[[get-global-retention-api-query-parms]]
+==== {api-query-parms-title}
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=local]
+
+include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=master-timeout]
+
+[[get-global-retention-api-response-body]]
+==== {api-response-body-title}
+
+`default_retention`::
+(Optional, string)
+The default retention that will apply to any data stream managed by data stream lifecycle that does not have a retention
+defined on the data stream level.
+
+`max_retention`::
+(Optional, string)
+The max retention that will apply to all data streams managed by data stream lifecycle. The max retention will override the
+retention of a data stream whose retention exceeds the max retention.
+
+
+[[data-streams-get-global-retention-example]]
+==== {api-examples-title}
+
+////
+
+[source,console]
+--------------------------------------------------
+PUT _data_stream/_global_retention
+{
+  "default_retention": "7d",
+  "max_retention": "90d"
+}
+--------------------------------------------------
+// TESTSETUP
+
+[source,console]
+--------------------------------------------------
+DELETE _data_stream/_global_retention
+--------------------------------------------------
+// TEARDOWN
+
+////
+
+Let's retrieve the global retention:
+
+[source,console]
+--------------------------------------------------
+GET _data_stream/_global_retention
+--------------------------------------------------
+
+The response will look like the following:
+
+[source,console-result]
+--------------------------------------------------
+{
+  "default_retention": "7d",    <1>
+  "max_retention": "90d"        <2>
+}
+--------------------------------------------------
+<1> 7 days retention will be applied to any data stream that does not have retention set in its lifecycle.
+<2> 90 days retention will be applied to all data streams that have retention that exceeds the 90 days, this
+applies to data streams that have infinite retention too.

docs/reference/data-streams/lifecycle/apis/put-global-retention.asciidoc

@@ -0,0 +1,131 @@
+[[data-streams-put-global-retention]]
+=== Update the global retention of data streams
+++++
+<titleabbrev>Update Data Stream Global Retention</titleabbrev>
+++++
+
+preview::[]
+
+Updates the global retention configuration that applies on every data stream managed by <<data-stream-lifecycle,data stream lifecycle>>.
+
+[[put-global-retention-api-prereqs]]
+==== {api-prereq-title}
+
+** If the {es} {security-features} are enabled, you must have the `manage_data_stream_global_retention` <<privileges-list-cluster,cluster privilege>> to use this API.
+
+[[data-streams-put-global-retention-request]]
+==== {api-request-title}
+
+`PUT _data_stream/_global_retention`
+
+[[data-streams-put-global-retention-desc]]
+==== {api-description-title}
+
+Updates the global retention configuration that is applied on data streams managed by data stream lifecycle.
+
+[role="child_attributes"]
+[[put-global-retention-api-query-parms]]
+==== {api-query-parms-title}
+
+`dry_run`::
+(Boolean) Signals that the request should determine the effect of the provided configuration without updating the
+global retention settings. The default value is `false`, which means the configuration provided will be applied.
+
+[[put-global-retention-api-request-body]]
+==== {api-request-body-title}
+
+`default_retention`::
+(Optional, string)
+The default retention that will apply to any data stream managed by data stream lifecycle that does not have a retention
+defined on the data stream level.
+
+`max_retention`::
+(Optional, string)
+The max retention that will apply to all data streams managed by data stream lifecycle. The max retention will override the
+retention of a data stream which retention exceeds the max retention.
+
+[[put-global-retention-api-response-body]]
+==== {api-response-body-title}
+
+`acknowledged`::
+(boolean)
+True, if the global retention has been updated to the provided values. False, if it fails or if it was a dry run.
+
+`dry_run`::
+(boolean)
+True, if this was a dry run, false otherwise.
+
+`affected_data_streams`::
+(array of objects)
+Contains information about the data streams affected by the change.
++
+.Properties of objects in `affected_data_streams`
+[%collapsible%open]
+====
+`name`::
+(string)
+Name of the data stream.
+`previous_effective_retention`::
+(string)
+The retention that was effective before the change of this request. `infinite` if there was no retention applicable.
+`new_effective_retention`::
+(string)
+The retention that is or would be effective after this request. `infinite` if there is no retention applicable.
+====
+
+[[data-streams-put-global-retention-example]]
+==== {api-examples-title}
+
+////
+[source,console]
+----
+PUT /_index_template/template
+{
+  "index_patterns": ["my-data-stream*"],
+  "template": {
+    "lifecycle": {}
+  },
+  "data_stream": { }
+}
+
+PUT /_data_stream/my-data-stream
+----
+// TESTSETUP
+////
+
+////
+[source,console]
+----
+DELETE /_data_stream/my-data-stream*
+DELETE /_index_template/template
+DELETE /_data_stream/_global_retention
+----
+// TEARDOWN
+////
+
+Let's update the global retention:
+[source,console]
+--------------------------------------------------
+PUT _data_stream/_global_retention
+{
+  "default_retention": "7d",
+  "max_retention": "90d"
+}
+--------------------------------------------------
+
+The response will look like the following:
+
+[source,console-result]
+--------------------------------------------------
+{
+  "acknowledged": true,
+  "dry_run": false,
+  "affected_data_streams": [
+    {
+      "name": "my-data-stream",
+      "previous_effective_retention": "infinite",
+      "new_effective_retention": "7d"
+    }
+  ]
+}
+--------------------------------------------------