Improve documentation for X-Opaque-ID (#46167) (#46333)

this field can be present in search slow logs and deprecation logs. The
docs describes how to enable this functionality and what expect in logs.
closes #44851

Author: pgomulka

docs/reference/index-modules/slowlog.asciidoc

@@ -83,6 +83,41 @@ logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref
 logger.index_search_slowlog_rolling.additivity = false
 --------------------------------------------------
 
+[float]
+==== Identifying search slow log origin
+
+It is often useful to identify what triggered a slow running query. If a call was initiated with an `X-Opaque-ID` header, then the user ID
+is included in Search Slow logs as an additional **id** field (scroll to the right).
+[source,txt]
+---------------------------
+[2030-08-30T11:59:37,786][WARN ][i.s.s.query              ] [node-0] [index6][0] took[78.4micros], took_millis[0], total_hits[0 hits], stats[], search_type[QUERY_THEN_FETCH], total_shards[1], source[{"query":{"match_all":{"boost":1.0}}}], id[MY_USER_ID],
+---------------------------
+// NOTCONSOLE
+The user ID is also included in JSON logs.
+[source,js]
+---------------------------
+{
+  "type": "index_search_slowlog",
+  "timestamp": "2030-08-30T11:59:37,786+02:00",
+  "level": "WARN",
+  "component": "i.s.s.query",
+  "cluster.name": "distribution_run",
+  "node.name": "node-0",
+  "message": "[index6][0]",
+  "took": "78.4micros",
+  "took_millis": "0",
+  "total_hits": "0 hits",
+  "stats": "[]",
+  "search_type": "QUERY_THEN_FETCH",
+  "total_shards": "1",
+  "source": "{\"query\":{\"match_all\":{\"boost\":1.0}}}",
+  "id": "MY_USER_ID",
+  "cluster.uuid": "Aq-c-PAeQiK3tfBYtig9Bw",
+  "node.id": "D7fUYfnfTLa2D7y-xw6tZg"
+}
+---------------------------
+// NOTCONSOLE
+
 [float]
 [[index-slow-log]]
 === Index Slow log

docs/reference/setup/logging-config.asciidoc

@@ -214,6 +214,24 @@ files (four rolled logs, and the active log).
 You can disable it in the `config/log4j2.properties` file by setting the deprecation
 log level to `error`.
 
+You can identify what is triggering deprecated functionality if `X-Opaque-Id` was used as an HTTP header.
+The user ID is included in the `X-Opaque-ID` field in deprecation JSON logs.
+[source,js]
+---------------------------
+{
+  "type": "deprecation",
+  "timestamp": "2019-08-30T12:07:07,126+02:00",
+  "level": "WARN",
+  "component": "o.e.d.r.a.a.i.RestCreateIndexAction",
+  "cluster.name": "distribution_run",
+  "node.name": "node-0",
+  "message": "[types removal] Using include_type_name in create index requests is deprecated. The parameter will be removed in the next major version.",
+  "x-opaque-id": "MY_USER_ID",
+  "cluster.uuid": "Aq-c-PAeQiK3tfBYtig9Bw",
+  "node.id": "D7fUYfnfTLa2D7y-xw6tZg"
+}
+---------------------------
+// NOTCONSOLE
 
 [float]
 [[json-logging]]