Improve docs for search preferences (#32159)

Today it is unclear what guarantees are offered by the search preference
feature, and we claim a guarantee that is stronger than what we really offer:

> A custom value will be used to guarantee that the same shards will be used
> for the same custom value.

This commit clarifies this documentation.

Forward-port of #32098 to `master`.

Author: DaveCTurner

docs/reference/search/request/preference.asciidoc

@@ -1,38 +1,55 @@
 [[search-request-preference]]
 === Preference
 
-Controls a `preference` of which shard copies on which to execute the
-search. By default, the operation is randomized among the available shard 
-copies, unless allocation awareness is used.
+Controls a `preference` of the shard copies on which to execute the search.  By
+default, Elasticsearch selects from the available shard copies in an
+unspecified order, taking the <<allocation-awareness,allocation awareness>> and
+<<search-adaptive-replica,adaptive replica selection>> configuration into
+account. However, it may sometimes be desirable to try and route certain
+searches to certain sets of shard copies, for instance to make better use of
+per-copy caches.
 
 The `preference` is a query string parameter which can be set to:
 
 [horizontal]
-`_local`:: 
-	The operation will prefer to be executed on a local
-	allocated shard if possible.
+`_only_local`::
+	The operation will be executed only on shards allocated to the local
+	node.
+
+`_local`::
+	The operation will be executed on shards allocated to the local node if
+	possible, and will fall back to other shards if not.
 
 `_prefer_nodes:abc,xyz`::
-	Prefers execution on the nodes with the provided
-	node ids (`abc` or `xyz` in this case) if applicable.
+	The operation will be executed on nodes with one of the provided node
+	ids (`abc` or `xyz` in this case) if possible. If suitable shard copies
+	exist on more than one of the selected nodes then the order of
+	preference between these copies is unspecified.
 
-`_shards:2,3`:: 
-	Restricts the operation to the specified shards. (`2`
-	and `3` in this case). This preference can be combined with other
-	preferences but it has to appear first: `_shards:2,3|_local`
+`_shards:2,3`::
+	Restricts the operation to the specified shards. (`2` and `3` in this
+	case).  This preference can be combined with other preferences but it
+	has to appear first: `_shards:2,3|_local`
 
-`_only_nodes`::
-    Restricts the operation to nodes specified in <<cluster,node specification>>
+`_only_nodes:abc*,x*yz,...`::
+	Restricts the operation to nodes specified according to the
+	<<cluster,node specification>>. If suitable shard copies exist on more
+	than one of the selected nodes then the order of preference between
+	these copies is unspecified.
 
-Custom (string) value:: 
-	A custom value will be used to guarantee that
-	the same shards will be used for the same custom value. This can help
-	with "jumping values" when hitting different shards in different refresh
-	states. A sample value can be something like the web session id, or the
-	user name.
+Custom (string) value::
+	Any value that does not start with `_`. If two searches both give the same
+	custom string value for their preference and the underlying cluster state
+	does not change then the same ordering of shards will be used for the
+	searches. This does not guarantee that the exact same shards will be used
+	each time: the cluster state, and therefore the selected shards, may change
+	for a number of reasons including shard relocations and shard failures, and
+	nodes may sometimes reject searches causing fallbacks to alternative nodes.
+	However, in practice the ordering of shards tends to remain stable for long
+	periods of time. A good candidate for a custom preference value is something
+	like the web session id or the user name.
 
-For instance, use the user's session ID to ensure consistent ordering of results
-for the user:
+For instance, use the user's session ID `xyzabc123` as follows:
 
 [source,js]
 ------------------------------------------------
@@ -47,3 +64,9 @@ GET /_search?preference=xyzabc123
 ------------------------------------------------
 // CONSOLE
 
+NOTE: The `_only_local` preference guarantees only to use shard copies on the
+local node, which is sometimes useful for troubleshooting. All other options do
+not _fully_ guarantee that any particular shard copies are used in a search,
+and on a changing index this may mean that repeated searches may yield
+different results if they are executed on different shard copies which are in
+different refresh states.