diff --git a/docs/user/dql/vector-search.rst b/docs/user/dql/vector-search.rst new file mode 100644 index 00000000000..3c16e918681 --- /dev/null +++ b/docs/user/dql/vector-search.rst @@ -0,0 +1,283 @@ + +============= +Vector Search +============= + +.. rubric:: Table of contents + +.. contents:: + :local: + :depth: 2 + +Introduction +============ + +The ``vectorSearch()`` table function runs a k-NN query against a ``knn_vector`` +field and exposes the matching documents as a relation in the ``FROM`` clause. +It relies on the OpenSearch `k-NN plugin +`_ — the target index must +map the vector field as ``knn_vector`` and the index must be created with +``index.knn: true``. + +Relevance is expressed through the OpenSearch ``_score`` metadata field, and +results are returned ordered by ``_score DESC`` by default. + +vectorSearch +============ + +Description +----------- + +``vectorSearch(table='', field='', vector='', option='')`` + +All four arguments are required and must be passed by name as string +literals. Positional arguments, or a mix of positional and named +arguments, are not supported. For example, the following is invalid:: + + FROM vectorSearch('my_vectors', field='embedding', + vector='[0.1,0.2]', option='k=5') AS v + +A table alias is required. Projected fields are referenced through the +alias (``v._id``, ``v._score``, ``v.category``). + +If the ``opensearch-knn`` plugin is not installed on the target cluster, +query execution fails with a ``vectorSearch() requires the k-NN plugin`` +error. ``_explain`` continues to work without the plugin. + +Arguments +--------- + +- ``table`` — single concrete index or alias to search. Wildcards + (``*``), comma-separated multi-index targets, ``_all``, ``.``, and + ``..`` are not supported. The target index must have + ``index.knn: true`` and map the target field as ``knn_vector``. +- ``field`` — name of the ``knn_vector`` field. +- ``vector`` — query vector as a JSON-style array of numbers, passed as a + string (for example, ``'[0.1, 0.2, 0.3]'``). Components must be + comma-separated finite numbers. Semicolon, colon, and pipe separators + are not supported, and empty components (for example, ``'[1.0,,2.0]'`` + or ``'[1.0,]'``) return an error. The vector dimension must match the + ``knn_vector`` mapping on the target index. +- ``option`` — comma-separated ``key=value`` pairs. Exactly one of ``k``, + ``max_distance``, or ``min_score`` is required. ``filter_type`` is + optional. + +Supported option keys +--------------------- + +Option keys are lower-case and case-sensitive. ``K=5`` or +``Filter_Type=post`` returns an "Unknown option key" error. + +- ``k`` — top-k mode. Integer between 1 and 10000. The query returns up to + ``k`` nearest neighbors. +- ``max_distance`` — radial mode. Non-negative number. Returns all + documents within the given distance of the query vector. ``LIMIT`` is + required. +- ``min_score`` — radial mode. Non-negative number. Returns all documents + with score at or above the given threshold. ``LIMIT`` is required. +- ``filter_type`` — ``post`` or ``efficient``. Controls how a ``WHERE`` + clause is applied. See `Filtering`_. + +``k``, ``max_distance``, and ``min_score`` are mutually exclusive; specify +exactly one. + +Native k-NN tuning options (for example, ``method_parameters.ef_search``, +``method_parameters.nprobes``, ``rescore.oversample_factor``) are not +supported through ``vectorSearch()`` and return an "Unknown option +key" error. + +Syntax +------ + +:: + + SELECT + FROM vectorSearch( + table='', + field='', + vector='', + option='' + ) AS + [WHERE ] + [ORDER BY ._score DESC] + [LIMIT ] + +Example 1: Top-k +---------------- + +Return the five nearest neighbors of a query vector:: + + POST /_plugins/_sql + { + "query" : """ + SELECT v._id, v._score + FROM vectorSearch( + table='my_vectors', + field='embedding', + vector='[0.1, 0.2, 0.3]', + option='k=5' + ) AS v + """ + } + +In top-k mode, the request size defaults to ``k``; adding ``LIMIT n`` further +reduces the row count, but ``n`` must not exceed ``k``. + +Example 2: Radial search (``max_distance``) +------------------------------------------- + +Return every document within a maximum distance of the query vector. +``LIMIT`` is required for radial searches — without it the result set is +unbounded:: + + POST /_plugins/_sql + { + "query" : """ + SELECT v._id, v._score + FROM vectorSearch( + table='my_vectors', + field='embedding', + vector='[0.1, 0.2, 0.3]', + option='max_distance=0.5' + ) AS v + LIMIT 100 + """ + } + +Example 3: Radial search (``min_score``) +---------------------------------------- + +Return every document whose score is at least the given threshold:: + + POST /_plugins/_sql + { + "query" : """ + SELECT v._id, v._score + FROM vectorSearch( + table='my_vectors', + field='embedding', + vector='[0.1, 0.2, 0.3]', + option='min_score=0.8' + ) AS v + LIMIT 100 + """ + } + +Filtering +========= + +A ``WHERE`` clause on non-vector fields of the ``vectorSearch()`` alias is +pushed down to OpenSearch when it can be translated to an OpenSearch filter. +Two placement strategies are available via the ``filter_type`` option: + +- ``post`` — the ``WHERE`` predicate is applied as a non-scoring + ``bool.filter`` alongside the k-NN query. The k-NN query runs first and + its results are then filtered. +- ``efficient`` — the ``WHERE`` predicate is embedded directly inside the + k-NN query (``knn.filter``), enabling pre-filtering during the ANN search. + See the `k-NN filtering guide `_ + for engine and method requirements. + +Behavior depends on whether ``filter_type`` is specified: + +- **Omitted** — pushdown is attempted using the ``post`` placement. + Predicates that translate to native OpenSearch queries are pushed down as a + ``bool.filter`` alongside the k-NN query. Predicates that do not have a + native equivalent (for example, arithmetic or function calls on indexed + fields) are pushed down as an OpenSearch script query and evaluated + server-side. Only when predicate translation itself fails does the engine + fall back to evaluating the ``WHERE`` clause in memory after the k-NN + results are returned. A query with no ``WHERE`` clause is valid. +- **Explicit ``post``** — a ``WHERE`` clause is required and must be + translatable to an OpenSearch filter query. If the ``WHERE`` clause is + missing or cannot be translated, the query fails with an error. + Specifying ``filter_type=post`` explicitly is useful when the query + should fail with an error instead of silently falling back to + in-memory filtering. +- **Explicit ``efficient``** — a ``WHERE`` clause is required and must + compile to a filter shape that can be embedded under ``knn.filter``. + ``efficient`` supports simple native filters: ``term``, ``range``, + ``wildcard``, ``exists``, full-text family (``match``, ``match_phrase``, + ``match_phrase_prefix``, ``match_bool_prefix``, ``multi_match``, + ``query_string``, ``simple_query_string``), and boolean combinations of + those filters. Predicates that compile to script queries (arithmetic, + function calls, ``CASE``, date math), nested predicates, and other + query shapes are not supported in this mode and return an error. + +Example 4: Implicit pushdown (no ``filter_type``) +------------------------------------------------- + +:: + + POST /_plugins/_sql + { + "query" : """ + SELECT v._id, v._score, v.category + FROM vectorSearch( + table='my_vectors', + field='embedding', + vector='[0.1, 0.2, 0.3]', + option='k=10' + ) AS v + WHERE v.category = 'books' + """ + } + +Example 5: Efficient (pre-)filtering +------------------------------------ + +:: + + POST /_plugins/_sql + { + "query" : """ + SELECT v._id, v._score, v.category + FROM vectorSearch( + table='my_vectors', + field='embedding', + vector='[0.1, 0.2, 0.3]', + option='k=10,filter_type=efficient' + ) AS v + WHERE v.category = 'books' + """ + } + +Scoring, sorting, and limits +============================ + +- ``vectorSearch()`` exposes the OpenSearch ``_score`` metadata field on the + alias. Select it as ``._score``. +- ``_score`` can be selected and referenced in ``ORDER BY``, but it cannot + appear in ``WHERE``. Use ``option='min_score=...'`` for score-threshold + vector search. +- Results are returned in ``_score DESC`` order by default. The only + supported ``ORDER BY`` expression is ``._score DESC``. +- In top-k mode (``k=N``), ``LIMIT n`` is optional; when present, ``n`` must + be ``≤ k``. +- In radial mode (``max_distance`` or ``min_score``), ``LIMIT`` is required. +- ``OFFSET`` is not supported on ``vectorSearch()``. Use ``LIMIT`` only. + +Limitations +=========== + +The following are not supported on ``vectorSearch()``: + +- ``GROUP BY`` and aggregations over a ``vectorSearch()`` relation are + not supported and return an error. +- An outer ``WHERE`` clause applied to a ``vectorSearch()`` subquery is + not supported and returns an error, because the predicate would be + evaluated only after the top-k rows have been selected by vector + distance and can silently yield zero rows. Place the predicate inside + the subquery, directly on the ``vectorSearch()`` alias, so that it + participates in ``WHERE`` pushdown. +- ``JOIN`` between a ``vectorSearch()`` relation and another relation is + not supported. +- ``UNION`` / ``INTERSECT`` / ``EXCEPT`` combining a ``vectorSearch()`` + relation with another relation is not supported. +- Multiple ``vectorSearch()`` calls in the same query are not supported. +- The query vector must be supplied as a literal. Parameterized vectors + (for example, values bound from another column) are not supported. +- Indexes that define a user field named ``_score`` cannot be queried + with ``vectorSearch()`` because ``_score`` is reserved for the + synthetic vector score exposed on the alias. Rename the field or query + the index with a plain ``SELECT``. diff --git a/docs/user/index.rst b/docs/user/index.rst index bb4b6399198..32ce39ed93d 100644 --- a/docs/user/index.rst +++ b/docs/user/index.rst @@ -43,6 +43,8 @@ OpenSearch SQL enables you to extract insights out of OpenSearch using the famil - `Window Functions `_ + - `Vector Search `_ + * **Beyond SQL** - `PartiQL (JSON) Support `_