From cec887d29c6b429f161e1de589fc281303d20b11 Mon Sep 17 00:00:00 2001 From: Simon Dew Date: Sat, 11 Oct 2025 22:39:54 +0100 Subject: [PATCH 01/14] Explain how the appropriate index is selected --- modules/vector-index/pages/composite-vector-index.adoc | 6 ++++++ modules/vector-index/pages/hyperscale-vector-index.adoc | 4 ++-- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/modules/vector-index/pages/composite-vector-index.adoc b/modules/vector-index/pages/composite-vector-index.adoc index 0755c0c27..6406976c5 100644 --- a/modules/vector-index/pages/composite-vector-index.adoc +++ b/modules/vector-index/pages/composite-vector-index.adoc @@ -194,9 +194,15 @@ The first of these functions is faster, but less precise. The second is more precise, but slower. Which you choose depends on your use case. +To use your Composite Vector index, use the `SELECT` statement with an `ORDER BY` clause containing a call to the `APPROX_VECTOR_DISTANCE` function. +This function selects the Composite Vector index when you query the vector key that the index covers. + You should also use a `LIMIT` clause to return just the number of vectors you need. The query pushes the `LIMIT` clause down into the index scan so that the scan ends after finding the required number of matches. +You can also perform vector comparisons using the VECTOR_DISTANCE function. +This function does not select a Hyperscale Vector index or a Composite Vector Index, but performs a brute-force comparison. + === Query RGB Values Querying the RGB values in `rgb.colorvect_l2` requires a vector with only three values. diff --git a/modules/vector-index/pages/hyperscale-vector-index.adoc b/modules/vector-index/pages/hyperscale-vector-index.adoc index 7e2ba600f..4b7c5660b 100644 --- a/modules/vector-index/pages/hyperscale-vector-index.adoc +++ b/modules/vector-index/pages/hyperscale-vector-index.adoc @@ -160,8 +160,8 @@ In most cases, you'll not use these settings in a production environment. == Query with a Hyperscale Vector Index -To use your Hyperscale Vector index, use the `SELECT` statement with a `ORDER BY` clause containing a call to the `APPROX_VECTOR_DISTANCE` function. -This function uses the Hyperscale Vector index when you query the vector key the index covers. +To use your Hyperscale Vector index, use the `SELECT` statement with an `ORDER BY` clause containing a call to the xref:n1ql:n1ql-language-reference/vectorfun.adoc#approx_vector_distance[APPROX_VECTOR_DISTANCE()] function. +The query selects the Hyperscale Vector index when this function includes the vector key that the index covers. A typical query looks like this: From 07022b24039249dfa228ef2ffa364caa1e62283e Mon Sep 17 00:00:00 2001 From: Simon Dew Date: Sat, 11 Oct 2025 22:44:03 +0100 Subject: [PATCH 02/14] Explain all ANN_DISTANCE arguments in Hyperscale Vector index guide --- .../pages/hyperscale-vector-index.adoc | 33 ++++++++++--------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/modules/vector-index/pages/hyperscale-vector-index.adoc b/modules/vector-index/pages/hyperscale-vector-index.adoc index 4b7c5660b..287aeac16 100644 --- a/modules/vector-index/pages/hyperscale-vector-index.adoc +++ b/modules/vector-index/pages/hyperscale-vector-index.adoc @@ -167,19 +167,18 @@ A typical query looks like this: [source,sqlpp] ---- -SELECT - FROM > +SELECT + FROM ORDER BY APPROX_VECTOR_DISTANCE( - , - , - , - ] + , + , + , + , + , + ) LIMIT ; ---- -NOTE: The previous query example shows just a subset of the most common parameters for the `APPROX_VECTOR_DISTANCE` function. -See xref:n1ql:n1ql-language-reference/vectorfun.adoc#approx_vector_distance[APPROX_VECTOR_DISTANCE()] for the full set of parameters available in this function. - The `APPROX_VECTOR_DISTANCE` parameters shown in the example are: * `collection-vector-column` is the name of the key containing the vector in the collection. @@ -189,17 +188,21 @@ It can be an array of floating point numbers or a base64 encoded string. This value should match the distance metric you used when you created the index. * `centroids-to-probe` is an optional integer value that sets the number of centroids to probe for matching vectors. By default, the vector search only probes a single centroid. +* `rerank` is an optional Boolean that can only be used when `centroids-to-probe` is specified. +It specifies whether the function should use full vectors to achieve better results. +For more information, see xref:vector-index:hyperscale-reranking.adoc[]. +* `topNScan` is an optional positive integer that can only be used when `centroids-to-probe` and `rerank` are specified. +If specified, it sets the number of records to scan. +Also use a `LIMIT` clause to return just the number of results you need. +The query pushes the `LIMIT` clause down into the index scan so that the scan ends after finding the number of matches you need. -NOTE: You can also call the function xref:n1ql:n1ql-language-reference/vectorfun.adoc#vector_distance[VECTOR_DISTANCE()] to find similar vectors. -However, this function does not use the Hyperscale Vector index to perform the vector search. +NOTE: You can also call the function xref:n1ql:n1ql-language-reference/vectorfun.adoc#vector_distance[VECTOR_DISTANCE()] to find similar vectors. +However, this function does not use the Hyperscale Vector index to perform the vector search. Instead, it performs a brute-force search for similar vectors. -It's useful to measure the recall of your Hyperscale Vector index. +It's useful to measure the recall of your Hyperscale Vector index. See xref:vector-index:vector-index-best-practices.adoc#recall-accuracy[Determine Recall Rate] for more information about measuring recall. -Also use a `LIMIT` clause to return just the number of results you need. -The query pushes the `LIMIT` clause down into the index scan so that the scan ends after finding the number of matches you need. - [#query-example] === Hyperscale Vector Index Query Example From f1454cbd4b74daa89b9713abad24728d0049d24c Mon Sep 17 00:00:00 2001 From: Simon Dew Date: Sat, 11 Oct 2025 22:44:45 +0100 Subject: [PATCH 03/14] Index Service not Query Service --- modules/vector-index/pages/composite-vector-index.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/vector-index/pages/composite-vector-index.adoc b/modules/vector-index/pages/composite-vector-index.adoc index 6406976c5..5c4fdb1bb 100644 --- a/modules/vector-index/pages/composite-vector-index.adoc +++ b/modules/vector-index/pages/composite-vector-index.adoc @@ -16,9 +16,9 @@ The added vector column lets your application perform a query using both the ind The Composite Vector index's single vector column enables semantic and similarity searches within your {sqlpp} queries. When creating the index, you use a `VECTOR` key attribute to identify the key that contains the embedded vectors. -When your query contains an embedded vector, the Query Service uses any non-vector predicates in the query to filter index entries. -Then it performs a vector similarity search to locate semantically related vectors. -Handling the non-vector predicates first reduces the number of vector similarity comparisons the Query Service must do to find similar vectors. +When your query contains an embedded vector, the Index Service uses any non-vector predicates in the query to filter index entries. +Then it performs a vector similarity search to locate semantically related vectors. +Handling the non-vector predicates first reduces the number of vector similarity comparisons the Index Service must do to find similar vectors. == Prerequisites From 773307b80a5f054f2d5c879ac1c7c5c6d3530aa4 Mon Sep 17 00:00:00 2001 From: Simon Dew Date: Sat, 11 Oct 2025 22:45:48 +0100 Subject: [PATCH 04/14] ORDER BY not GROUP BY --- modules/vector-index/pages/composite-vector-index.adoc | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/modules/vector-index/pages/composite-vector-index.adoc b/modules/vector-index/pages/composite-vector-index.adoc index 5c4fdb1bb..50a91ebdb 100644 --- a/modules/vector-index/pages/composite-vector-index.adoc +++ b/modules/vector-index/pages/composite-vector-index.adoc @@ -185,10 +185,7 @@ If successful, Couchbase {product-name} responds with: After Couchbase {product-name} creates the index, it begins training it. Depending on your system, this training can take several seconds. -== Query with a Composite Vector Index Column - -You query embedded vector attributes that you have indexed in a Composite Vector index to find similar vectors and therefore similar semantic content. -To find the most similar vectors, use a `GROUP BY` clause in your query to return the most relevant vectors first. +To find the most similar vectors, use an `ORDER BY` clause in your query to return the most relevant vectors first. In this clause, call one of two functions that actually performs the vector comparisons: `APPROX_VECTOR_DISTANCE` or `VECTOR_DISTANCE`. The first of these functions is faster, but less precise. The second is more precise, but slower. From 5c1841f5b2935c54929a49ddc249145ee22e74e6 Mon Sep 17 00:00:00 2001 From: Simon Dew Date: Sat, 11 Oct 2025 22:46:43 +0100 Subject: [PATCH 05/14] Similarity is optional --- modules/n1ql/pages/n1ql-language-reference/createindex.adoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/modules/n1ql/pages/n1ql-language-reference/createindex.adoc b/modules/n1ql/pages/n1ql-language-reference/createindex.adoc index c7c38b4d5..b683c151c 100644 --- a/modules/n1ql/pages/n1ql-language-reference/createindex.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/createindex.adoc @@ -473,7 +473,7 @@ The embedded model you use to embed the vectors determines the number of dimensi |Integer |**similarity** + -__required__ +__optional__ | Sets the distance metric to use when comparing vectors during index creation. Couchbase {product-name} uses the following strings to represent the distance metrics: @@ -487,6 +487,8 @@ L2_SQUARED:: EUCLIDEAN_SQUARED:: xref:vector-index:vectors-and-indexes-overview.adoc#euclidean-squared[Euclidean Squared Distance] For the greatest accuracy, use the distance metric you plan to use to query the data. + +**Default:** `L2_SQUARED` |String |**description** + From 3a2ae5b410a4f7df6267f1aa8bcfb966ebe817b4 Mon Sep 17 00:00:00 2001 From: Simon Dew Date: Sat, 11 Oct 2025 22:48:08 +0100 Subject: [PATCH 06/14] Description is optional --- .../n1ql/pages/n1ql-language-reference/createindex.adoc | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/modules/n1ql/pages/n1ql-language-reference/createindex.adoc b/modules/n1ql/pages/n1ql-language-reference/createindex.adoc index b683c151c..aedfba67b 100644 --- a/modules/n1ql/pages/n1ql-language-reference/createindex.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/createindex.adoc @@ -395,7 +395,7 @@ Use the WITH clause to specify additional options. expr:: An object with the following properties. -[options="header", cols="1a,4a,1a"] +[options="header", cols="3a,8a,2a"] |=== |Name|Description|Schema @@ -461,7 +461,7 @@ If the value of this property is not less than the number of index nodes in the Composite vector indexes support the following additional options. -[options="header", cols="1a,4a,1a"] +[options="header", cols="3a,8a,2a"] |=== |Name|Description|Schema //tag::index-with[] @@ -492,7 +492,7 @@ For the greatest accuracy, use the distance metric you plan to use to query the |String |**description** + -__required__ +__optional__ | The settings for the quantization and index algorithms. The string is made up of the following settings: @@ -503,7 +503,9 @@ PQ:: For product quantization -- the number of subquantizers, and the number of For more information, see {algo-settings}[Quantization and Centroid Settings]. +[%hardbreaks] **Pattern:** `^IVF[0-9]*,(SQ[468]|PQ[0-9]+x[0-9]+)$` +**Default:** `IVF,SQ8` |String |**scan_nprobes** + From 04b0f67ad79c76df72b396b233ffd0bb7cf24781 Mon Sep 17 00:00:00 2001 From: Simon Dew Date: Sat, 11 Oct 2025 22:51:03 +0100 Subject: [PATCH 07/14] Composite Vector index with leading scalar key --- .../n1ql-language-reference/createindex.adoc | 14 ++++++++ .../examples/gsi-vector-idx-examples.sqlpp | 6 ++++ .../pages/composite-vector-index.adoc | 34 ++++++++++++++++++- 3 files changed, 53 insertions(+), 1 deletion(-) diff --git a/modules/n1ql/pages/n1ql-language-reference/createindex.adoc b/modules/n1ql/pages/n1ql-language-reference/createindex.adoc index aedfba67b..2ddd1c63c 100644 --- a/modules/n1ql/pages/n1ql-language-reference/createindex.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/createindex.adoc @@ -760,6 +760,20 @@ include::vector-index:example$gsi-vector-idx-examples.sqlpp[tag=create-vectors-i ---- ==== +[[ex-create-colors-idx]] +.Create a composite vector index with a scalar leading key +==== +For this example, you must install the vector sample data as described in {prerequisites}[Prerequisites]. +The path to the required keyspace is specified by the query, so you do not need to set the query context. + +Create a composite vector index that indexes the scalar `color` and `brightness` fields, as well as the vector field named `embedding-vector-dot`. + +[source,sqlpp] +---- +include::vector-index:example$gsi-vector-idx-examples.sqlpp[tag=create-colors-idx] +---- +==== + == Related Links * xref:indexes:indexing-overview.adoc[] diff --git a/modules/vector-index/examples/gsi-vector-idx-examples.sqlpp b/modules/vector-index/examples/gsi-vector-idx-examples.sqlpp index aae249659..700aedccf 100644 --- a/modules/vector-index/examples/gsi-vector-idx-examples.sqlpp +++ b/modules/vector-index/examples/gsi-vector-idx-examples.sqlpp @@ -32,6 +32,12 @@ CREATE INDEX `color_desc_idx` ON `vector-sample`.`color`.`rgb` WITH { "dimension":1536, "similarity":"DOT", "description":" IVF,SQ8" } /* end::create-vectors-idx[] */ +/* tag::create-colors-idx[] */ +CREATE INDEX `color_name_idx` ON `vector-sample`.`color`.`rgb` + (color, brightness, `embedding_vector_dot` VECTOR) + WITH { "dimension":1536, "similarity":"DOT", "description":" IVF,SQ8" } +/* end::create-colors-idx[] */ + /* tag::query-rgb-idx[] */ SELECT b.color, b.colorvect_l2, b.brightness from `rgb` AS b ORDER BY APPROX_VECTOR_DISTANCE(b.colorvect_l2,[128,128,128],"L2") diff --git a/modules/vector-index/pages/composite-vector-index.adoc b/modules/vector-index/pages/composite-vector-index.adoc index 50a91ebdb..a8772a469 100644 --- a/modules/vector-index/pages/composite-vector-index.adoc +++ b/modules/vector-index/pages/composite-vector-index.adoc @@ -64,7 +64,10 @@ include::vector-index:partial$create-composite-index-capella-ui.adoc[opts=option Creating a Composite Vector index is similar to creating a non-vector GSI index. See xref:guides:create-index.adoc[] for an overview of creating indexes. -In the `CREATE INDEX` statement to create the Composite Vector index, add the `VECTOR` lead key attribute after the vector's key name to declare it as an embedded vector. +In the `CREATE INDEX` statement to create the Composite Vector index, add the `VECTOR` key attribute after the vector's key name to declare it as an embedded vector. + +The index key that refers to a vector field may be the only index key. +If there are multiple index keys, the index key referring to the vector field may be any of the index keys, including the leading index key. You must also use the xref:n1ql:n1ql-language-reference/createindex.adoc#index-with[WITH] clause to specify some additional information for the vector column. The format for this clause with the most commonly used parameters is: @@ -185,6 +188,35 @@ If successful, Couchbase {product-name} responds with: After Couchbase {product-name} creates the index, it begins training it. Depending on your system, this training can take several seconds. +==== Create a Composite Vector Index with a Scalar Leading Key + +If a Composite Vector index has multiple index keys, the leading index key may be a scalar field, rather than the vector field. +This enables the Index Service to pre-filter using a scalar field. +This is a key advantage that a Composite Vector index has over a Hyperscale Vector index with included scalar fields. + +The following example creates a Composite Vector index that indexes the scalar `color` and brightness fields, as well as the embedded vectors in the `embedding_vector_dot` field. + +[source,sqlpp] +---- +include::vector-index:example$gsi-vector-idx-examples.sqlpp[tag=create-colors-idx] +---- + +If successful, Couchbase {product-name} responds with: + +[source, json] +---- +[ + { + "id": "45222612ffac4e98", + "name": "color_name_idx", + "state": "online" + } +] +---- + +After Couchbase {product-name} creates the index, it begins training it. +Depending on your system, this training can take several seconds. + To find the most similar vectors, use an `ORDER BY` clause in your query to return the most relevant vectors first. In this clause, call one of two functions that actually performs the vector comparisons: `APPROX_VECTOR_DISTANCE` or `VECTOR_DISTANCE`. The first of these functions is faster, but less precise. From 1e1ce89e43873f06ed310d24ce5655d90fafe536 Mon Sep 17 00:00:00 2001 From: Simon Dew Date: Sat, 11 Oct 2025 23:34:50 +0100 Subject: [PATCH 08/14] HVI and CVI are supported on Windows --- modules/n1ql/pages/n1ql-language-reference/createindex.adoc | 2 -- .../n1ql-language-reference/vector-index-no-windows.adoc | 5 ----- 2 files changed, 7 deletions(-) delete mode 100644 modules/n1ql/partials/n1ql-language-reference/vector-index-no-windows.adoc diff --git a/modules/n1ql/pages/n1ql-language-reference/createindex.adoc b/modules/n1ql/pages/n1ql-language-reference/createindex.adoc index 2ddd1c63c..e181ca701 100644 --- a/modules/n1ql/pages/n1ql-language-reference/createindex.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/createindex.adoc @@ -73,8 +73,6 @@ If the data set changes dramatically, you must drop and rebuild the index to upd To execute the {doctitle} statement, you must have the `Query Manage Index` privilege granted on the keyspace. For more information about user roles, see {authorization-overview}[Authorization]. - -include::partial$n1ql-language-reference/vector-index-no-windows.adoc[] //end::prerequisites[] == Syntax diff --git a/modules/n1ql/partials/n1ql-language-reference/vector-index-no-windows.adoc b/modules/n1ql/partials/n1ql-language-reference/vector-index-no-windows.adoc deleted file mode 100644 index bf7ee376e..000000000 --- a/modules/n1ql/partials/n1ql-language-reference/vector-index-no-windows.adoc +++ /dev/null @@ -1,5 +0,0 @@ -[IMPORTANT] --- -You cannot use Hyperscale Vector Indexes or Composite Vector Indexes on Windows platforms. -You can use Hyperscale Vector Indexes and Composite Vector Indexes in Couchbase Server version 8.0 and later on Linux and macOS. --- \ No newline at end of file From f336eea4d16d192e8eec759e5fb4a1f7ad613a1a Mon Sep 17 00:00:00 2001 From: Simon Dew Date: Sat, 11 Oct 2025 23:36:58 +0100 Subject: [PATCH 09/14] Include persist_full_vector --- modules/n1ql/pages/n1ql-language-reference/createindex.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/n1ql/pages/n1ql-language-reference/createindex.adoc b/modules/n1ql/pages/n1ql-language-reference/createindex.adoc index e181ca701..d3d7c7b6b 100644 --- a/modules/n1ql/pages/n1ql-language-reference/createindex.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/createindex.adoc @@ -524,7 +524,7 @@ Otherwise, the default value is 10% of the index count, or 10{nbsp}× the n **Maximum:** `1000000` |Integer -ifeval::["{docname}" == "createvectorindex"] +ifeval::["{docname}" == "n1ql-language-reference/createvectorindex"] |**persist_full_vector** + __optional__ | If `true` (the default), the index stores the full vector value in addition to the quantized value. From 627a35c4a5be8ffa8b0fccd4ea21be906afa07f2 Mon Sep 17 00:00:00 2001 From: Simon Dew Date: Sat, 11 Oct 2025 23:37:29 +0100 Subject: [PATCH 10/14] Remove trailing spaces --- .../n1ql-language-reference/vectorfun.adoc | 4 +- .../pages/composite-vector-index.adoc | 91 ++++++++++--------- .../pages/hyperscale-vector-index.adoc | 58 ++++++------ 3 files changed, 78 insertions(+), 75 deletions(-) diff --git a/modules/n1ql/pages/n1ql-language-reference/vectorfun.adoc b/modules/n1ql/pages/n1ql-language-reference/vectorfun.adoc index 9f82a3d19..ad7fbac73 100644 --- a/modules/n1ql/pages/n1ql-language-reference/vectorfun.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/vectorfun.adoc @@ -60,9 +60,9 @@ If the Query Service selects a hyperscale vector index or composite vector index If an invalid value is provided, defaults to `1`. rerank:: [Optional; can only be used when `nprobes` is specified] -A boolean. +A Boolean. If `false`, the function uses quantized vectors. -It `true`, the function uses full vectors to reorder the results. +If `true`, the function uses full vectors to reorder the results. The default is `false`. topNScan:: [Optional; can only be used when `nprobes` and `rerank` are specified] diff --git a/modules/vector-index/pages/composite-vector-index.adoc b/modules/vector-index/pages/composite-vector-index.adoc index a8772a469..02942e5a4 100644 --- a/modules/vector-index/pages/composite-vector-index.adoc +++ b/modules/vector-index/pages/composite-vector-index.adoc @@ -3,15 +3,15 @@ :page-ui-name: {ui-name} :page-product-name: {product-name} :description: A Composite Vector index is a Global Secondary Index (GSI) with a single vector column that combines scalar queries with semantic search. -:stem: +:stem: :page-toclevels: 3 :index-name: Composite Vector index :index-name-plural: Composite Vector indexes {description} -The added vector column lets your application perform a query using both the index's scalar, array, and object index entries to pre-filter the dataset before performing a vector similarity search. +The added vector column lets your application perform a query using both the index's scalar, array, and object index entries to pre-filter the dataset before performing a vector similarity search. -== How the Composite Vector Index's Vector Column Works +== How the Composite Vector Index's Vector Column Works The Composite Vector index's single vector column enables semantic and similarity searches within your {sqlpp} queries. When creating the index, you use a `VECTOR` key attribute to identify the key that contains the embedded vectors. @@ -22,10 +22,10 @@ Handling the non-vector predicates first reduces the number of vector similarity == Prerequisites -* You must have the Index Service enabled on at least one node in your cluster. +* You must have the Index Service enabled on at least one node in your cluster. For more information about how to deploy a new node and Services on your database, see xref:manage:manage-nodes/node-management-overview.adoc[]. -* You must have a bucket with scopes and collections in your database. +* You must have a bucket with scopes and collections in your database. For more information about how to create a bucket, see xref:manage:manage-buckets/create-bucket.adoc[]. * Your account must have the xref:learn:security/roles.adoc#query-manage-index[Query Manage Index] or an administrator role to be able to create an index. @@ -35,7 +35,7 @@ You can add a single vector to a Composite Vector index. If your documents contain multiple embedded vectors, you can create multiple indexes -- one for each vector attribute. + Embeddings can be an array of floating point numbers or a base64 encoded string. -Couchbase {product-name} does not embed vectors itself. +Couchbase {product-name} does not embed vectors itself. You must use an external embedding model to embed vectors into your data and add them to your documents. + [TIP] @@ -43,13 +43,13 @@ You must use an external embedding model to embed vectors into your data and add include::vector-search:partial$download-sample-partial.adoc[] -- -* You must know the number of dimensions the vector contains. +* You must know the number of dimensions the vector contains. The embedding model you use to embed the vectors may determine this value for you. For example, OpenAI API's `text-embedding-ada-002` embedding model that embedded the sample data demonstrated later in this page creates vectors that have 1536 dimensions. -* You must decide what distance metric and quantization you want your index to use. -The metrics affect how the index compares vectors. -The quantization determines how much memory your index uses and the amount of processing Couchbase {product-name} must perform to train and search them. +* You must decide what distance metric and quantization you want your index to use. +The metrics affect how the index compares vectors. +The quantization determines how much memory your index uses and the amount of processing Couchbase {product-name} must perform to train and search them. See xref:vector-index:vectors-and-indexes-overview.adoc#vector_similarity[Vector Similarity Metrics] and xref:vector-index:vectors-and-indexes-overview.adoc#quantization[Quantization] for more information. @@ -58,7 +58,7 @@ See xref:vector-index:vectors-and-indexes-overview.adoc#vector_similarity[Vector //// include::vector-index:partial$create-composite-index-capella-ui.adoc[opts=optional] -== Create a Composite Vector Index Using SQL++ +== Create a Composite Vector Index Using {sqlpp} //// == Create a Composite Vector Index @@ -69,7 +69,7 @@ In the `CREATE INDEX` statement to create the Composite Vector index, add the `V The index key that refers to a vector field may be the only index key. If there are multiple index keys, the index key referring to the vector field may be any of the index keys, including the leading index key. -You must also use the xref:n1ql:n1ql-language-reference/createindex.adoc#index-with[WITH] clause to specify some additional information for the vector column. +You must also use the xref:n1ql:n1ql-language-reference/createindex.adoc#index-with[WITH] clause to specify some additional information for the vector column. The format for this clause with the most commonly used parameters is: [source,sqlpp] @@ -91,7 +91,7 @@ include::partial$distance-metric-list.adoc[] + For the greatest accuracy, use the distance function you plan to use when querying vector data. -* `centroids_and_quantization` is a string containing the settings for the quantization and index algorithms. +* `centroids_and_quantization` is a string containing the settings for the quantization and index algorithms. See <<#algo_settings>> in the next section for more information. For a full list of the parameters that affect a Composite Vector index, see xref:n1ql:n1ql-language-reference/createindex.adoc[]. @@ -102,7 +102,7 @@ include::partial$index-algorithm-settings.adoc[] === Examples The following examples show you how to create two Composite Vector index with a vector column using sample data. -They both use the data from the `color_data_2vectors.zip` file mentioned earlier. +They both use the data from the `color_data_2vectors.zip` file mentioned earlier. The following query gets a single document from the `rgb` collection in the `vector-sample` bucket's `color` scope. It truncates the `embedding_vector_dot` attribute to the first four values to improve readability. @@ -121,8 +121,8 @@ include::vector-index:example$composite-vector-data.json[tag=sample-doc] ==== Index the RGB Values -The `rgb.json` file's `colorvect_l2` attribute defines an array containing the RGB values for the entry's color. -While this technically is not an embedded vector, you can still create a vector index column for this array. +The `rgb.json` file's `colorvect_l2` attribute defines an array containing the RGB values for the entry's color. +While this technically is not an embedded vector, you can still create a vector index column for this array. The following example creates a Composite Vector index for this attribute as an embedded vector as well as the color's name and brightness. [source,sqlpp] @@ -133,14 +133,14 @@ include::vector-index:example$gsi-vector-idx-examples.sqlpp[tag=create-rgb-idx] In this example: * The number of dimensions is 3, because there are three values in the array containing the RGB value. -* The similarity function is `L2`. +* The similarity function is `L2`. This function works well to find related vectors which are close by the search vector. In this example, finding similar colors depends more on proximity than the magnitude or alignment of the vectors. See xref:vectors-and-indexes-overview.adoc#vector_similarity[Vector Similarity] for a comparison of similarity functions. * The `description` lets Couchbase {product-name} decide the number of centroids for the IVF algorithm. It also chooses to use Scalar Quantization with an 8-bit index, splitting each dimension into 256 bins. -This setting does not actually save any space in the index, as each of the RGB dimensions are already 8-bit values. -However, in this example, memory use is not a concern as the dataset is small. +This setting does not actually save any space in the index, as each of the RGB dimensions are already 8-bit values. +However, in this example, memory use is not a concern as the dataset is small. The result of running example is: @@ -162,14 +162,14 @@ The data sample shown in <<#examples>> truncated this attribute to several value The embedded vector contains 1536 dimensions. The following example creates a Composite Vector index that indexes the embedded vectors in the `embedding_vector_dot` as well as indexing the scalar `color` that contains the color's name and `brightness.` - + [source,sqlpp] ---- include::vector-index:example$gsi-vector-idx-examples.sqlpp[tag=create-vectors-idx] ---- -This example uses the Dot Product similarity function. -This function works better with the embedded text content than the Euclidean function used in the previous example. +This example uses the Dot Product similarity function. +This function works better with the embedded text content than the Euclidean function used in the previous example. It also uses the same algorithms as the previous example--Couchbase {product-name} chooses the number of centroids, and uses SQ quantization with 256 bins. If successful, Couchbase {product-name} responds with: @@ -185,7 +185,7 @@ If successful, Couchbase {product-name} responds with: ] ---- -After Couchbase {product-name} creates the index, it begins training it. +After Couchbase {product-name} creates the index, it begins training it. Depending on your system, this training can take several seconds. ==== Create a Composite Vector Index with a Scalar Leading Key @@ -217,6 +217,9 @@ If successful, Couchbase {product-name} responds with: After Couchbase {product-name} creates the index, it begins training it. Depending on your system, this training can take several seconds. +== Query with a Composite Vector Index + +You query embedded vector attributes to find similar vectors, and therefore similar semantic content. To find the most similar vectors, use an `ORDER BY` clause in your query to return the most relevant vectors first. In this clause, call one of two functions that actually performs the vector comparisons: `APPROX_VECTOR_DISTANCE` or `VECTOR_DISTANCE`. The first of these functions is faster, but less precise. @@ -226,7 +229,7 @@ Which you choose depends on your use case. To use your Composite Vector index, use the `SELECT` statement with an `ORDER BY` clause containing a call to the `APPROX_VECTOR_DISTANCE` function. This function selects the Composite Vector index when you query the vector key that the index covers. -You should also use a `LIMIT` clause to return just the number of vectors you need. +You should also use a `LIMIT` clause to return just the number of vectors you need. The query pushes the `LIMIT` clause down into the index scan so that the scan ends after finding the required number of matches. You can also perform vector comparisons using the VECTOR_DISTANCE function. @@ -243,13 +246,13 @@ The following example finds colors that are similar to gray, which has an RGB va include::vector-index:example$gsi-vector-idx-examples.sqlpp[tag=query-rgb-idx] ---- -The query uses the `APPROX_VECTOR_DISTANCE` function to sort the results. +The query uses the `APPROX_VECTOR_DISTANCE` function to sort the results. You pass it the vector column to search, the vector to search for (in this case, the array `128, 128, 128`) and the distance function. For the best accuracy, use the same distance function you specified when creating the Composite Vector index (in this case, `L2`). The query pushes the `LIMIT` clause down into the index scan, so once it finds the 5 entries that satisfy the query, it exits. -The top result is the entry for gray. +The top result is the entry for gray. The other results are all shades of gray: [source,json] @@ -368,7 +371,7 @@ To query the `color_desc_idx` Composite Vector index containing the embedded vec In a production environment, your application calls the same embedding model it called to generate the embedded vectors in your documents to generate a vector for the query value. For this example, you can use embedded vectors in the `rgb_questions.json` file that's in the `color_data_2vectors.zip` file. -This file contains a `question` attribute containing a search prompt for a particular color. +This file contains a `question` attribute containing a search prompt for a particular color. The following query gets a single document from the `rgb_questions` collection in the `vector-sample` bucket's `color` scope. It truncates the `couchbase_search_query.knn.vector` attribute to the first four values to improve readability. @@ -387,7 +390,7 @@ include::vector-index:example$composite-vector-data.json[tag=sample-doc-question The `couchbase_search_query.knn.vector` attribute contains the embedded vector for the `question` attribute. -This example queries the `embedding_vector_dot` column. +This example queries the `embedding_vector_dot` column. It appears here with most of the 1536 vectors omitted: [source,sqlpp] @@ -395,7 +398,7 @@ It appears here with most of the 1536 vectors omitted: include::vector-index:example$query-vectors-idx-whole.sqlpp[tags=!query-vectors-idx-truncated] ---- -You can click the btn:[View] button to see and copy the entire query with all the vectors. +You can click btn:[View] to see and copy the entire query with all the vectors. Another option is to import the `rgb_questions.json` file into another collection in the `vector-sample` bucket's `color` scope named `rgb-questions`. Then you can use a subquery to get the vectors for the question and use it in your query of the `rgb` collection's `embedding_vector_dot` attribute: @@ -412,37 +415,37 @@ In either case, the results of the query are the same: [ { "color": "cantalope", - "description": "The color cantaloupe is a soft and soothing shade that evokes feelings of calmness - and relaxation. It is a refreshing hue that brings to mind the juicy and sweet fruit it is named - after. This delicate color is a pale orange with hints of pink, giving it a subtle and gentle + "description": "The color cantaloupe is a soft and soothing shade that evokes feelings of calmness + and relaxation. It is a refreshing hue that brings to mind the juicy and sweet fruit it is named + after. This delicate color is a pale orange with hints of pink, giving it a subtle and gentle appearance. It is a perfect color for creating a peaceful and tranquil atmosphere." }, { "color": "papaya whip", - "description": "Papaya whip is a soft and mellow color that can be described as a light shade of - peach or coral. It has a calming and soothing effect, similar to the tropical fruit it is named - after. This color is perfect for creating a warm and inviting atmosphere, and it pairs well with - other pastel shades or neutral tones. Papaya whip is a versatile color that can be used in both + "description": "Papaya whip is a soft and mellow color that can be described as a light shade of + peach or coral. It has a calming and soothing effect, similar to the tropical fruit it is named + after. This color is perfect for creating a warm and inviting atmosphere, and it pairs well with + other pastel shades or neutral tones. Papaya whip is a versatile color that can be used in both fashion and interior design, adding a touch of elegance and sophistication to any space." }, { "color": "apricot", - "description": "Apricot is a warm and inviting color, reminiscent of the soft glow of a sunset. It - has the ability to soften the harshness of other colors and enliven any space it is used in. It is a + "description": "Apricot is a warm and inviting color, reminiscent of the soft glow of a sunset. It + has the ability to soften the harshness of other colors and enliven any space it is used in. It is a delicate and soothing hue, perfect for creating a cozy and welcoming atmosphere." } ] ---- -The second result, the color papaya whip, matches the `rgb_questions` collection's `wanted_similar_color_from_search` attribute. +The second result, the color papaya whip, matches the `rgb_questions` collection's `wanted_similar_color_from_search` attribute. === Adding a Scalar Using additional scalar fields in your search can improve your results and reduce the overhead of performing a vector search. -For example, filtering on an additional scalar field reduces the number of vectors an index scan has to compare. +For example, filtering on an additional scalar field reduces the number of vectors an index scan has to compare. Searching for scalar values requires less resources than vector searches. -The example that created the `color_desc_idx` index added fields in addition to the `embedding_vector_dot` vector field. +The example that created the `color_desc_idx` index added fields in addition to the `embedding_vector_dot` vector field. The following example adds a filter based on the `brightness` field to reduce the number of vectors that get compared and also improve the results. The version of the query that performs a subquery of the `rgb-questions` to get the vector value is: @@ -452,16 +455,16 @@ The version of the query that performs a subquery of the `rgb-questions` to get include::vector-index:example$gsi-vector-idx-examples.sqlpp[tag=query-vectors-idx-subquery-filtered] ---- -The truncated version of the query is: +The truncated version of the query is: [source,sqlpp] ---- include::vector-index:example$query-vectors-idx-filtered-whole.sqlpp[tags=!query-vectors-idx-filtered-truncated] ---- -You can click the btn:[View] button to see and copy the entire query with all the vectors. +You can click btn:[View] to see and copy the entire query with all the vectors. -The results of this query moves the papaya whip entry to the top. +The results of this query moves the papaya whip entry to the top. [source,json] ---- diff --git a/modules/vector-index/pages/hyperscale-vector-index.adoc b/modules/vector-index/pages/hyperscale-vector-index.adoc index 287aeac16..da6d8b42f 100644 --- a/modules/vector-index/pages/hyperscale-vector-index.adoc +++ b/modules/vector-index/pages/hyperscale-vector-index.adoc @@ -3,7 +3,7 @@ :page-ui-name: {ui-name} :page-product-name: {product-name} :description: Hyperscale Vector Indexes are optimized to index a single vector column. They offer the highest performance of any index when it comes to vector data. -:stem: +:stem: :page-toclevels: 3 :index-name: Hyperscale Vector index :index-name-plural: Hyperscale Vector indexes @@ -16,26 +16,26 @@ Because they provide the best performance, consider testing a Hyperscale Vector If you find theirs performance does not meet your needs, then test using a Composite Vector Index or a Search Vector Index. -== How the Hyperscale Vector Index Works +== How the Hyperscale Vector Index Works -The Hyperscale Vector Index primarily relies on data stored in an optimized format on disk. -By relying on disk storage, they have lower memory requirements than other index types. +The Hyperscale Vector Index primarily relies on data stored in an optimized format on disk. +By relying on disk storage, they have lower memory requirements than other index types. -You add a single vector column to a Hyperscale Vector Index. +You add a single vector column to a Hyperscale Vector Index. The vector value can be an array of floating point values. -You can nest the array in another array as long as you use an xref:n1ql:n1ql-language-reference/unnest.adoc[UNNEST clause] to extract the it from the containing array. -You can also store the vector in a BASE64 string. +You can nest the array in another array as long as you use an xref:n1ql:n1ql-language-reference/unnest.adoc[UNNEST clause] to extract the it from the containing array. +You can also store the vector in a BASE64 string. == Prerequisites Hyperscale Vector Indexes have the following requirements: -* You must have the Index Service enabled on at least one node in your cluster. +* You must have the Index Service enabled on at least one node in your cluster. For more information about how to deploy a new node and Services on your database, see xref:manage:manage-nodes/node-management-overview.adoc[]. * Your account must have the xref:learn:security/roles.adoc#query-manage-index[Query Manage Index] or an administrator role to be able to create an index. -* You must have a bucket your database. +* You must have a bucket your database. For more information about how to create a bucket, see xref:manage:manage-buckets/create-bucket.adoc[]. * You have documents in a collection that contain one or more vector embeddings. @@ -43,7 +43,7 @@ You can add a single vector to a Hyperscale Vector Index. If your documents contain multiple embedded vectors, you can create multiple indexes -- one for each vector attribute. + Embeddings can be an array of floating point numbers or a base64 encoded string. -Couchbase {product-name} does not embed vectors itself. +Couchbase {product-name} does not embed vectors itself. You must use an external embedding model to embed vectors into your data and add them to your documents. + [TIP] @@ -51,17 +51,17 @@ You must use an external embedding model to embed vectors into your data and add include::vector-search:partial$download-sample-partial.adoc[] -- -* The vectors you add to an index must contain the same number of dimensions. +* The vectors you add to an index must contain the same number of dimensions. Also the values in the vector must be 32-bit floating point numbers. -If a vector does not meet both of these requirements, the vector index treats it as a NULL value and the document is not added to the index. +If a vector does not meet both of these requirements, the vector index treats it as a NULL value and the document is not added to the index. -* You must know the number of dimensions the vector contains. -The embedding model you use to embed the vectors may determine this value for you. +* You must know the number of dimensions the vector contains. +The embedding model you use to embed the vectors may determine this value for you. For example, OpenAI API's `text-embedding-ada-002` embedding model creates vectors that have 1536 dimensions. * You must decide whether you want to use the default distance metric and quantization for your index. -By default, a Hyperscale Vector index uses the xref:vector-index:vectors-and-indexes-overview.adoc#euclidean-squared[Euclidean distance squared] metric and xref:vector-index:vectors-and-indexes-overview.adoc#sq[Scalar Quantization (SQ)] with 8 bits per vector dimension (`SQ8`). -The metrics affect how the index compares vectors. +By default, a Hyperscale Vector index uses the xref:vector-index:vectors-and-indexes-overview.adoc#euclidean-squared[Euclidean distance squared] metric and xref:vector-index:vectors-and-indexes-overview.adoc#sq[Scalar Quantization (SQ)] with 8 bits per vector dimension (`SQ8`). +The metrics affect how the index compares vectors. The quantization determines how much memory your index uses and the amount of processing Couchbase {product-name} must perform to train and search them. See xref:vector-index:vectors-and-indexes-overview.adoc#vector_similarity[Vector Similarity Metrics] and xref:vector-index:vectors-and-indexes-overview.adoc#quantization[Quantization] for more information. @@ -75,9 +75,9 @@ include::vector-index:partial$create-hyperscale-index-capella-ui.adoc[opts=optio == Create a Hyperscale Vector Index Use the `CREATE VECTOR INDEX` statement to create a Hyperscale Vector Index. -This statement is similar to the `CREATE INDEX` statement that you use to create Global Secondary Indexes (GSI). +This statement is similar to the `CREATE INDEX` statement that you use to create Global Secondary Indexes (GSI). See xref:guides:create-index.adoc[] for an overview of creating indexes. -You must also supply a `WITH` clause to set some additional information about the vector column that the Hyperscale index needs. +You must also supply a `WITH` clause to set some additional information about the vector column that the Hyperscale index needs. The following syntax shows the minimum required parameters to create a Hyperscale Vector index: @@ -91,7 +91,7 @@ CREATE VECTOR INDEX `` }; ---- -NOTE: This syntax for the `CREATE VECTOR INDEX` shows the minimum required parameters to get you started. +NOTE: This syntax for the `CREATE VECTOR INDEX` shows the minimum required parameters to get you started. For the full syntax, see xref:n1ql:n1ql-language-reference/createvectorindex.adoc[] in the SQL++ for Query Reference. The parameters in this statement are: @@ -99,7 +99,7 @@ The parameters in this statement are: * `index_name` is a string that sets the name of the index. * `collection` is the path of the collection to index. * `key_name` is the name of the key containing the vector that you want to index. -The key value must be an array of floating point numbers or a base64 encoded string. +The key value must be an array of floating point numbers or a base64 encoded string. * `dimensions` The number of dimensions in the vector as an integer. The embedded model you use to embed the vectors determines the number of dimensions in the vector. * `similarity_metric` is a string that sets the distance metric to use when comparing vectors during index creation. @@ -108,7 +108,7 @@ include::partial$distance-metric-list.adoc[] + For the greatest accuracy, use the distance metric you plan to use to query the data. -* `centroids_and_quantization` is a string containing the settings for the quantization and index algorithms. +* `centroids_and_quantization` is a string containing the settings for the quantization and index algorithms. See <<#algo_settings>> in the next section for more information. include::partial$index-algorithm-settings.adoc[] @@ -146,14 +146,14 @@ include::vector-index:example$hyperscale-idx-examples.sqlpp[tag=create-rgb-idx] The key pieces of this example are: -* The `CREATE VECTOR INDEX` statement creates a Hyperscale Vector index. +* The `CREATE VECTOR INDEX` statement creates a Hyperscale Vector index. This differs from the `CREATE INDEX` statement used to create a Global Secondary Index (GSI). * The `WITH` clause defines several settings specific to Hyperscale Vector indexes: ** It uses the Euclidean distance (`l2`) similarity function when locating centroids. This function has high accuracy, which matters in a dataset with only 153 documents. ** Also, because the dataset is so small, the example sets the `description` to `IVF8,SQ4`. This value has the xref:vector-index:vectors-and-indexes-overview.adoc#IVF[inverted file algorithm] use just 8 centroids. -It also uses 4-bit xref:vector-index:vectors-and-indexes-overview.adoc#sq[scalar quantization]. +It also uses 4-bit xref:vector-index:vectors-and-indexes-overview.adoc#sq[scalar quantization]. These settings limit the fragmentation of the small dataset. In most cases, you'll not use these settings in a production environment. @@ -184,7 +184,7 @@ The `APPROX_VECTOR_DISTANCE` parameters shown in the example are: * `collection-vector-column` is the name of the key containing the vector in the collection. * `search-vector-value` is the vector value to search for in the collection column. It can be an array of floating point numbers or a base64 encoded string. -* `distance-metric` is the distance metric to use when comparing the vectors. +* `distance-metric` is the distance metric to use when comparing the vectors. This value should match the distance metric you used when you created the index. * `centroids-to-probe` is an optional integer value that sets the number of centroids to probe for matching vectors. By default, the vector search only probes a single centroid. @@ -211,7 +211,7 @@ In actual use, your application generates a vector for the query value using the To avoid the complication of calling an embedding model, this example uses embedded vectors in the `rgb_questions.json` file that's included in `color_data_2vectors.zip`. For this example, the contents of this file are loaded into a collection named `vector-sample.color.rgb-questions`. -This collection contains a `question` attribute which is a search prompt for a particular color. +This collection contains a `question` attribute which is a search prompt for a particular color. The `couchbase_search_query.knn.vector` attribute contains the embedded vector for the `question` attribute. The following query lists several attributes from a document in the collection. It truncates the `couchbase_search_query.knn.vector` attribute to just the first 4 dimensions of the vector for readability: @@ -230,7 +230,7 @@ include::vector-index:example$hyperscale-idx-data.json[tag=query-rgb-questions-o To use the embedded vector, you need to include the `couchbase_search_query.knn.vector` attribute in your query's the `SELECT` statement. You can either directly cut & paste the entire array into your query or use a subquery to retrieve it from the `vector-sample.color.rgb-questions` collection. -The following example uses a subquery to get the vector, and also includes the `wanted_similar_color_from_search` attribute in the output which shows you the color that the query should return. +The following example uses a subquery to get the vector, and also includes the `wanted_similar_color_from_search` attribute in the output which shows you the color that the query should return. [source,sqlpp] ---- @@ -244,10 +244,10 @@ The parameters the example passes to this function are: * The `couchbase_search_query.knn[0].vector` is the vector search value to compare to the vectors in the index. This value is the result of the subquery that gets the vector from the `rgb-questions` collection. * `l2` is the distance metric to use. -* `4` is the number of centroids to probe for matching vectors. +* `4` is the number of centroids to probe for matching vectors. This value defaults to `1`. This example sets this value to `4` because the dataset is small. -In a small dataset, it's more likely that relevant vectors are not associated with the same centroid. +In a small dataset, it's more likely that relevant vectors are not associated with the same centroid. This parameter broadens the search beyond a single centroid to find more relevant vectors. If you re-run the query with the default value of `1`, you'll see that the results are less relevant. @@ -258,7 +258,7 @@ The result of running the query is: include::vector-index:example$hyperscale-idx-data.json[tag=query-rgb-color] ---- -The second result, `sky blue`, matches the `wanted_similar_color_from_search` attribute as the most relevant color to the question posed in the `question` attribute. +The second result, `sky blue`, matches the `wanted_similar_color_from_search` attribute as the most relevant color to the question posed in the `question` attribute. All of the results are still relevant to the question. === Getting the Vector Distance From a0c0525d488db924ad0a153057996fbb061a70b5 Mon Sep 17 00:00:00 2001 From: Simon Dew Date: Mon, 13 Oct 2025 10:52:00 +0100 Subject: [PATCH 11/14] Scan for typos --- .../pages/composite-vector-index.adoc | 19 ++- .../vector-index/pages/hyperscale-filter.adoc | 10 +- .../pages/hyperscale-vector-index.adoc | 9 +- .../pages/use-vector-indexes.adoc | 29 ++-- .../pages/vector-index-best-practices.adoc | 8 +- .../pages/vectors-and-indexes-overview.adoc | 131 +++++++++--------- .../partials/index-algorithm-settings.adoc | 40 +++--- 7 files changed, 122 insertions(+), 124 deletions(-) diff --git a/modules/vector-index/pages/composite-vector-index.adoc b/modules/vector-index/pages/composite-vector-index.adoc index 02942e5a4..574143b26 100644 --- a/modules/vector-index/pages/composite-vector-index.adoc +++ b/modules/vector-index/pages/composite-vector-index.adoc @@ -8,6 +8,7 @@ :index-name: Composite Vector index :index-name-plural: Composite Vector indexes +[abstract] {description} The added vector column lets your application perform a query using both the index's scalar, array, and object index entries to pre-filter the dataset before performing a vector similarity search. @@ -81,7 +82,7 @@ WITH {"dimension": , ---- NOTE: The `WITH` clause can contain other parameters that affect how the index processes vectors. -For a full list of these parameters, see xref:n1ql:n1ql-language-reference/createindex.adoc#index-with[WITH Clause] in the {sqlpp} for Query Reference. +For a full list of the parameters that affect a Composite Vector index, see xref:n1ql:n1ql-language-reference/createindex.adoc[] in the {sqlpp} for Query Reference. * `dimensions` is an integer value that sets the number of dimensions in the vector. This value is set by the embedded model you used to embed the vectors. @@ -94,8 +95,6 @@ For the greatest accuracy, use the distance function you plan to use when queryi * `centroids_and_quantization` is a string containing the settings for the quantization and index algorithms. See <<#algo_settings>> in the next section for more information. -For a full list of the parameters that affect a Composite Vector index, see xref:n1ql:n1ql-language-reference/createindex.adoc[]. - include::partial$index-algorithm-settings.adoc[] [#examples] @@ -161,7 +160,7 @@ The `embedding_vector_dot` attribute contains the embedded vectors for the text The data sample shown in <<#examples>> truncated this attribute to several values. The embedded vector contains 1536 dimensions. -The following example creates a Composite Vector index that indexes the embedded vectors in the `embedding_vector_dot` as well as indexing the scalar `color` that contains the color's name and `brightness.` +The following example creates a Composite Vector index that indexes the embedded vectors in the `embedding_vector_dot` as well as indexing the scalar `color` that contains the color's name and `brightness`. [source,sqlpp] ---- @@ -170,7 +169,7 @@ include::vector-index:example$gsi-vector-idx-examples.sqlpp[tag=create-vectors-i This example uses the Dot Product similarity function. This function works better with the embedded text content than the Euclidean function used in the previous example. -It also uses the same algorithms as the previous example--Couchbase {product-name} chooses the number of centroids, and uses SQ quantization with 256 bins. +It also uses the same algorithms as the previous example -- Couchbase {product-name} chooses the number of centroids, and uses SQ quantization with 256 bins. If successful, Couchbase {product-name} responds with: @@ -229,7 +228,7 @@ Which you choose depends on your use case. To use your Composite Vector index, use the `SELECT` statement with an `ORDER BY` clause containing a call to the `APPROX_VECTOR_DISTANCE` function. This function selects the Composite Vector index when you query the vector key that the index covers. -You should also use a `LIMIT` clause to return just the number of vectors you need. +You should use a `LIMIT` clause to return just the number of vectors you need. The query pushes the `LIMIT` clause down into the index scan so that the scan ends after finding the required number of matches. You can also perform vector comparisons using the VECTOR_DISTANCE function. @@ -390,7 +389,7 @@ include::vector-index:example$composite-vector-data.json[tag=sample-doc-question The `couchbase_search_query.knn.vector` attribute contains the embedded vector for the `question` attribute. -This example queries the `embedding_vector_dot` column. +This example queries the `embedding_vector_dot` column. It appears here with most of the 1536 vectors omitted: [source,sqlpp] @@ -398,7 +397,7 @@ It appears here with most of the 1536 vectors omitted: include::vector-index:example$query-vectors-idx-whole.sqlpp[tags=!query-vectors-idx-truncated] ---- -You can click btn:[View] to see and copy the entire query with all the vectors. +Click btn:[icon:github[\] View] to see and copy the entire query with all the vectors. Another option is to import the `rgb_questions.json` file into another collection in the `vector-sample` bucket's `color` scope named `rgb-questions`. Then you can use a subquery to get the vectors for the question and use it in your query of the `rgb` collection's `embedding_vector_dot` attribute: @@ -462,9 +461,9 @@ The truncated version of the query is: include::vector-index:example$query-vectors-idx-filtered-whole.sqlpp[tags=!query-vectors-idx-filtered-truncated] ---- -You can click btn:[View] to see and copy the entire query with all the vectors. +Click btn:[icon:github[\] View] to see and copy the entire query with all the vectors. -The results of this query moves the papaya whip entry to the top. +The results show that this query moves the papaya whip entry to the top. [source,json] ---- diff --git a/modules/vector-index/pages/hyperscale-filter.adoc b/modules/vector-index/pages/hyperscale-filter.adoc index 554bde79d..a2a752e2b 100644 --- a/modules/vector-index/pages/hyperscale-filter.adoc +++ b/modules/vector-index/pages/hyperscale-filter.adoc @@ -3,18 +3,18 @@ :page-ui-name: {ui-name} :page-product-name: {product-name} :description: You can reduce the number of vectors a query that uses a Hyperscale Vector index by adding scalar values. -:stem: +:stem: :page-toclevels: 3 :index-name: Hyperscale Vector index :index-name-plural: Hyperscale Vector indexes [abstract] {description} -A Hyperscale Vector index has a single column that indexes the vector. +A Hyperscale Vector index has a single column that indexes the vector. However, you can include scalar values in the index that you can use to filter the vector search. The index stores these included values along with the vector, but they're not indexed. -In your queries that use a Hypserscale Vector index, you add the included scalar values as predicates in the `WHERE` clause. +In your queries that use a Hyperscale Vector index, you add the included scalar values as predicates in the `WHERE` clause. During the index scan, Couchbase {product-name} uses the included scalar values to filter out vectors that do not meet the filter restriction. If the filter matches the entry, Couchbase {product-name} performs the more expensive vector comparison operation to determine the distance between the vector value and the search vector. @@ -52,6 +52,6 @@ The result of running the query is: include::vector-index:example$hyperscale-idx-data.json[tag=query-rgb-brightness] ---- -Querying with the `brightness` attribute changes makes `sky blue` the top result compared to the results from the example in xref:vector-index:hyperscale-vector-index.adoc#query-with-a-hyperscale-vector-index[Query with a Hyperscale Vector Index]. +Querying with the `brightness` attribute changes makes `sky blue` the top result compared to the results from the example in xref:vector-index:hyperscale-vector-index.adoc#query-with-a-hyperscale-vector-index[Query with a Hyperscale Vector Index]. The restriction can also make the query faster. -On a laptop running a three-node cluster in Docker containers, the query ran in (43{nbsp}ms verses 219{nbsp}ms for the query without the filter. +On a laptop running a three-node cluster in Docker containers, the query ran in 43{nbsp}ms versus 219{nbsp}ms for the query without the filter. diff --git a/modules/vector-index/pages/hyperscale-vector-index.adoc b/modules/vector-index/pages/hyperscale-vector-index.adoc index da6d8b42f..0d3bf9330 100644 --- a/modules/vector-index/pages/hyperscale-vector-index.adoc +++ b/modules/vector-index/pages/hyperscale-vector-index.adoc @@ -13,8 +13,7 @@ They can scale up to a billion documents containing vectors with a large number of dimensions. Because they provide the best performance, consider testing a Hyperscale Vector index for your application before resorting to the other types of indexes. -If you find theirs performance does not meet your needs, then test using a Composite Vector Index or a Search Vector Index. - +If you find their performance does not meet your needs, then test using a Composite Vector Index or a Search Vector Index. == How the Hyperscale Vector Index Works @@ -23,7 +22,7 @@ By relying on disk storage, they have lower memory requirements than other index You add a single vector column to a Hyperscale Vector Index. The vector value can be an array of floating point values. -You can nest the array in another array as long as you use an xref:n1ql:n1ql-language-reference/unnest.adoc[UNNEST clause] to extract the it from the containing array. +You can nest the array in another array as long as you use an xref:n1ql:n1ql-language-reference/unnest.adoc[UNNEST clause] to extract it from the containing array. You can also store the vector in a BASE64 string. == Prerequisites @@ -228,8 +227,8 @@ The output of the query looks like this: include::vector-index:example$hyperscale-idx-data.json[tag=query-rgb-questions-output] ---- -To use the embedded vector, you need to include the `couchbase_search_query.knn.vector` attribute in your query's the `SELECT` statement. -You can either directly cut & paste the entire array into your query or use a subquery to retrieve it from the `vector-sample.color.rgb-questions` collection. +To use the embedded vector, you need to include the `couchbase_search_query.knn.vector` attribute in your query's `SELECT` statement. +You can either directly copy and paste the entire array into your query or use a subquery to retrieve it from the `vector-sample.color.rgb-questions` collection. The following example uses a subquery to get the vector, and also includes the `wanted_similar_color_from_search` attribute in the output which shows you the color that the query should return. [source,sqlpp] diff --git a/modules/vector-index/pages/use-vector-indexes.adoc b/modules/vector-index/pages/use-vector-indexes.adoc index 69d57a827..6a591e1da 100644 --- a/modules/vector-index/pages/use-vector-indexes.adoc +++ b/modules/vector-index/pages/use-vector-indexes.adoc @@ -5,6 +5,7 @@ :description: Use Couchbase {page-product-name}'s vector indexes to find documents based on content similarity or semantic meaning. :page-toclevels: 3 +[abstract] {description} Couchbase {product-name} supports three types of vector indexes. @@ -23,12 +24,12 @@ The three types of Couchbase {product-name} vector indexes are: Hyperscale Vector Indexes:: + -- -* Specifically designed for vector searches -* Perform vector similarity and semantic searches faster than the other types of indexes -* Designed to scale to billions of vectors -* Most of the index resides in a highly optimized format on disk -* High accuracy even for vectors with a large number of dimensions -* Supports concurrent searches and inserts for datasets that are constantly changing +* Specifically designed for vector searches. +* Perform vector similarity and semantic searches faster than the other types of indexes. +* Designed to scale to billions of vectors. +* Most of the index resides in a highly optimized format on disk. +* High accuracy even for vectors with a large number of dimensions. +* Supports concurrent searches and inserts for datasets that are constantly changing. -- + @@ -41,11 +42,11 @@ See xref:vector-index:hyperscale-vector-index.adoc[] for more information. Composite Vector Indexes:: + -- -* Combines a standard xref:learn:services-and-indexes/indexes/indexing-and-query-perf.adoc#secondary-index[Global Secondary index (GSI)] with a single vector column +* Combine a standard xref:learn:services-and-indexes/indexes/indexing-and-query-perf.adoc#secondary-index[Global Secondary index (GSI)] with a single vector column. * Designed for searches using a single vector value along with standard scalar values that filter out large portions of the dataset. The scalar attributes in a query reduce the number of vectors the Couchbase {product-name} has to compare when performing a vector search to find similar vectors. * Consume a moderate amount of memory and can index billions of documents. -* Work well for cases where your queries are highly selective -- returning a small number of results from a large dataset +* Work well for cases where your queries are highly selective -- returning a small number of results from a large dataset. -- + @@ -55,12 +56,12 @@ Use Composite Vector indexes when you want to perform searches of documents usin To learn how to use Composite Vector indexes, see xref:vector-index:composite-vector-index.adoc[]. -Search Vector Index:: +Search Vector Indexes:: + -- -* Combines a Couchbase {product-name} Search index with a single vector column -* Allows hybrid searches that combine vector searches with Full-Text Search (FTS) and geospatial searches. -* Can index millions of documents +* Combine a Couchbase {product-name} Search index with a single vector column. +* Allow hybrid searches that combine vector searches with Full-Text Search (FTS) and geospatial searches. +* Can index millions of documents. -- + @@ -200,11 +201,11 @@ They apply scalar filters before performing a vector similarity search. This is ideal for workflows where you want to exclude large portions of the dataset to reduce the number of vectors that the vector search has to compare. Composite Vector indexes work well for applications such as: -Content Recommendations:: +Content recommendations:: Viewers often want hard constraints on their searches, such as genre, language, or favorites actors or directors. Composite Vector indexes can filter using these constraints before performing a vector similarity search on fields such as the plot description or reviews. -Job Searches:: +Job searches:: Job seekers often want to filter jobs based on location, salary, or job title. Composite Vector indexes can filter using these constraints before performing a vector similarity search on fields such as the job description or required skills. diff --git a/modules/vector-index/pages/vector-index-best-practices.adoc b/modules/vector-index/pages/vector-index-best-practices.adoc index 79ed31ef8..84b93e85a 100644 --- a/modules/vector-index/pages/vector-index-best-practices.adoc +++ b/modules/vector-index/pages/vector-index-best-practices.adoc @@ -38,9 +38,9 @@ However, it does improve the queries per second (QPS) that the index can handle. It also lowers the latency of queries that use the index. Hyperscale Vector indexes tend to perform better with larger centroids (smaller `nList` values). -This performance gain has to be balanced against tyhe possibility of increased disk I/O due to centroids having more vectors associated with them. +This performance gain has to be balanced against the possibility of increased disk I/O due to centroids having more vectors associated with them. -*Recommendation:* The best practice for the `nList` depend on your vector index type: +*Recommendation:* The best practice for the `nList` depends on your vector index type: * If you find that your Composite Vector index query throughput and latency is not meeting your needs, you can try rebuilding the index with a larger `nList` value. @@ -84,8 +84,8 @@ These changes do not affect the QPS or latency of queries that use the index. The number of replicas you set affects the fault tolerance, query throughput, and memory footprint of the index. You set this value using the `num_replicas` key in the `WITH` clause when you create or alter your index. -Tests show that increasing the number of replicas learly increases the QPS and linerly reduces the latency of queries that use the index. -However, it also linearly increased the memory footprint of the index. +Tests show that increasing the number of replicas linearly increases the QPS and linearly reduces the latency of queries that use the index. +However, it also linearly increases the memory footprint of the index. Adding replicas does not affect the time it takes to build the index. *Recommendation:* if you find that your query throughput or latency is not meeting your needs, you can try increasing the number of replicas for the index if you can afford the additional memory usage. diff --git a/modules/vector-index/pages/vectors-and-indexes-overview.adoc b/modules/vector-index/pages/vectors-and-indexes-overview.adoc index 10a346721..47e35666d 100644 --- a/modules/vector-index/pages/vectors-and-indexes-overview.adoc +++ b/modules/vector-index/pages/vectors-and-indexes-overview.adoc @@ -6,19 +6,18 @@ :description: This page is a high-level overview of vectors and how they work in indexes. :page-toclevels: 3 -[abstract] +[abstract] {description} Couchbase {product-name} supports several types of vector indexes. These indexes let you add similarity searches to your applications without relying on external services. -This page provides an overview of vectors. If you're already familiar with vectors, you can skip to the next page, xref:vector-index:use-vector-indexes.adoc[] for information about how to use vector indexes in your application. == About Vectors Vectors are a numerical representation of complex unstructured data such as text, images, or audio. They distill this complex data into an array of floating-point values called dimensions. -The dimensions in vectors capture features of the data in a way that makes them easy to compare mathematically. -Similar data, such as text about the same topic or images of a similar subject, have similar vectors. +The dimensions in vectors capture features of the data in a way that makes them easy to compare mathematically. +Similar data, such as text about the same topic or images of a similar subject, have similar vectors. Similar vectors are close together the multi-dimensional space formed by all of the vectors in the dataset. Vectors for dissimilar data (articles discussing two different topics for example) have vectors that are further apart in this multi-dimensional space. @@ -29,27 +28,27 @@ This similarity goes beyond just finding parts of the data that's identical (for The vectors represent features such as the meaning of text rather than just superficial similarities. When two pieces of data have similar vectors, they have semantically related (having similar meanings or subjects). -For example, suppose you have three text articles. +For example, suppose you have three text articles. The first is about feral cats, the second about domestic cats, and the third is about the UNIX/Linux `cat` command. A text search for the term "cat" could return all three articles at the top of its search results due to their frequent use of the search term. -However, that does not mean that the articles are all semantically related. -By generating vectors for the articles and comparing them, you can determine any semantic relationship between them. +However, that does not mean that the articles are all semantically related. +By generating vectors for the articles and comparing them, you can determine any semantic relationship between them. -Vectors can be similar in a variety of ways. +Vectors can be similar in a variety of ways. They may: * Point in similar directions. -* Be close to each other in the vector space made up of all the dimensions in the vectors. -* Have a similar magnitude (length). +* Be close to each other in the vector space made up of all the dimensions in the vectors. +* Have a similar magnitude (length). image::vector-space-example.svg["Three-dimensional plot showing two vectors that are close by and parallel and a third vector that's away and points in another direction"] -Because the first two articles are about cats, their vectors point in a similar direction and are in a similar location in space. +Because the first two articles are about cats, their vectors point in a similar direction and are in a similar location in space. The third article, despite using the word "cat" as frequently as the two other articles, has a vector that has significant differences from the other two. -NOTE: Some diagrams in this document (and in other discussions available on the web) show vectors in three dimensions. -These diagrams are a simplification. -The vectors used in AI applications have hundreds to thousands of dimensions. +NOTE: Some diagrams in this document (and in other discussions available on the web) show vectors in three dimensions. +These diagrams are a simplification. +The vectors used in AI applications have hundreds to thousands of dimensions. They're easy for computers to handle but are rather difficult for humans to visualize. In addition, the vector dimension values in the diagrams show only 4 decimal places to conserve space. The floating point dimensional values of actual vectors often use 6 or 7 decimal places. @@ -58,7 +57,7 @@ When viewing these diagrams, remember that they only scratch the surface of the === Using Vectors to Find Similar Complex Data Vectors let you search for similarity in data that's hard to search using traditional techniques. -Image data, for example, is just a string of raw binary values. +Image data, for example, is just a string of raw binary values. Matching portions of this data within two image files does not mean they're of the same subject, or even have any resemblance. Also, comparing megabytes of raw image data is computationally expensive. Generating vectors of the images distills features of the image into a smaller, more manageable amount of data. @@ -70,11 +69,11 @@ Comparing the vectors is more manageable and actually results in finding similar Embedding is the process of generating vectors that represent a piece of complex unstructured data. You use an embedding model that's specific to the data's type to generate a vector to represent it. -For example, to generate vectors from text, you can use a use models such as https://en.wikipedia.org/wiki/Word2vec[Word2Vec^], https://en.wikipedia.org/wiki/GloVe[Global Vectors for Word Representation (GloVe)^], or https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2[all-MiniLM-L6-v2^]. -These models take into account the context around each potion of the text when generating a vector. -This context is what captures the semantic meaning of the text and embeds it in the vector's dimensions. +For example, to generate vectors from text, you can use models such as https://en.wikipedia.org/wiki/Word2vec[Word2Vec^], https://en.wikipedia.org/wiki/GloVe[Global Vectors for Word Representation (GloVe)^], or https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2[all-MiniLM-L6-v2^]. +These models take into account the context around each portion of the text when generating a vector. +This context is what captures the semantic meaning of the text and embeds it in the vector's dimensions. -To generate vectors from images, you use different embedding models such as a https://en.wikipedia.org/wiki/Residual_neural_network[Residual Neural Network (ResNet)^]. +To generate vectors from images, you use different embedding models such as a https://en.wikipedia.org/wiki/Residual_neural_network[Residual Neural Network (ResNet)^]. You can embed audio using a model such as OpenL3. image::embedding-vectors.svg["Diagram showing text, image, and audio data being transformed by embedding models into a string of floating point numbers that represent vectors."] @@ -87,7 +86,7 @@ Beyond choosing an embedding model that's appropriate for your data, you general === Embedding Model Compatibility -You can only compare the similarity of vectors generated by the same embedding model. +You can only compare the similarity of vectors generated by the same embedding model. For example, you cannot find the similarity of a vector generated by Word2Vec and one generated by GloVe. You also cannot compare a vector for a piece of text generated by Word2Vec with the vector for an image generated by ResNet. @@ -110,7 +109,7 @@ You can use external embedding models or https://www.couchbase.com/products/ai-s [#vector_similarity] == Vector Similarity Metrics -Once you have embedded vectors in your database, you find documents with a similar meaning by finding similar vectors. +Once you have embedded vectors in your database, you find documents with a similar meaning by finding similar vectors. You use metrics to (also referred to as distance functions) to find similar vectors. When you create an index or perform a vector search, you must specify which metric to use to compare vectors. Each metric works best for certain applications and types of data. @@ -118,17 +117,17 @@ Each metric works best for certain applications and types of data. Couchbase {product-name} supports four metrics: [#euclidean] -=== Euclidean Distance +=== Euclidean Distance Euclidean Distance (also known as L2) calculates the geometric distance between two vectors by finding the distance between the individual dimensions in the vectors. -This method is most sensitive to the distance between the vectors in space, rather than their alignment. -It's also sensitive to the scale of the vectors, where the length of one vector verses the other affects the relationship between the vectors. +This method is most sensitive to the distance between the vectors in space, rather than their alignment. +It's also sensitive to the scale of the vectors, where the length of one vector versus the other affects the relationship between the vectors. Use this method when the actual distance of the vectors and their magnitudes are important. This method is useful if the distance between vectors represents a real-world value. - + image::euclidean-distance-example.svg["Three-dimensional plot showing two vectors with points along each vector joined by dotted lines, indicating the summing of corresponding points."] -NOTE: When you select Euclidean Distance or L2 as the metric for a vector index, Couchbase {product-name} internally uses the <<#euclidean-squared>> metric (explained in the next section) to perform vector comparisons. +NOTE: When you select Euclidean Distance or L2 as the metric for a vector index, Couchbase {product-name} internally uses the <<#euclidean-squared>> metric (explained in the next section) to perform vector comparisons. This approach improves performance because it avoids performing a computationally expensive square root operation. Vector searches using the Euclidean Squared metric return the same relevant vectors and ranking of results as Euclidean Distance. If your query materializes or projects the actual distance between vectors, Couchbase {product-name} calculates the actual Euclidean Distance. @@ -136,7 +135,7 @@ For example, if your query returns the distance between vectors as a column, Cou Euclidean Distance is useful for tasks such as: -* 3D motion capture where you're detecting similar positions or trajectories of joints, objects, where finding real-world values for thresholds is important. +* 3D motion capture where you're detecting similar positions or trajectories of joints, objects, where finding real-world values for thresholds is important. * Geographic or spatial searches where you care about exact (and often real-world) distances. * Other cases where you use the results as filters in calculations that require the actual distance between the vectors. @@ -144,7 +143,7 @@ NOTE: Only Hyperscale Vector and Composite Vector indexes support this metric. Search Vector Indexes do not support it. [#euclidean-squared] -=== Euclidean Squared Distance +=== Euclidean Squared Distance Euclidean Squared Distance (also known as L2 Squared or L2^2^) is similar to Euclidean Distance. @@ -155,17 +154,17 @@ Euclidean Distance Formula:: latexmath:[L2(x, y) = \sqrt{\sum_{i=1}^n (x_i - y_i)^2}] -Euclidean Squared Distance Formula:: +Euclidean Squared Distance Formula:: latexmath:[L2^2(x, y) = \sum_{i=1}^n (x_i - y_i)^2] Because it does not take the square root of the sums, Euclidean Squared Distance is faster to calculate than Euclidean Distance. -However, it does not represent the actual distance between the vectors. -The results of a vector search using L2 Squared as a metric always returns the same rankings that an L2 search returns. +However, it does not represent the actual distance between the vectors. +The results of a vector search using L2 Squared as a metric always returns the same rankings that an L2 search returns. In cases where the dimensions of the vectors represent real-world values, L2 is more intuitive to use because it returns the actual distance between the vectors. -Use this method when you need higher performance than Euclidean Distance. -It's better in cases when comparing literal distances is less important than performance and where having the ranking of search results is sufficient. +Use this method when you need higher performance than Euclidean Distance. +It's better in cases when comparing literal distances is less important than performance and where having the ranking of search results is sufficient. For example: @@ -181,14 +180,14 @@ Search Vector Indexes do not support it. [#dot] === Dot Product -This metric finds related vectors by comparing the magnitude (length) and alignment of the vectors. +This metric finds related vectors by comparing the magnitude (length) and alignment of the vectors. Here, the proximity of the vectors in space is less important than whether they point in a similar direction and have the a similar magnitude. Vectors that point in similar directions (have a low angle between them) and have similar magnitude are strongly associated with each other. This method uses the similarity of the vectors' magnitudes to rank their relation. - + image::dot-product-example.svg["Three-dimensional plot showing two vectors labeled with their magnitudes and the angle between them labeled with the theta symbol."] - -Use this method for tasks where ranking confidence is key. + +Use this method for tasks where ranking confidence is key. The magnitude of the vectors is important when determining the strength of the relationship. For example: @@ -199,9 +198,9 @@ For example: [#cosine] === Cosine Similarity -This metric is similar to the dot product. +This metric is similar to the dot product. However, it normalizes the vectors (making them the same length) during comparison. -Normalization means their magnitudes are not taken into account, just their alignment. +Normalization means their magnitudes are not taken into account, just their alignment. This normalization makes this method better for comparing textual data because it's less sensitive to the length of the data. The magnitude of a vector generated by some text embedding models depend on the length of the source text. Normalizing vector magnitudes emphasizes the semantic meaning when performing comparisons. @@ -219,19 +218,19 @@ Use this method when you're performing searches that rely on semantic meaning, s [#index-algorithms] == Index Formats -Couchbase {product-name} supports two formats that the Vector indexes use when storing their data. +Couchbase {product-name} supports two formats that the Vector indexes use when storing their data. These formats control how the index stores the vectors. Couchbase {product-name} automatically chooses which algorithm to use, based on the type of the index and the size of the dataset. However, you can set parameters when you create the index that affect how the algorithm organizes data. === Flat Index -This algorithm just stores the full vector value in the index without performing any sort of optimization. +This algorithm just stores the full vector value in the index without performing any sort of optimization. Searches using this index use a brute-force method to find similar vectors. Adding new vectors to a flat index is fast and the index does not need training. -It also offers high precision because search compares the full vector values. +It also offers high precision because search compares the full vector values. However, searching a flat index is inefficient. The search must compare every vector in the index to find matches. -You should only use it for small data sets or for testing. +You should only use it for small data sets or for testing. NOTE: Search Vector Indexes use a flat index when indexing datasets with 1000 or fewer vectors. Hyperscale Vector and Composite Vector indexes only support the next algorithm, IVF. @@ -241,7 +240,7 @@ Hyperscale Vector and Composite Vector indexes only support the next algorithm, This algorithm partitions the vector space to limit the number of vectors that a search must compare. An IVF index trains itself during creation using a set of parameters you pass to it: -. It samples vectors in the dataset to locate clusters of similar vectors. +. It samples vectors in the dataset to locate clusters of similar vectors. . For each cluster it finds, it chooses a representative vector (called a centroid). . It creates a data structure called an inverted list that associates all vectors with their closest centroid. @@ -270,7 +269,7 @@ Search Vector Indexes automatically uses IVF when the indexing datasets larger t Quantization simplifies vector data so it consumes less space in memory and on disk. Reducing the amount of data in a vector also makes vector comparison faster. -However, quantization does reduce the accuracy of a vector. +However, quantization does reduce the accuracy of a vector. It's similar to using a lossy image compression format, such as JPEG, on an picture to reduce its data size. It trades off some detail and accuracy for a smaller, more manageable data size. @@ -290,18 +289,18 @@ PQ processes vectors by following these steps: For example, suppose the vectors contain 1024 dimensions. Then PQ may break the vectors into 128 subspaces, each containing 8 dimensions. . It finds groups of centroids in each subspace and adds them to a data structure called a codebook. -The codebook assigns each centroid an index number. +The codebook assigns each centroid an index number. . PQ then reduces the size of the vectors by replacing each subspace's dimensions with the index of the closest centroid in the codebook. image::pq-diagram.svg["Three-dimensional plot showing two vectors that are of equal length, with the angle between them labeled as theta."] PQ could quantize the vector in the previous example from 1024 64-bit floating point values to 128 8 bit or 16 bit integer values. -This data size reduction improves performance because integer operations are less computationally expensive than floating point operations. -It also reduces memory use because each vector requires less data to store (for example, from 64KB to 128 bytes). +This data size reduction improves performance because integer operations are less computationally expensive than floating point operations. +It also reduces memory use because each vector requires less data to store (for example, from 64KB to 128 bytes). A search of the PQ index performs the same steps on the vector you're searching for: - + . It breaks the search vector into subspaces and quantizes it using the codebook. . It then locates closest centroids to the quantized query vector. . It searches the vectors related to the matching centroids for vectors that relate to the search vector. @@ -320,11 +319,11 @@ Use PQ quantization when: Hyoerscale Vector and Composite Vector indexes support PQ quantization. -Search Vector Indexes do not support it. +Search Vector Indexes do not support it. [#sq] === Scalar Quantization (SQ) -Scalar Quantization is a simpler form of quantization than PQ. +Scalar Quantization is a simpler form of quantization than PQ. Instead of breaking the vector into subspaces, it quantizes each dimension of the vector independently. Its quantized vectors are larger, but are a more precise represenation than than those quantized using PQ. @@ -335,8 +334,8 @@ SQ follows these steps to process vectors: . It then partitions each dimension in the vector into equal segments (called bins) over the range of the dimension's values. For example, in the diagram, the first dimension's values range from 0 to 1, while the second dimension's range from 0.1 to 0.9. This means the second dimension's bins will only cover the range from 0.1 to 0.9. -. SQ assigns each bin an index value that's from 4 to 8 bits in length. -. It find a centroid within the range associated with the bin. +. SQ assigns each bin an index value that's from 4 to 8 bits in length. +. It finds a centroid within the range associated with the bin. It saves the centroid as a 16 or 32 bit floating point number. . SQ then quantizes the vectors by replacing their dimensional values with the index value of the bin the dimensional value falls into. @@ -345,12 +344,12 @@ image::sq-diagram.svg["Diagram showing each floating point dimension of a vector This quantization reduces the dimensions from 32 or 64 bit floating point values to 8 or less bit integers, reducing the memory needed for search. It also makes search faster because integer operations are computationally less expensive than floating point operations. -A search on an SQ index reassembles vectors before comparing them. +A search on an SQ index reassembles vectors before comparing them. For each of the vector's dimensions, SQ replaces the integer bin number with the floating point centroid associated with the bin. -This recreation is an approximation of the original vector, because each dimension in the vector is the bin's centroid value instead of the original dimensional value. +This recreation is an approximation of the original vector, because each dimension in the vector is the bin's centroid value instead of the original dimensional value. Because this method quantizes the vectors, it loses precision. -It's less precise than the PQ method (which also quantizes vectors) because it assumes that each dimension can be evenly divided. +It's less precise than the PQ method (which also quantizes vectors) because it assumes that each dimension can be evenly divided. Its accuracy suffers if your data has clusters. It's best suited for data that's more evenly distributed and does not have high correlations. For example, SQ may be a good fit for an IoT sensor dataset where the measured values (such as temperature) are evenly distributed over the range of measurements. @@ -358,7 +357,7 @@ For example, SQ may be a good fit for an IoT sensor dataset where the measured v SQ has a lower training overhead compared to PQ because it does not search for clusters of vectors. Its training just determines the range of values for each dimension. -Use SQ if you do not want the processing overhead of training a PQ index. +Use SQ if you do not want the processing overhead of training a PQ index. You can also use it if you have a smaller dataset (hundreds of thousands to millions of vectors). All three types of vector indexes support SQ quantization. @@ -372,7 +371,7 @@ Instead, they automatically choose whether to use quantization: * Search Vector Indexes do not use quantization for datasets smaller than 10000 vectors. * Search Vector Indexes automatically use 8-bit SQ quantization for datasets with 10000 vectors or larger. -When creating a Hyperscale Vector or Composite Index, you choose which quantization method to use when creating the index. +When creating a Hyperscale Vector or Composite Index, you choose which quantization method to use when creating the index. When deciding, consider the following: * Use SQ when you want higher accuracy than PQ, at the cost of lower compression and therefore more memory use. @@ -388,30 +387,30 @@ See xref:vector-index:vector-index-best-practices.adoc[] for more information ab [#index-training] == The Importance of Index Training -The IVF indexing algorithm and the PQ ans SQ quantizations rely on training the index on existing data. +The IVF indexing algorithm and the PQ and SQ quantizations rely on training the index on existing data. Both PQ and IVF locate centroids that represent clusters of vectors in vector space. The SQ quantization samples vectors to determine the range of values in each dimension. These training techniques reflect the content of dataset at the time you create the index. For this training to give you accurate search results, the data in your dataset must represent the full range of data you expect to search. If the dataset is not representative, the centroids that the PQ quantization or IVF indexing method selects may not accurately represent clusters of data in the dataset. -Having accurate centroids are key to these indexing methods returning accurate results. +Having accurate centroids are key to these indexing methods returning accurate results. For indexes using SQ quantization, the range of values in the dimensions may not accurately represent the data in the dataset. -If one or more dimensional values fall outside the range that SQ assign bins to, they get assigned closest bin which may not properly represent the value. +If one or more dimensional values fall outside the range that SQ assigns bins to, they get assigned to the closest bin, which may not properly represent the value. As your dataset changes, these inaccuracies can build, skewing results. -Over time, may find your vector search's recall (percentage of the nearest neighbors of the search vector returned by a query) decreases. +Over time, may find your vector search's recall (percentage of the nearest neighbors of the search vector returned by a query) decreases. Searches for similar vectors may miss relevant results or may rank less-related vectors higher than more relevant results. These inaccuracies can occur because the data in the dataset changes over time as you add and remove documents. The centroids identified by PQ and IVF may no longer adequately reflect the data in your database. New clusters of vectors may have developed, and old ones may have dissipated. For indexes using SQ quantization, the range of values in the dimensions may have changed. -To resolve these accuracy issues, you can retrain your indexes when you notice poor recall. +To resolve these accuracy issues, you can retrain your indexes when you notice poor recall. You can also choose to retrain indexes periodically or after you make significant changes to your dataset. -To retrain an index, you create a new index with the same (or tweaked) settings as the existing index, but with a different name. +To retrain an index, you create a new index with the same (or tweaked) settings as the existing index, but with a different name. After Couchbase {product-name} builds and trains the new index, you can drop the old index. -You should consider the need to retrain your indexes when choosing which quantization to use. -For example, the PQ quantization requires more resources to train than SQ. +You should consider the need to retrain your indexes when choosing which quantization to use. +For example, the PQ quantization requires more resources to train than SQ. If your dataset evolves rapidly, you may choose to not use PQ because you'll need to perform more frequent retraining. diff --git a/modules/vector-index/partials/index-algorithm-settings.adoc b/modules/vector-index/partials/index-algorithm-settings.adoc index 22771ff84..7bedc5df5 100644 --- a/modules/vector-index/partials/index-algorithm-settings.adoc +++ b/modules/vector-index/partials/index-algorithm-settings.adoc @@ -1,7 +1,7 @@ [#algo_settings] -=== Quantization and Centroid settings +=== Quantization and Centroid Settings -When creating an index that includes a vector field, you choose settings that affect how the index processes vectors. +When creating an index that includes a vector field, you choose settings that affect how the index processes vectors. The parameter named `description` is the primary setting that controls the quantization and the number of centroids Couchbase {product-name} to create the index. Using it, you control how the index subdivides the dataset to improve performance and how it quantizes vectors to reduce memory and processing requirements. @@ -16,15 +16,15 @@ The following sections describe the settings for centroids and quantization. ==== Number of Centroids -{index-name} uses several algorithms to organize its data to improve its performance. +{index-name} uses several algorithms to organize its data to improve its performance. One of these algorithms, xref:vector-index:vectors-and-indexes-overview.adoc#IVF[Inverted File (IVF)], has a setting you can adjust to control how it subdivides the dataset. The other algorithms that {index-name} uses do not have settings you can adjust. -The key setting for IVF is the number of centroids it allocates for the index. +The key setting for IVF is the number of centroids it allocates for the index. This setting controls how large the centroids are. Larger centroids have more vectors associated with them. -You can have Couchbase {product-name} choose a number of centroids for you by not providing a value after the `IVF` in your `description` parameter. +You can have Couchbase {product-name} choose a number of centroids for you by not providing a value after the `IVF` in your `description` parameter. It sets the number of centroids to the number of vectors in the dataset divided by 1000. You can manually set the number of centroids for the index by adding an integer value after the `IVF` in the `description` parameter. @@ -38,24 +38,24 @@ In this case, a vector search has to perform more comparisons, making the search However, having fewer centroids decreases the processing required to train the index. * A greater number of centroids results in a greater processing cost for training. -This increase is due to the training process having to search for more data cluster to identify more centroids. +This increase is due to the training process having to search for more data cluster to identify more centroids. However, it reduces the number of vectors associated with each centroid. This reduction makes search faster by limiting the number of vector comparisons during a search. endif::[] ifeval::['{index-name}' == 'Hyperscale Vector index'] -{index-name-plural} perform better with larger centroids (fewer centroids in the index). +{index-name-plural} perform better with larger centroids (fewer centroids in the index). They use algorithms to skip vectors that are far away from the search vector, making searches faster even with more vectors per centroid. Having fewer centroids can also speed up the index training process because Couchbase {product-name} has to identify fewer data clusters. However, having more vectors per centroid can result in more disk I/O during searches because each centroid has more data associated with it. When choosing the number of centroids for your index, consider the following guidelines: -* If the majority of your working data set fits into the bucket's memory quota, choose a smaller number of centroids for the index. +* If the majority of your working data set fits into the bucket's memory quota, choose a smaller number of centroids for the index. Having more of the working data set in memory reduces disk I/O during searches, making searches faster. Another option is to have the fastest possible storage such as a fast NVME connected to a high-speed PCIe interface. -* If your working data set is much larger than the bucket's memory quota, choose a larger number of centroids for the index. +* If your working data set is much larger than the bucket's memory quota, choose a larger number of centroids for the index. This setting reduces the number of vectors associated with each centroid, which can reduce disk I/O during searches. endif::[] @@ -65,7 +65,7 @@ See xref:vector-index:vector-index-best-practices.adoc#nlist[nList] for more gui ==== Quantization Setting -{index-name} always uses quantization to reduce the size of vectors stored in the index. +{index-name} always uses quantization to reduce the size of vectors stored in the index. You must choose whether the index uses xref:vector-index:vectors-and-indexes-overview.adoc#sq[Scalar Quantization (SQ)] or xref:vector-index:vectors-and-indexes-overview.adoc#pq[Product Quantization (PQ)]. See xref:vector-index:vector-index-best-practices.adoc#quantization[Quantization] for guidance on choosing the quantization method for the index. @@ -75,7 +75,7 @@ Each quantization method has additional settings explained in the following sect ===== SQ Settings -For SQ, you set the number of bits the SQ algorithm uses for the bin index value or the number of bits it uses to store the centroid for each bin. +For SQ, you set the number of bits the SQ algorithm uses for the bin index value, or the number of bits it uses to store the centroid for each bin. The values for SQ that Couchbase {product-name} supports are: [%autowidth] @@ -92,9 +92,9 @@ The values for SQ that Couchbase {product-name} supports are: | SQ uses an 8-bit index value splitting each vector dimension into 256 subspaces. // | `SQfp16` -// | SQ uses an 8-bit index value splitting each vector dimension into 256 subspaces. +// | SQ uses an 8-bit index value splitting each vector dimension into 256 subspaces. // However, it stores the centroid value as a 16-bit floating point number instead of a 32-bit floating point number. -// This cuts the memory used to store the centroids in half. +// This cuts the memory used to store the centroids in half. |=== @@ -105,12 +105,12 @@ See xref:vector-index:vectors-and-indexes-overview.adoc#sq[Scalar Quantization] If you choose to use PQ in your index, you must set two values: -* The number of subquantizers (number of subspaces PQ splits the vector's dimensions into) to use. +* The number of subquantizers (number of subspaces PQ splits the vector's dimensions into) to use. This value must be a divisor of the number of dimensions in the vector. -For example, if your vector has 99 dimensions, you can only use the values 3, 9, 11, 33, and 99 for the subquantizers. -Using any other value returns an error. +For example, if your vector has 99 dimensions, you can only use the values 3, 9, 11, 33, and 99 for the subquantizers. +Using any other value returns an error. -* The number of bits in the centroid's index value. +* The number of bits in the centroid's index value. This value sets the number centroids to find in each subspace. For example, setting this value to 8 has PQ store the index for the centroids in a byte. This results in SQ using 256 centroids per subspace. @@ -129,7 +129,7 @@ The format for the PQ settings is: pq-settings ::= 'PQ' subquantizers 'x' number-of-bits ---- -For example, `PQ32x8` has PQ break the vector's dimensions into 32 subspaces, each of which has 256 centroids. +For example, `PQ32x8` has PQ break the vector's dimensions into 32 subspaces, each of which has 256 centroids. See xref:vector-index:vectors-and-indexes-overview.adoc#pq[Product Quantization] for more information about how PQ works. ==== Algorithm Settings Examples @@ -141,11 +141,11 @@ The following table shows several `description` values along with an explanation | Setting | Effect | `IVF,SQ8` -| Couchbase {product-name} chooses the number of centroids the IVF algorithm uses. +| Couchbase {product-name} chooses the number of centroids the IVF algorithm uses. The index uses Scalar Quantization with an 8-bit index, meaning it breaks each of the vector's dimensions into 256 bins. | `IVF1024,PQ8x8` -| IVF uses 1024 centroids to divide the dataset. +| IVF uses 1024 centroids to divide the dataset. The index uses Product Quantization. PQ breaks the vector space into 8 subspaces, each of which uses 8-bits to represent centroids in the subspace. This settings means each subspace has 256 centroids. From 4b2b854d3d059f346b0f68805a574166e37f8e37 Mon Sep 17 00:00:00 2001 From: Simon Dew Date: Mon, 13 Oct 2025 12:06:07 +0100 Subject: [PATCH 12/14] Capitalization of Hyperscale Vector and Composite Vector --- .../pages/indexing-and-query-perf.adoc | 4 +-- .../altervectorindex.adoc | 12 +++---- .../n1ql-language-reference/build-index.adoc | 2 +- .../n1ql-language-reference/createindex.adoc | 34 +++++++++---------- .../createvectorindex.adoc | 20 +++++------ .../n1ql-language-reference/dropindex.adoc | 4 +-- .../dropvectorindex.adoc | 6 ++-- .../n1ql-language-reference/vectorfun.adoc | 20 +++++------ 8 files changed, 51 insertions(+), 51 deletions(-) diff --git a/modules/indexes/pages/indexing-and-query-perf.adoc b/modules/indexes/pages/indexing-and-query-perf.adoc index a5b7614ab..d6457a305 100644 --- a/modules/indexes/pages/indexing-and-query-perf.adoc +++ b/modules/indexes/pages/indexing-and-query-perf.adoc @@ -588,7 +588,7 @@ For further details and examples, see {covering-indexes}[]. [#hyperscale-vector-index] == Hyperscale Vector Index -Hyperscale vector indexes are a type of <> which contain a single vector field. +Hyperscale Vector indexes are a type of <> which contain a single vector field. They excel at indexing huge datasets that can scale into the billions of documents. They're optimized for pure vector searches, offering the highest performance of any index for your AI applications. @@ -597,7 +597,7 @@ For further details and examples, see {hyperscale-vector-index}[]. [#composite-vector-index] == Composite Vector Index -Composite vector indexes are a type of <> which contain a single vector field and one or more scalar fields. +Composite Vector indexes are a type of <> which contain a single vector field and one or more scalar fields. Your AI applications can use the index's scalar fields to filter the dataset before performing a vector similarity search. For further details and examples, see {composite-vector-index}[]. diff --git a/modules/n1ql/pages/n1ql-language-reference/altervectorindex.adoc b/modules/n1ql/pages/n1ql-language-reference/altervectorindex.adoc index 1386211c0..2aecc2c00 100644 --- a/modules/n1ql/pages/n1ql-language-reference/altervectorindex.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/altervectorindex.adoc @@ -66,7 +66,7 @@ For more information, see xref:n1ql:n1ql-intro/queriesandresults.adoc#query-cont .Create and move an index from one node to another ==== -Create a hyperscale vector index on node 172.19.0.2. +Create a Hyperscale Vector index on node 172.19.0.2. [source,sqlpp] ---- @@ -91,7 +91,7 @@ To check the node where the index is located, see xref:manage:manage-indexes/man .Create and move an index replica from one node to another ==== -Create a hyperscale vector index on node 172.19.0.2 with a replica on node 172.19.0.3, then move its replica from node 172.19.0.*3* to 172.19.0.*4*. +Create a Hyperscale Vector index on node 172.19.0.2 with a replica on node 172.19.0.3, then move its replica from node 172.19.0.*3* to 172.19.0.*4*. [source,sqlpp] ---- @@ -109,7 +109,7 @@ WITH {"action": "move", "nodes": ["172.19.0.2:8091", "172.19.0.4:8091"]}; .Increase the number of replicas ==== -Create a hyperscale vector index on node 172.19.0.2 with a replica on nodes 172.19.0.*3*, then increase the number of replicas to 2 and specify that new replicas may be placed on any available index nodes in the cluster. +Create a Hyperscale Vector index on node 172.19.0.2 with a replica on nodes 172.19.0.*3*, then increase the number of replicas to 2 and specify that new replicas may be placed on any available index nodes in the cluster. [source,sqlpp] ---- @@ -127,7 +127,7 @@ WITH {"action": "replica_count", "num_replica": 2}; .Increase the number of replicas and specify the nodes ==== -Create a hyperscale vector index on node 172.19.0.2 with a replica on node 172.19.0.3, then increase the number of replicas to 2, and specify that replicas may be placed on nodes 172.19.0.*3* and 172.19.0.*4*. +Create a Hyperscale Vector index on node 172.19.0.2 with a replica on node 172.19.0.3, then increase the number of replicas to 2, and specify that replicas may be placed on nodes 172.19.0.*3* and 172.19.0.*4*. [source,sqlpp] ---- @@ -149,7 +149,7 @@ WITH {"action": "replica_count", .Decrease the number of replicas ==== -Create a hyperscale vector index on node 172.19.0.2 with replicas on nodes 172.19.0.*3* and 172.19.0.*4*, then decrease the number of replicas to 1. +Create a Hyperscale Vector index on node 172.19.0.2 with replicas on nodes 172.19.0.*3* and 172.19.0.*4*, then decrease the number of replicas to 1. [source,sqlpp] ---- @@ -167,7 +167,7 @@ WITH {"action": "replica_count", "num_replica": 1}; .Drop a specific replica ==== -Create a hyperscale vector index with 2 replicas, and specify that nodes 172.19.0.2, 172.19.0.3, and 172.19.0.4 should be available for index and replica placement. +Create a Hyperscale Vector index with 2 replicas, and specify that nodes 172.19.0.2, 172.19.0.3, and 172.19.0.4 should be available for index and replica placement. Then delete replica 2. [source,sqlpp] diff --git a/modules/n1ql/pages/n1ql-language-reference/build-index.adoc b/modules/n1ql/pages/n1ql-language-reference/build-index.adoc index 5da90b935..22c6e53f6 100644 --- a/modules/n1ql/pages/n1ql-language-reference/build-index.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/build-index.adoc @@ -52,7 +52,7 @@ If any of the indexes specified by BUILD INDEX have already been built, BUILD IN When building an index which has automatic index replicas, all of the replicas are also built as part of the BUILD INDEX statement, without having to manually specify them. -Hyperscale vector indexes and composite vector indexes require a codebook for the vector field. +Hyperscale Vector indexes and Composite Vector indexes require a codebook for the vector field. The codebook is the result of sampling the dataset and is saved as part of the index metadata. The codebook is created as part of the BUILD INDEX process, and is not incrementally updated. diff --git a/modules/n1ql/pages/n1ql-language-reference/createindex.adoc b/modules/n1ql/pages/n1ql-language-reference/createindex.adoc index d3d7c7b6b..a552e0cf5 100644 --- a/modules/n1ql/pages/n1ql-language-reference/createindex.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/createindex.adoc @@ -4,7 +4,7 @@ :page-toclevels: 2 :imagesdir: ../../assets/images :keywords: secondary, index, placement -:description: The CREATE INDEX statement allows you to create secondary indexes and composite vector indexes. +:description: The CREATE INDEX statement allows you to create secondary indexes and Composite Vector indexes. :authorization-overview: xref:learn:security/authorization-overview.adoc :index-replication: xref:indexes:index-replication.adoc#index-replication @@ -35,8 +35,8 @@ The `CREATE INDEX` statement allows you to create secondary indexes. Secondary indexes contain a filtered or a full set of keys in a given keyspace. Secondary indexes are optional but increase query efficiency on a keyspace. -In Couchbase Server 8.0 and later, the `CREATE INDEX` statement also allows you to create composite vector indexes. -To create hyperscale vector indexes, use the {create-vector-index}[CREATE VECTOR INDEX] statement. +In Couchbase Server 8.0 and later, the `CREATE INDEX` statement also allows you to create Composite Vector indexes. +To create Hyperscale Vector indexes, use the {create-vector-index}[CREATE VECTOR INDEX] statement. //tag::purpose[] == Purpose @@ -57,7 +57,7 @@ In Couchbase Server Enterprise Edition, the recommended way to do this is using In Couchbase Server Community Edition, you need to create multiple identical indexes and place them using the `nodes` option. See <> for more details. -Hyperscale vector indexes and composite vector indexes require a codebook for the vector field. +Hyperscale Vector indexes and Composite Vector indexes require a codebook for the vector field. The codebook is the result of sampling the dataset and is saved as part of the index metadata. The codebook is created as part of the {build-index}[BUILD INDEX] process, and is not incrementally updated. @@ -208,7 +208,7 @@ include::partial$grammar/ddl.ebnf[tag=index-keys-and-attribs] image::n1ql-language-reference/index-keys-and-attribs.png["Syntax diagram: see source code listing", align=left] -Secondary indexes and composite vector indexes can have many keys. +Secondary indexes and Composite Vector indexes can have many keys. Each key may have index attributes, which define the behavior of the index key. [horizontal] @@ -233,7 +233,7 @@ image::n1ql-language-reference/index-key.png["Syntax diagram: see source code li The index key is a {sqlpp} {expression}[expression] referring to a field in the document, or an ARRAY expression on the field. -For a composite vector index, one index key must refer to a vector field in the document. +For a Composite Vector index, one index key must refer to a vector field in the document. The index key that refers to a vector field may be the only index key. If there are multiple index keys, the index key referring to the vector field may be any of the index keys, including the leading index key. @@ -279,7 +279,7 @@ See <>. include-missing:: (Optional) If the leading index key is a non-vector field, it may also include the `INCLUDE MISSING` clause. See <>. -include-vector:: (Optional) In a composite vector index, one index key must include the `VECTOR` keyword. +include-vector:: (Optional) In a Composite Vector index, one index key must include the `VECTOR` keyword. See <>. [[index-order]] @@ -340,7 +340,7 @@ Used to partition the index. Index partitioning helps increase the query performance by dividing and spreading a large index of documents across multiple nodes, horizontally scaling out an index as needed. For details, see {index-partitioning}[Index Partitioning]. -With hyperscale vector indexes and composite vector indexes, training is done for each index node independently, and the codebook is provided to all partitions on that node. +With Hyperscale Vector indexes and Composite Vector indexes, training is done for each index node independently, and the codebook is provided to all partitions on that node. If there are multiple partitions for an index on a node, training is only done once for all partitions. See {index-training}[The Importance of Index Training]. //end::index-partition[] @@ -457,7 +457,7 @@ If the value of this property is not less than the number of index nodes in the //end::index-with[] |=== -Composite vector indexes support the following additional options. +Composite Vector indexes support the following additional options. [options="header", cols="3a,8a,2a"] |=== @@ -548,7 +548,7 @@ See {index-partitioning}[]. [IMPORTANT] .Attention ==== -Do not create (or drop) secondary indexes, composite vector indexes, or hyperscale vector indexes when any Index service node is down, as this may result in duplicate index names. +Do not create (or drop) secondary indexes, Composite Vector indexes, or Hyperscale Vector indexes when any Index service node is down, as this may result in duplicate index names. ==== //end::usage[] @@ -606,7 +606,7 @@ The system automatically load-balances an index scan across the index and all it Adding index replicas enables you to scale scan throughput, in addition to providing high availability. ==== -With hyperscale vector indexes and composite vector indexes, training is done by each replica index independently, and the codebook is stored as part of index metadata. +With Hyperscale Vector indexes and Composite Vector indexes, training is done by each replica index independently, and the codebook is stored as part of index metadata. See {index-training}[The Importance of Index Training]. //end::index-replicas[] @@ -731,12 +731,12 @@ For more examples of indexes where the leading key may be missing, see xref:n1ql ==== [[ex-create-rgb-idx]] -.Create a composite vector index +.Create a Composite Vector index ==== For this example, you must install the vector sample data as described in {prerequisites}[Prerequisites]. The path to the required keyspace is specified by the query, so you do not need to set the query context. -Create a composite vector index that indexes the vector field named `colorvect_l2`, as well as the scalar `color` and `brightness` fields. +Create a Composite Vector index that indexes the vector field named `colorvect_l2`, as well as the scalar `color` and `brightness` fields. [source,sqlpp] ---- @@ -745,12 +745,12 @@ include::vector-index:example$gsi-vector-idx-examples.sqlpp[tag=create-rgb-idx] ==== [[ex-create-vectors-idx]] -.Create a composite vector index using embedded vectors +.Create a Composite Vector index using embedded vectors ==== For this example, you must install the vector sample data as described in {prerequisites}[Prerequisites]. The path to the required keyspace is specified by the query, so you do not need to set the query context. -Create a composite vector index that indexes the vector field named `embedding-vector-dot`, as well as the scalar `color` and `brightness` fields. +Create a Composite Vector index that indexes the vector field named `embedding-vector-dot`, as well as the scalar `color` and `brightness` fields. [source,sqlpp] ---- @@ -759,12 +759,12 @@ include::vector-index:example$gsi-vector-idx-examples.sqlpp[tag=create-vectors-i ==== [[ex-create-colors-idx]] -.Create a composite vector index with a scalar leading key +.Create a Composite Vector index with a scalar leading key ==== For this example, you must install the vector sample data as described in {prerequisites}[Prerequisites]. The path to the required keyspace is specified by the query, so you do not need to set the query context. -Create a composite vector index that indexes the scalar `color` and `brightness` fields, as well as the vector field named `embedding-vector-dot`. +Create a Composite Vector index that indexes the scalar `color` and `brightness` fields, as well as the vector field named `embedding-vector-dot`. [source,sqlpp] ---- diff --git a/modules/n1ql/pages/n1ql-language-reference/createvectorindex.adoc b/modules/n1ql/pages/n1ql-language-reference/createvectorindex.adoc index cb3d2e472..babcd9665 100644 --- a/modules/n1ql/pages/n1ql-language-reference/createvectorindex.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/createvectorindex.adoc @@ -4,7 +4,7 @@ :page-toclevels: 2 :imagesdir: ../../assets/images :keywords: secondary, index, placement -:description: The CREATE VECTOR INDEX statement allows you to create hyperscale vector indexes. +:description: The CREATE VECTOR INDEX statement allows you to create Hyperscale Vector indexes. :authorization-overview: xref:learn:security/authorization-overview.adoc :index-replication: xref:indexes:index-replication.adoc#index-replication @@ -33,9 +33,9 @@ // TEMP include::partial$n1ql-language-reference/horizontal-style.adoc[] -The `CREATE VECTOR INDEX` statement allows you to create hyperscale vector indexes. +The `CREATE VECTOR INDEX` statement allows you to create Hyperscale Vector indexes. -To create secondary indexes or composite vector indexes, use the {create-index}[CREATE INDEX] statement. +To create secondary indexes or Composite Vector indexes, use the {create-index}[CREATE INDEX] statement. include::createindex.adoc[tags=purpose] @@ -94,7 +94,7 @@ include::partial$grammar/ddl.ebnf[tags=index-key-and-attrib] image::n1ql-language-reference/index-key-and-attrib.png["Syntax diagram: see source code listing", align=left] -Hyperscale vector indexes only have one key, which must be a vector field. +Hyperscale Vector indexes only have one key, which must be a vector field. The index key takes one attribute, the VECTOR keyword. [horizontal] @@ -178,11 +178,11 @@ include::createindex.adoc[tags=defer-index-builds-by-default] To try the examples in this section, you must install the vector sample data as described in {prerequisites}[Prerequisites]. [[ex-create-rgb-idx]] -.Create a hyperscale vector index +.Create a Hyperscale Vector index ==== For this example, the path to the required keyspace is specified by the query, so you do not need to set the query context. -Create a hyperscale vector index for the vector column named `embedding-vector-dot`. +Create a Hyperscale Vector index for the vector column named `embedding-vector-dot`. [source,sqlpp] ---- @@ -191,11 +191,11 @@ include::vector-index:example$hyperscale-idx-examples.sqlpp[tag=create-rgb-idx] ==== [[ex-create-idx-brightness]] -.Create a hyperscale vector index with included scalar values +.Create a Hyperscale Vector index with included scalar values ==== For this example, the path to the required keyspace is specified by the query, so you do not need to set the query context. -Create a hyperscale vector index for the vector column named `embedding-vector-dot`, including the scalar `brightness` field. +Create a Hyperscale Vector index for the vector column named `embedding-vector-dot`, including the scalar `brightness` field. [source,sqlpp] ---- @@ -204,11 +204,11 @@ include::vector-index:example$hyperscale-idx-examples.sqlpp[tag=create-idx-brigh ==== [[ex-create-rgb-no-persist]] -.Create a hyperscale vector index with no reranking +.Create a Hyperscale Vector index with no reranking ==== For this example, the path to the required keyspace is specified by the query, so you do not need to set the query context. -Create a hyperscale vector index from the example RGB dataset that does not persist the full vector value. +Create a Hyperscale Vector index from the example RGB dataset that does not persist the full vector value. [source,sqlpp] ---- diff --git a/modules/n1ql/pages/n1ql-language-reference/dropindex.adoc b/modules/n1ql/pages/n1ql-language-reference/dropindex.adoc index 155e69a1d..a0c66381e 100644 --- a/modules/n1ql/pages/n1ql-language-reference/dropindex.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/dropindex.adoc @@ -1,5 +1,5 @@ = DROP INDEX -:description: The DROP INDEX statement allows you to drop a secondary index, a composite vector index, or a hyperscale vector index. +:description: The DROP INDEX statement allows you to drop a secondary index, a Composite Vector index, or a Hyperscale Vector index. :page-topic-type: reference :page-partial: :page-toclevels: 2 @@ -305,7 +305,7 @@ If you drop an index which is scheduled for background creation, a warning messa [IMPORTANT] .Attention ==== -Do not drop (or create) secondary indexes, composite vector indexes, or hyperscale vector indexes when any Index service node is down, as this may result in duplicate index names. +Do not drop (or create) secondary indexes, Composite Vector indexes, or Hyperscale Vector indexes when any Index service node is down, as this may result in duplicate index names. ==== //end::usage[] diff --git a/modules/n1ql/pages/n1ql-language-reference/dropvectorindex.adoc b/modules/n1ql/pages/n1ql-language-reference/dropvectorindex.adoc index 38cf96d94..56673a954 100644 --- a/modules/n1ql/pages/n1ql-language-reference/dropvectorindex.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/dropvectorindex.adoc @@ -1,5 +1,5 @@ = DROP VECTOR INDEX -:description: The DROP VECTOR INDEX statement allows you to drop a hyperscale vector index, a composite vector index, or a secondary index. +:description: The DROP VECTOR INDEX statement allows you to drop a Hyperscale Vector index, a Composite Vector index, or a secondary index. :page-topic-type: reference :page-status: Couchbase Server 8.0 :page-toclevels: 2 @@ -52,11 +52,11 @@ include::dropindex.adoc[tags=usage] To try the examples in this section, you must install the vector sample data as described in {prerequisites}[Prerequisites]. [[ex-1]] -.Drop a hyperscale vector index +.Drop a Hyperscale Vector index ==== For this example, the path to the required keyspace is specified by the query, so you do not need to set the query context. -Drop the hyperscale vector index called `color_desc_hyperscale`, if it exists. +Drop the Hyperscale Vector index called `color_desc_hyperscale`, if it exists. [source,sqlpp] ---- diff --git a/modules/n1ql/pages/n1ql-language-reference/vectorfun.adoc b/modules/n1ql/pages/n1ql-language-reference/vectorfun.adoc index ad7fbac73..7d62cc8ba 100644 --- a/modules/n1ql/pages/n1ql-language-reference/vectorfun.adoc +++ b/modules/n1ql/pages/n1ql-language-reference/vectorfun.adoc @@ -23,16 +23,16 @@ This function has an alias <>. Finds the approximate distance between a provided vector and the content of a specified field that contains vector embeddings. -This function works best with a hyperscale vector index or composite vector index. +This function works best with a Hyperscale Vector index or Composite Vector index. If a query contains this function, and all of the following are true: -* The cluster has a hyperscale vector index or a composite vector index with a vector index key which is the same as the vector field referenced by the function +* The cluster has a Hyperscale Vector index or a Composite Vector index with a vector index key which is the same as the vector field referenced by the function * The vector index key uses a similarity setting which is the same as the distance metric referenced by the function * The vector index key has the same dimension as the vector provided by the function -… then the Query optimizer selects that hyperscale vector index or composite vector index for use with the query containing this function. +… then the Query optimizer selects that Hyperscale Vector index or Composite Vector index for use with the query containing this function. This function is faster, but less precise than <>. You should use this function in your production queries. @@ -45,7 +45,7 @@ The field must contain an array of floating point numbers, or a base64 encoded s queryvec:: An array of floating point numbers, or a base64 encoded string, representing the vector value to search for in the vector field. metric:: A string representing the distance metric to use when comparing the vectors. -To select a hyperscale vector index or composite vector index for the query, the distance metric should match the `similarity` setting that you used when you created the index. +To select a Hyperscale Vector index or Composite Vector index for the query, the distance metric should match the `similarity` setting that you used when you created the index. + [horizontal.compact] COSINE;; xref:vector-index:vectors-and-indexes-overview.adoc#cosine[Cosine Similarity] @@ -56,7 +56,7 @@ L2_SQUARED;; EUCLIDEAN_SQUARED;; xref:vector-index:vectors-and-indexes-overview.adoc#euclidean-squared[Euclidean Squared Distance] nprobes:: [Optional] An integer representing the number of centroids to probe for matching vectors. -If the Query Service selects a hyperscale vector index or composite vector index for the query, this option defaults to the `scan_nprobes` setting that you used when you created the index. +If the Query Service selects a Hyperscale Vector index or Composite Vector index for the query, this option defaults to the `scan_nprobes` setting that you used when you created the index. If an invalid value is provided, defaults to `1`. rerank:: [Optional; can only be used when `nprobes` is specified] @@ -66,7 +66,7 @@ If `true`, the function uses full vectors to reorder the results. The default is `false`. topNScan:: [Optional; can only be used when `nprobes` and `rerank` are specified] -This option only applies if using a hyperscale vector index. +This option only applies if using a Hyperscale Vector index. A positive integer representing the number of records to scan. The default is `0`, meaning the function uses the indexer default. @@ -80,9 +80,9 @@ To try the examples in this section, you must do the following: * Install the `rgb` and `rgb-questions` collections from the supplied vector sample, as described in xref:vector-index:hyperscale-vector-index.adoc#prerequisites[Prerequisites]. -* Create a composite vector index in the `rbg` collection on the field named `colorvect_l2`, as described in xref:n1ql:n1ql-language-reference/createindex.adoc#ex-create-rgb-idx[CREATE INDEX Example 6]. +* Create a Composite Vector index in the `rbg` collection on the field named `colorvect_l2`, as described in xref:n1ql:n1ql-language-reference/createindex.adoc#ex-create-rgb-idx[CREATE INDEX Example 6]. -* Create a hyperscale vector index in the `rbg` collection on the field named `embedding-vector-dot`, as described in xref:n1ql:n1ql-language-reference/createvectorindex.adoc#ex-create-rgb-idx[CREATE VECTOR INDEX Example 1]. +* Create a Hyperscale Vector index in the `rbg` collection on the field named `embedding-vector-dot`, as described in xref:n1ql:n1ql-language-reference/createvectorindex.adoc#ex-create-rgb-idx[CREATE VECTOR INDEX Example 1]. [#approx_vector_distance_ex_simple] .APPROX_VECTOR_DISTANCE() Example 1 @@ -137,7 +137,7 @@ The results show how the distance changes as the similarity decreases. ---- Compare this with the result of <>. -In this case, the results are identical because the query is not using a hyperscale vector index or composite vector index. +In this case, the results are identical because the query is not using a Hyperscale Vector index or Composite Vector index. ==== [#approx_vector_distance_ex_rbg] @@ -544,7 +544,7 @@ This function has an alias <>. Finds the exact distance between a provided vector and the content of a specified field that contains vector embeddings. -This function does not use a hyperscale vector index or composite vector index to perform the comparison. +This function does not use a Hyperscale Vector index or Composite Vector index to perform the comparison. Instead, it performs a brute-force search for similar vectors. This function is slower, but more precise than <>. From 1e62b85f847eb3f1990584942be77b0b6d37462b Mon Sep 17 00:00:00 2001 From: Simon Dew Date: Mon, 13 Oct 2025 12:19:12 +0100 Subject: [PATCH 13/14] Examples on this page --- modules/vector-index/pages/composite-vector-index.adoc | 8 +++----- modules/vector-index/pages/hyperscale-vector-index.adoc | 9 ++++----- 2 files changed, 7 insertions(+), 10 deletions(-) diff --git a/modules/vector-index/pages/composite-vector-index.adoc b/modules/vector-index/pages/composite-vector-index.adoc index 574143b26..667de7633 100644 --- a/modules/vector-index/pages/composite-vector-index.adoc +++ b/modules/vector-index/pages/composite-vector-index.adoc @@ -38,11 +38,6 @@ If your documents contain multiple embedded vectors, you can create multiple ind Embeddings can be an array of floating point numbers or a base64 encoded string. Couchbase {product-name} does not embed vectors itself. You must use an external embedding model to embed vectors into your data and add them to your documents. -+ -[TIP] --- -include::vector-search:partial$download-sample-partial.adoc[] --- * You must know the number of dimensions the vector contains. The embedding model you use to embed the vectors may determine this value for you. @@ -53,6 +48,9 @@ The metrics affect how the index compares vectors. The quantization determines how much memory your index uses and the amount of processing Couchbase {product-name} must perform to train and search them. See xref:vector-index:vectors-and-indexes-overview.adoc#vector_similarity[Vector Similarity Metrics] and xref:vector-index:vectors-and-indexes-overview.adoc#quantization[Quantization] for more information. +=== Examples on this Page + +include::vector-search:partial$download-sample-partial.adoc[] [#create-index] // TODO: For uptake in Couchbase Capella diff --git a/modules/vector-index/pages/hyperscale-vector-index.adoc b/modules/vector-index/pages/hyperscale-vector-index.adoc index 0d3bf9330..aa8f52018 100644 --- a/modules/vector-index/pages/hyperscale-vector-index.adoc +++ b/modules/vector-index/pages/hyperscale-vector-index.adoc @@ -44,11 +44,6 @@ If your documents contain multiple embedded vectors, you can create multiple ind Embeddings can be an array of floating point numbers or a base64 encoded string. Couchbase {product-name} does not embed vectors itself. You must use an external embedding model to embed vectors into your data and add them to your documents. -+ -[TIP] --- -include::vector-search:partial$download-sample-partial.adoc[] --- * The vectors you add to an index must contain the same number of dimensions. Also the values in the vector must be 32-bit floating point numbers. @@ -64,6 +59,10 @@ The metrics affect how the index compares vectors. The quantization determines how much memory your index uses and the amount of processing Couchbase {product-name} must perform to train and search them. See xref:vector-index:vectors-and-indexes-overview.adoc#vector_similarity[Vector Similarity Metrics] and xref:vector-index:vectors-and-indexes-overview.adoc#quantization[Quantization] for more information. +=== Examples on this Page + +include::vector-search:partial$download-sample-partial.adoc[] + [#create-index] // TODO: For uptake in Couchbase Capella //// From 1b675c1ba3ae1b206ea0c49603668199bf8c202c Mon Sep 17 00:00:00 2001 From: Simon Dew Date: Mon, 13 Oct 2025 12:20:05 +0100 Subject: [PATCH 14/14] Remove feature branch preview config --- preview/DOC-12294-GSI-vector-index.yml | 5 ----- 1 file changed, 5 deletions(-) delete mode 100644 preview/DOC-12294-GSI-vector-index.yml diff --git a/preview/DOC-12294-GSI-vector-index.yml b/preview/DOC-12294-GSI-vector-index.yml deleted file mode 100644 index c28497293..000000000 --- a/preview/DOC-12294-GSI-vector-index.yml +++ /dev/null @@ -1,5 +0,0 @@ -sources: - docs-server: - branches: [DOC-12565_vector_search_concepts] -override: - startPage: server:introduction:intro.adoc