Upgrade Elasticsearch to Apache Lucene 10.4

benchmarks/src/main/java/org/elasticsearch/benchmark/index/codec/tsdb/TSDBDocValuesMergeBenchmark.java

@@ -26,7 +26,7 @@
 import org.apache.lucene.util.BytesRef;
 import org.elasticsearch.benchmark.Utils;
 import org.elasticsearch.cluster.metadata.DataStream;
-import org.elasticsearch.index.codec.Elasticsearch92Lucene103Codec;
+import org.elasticsearch.index.codec.Elasticsearch93Lucene104Codec;
 import org.elasticsearch.index.codec.tsdb.BinaryDVCompressionMode;
 import org.elasticsearch.index.codec.tsdb.es819.ES819Version3TSDBDocValuesFormat;
 import org.openjdk.jmh.annotations.Benchmark;
@@ -266,7 +266,7 @@ private static IndexWriterConfig createIndexWriterConfig(boolean optimizedMergeE
             true,
             NUMERIC_LARGE_BLOCK_SHIFT
         );
-        config.setCodec(new Elasticsearch92Lucene103Codec() {
+        config.setCodec(new Elasticsearch93Lucene104Codec() {
             @Override
             public DocValuesFormat getDocValuesFormatForField(String field) {
                 return docValuesFormat;

benchmarks/src/main/java/org/elasticsearch/benchmark/vector/scorer/BenchmarkUtils.java

@@ -9,11 +9,11 @@
 
 package org.elasticsearch.benchmark.vector.scorer;
 
+import org.apache.lucene.backward_codecs.lucene99.OffHeapQuantizedByteVectorValues;
 import org.apache.lucene.codecs.hnsw.FlatVectorScorerUtil;
 import org.apache.lucene.codecs.lucene95.OffHeapByteVectorValues;
 import org.apache.lucene.codecs.lucene95.OffHeapFloatVectorValues;
 import org.apache.lucene.codecs.lucene99.Lucene99ScalarQuantizedVectorScorer;
-import org.apache.lucene.codecs.lucene99.OffHeapQuantizedByteVectorValues;
 import org.apache.lucene.index.ByteVectorValues;
 import org.apache.lucene.index.FloatVectorValues;
 import org.apache.lucene.index.VectorSimilarityFunction;

build-tools-internal/version.properties

@@ -1,5 +1,5 @@
 elasticsearch     = 9.4.0
-lucene            = 10.3.2
+lucene            = 10.4.0
 
 bundled_jdk_vendor = openjdk
 bundled_jdk = 25.0.2+10@b1e0dfa218384cb9959bdcb897162d4e

docs/Versions.asciidoc

@@ -1,8 +1,8 @@
 
 include::{docs-root}/shared/versions/stack/{source_branch}.asciidoc[]
 
-:lucene_version:        10.3.2
-:lucene_version_path:   10_3_2
+:lucene_version:        10.4.0
+:lucene_version_path:   10_4_0
 :jdk:                   11.0.2
 :jdk_major:             11
 :build_type:            tar

docs/changelog/141074.yaml

@@ -0,0 +1,5 @@
+pr: 141074
+summary: Add flat_index_threshold parameter for hnsw dense_vector fields
+area: Vector Search
+type: enhancement
+issues: []

docs/changelog/141882.yaml

@@ -0,0 +1,5 @@
+area: Search
+issues: []
+pr: 141882
+summary: Upgrade Elasticsearch to Apache Lucene 10.4
+type: upgrade

docs/reference/elasticsearch/mapping-reference/dense-vector.md

@@ -344,7 +344,7 @@ This configuration is appropriate when full source fidelity is required, such as
 ## Automatically quantize vectors for kNN search [dense-vector-quantization]
 
 The `dense_vector` field type supports quantization to reduce the memory footprint required when [searching](docs-content://solutions/search/vector/knn.md#approximate-knn) `float` vectors. The supported vector quantization strategies for `dense_vector` kNN indexing are:
-- [`int8`](#dense-vector-quantization-int8) 
+- [`int8`](#dense-vector-quantization-int8)
 - [`int4`](#dense-vector-quantization-int4)
 - [`bbq`](#dense-vector-quantization-bbq), available as:
   - [`bbq_hnsw`](/reference/elasticsearch/mapping-reference/bbq.md#bbq-hnsw)
@@ -573,9 +573,6 @@ $$$dense-vector-index-options$$$
 `ef_construction`
 :   (Optional, integer) The number of candidates to track while assembling the list of nearest neighbors for each new node. Defaults to `100`. Only applicable to `hnsw`, `int8_hnsw`, `int4_hnsw` and `bbq_hnsw` index types.
 
-`confidence_interval`
-:   (Optional, float) Only applicable to `int8_hnsw`, `int4_hnsw`, `int8_flat`, and `int4_flat` index types. The confidence interval to use when quantizing the vectors. Can be any value between and including `0.90` and `1.0` or exactly `0`. When the value is `0`, this indicates that dynamic quantiles should be calculated for optimized quantization. When between `0.90` and `1.0`, this value restricts the values used when calculating the quantization thresholds. For example, a value of `0.95` will only use the middle 95% of the values when calculating the quantization thresholds (e.g. the highest and lowest 2.5% of values will be ignored). Defaults to `1/(dims + 1)` for `int8` quantized vectors and `0` for `int4` for dynamic quantile calculation.
-
 `default_visit_percentage` {applies_to}`stack: ga 9.2`
 :   (Optional, integer) Only applicable to `bbq_disk`. Must be between 0 and 100.  0 will default to using `num_candidates` for calculating the percent visited. Increasing `default_visit_percentage` tends to improve the accuracy of the final results. Defaults to ~1% per shard for every 1 million vectors.
 
@@ -734,7 +731,7 @@ flat --> int8_flat --> int4_flat --> hnsw --> int8_hnsw --> int4_hnsw
 :::
 ::::
 
-For updating all HNSW types (`hnsw`, `int8_hnsw`, `int4_hnsw`, `bbq_hnsw`) the number of connections `m` must either stay the same or increase. For the scalar quantized formats  `int8_flat`, `int4_flat`, `int8_hnsw` and `int4_hnsw` the `confidence_interval` must always be consistent (once defined, it cannot change).
+For updating all HNSW types (`hnsw`, `int8_hnsw`, `int4_hnsw`, `bbq_hnsw`) the number of connections `m` must either stay the same or increase.
 
 Updating `type` in `index_options` will fail in all other scenarios.
 

docs/reference/elasticsearch/mapping-reference/semantic-text-setup-configuration.md

@@ -13,14 +13,14 @@
 
 ## Configure {{infer}} endpoints [configure-inference-endpoints]
 
-You can configure {{infer}} endpoints for `semantic_text` fields in the following ways: 
+You can configure {{infer}} endpoints for `semantic_text` fields in the following ways:
 
 - [Use ELSER on EIS](#using-elser-on-eis)
 - [Use default and preconfigured endpoints](#default-and-preconfigured-endpoints)
 - [Use a custom {{infer}} endpoint](#using-custom-endpoint)
 
 :::{note}
-If you use a [custom {{infer}} endpoint](#using-custom-endpoint) through your ML node and not through Elastic {{infer-cap}} Service (EIS), the recommended method is to [use dedicated endpoints for ingestion and search](#dedicated-endpoints-for-ingestion-and-search). 
+If you use a [custom {{infer}} endpoint](#using-custom-endpoint) through your ML node and not through Elastic {{infer-cap}} Service (EIS), the recommended method is to [use dedicated endpoints for ingestion and search](#dedicated-endpoints-for-ingestion-and-search).
 
 {applies_to}`stack: ga 9.1.0` If you use EIS, you don't have to set up dedicated endpoints.
 :::
@@ -195,9 +195,9 @@
 
 ### Use dedicated endpoints for ingestion and search [dedicated-endpoints-for-ingestion-and-search]
 
-If you use a [custom {{infer}} endpoint](#using-custom-endpoint) through your ML node and not through Elastic {{infer-cap}} Service, the recommended way to use `semantic_text` is by having dedicated {{infer}} endpoints for ingestion and search. 
+If you use a [custom {{infer}} endpoint](#using-custom-endpoint) through your ML node and not through Elastic {{infer-cap}} Service, the recommended way to use `semantic_text` is by having dedicated {{infer}} endpoints for ingestion and search.
 
 This ensures that search speed remains unaffected by ingestion workloads, and vice versa. After creating dedicated {{infer}} endpoints for both, you can reference them using the `inference_id`
 and `search_inference_id` parameters when setting up the index mapping for an index that uses the `semantic_text` field.
 
 ```console
@@ -232,7 +232,7 @@
   "mappings": {
     "properties": {
       "content": {
-        "type": "semantic_text", 
+        "type": "semantic_text",
         "index_options": {
           "sparse_vector": {
             "prune": true, <1>
@@ -271,8 +271,7 @@
           "dense_vector": {
             "type": "int8_hnsw", <1>
             "m": 15, <2>
-            "ef_construction": 90, <3>
-            "confidence_interval": 0.95 <4>
+            "ef_construction": 90 <3>
           }
         }
       }
@@ -283,5 +282,4 @@
 1. (Optional) Selects the `int8_hnsw` vector quantization strategy. Learn about [default quantization types](/reference/elasticsearch/mapping-reference/dense-vector.md#default-quantization-types).
 2. (Optional) Sets `m` to 15 to control how many neighbors each node connects to in the HNSW graph. Default is `16`.
 3. (Optional) Sets `ef_construction` to 90 to control how many candidate neighbors are considered during graph construction. Default is `100`.
-4. (Optional) Sets `confidence_interval` to 0.95 to limit the value range used during quantization and balance accuracy with memory efficiency.