Upgrade Elasticsearch to Apache Lucene 10.4

[benwtrent]

This updates to new version of Apache Lucene.

And provides some significant performance improvements for both relevance lexical, logs, metrics, and vector search.

[github-actions]

Vale Linting Results

Summary: 1 warning found

⚠️ Warnings (1)

File	Line	Rule	Message
docs/reference/elasticsearch/mapping-reference/semantic-text-setup-configuration.md	200	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'and the reverse' instead of 'vice versa'.

---

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elasticsearchmachine]

Library update summary (dependency bump)

This PR upgrades Elasticsearch’s embedded Apache Lucene dependency set and refreshes the related build verification metadata; most of the code churn is adapting codecs/vectors/tests from lucene103 to lucene104.

Update Summary

Library	Old	New
Apache Lucene (org.apache.lucene:*)	10.3.2	10.4.0

Version Differences

Library	Change
Apache Lucene (org.apache.lucene:*)	10.3.2 → 10.4.0

Paths touched (high level)

• Version source: build-tools-internal/version.properties
• Dependency verification: gradle/verification-metadata.xml
• Docs/changelog: docs/Versions.asciidoc, docs/changelog/141074.yaml, plus a couple mapping docs updates under docs/reference/...
• Server + codecs/vectors: broad updates under server/src/main/java/org/elasticsearch/index/codec/** and related tests under server/src/test/** (notably new/updated Lucene104* codec references)
• SIMD + GPU/vector plumbing: libs/simdvec/**, libs/gpu-codec/**
• REST/YAML and QA coverage: rest-api-spec/** YAML tests, qa/vector/**, qa/rolling-upgrade/**
• X-Pack: inference semantic text mapping/tests and some frozen engine bits under x-pack/plugin/**

Key observations

• Scope: while this is a single library bump, it’s a wide surface-area update (codecs, vector formats, REST tests, QA), consistent with Lucene’s lucene104 package/format changes.
• Release notes highlights (Lucene 10.4.0) (from https://lucene.apache.org/core/10_4_0/changes/Changes.html):
  • Vectors/codecs: adds/extends Lucene104*ScalarQuantizedVectorsFormat and Lucene104*HnswScalarQuantizedVectorsFormat (incl. asymmetric quantization variants) and related vector scoring/rescoring/bulk-scoring APIs.
  • Performance: multiple optimizations around HNSW indexing/search (bulk scoring, diversity checks, graph building), plus postings/docvalues/query hot-path improvements.
  • APIs: additions like NumericDocValues#longValues, DocIdStream#intoArray, and DirectoryReader.open(...) accepting an ExecutorService.
• Follow-ups to watch: PR discussion already flags a potential GPU impact with int8/HNSW quantization format shifts; worth keeping an eye on GPU-related tests/benchmarks and any downstream consumers.

[iverase]

I read through all the changes and everything looks good.

LGTM!

[mayya-sharipova]

I've checked a part about hnsw threshold (I am not familiar about confidence).

So that part LGTM!

[elasticsearchmachine]

Hi @benwtrent, I've created a changelog YAML for you.