Squashed commit of the following:

commit e6b8a64
Author: Lorenzo Dematte <[email protected]>
Date:   Tue Jun 10 12:02:16 2025 +0200

    PR comments

commit ad0902e
Author: Moritz Mack <[email protected]>
Date:   Fri Jun 6 13:33:37 2025 +0200

    Update ReproduceInfoPrinter to correctly print a reproduction for lucene / BC upgrade tests. Relates to ES-12005

commit 78b4168
Author: Alexander Spies <[email protected]>
Date:   Fri Jun 6 11:37:53 2025 +0200

    ESQL: Throw ISE instead of IAE for illegal block in page (elastic#128960)

    IAE gets reported as a 400 status code, but that's inappropriate when
    inconsistent pages are always bugs, and should be reported with a 500.
    Throw ISE instead.

commit 29e68bd
Author: Aurélien FOUCRET <[email protected]>
Date:   Fri Jun 6 11:24:23 2025 +0200

    [ES|QL] Fix test releases for telemetry. (elastic#128971)

commit 1a76bc2
Author: Bogdan Pintea <[email protected]>
Date:   Fri Jun 6 11:01:49 2025 +0200

    ESQL: Workaround for RLike handling of empty lang pattern (elastic#128895)

    Lucene's `org.apache.lucene.util.automaton.Operations#getSingleton` fails with an Automaton for a `REGEXP_EMPTY` `RegExp`. This adds a workaround for that, to check the type of automaton before calling into that failing method.

    Closes elastic#128813

commit e24fd32
Author: Aurélien FOUCRET <[email protected]>
Date:   Fri Jun 6 10:25:28 2025 +0200

    [ES|QL] Enable the completion command as a tech preview feature (elastic#128948)

commit 3f03775
Author: Niels Bauman <[email protected]>
Date:   Fri Jun 6 09:00:24 2025 +0200

    Remove non-test usages of `Metadata.Builder#putCustom` (elastic#128801)

    This removes all non-test usages of
    ```
    Metadata.Builder.putCustom(String type, ProjectCustom custom)
    ```
    And replaces it with appropriate calls to the equivalent method on
    `ProjectMetadata.Builder`.

    In most cases this _does not_ make the code project aware, but does
    reduce the number of deprecated methods in use.

commit 330d127
Author: Niels Bauman <[email protected]>
Date:   Fri Jun 6 08:07:59 2025 +0200

    Make utility methods in `IndexLifecycleTransition` project-aware (elastic#128930)

    Modifies the methods to work with a project scope rather than a cluster
    scope.
    This is part of an iterative process to make ILM project-aware.

commit 1b5720d
Author: Aurélien FOUCRET <[email protected]>
Date:   Fri Jun 6 07:50:52 2025 +0200

    [ES|QL] Fix test releases for LookupJoinTypesIT. (elastic#128985)

commit 40cf2d3
Author: Tim Vernum <[email protected]>
Date:   Fri Jun 6 13:09:31 2025 +1000

    Add "extension" attribute validation to IdP SPs (elastic#128805)

    This extends the change from elastic#128176 to validate the "custom
    attributes" on a per Service Provider basis.

    Each Service Provider (whether registered or wildcard based) has a
    field "attributes.extensions" which is a list of attribute names that
    may be provided by the caller of "/_idp/saml/init".

    Service Providers that have not be configured with extension
    attributes will reject any custom attributes in SAML init.

    This necessitates a new field in the service provider index (but only
    if the new `extensions` attribute is set).
    The template has been updated, but there is no data migration because
    the `saml-service-provider` index does not exist in any of the
    environments into which we wish to deploy this change.

commit 496fb2d
Author: Jordan Powers <[email protected]>
Date:   Thu Jun 5 19:50:09 2025 -0700

    Skip UTF8 to UTF16 conversion during document indexing (elastic#126492)

    When parsing documents, we receive the document as UTF-8 encoded data which
    we then parse and convert the fields to java-native UTF-16 encoded Strings.
    We then convert these strings back to UTF-8 for storage in lucene.

    This patch skips the redundant conversion, instead passing lucene a
    direct reference to the received UTF-8 bytes when possible.

commit c34f8b6
Author: Tim Vernum <[email protected]>
Date:   Fri Jun 6 12:03:25 2025 +1000

    Improve cache invalidation in IdP SP cache (elastic#128890)

    The Identity Provider's Service Provider cache had two issues:

    1. It checked for identity based on sequence numbers, but didn't
       include the `seq_no_primary_term` parameter on searches, which
       means the sequence would always by `-2`
    2. It didn't track whether the index was deleted, which means it
       could be caching values from an old version of the index

    This commit fixes both of these issues.

    In practice neither issue was a problem because there are no
    deployments that use index-based service providers, however the 2nd
    issue did cause some challenges for testing.

commit 923f029
Author: Nhat Nguyen <[email protected]>
Date:   Thu Jun 5 18:09:58 2025 -0700

    Fix block loader with missing ignored source (elastic#129006)

    We miss appending null when ignored_source is not available. Our
    randomized tests already cover this case, but we do not check it when
    loading fields.

    I labelled this non-issue for an unreleased bug.

    Closes elastic#128959
    Relates elastic#119546

commit 0f8178a
Author: Bogdan Pintea <[email protected]>
Date:   Fri Jun 6 02:02:24 2025 +0200

    ESQL: Forward port 8.19 RegexMatch serialization change version (elastic#128979)

    Fwd port ESQL_REGEX_MATCH_WITH_CASE_INSENSITIVITY_8_19.

    Related: elastic#128919.

commit ee716f1
Author: Simon Chase <[email protected]>
Date:   Thu Jun 5 15:20:08 2025 -0700

    transport: edit TransportConnectionListener for close exceptions (elastic#129015)

    The TransportConnectionListener interface has previously included the
    Transport.Connection being closed and unregistered in its onNodeDisconnected
    callback. This is not in use, and can be removed as it is also available in the
    onConnectionClosed callback. It is being replaced with a Nullable exception that
    caused the close. This is being used in pending work (ES-11448) to differentiate
    network issues from node restarts.

    Closes ES-12007

commit aceaf23
Merge: f18f4ee 159c57f
Author: elasticsearchmachine <[email protected]>
Date:   Thu Jun 5 19:27:54 2025 +0000

    Merge patch/serverless-fix into main

commit 159c57f
Author: Rene Groeschke <[email protected]>
Date:   Tue Jun 3 11:56:27 2025 +0200

    Prepare serverless patch including elastic#128784 elastic#128740 (elastic#128807)

    * Change default for vector.rescoring.directio to false (elastic#128784)

    On serverless (and potentially elsewhere), direct IO is not available, which can cause BBQ shards to fail to read with org.apache.lucene.CorruptIndexException when this setting is true.

    * Optimize sparse vector stats collection (elastic#128740)

    This change improves the performance of sparse vector statistics gathering by using the document count of terms directly, rather than relying on the field name field to compute stats.
    By avoiding per-term disk/network reads and instead leveraging statistics already loaded into leaf readers at index opening, we expect to significantly reduce overhead.

    Relates to elastic#128583

    ---------

    Co-authored-by: Dave Pifke <[email protected]>
    Co-authored-by: Jim Ferenczi <[email protected]>

Author: ldematte

benchmarks/src/main/java/org/elasticsearch/benchmark/xcontent/OptimizedTextBenchmark.java

@@ -0,0 +1,108 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the "Elastic License
+ * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
+ * Public License v 1"; you may not use this file except in compliance with, at
+ * your election, the "Elastic License 2.0", the "GNU Affero General Public
+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+package org.elasticsearch.benchmark.xcontent;
+
+import org.elasticsearch.benchmark.index.mapper.MapperServiceFactory;
+import org.elasticsearch.common.UUIDs;
+import org.elasticsearch.common.bytes.BytesReference;
+import org.elasticsearch.common.logging.LogConfigurator;
+import org.elasticsearch.index.mapper.MapperService;
+import org.elasticsearch.index.mapper.SourceToParse;
+import org.elasticsearch.xcontent.XContentBuilder;
+import org.elasticsearch.xcontent.XContentFactory;
+import org.elasticsearch.xcontent.XContentType;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.infra.Blackhole;
+
+import java.io.IOException;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Benchmark to measure indexing performance of keyword fields. Used to measure performance impact of skipping
+ * UTF-8 to UTF-16 conversion during document parsing.
+ */
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MILLISECONDS)
+@State(Scope.Benchmark)
+@Fork(1)
+@Threads(1)
+@Warmup(iterations = 1)
+@Measurement(iterations = 5)
+public class OptimizedTextBenchmark {
+    static {
+        // For Elasticsearch900Lucene101Codec:
+        LogConfigurator.loadLog4jPlugins();
+        LogConfigurator.configureESLogging();
+        LogConfigurator.setNodeName("test");
+    }
+
+    /**
+     * Total number of documents to index.
+     */
+    @Param("1048576")
+    private int nDocs;
+
+    private MapperService mapperService;
+    private SourceToParse[] sources;
+
+    private String randomValue(int length) {
+        final String CHARS = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
+        Random random = new Random();
+        StringBuilder builder = new StringBuilder(length);
+        for (int i = 0; i < length; i++) {
+            builder.append(CHARS.charAt(random.nextInt(CHARS.length())));
+        }
+        return builder.toString();
+    }
+
+    @Setup(Level.Trial)
+    public void setup() throws IOException {
+        mapperService = MapperServiceFactory.create("""
+            {
+                "_doc": {
+                    "dynamic": false,
+                    "properties": {
+                        "field": {
+                            "type": "keyword"
+                        }
+                    }
+                }
+            }
+            """);
+
+        sources = new SourceToParse[nDocs];
+        for (int i = 0; i < nDocs; i++) {
+            XContentBuilder b = XContentFactory.jsonBuilder();
+            b.startObject().field("field", randomValue(8)).endObject();
+            sources[i] = new SourceToParse(UUIDs.randomBase64UUID(), BytesReference.bytes(b), XContentType.JSON);
+        }
+    }
+
+    @Benchmark
+    public void indexDocuments(final Blackhole bh) {
+        final var mapper = mapperService.documentMapper();
+        for (int i = 0; i < nDocs; i++) {
+            bh.consume(mapper.parse(sources[i]));
+        }
+    }
+}

docs/changelog/128805.yaml

@@ -0,0 +1,5 @@
+pr: 128805
+summary: Add "extension" attribute validation to IdP SPs
+area: IdentityProvider
+type: enhancement
+issues: []

docs/changelog/128890.yaml

@@ -0,0 +1,5 @@
+pr: 128890
+summary: Improve cache invalidation in IdP SP cache
+area: IdentityProvider
+type: bug
+issues: []

docs/changelog/128895.yaml

@@ -0,0 +1,6 @@
+pr: 128895
+summary: Workaround for RLike handling of empty lang pattern
+area: ES|QL
+type: bug
+issues:
+ - 128813

docs/changelog/128948.yaml

@@ -0,0 +1,6 @@
+pr: 128948
+summary: ES|QL - Add COMPLETION command as a tech preview feature
+area: ES|QL
+type: feature
+issues:
+- 124405

docs/changelog/128960.yaml

@@ -0,0 +1,5 @@
+pr: 128960
+summary: Throw ISE instead of IAE for illegal block in page
+area: ES|QL
+type: bug
+issues: []

libs/x-content/impl/src/main/java/org/elasticsearch/xcontent/provider/json/ESJsonFactory.java

@@ -0,0 +1,74 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the "Elastic License
+ * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
+ * Public License v 1"; you may not use this file except in compliance with, at
+ * your election, the "Elastic License 2.0", the "GNU Affero General Public
+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+package org.elasticsearch.xcontent.provider.json;
+
+import com.fasterxml.jackson.core.JsonEncoding;
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonFactoryBuilder;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.io.IOContext;
+import com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper;
+import com.fasterxml.jackson.core.sym.ByteQuadsCanonicalizer;
+
+import java.io.IOException;
+
+public class ESJsonFactory extends JsonFactory {
+    ESJsonFactory(JsonFactoryBuilder b) {
+        super(b);
+    }
+
+    @Override
+    protected JsonParser _createParser(byte[] data, int offset, int len, IOContext ctxt) throws IOException {
+        if (len > 0
+            && Feature.CHARSET_DETECTION.enabledIn(_factoryFeatures)
+            && Feature.CANONICALIZE_FIELD_NAMES.enabledIn(_factoryFeatures)) {
+            var bootstrap = new ByteSourceJsonBootstrapper(ctxt, data, offset, len);
+            var encoding = bootstrap.detectEncoding();
+            if (encoding == JsonEncoding.UTF8) {
+                boolean invalidBom = false;
+                int ptr = offset;
+                // Skip over the BOM if present
+                if ((data[ptr] & 0xFF) == 0xEF) {
+                    if (len < 3) {
+                        invalidBom = true;
+                    } else if ((data[ptr + 1] & 0xFF) != 0xBB) {
+                        invalidBom = true;
+                    } else if ((data[ptr + 2] & 0xFF) != 0xBF) {
+                        invalidBom = true;
+                    } else {
+                        ptr += 3;
+                    }
+                }
+                if (invalidBom == false) {
+                    ByteQuadsCanonicalizer can = _byteSymbolCanonicalizer.makeChild(_factoryFeatures);
+                    return new ESUTF8StreamJsonParser(
+                        ctxt,
+                        _parserFeatures,
+                        null,
+                        _objectCodec,
+                        can,
+                        data,
+                        ptr,
+                        offset + len,
+                        ptr - offset,
+                        false
+                    );
+                }
+            }
+        }
+        return new ByteSourceJsonBootstrapper(ctxt, data, offset, len).constructParser(
+            _parserFeatures,
+            _objectCodec,
+            _byteSymbolCanonicalizer,
+            _rootCharSymbols,
+            _factoryFeatures
+        );
+    }
+}

libs/x-content/impl/src/main/java/org/elasticsearch/xcontent/provider/json/ESJsonFactoryBuilder.java

@@ -0,0 +1,20 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the "Elastic License
+ * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
+ * Public License v 1"; you may not use this file except in compliance with, at
+ * your election, the "Elastic License 2.0", the "GNU Affero General Public
+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+package org.elasticsearch.xcontent.provider.json;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonFactoryBuilder;
+
+public class ESJsonFactoryBuilder extends JsonFactoryBuilder {
+    @Override
+    public JsonFactory build() {
+        return new ESJsonFactory(this);
+    }
+}

libs/x-content/impl/src/main/java/org/elasticsearch/xcontent/provider/json/ESUTF8StreamJsonParser.java

@@ -0,0 +1,118 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the "Elastic License
+ * 2.0", the "GNU Affero General Public License v3.0 only", and the "Server Side
+ * Public License v 1"; you may not use this file except in compliance with, at
+ * your election, the "Elastic License 2.0", the "GNU Affero General Public
+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+package org.elasticsearch.xcontent.provider.json;
+
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.core.ObjectCodec;
+import com.fasterxml.jackson.core.SerializableString;
+import com.fasterxml.jackson.core.io.IOContext;
+import com.fasterxml.jackson.core.json.UTF8StreamJsonParser;
+import com.fasterxml.jackson.core.sym.ByteQuadsCanonicalizer;
+
+import org.elasticsearch.xcontent.Text;
+import org.elasticsearch.xcontent.XContentString;
+
+import java.io.IOException;
+import java.io.InputStream;
+
+public class ESUTF8StreamJsonParser extends UTF8StreamJsonParser {
+    protected int stringEnd = -1;
+
+    public ESUTF8StreamJsonParser(
+        IOContext ctxt,
+        int features,
+        InputStream in,
+        ObjectCodec codec,
+        ByteQuadsCanonicalizer sym,
+        byte[] inputBuffer,
+        int start,
+        int end,
+        int bytesPreProcessed,
+        boolean bufferRecyclable
+    ) {
+        super(ctxt, features, in, codec, sym, inputBuffer, start, end, bytesPreProcessed, bufferRecyclable);
+    }
+
+    /**
+     * Method that will try to get underlying UTF-8 encoded bytes of the current string token.
+     * This is only a best-effort attempt; if there is some reason the bytes cannot be retrieved, this method will return null.
+     * Currently, this is only implemented for ascii-only strings that do not contain escaped characters.
+     */
+    public Text getValueAsText() throws IOException {
+        if (_currToken == JsonToken.VALUE_STRING && _tokenIncomplete) {
+            if (stringEnd > 0) {
+                final int len = stringEnd - 1 - _inputPtr;
+                // For now, we can use `len` for `stringLength` because we only support ascii-encoded unescaped strings,
+                // which means each character uses exactly 1 byte.
+                return new Text(new XContentString.UTF8Bytes(_inputBuffer, _inputPtr, len), len);
+            }
+            return _finishAndReturnText();
+        }
+        return null;
+    }
+
+    protected Text _finishAndReturnText() throws IOException {
+        int ptr = _inputPtr;
+        if (ptr >= _inputEnd) {
+            _loadMoreGuaranteed();
+            ptr = _inputPtr;
+        }
+
+        int startPtr = ptr;
+        final int[] codes = INPUT_CODES_UTF8;
+        final int max = _inputEnd;
+        final byte[] inputBuffer = _inputBuffer;
+        while (ptr < max) {
+            int c = inputBuffer[ptr] & 0xFF;
+            if (codes[c] != 0) {
+                if (c == INT_QUOTE) {
+                    stringEnd = ptr + 1;
+                    final int len = ptr - startPtr;
+                    // For now, we can use `len` for `stringLength` because we only support ascii-encoded unescaped strings,
+                    // which means each character uses exactly 1 byte.
+                    return new Text(new XContentString.UTF8Bytes(inputBuffer, startPtr, len), len);
+                }
+                return null;
+            }
+            ++ptr;
+        }
+        return null;
+    }
+
+    @Override
+    public JsonToken nextToken() throws IOException {
+        if (_currToken == JsonToken.VALUE_STRING && _tokenIncomplete && stringEnd > 0) {
+            _inputPtr = stringEnd;
+            _tokenIncomplete = false;
+        }
+        stringEnd = -1;
+        return super.nextToken();
+    }
+
+    @Override
+    public boolean nextFieldName(SerializableString str) throws IOException {
+        if (_currToken == JsonToken.VALUE_STRING && _tokenIncomplete && stringEnd > 0) {
+            _inputPtr = stringEnd;
+            _tokenIncomplete = false;
+        }
+        stringEnd = -1;
+        return super.nextFieldName(str);
+    }
+
+    @Override
+    public String nextFieldName() throws IOException {
+        if (_currToken == JsonToken.VALUE_STRING && _tokenIncomplete && stringEnd > 0) {
+            _inputPtr = stringEnd;
+            _tokenIncomplete = false;
+        }
+        stringEnd = -1;
+        return super.nextFieldName();
+    }
+}

libs/x-content/impl/src/main/java/org/elasticsearch/xcontent/provider/json/JsonXContentImpl.java

@@ -11,7 +11,6 @@
 
 import com.fasterxml.jackson.core.JsonEncoding;
 import com.fasterxml.jackson.core.JsonFactory;
-import com.fasterxml.jackson.core.JsonFactoryBuilder;
 import com.fasterxml.jackson.core.JsonGenerator;
 import com.fasterxml.jackson.core.JsonParser;
 
@@ -47,7 +46,7 @@ public static final XContent jsonXContent() {
     }
 
     static {
-        jsonFactory = XContentImplUtils.configure(new JsonFactoryBuilder());
+        jsonFactory = XContentImplUtils.configure(new ESJsonFactoryBuilder());
         jsonFactory.configure(JsonGenerator.Feature.QUOTE_FIELD_NAMES, true);
         jsonFactory.configure(JsonParser.Feature.ALLOW_COMMENTS, true);
         jsonFactory.configure(JsonFactory.Feature.FAIL_ON_SYMBOL_HASH_OVERFLOW, false); // this trips on many mappings now...