ESQL: Enable new data types with created version by nik9000 · Pull Request #136327 · elastic/elasticsearch

nik9000 · 2025-10-09T20:45:52Z

Enables dense_vector and aggregate_metric_double if all nodes in all of the target clusters support it.

Relates to #135193

Enables `dense_vector` and `aggregate_metric_double` if all nodes in all of the target clusters support it. Relates to elastic#135193

github-actions · 2025-10-09T20:47:28Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

nik9000 · 2025-10-21T22:56:06Z

...qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/AllSupportedFieldsTestCase.java

-        if (Build.current().isSnapshot()) {
-            // We only test behavior in release builds. Snapshot builds will have data types enabled that are still under construction.
-            return List.of();
-        }


Moved this.

elasticsearchmachine · 2025-10-21T22:58:14Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-10-22T12:57:44Z

Hi @nik9000, I've created a changelog YAML for you.

alex-spies

Nice, thanks @nik9000 ! I have only minor comments, and a suggestion for a follow-up (more tests!) that we can tackle separately.

alex-spies · 2025-10-22T13:21:05Z

...qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/AllSupportedFieldsTestCase.java

    }

+    @Before
+    public void onlyOnSnapshot() throws IOException {


The name seems to suggest the opposite of the actual method body?

Suggested change

public void onlyOnSnapshot() throws IOException {

public void onlyOnProductionBuilds() throws IOException {

alex-spies · 2025-10-22T15:19:04Z

...qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/AllSupportedFieldsTestCase.java

+     * Tests fetching {@code dense_vector} if possible. Uses the {@code dense_vector_agg_metric_double_if_fns}
+     * work around if required.
+     */
    public final void testFetchDenseVector() throws IOException {


We could use a similar test for aggregate metric double, no?

alex-spies · 2025-10-22T15:22:58Z

...qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/AllSupportedFieldsTestCase.java

    }

+    @Before
+    public void onlyOnSnapshot() throws IOException {


Interesting! I thought that running tests in release mode will use release builds of all bwc versions as well. Is that not the case?

Can we add a comment to explain this? Also, should we double check our release tests whether they always should use release builds of bwc versions?

I'll add a comment, yeah. If you run:

./gradlew -p x-pack/plugin/esql/qa/server/multi-clusters/ clean v9.1.5#newToOld -Dbuild.snapshot=false -Dlicense.key="x-pack/license-tools/src/test/resources/public.key" -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.class='*All*'

while 9.1.5 is the tip of the 9.1 branch then we'll compile 9.1 and run it as a snapshot.

I'll ask around if this is intended.

I'll ask around if this is intended.

Well, not "intended", but not something we're likely to fix right now.

...qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/AllSupportedFieldsTestCase.java

alex-spies · 2025-10-22T17:40:29Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/IndexResolver.java

+     *                               production but tests need more control
+     * @param useAggregateMetricDoubleWhenNotSupported does the query itself force us to use {@code aggregate_metric_double} fields
+     *                                                 even if the remotes don't report that they support the type? This exists because
+     *                                                 some remotes <strong>do</strong> support {@code aggregate_metric_double} without


We can be more precise here: the support is precisely on 9.2.0; we can link to the data type's supported transport version so that hackers can use git blame to confirm that.

Being precise allows us to refactor and clean this in 9.5, when we don't have to expect a mixed cluster state with 9.2.0 (or only during a rolling upgrade)

alex-spies · 2025-10-22T17:40:38Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/IndexResolver.java

+     *                                                 reporting that they do. And, for a while, we used the query itself to opt into
+     *                                                 reading these fields.
+     * @param useDenseVectorWhenNotSupported does the query itself force us to use {@code dense_vector} fields even if the remotes don't
+     *                                       report that they support the type? This exists because some remotes <strong>do</strong>


alex-spies · 2025-10-22T17:44:04Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/IndexResolver.java

+        boolean useDenseVectorWhenNotSupported
+    ) {
+        /**
+         * The {@link #minTransportVersion}, but if any remote didn't tell us the version we assume


Can we say that, if a remote (cluster) didn't tell us the min version, it's at most on 9.1, because if all nodes where on 9.2.0+, the min transport version would be part of the field caps response?

I could add that to the comment, but this is lowest version it could possibly be, not the highest. At least, that's what I think it should be.

What you say is correct! I meant that it'd just be helpful to understand that the min transport version was added to the FC response in 9.2.0. Which means that if we didn't receive it, there is at least one cluster with at least one node that is on a previous version.

Cool! I've pushed some updated comments.

alex-spies · 2025-10-22T17:44:37Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/IndexResolver.java

+        boolean typeSupported = type.supportedVersion()
+            .supports(fieldsInfo.effectiveMinTransportVersion(), fieldsInfo.currentBuildIsSnapshot)
+            || switch (type) {
+                case AGGREGATE_METRIC_DOUBLE -> fieldsInfo.useAggregateMetricDoubleWhenNotSupported;
+                case DENSE_VECTOR -> fieldsInfo.useDenseVectorWhenNotSupported;
+                default -> false;
+            };
+        if (false == typeSupported) {
+            type = UNSUPPORTED;
+        }


Yeah, this is the main change and looks right!

alex-spies · 2025-10-22T17:46:45Z

...qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/AllSupportedFieldsTestCase.java

This change effectively adds support to agg metric double and dense vector fields in lookup indices and enrich policies.

Let's add a test case for using all kinds of types as lookup/enrich fields, right? (Not match fields, we have separate tests for that.) In a follow-up that is, this should be fine to merge without.

nik9000 · 2025-10-23T15:12:20Z

Follow ups we need:

~~Let's add a test case for using all kinds of types as lookup/enrich fields, right? (Not match fields, we have separate tests for that.)~~ ESQL: Fix enrich and lookup join resolution based on min transport version #137431
We could use a similar test for aggregate metric double, no? ESQL: Handle release of 9.2 in test #137070
Integrate with Include min transport version into FC response #136887
~~Create a pragma to enable under construction types (sounds kind of difficult, but probably worth it)~~ Update from @alex-spies : not sure a pragma is the right way to go, after all, but ESQL: Add createdVersion for types #137432 should enable us to test the behavior in SNAPSHOT, too, not just in release builds. Updating AllSupportedFieldsTestCase to also run in SNAPSHOT is part of ESQL: version-aware planning follow-ups and testing improvements #137534

…able

elasticsearchmachine · 2025-10-23T20:24:27Z

💔 Backport failed

Status	Branch	Result
❌	9.2	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 136327

I'd made a mistake in elastic#136327 when writing the test for fetching fields that the release of 9.2.0 revealed. This fixes it and adds one more test that we needed from elastic#136327.

Enables `dense_vector` and `aggregate_metric_double` if all nodes in all of the target clusters support it. Relates to elastic#135193

nik9000 · 2025-10-23T22:15:22Z

Backport: #137073

Enables `dense_vector` and `aggregate_metric_double` if all nodes in all of the target clusters support it. Relates to #135193

I'd made a mistake in #136327 when writing the test for fetching fields that the release of 9.2.0 revealed. This fixes it and adds one more test that we needed from #136327.

I'd made a mistake in elastic#136327 when writing the test for fetching fields that the release of 9.2.0 revealed. This fixes it and adds one more test that we needed from elastic#136327.

I'd made a mistake in elastic#136327 when writing the test for fetching fields that the release of 9.2.0 revealed. This fixes it and adds one more test that we needed from elastic#136327. (cherry picked from commit cd3839c) # Conflicts: # server/src/main/resources/transport/upper_bounds/9.3.csv # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlResolveFieldsResponse.java

I'd made a mistake in #136327 when writing the test for fetching fields that the release of 9.2.0 revealed. This fixes it and adds one more test that we needed from #136327. (cherry picked from commit cd3839c) # Conflicts: # server/src/main/resources/transport/upper_bounds/9.3.csv # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlResolveFieldsResponse.java Co-authored-by: Nik Everett <nik9000@gmail.com>

Enables `dense_vector` and `aggregate_metric_double` if all nodes in all of the target clusters support it. Relates to elastic#135193

I'd made a mistake in elastic#136327 when writing the test for fetching fields that the release of 9.2.0 revealed. This fixes it and adds one more test that we needed from elastic#136327.

BASE=440faee03ee04b4ab8dbf2e41afcc3cc6f0a26c0 HEAD=b86a0963774cbc1ccdf27e4fd19e5e312d736e33 Branch=main

* Basic support for DATE_RANGE field type in ESQL * [CI] Update transport version definitions * Update docs/changelog/133309.yaml * Suppres two more tests * Two more tests fixes * Two more tests * Some more infra for new tests * Some more fixes for tests * And some more tests * And some more tests * Fix capanility check in CSV tests and some more fixes * Block lookup join * Block lookup join * Relaxed the previous transport version check, which was in fact always false - replace it with something that can work for now. * Fixed a typo * Style fixes * More fixes: block lookup join. Delete date_range from unsupported yaml test, it's not unsupported anymore and there is a normal yaml test for it. * Supress RestEsqlIT because complex queries like this are not supported yet * Removed unused field * Bring back deleted yaml tests * Refactor DateRange Block type (and element type) to LongRange * Review fixes * small fixes * Added identity option to TO_DATE_RANGE, plus more tests * [CI] Update transport version definitions * Fix irregular spaces (#137014) * Fix irregular spaces * Update analysis-keyword-repeat-tokenfilter.md * Update search-suggesters.md * Update search-profile.md * Test utility for `POST _features/_reset` (#137133) We call `POST _features/_reset` in various places in the tests, only sometimes asserting that the response is `200 OK`. This commit extracts a utility that makes it harder to miss this assertion. It also adds the `?error_trace` parameter so that on failure the details are visible in the response, and `?master_timeout=-1` to avoid spurious timeouts if CI is running a little slow. * Revert "Test utility for `POST _features/_reset` (#137133)" This reverts commit df67e27. * Mute org.elasticsearch.xpack.esql.qa.multi_node.EsqlSpecIT test {csv-spec:lookup-join-expression.LookupJoinExpressionWithTerm} #137157 * Mute org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT test {csv-spec:lookup-join-expression.LookupJoinExpressionWithTerm} #137157 * Mute org.elasticsearch.xpack.esql.qa.single_node.GenerativeForkIT test {csv-spec:lookup-join-expression.LookupJoinExpressionWithTerm} #137160 * Mute org.elasticsearch.cluster.routing.allocation.decider.WriteLoadConstraintDeciderIT testCanRemainNotPreferredIsIgnoredWhenAllOtherNodesReturnNotPreferred #137162 * Try bulk load sorted ordinals across ranges (#137076) This change implements the TODO for loading sorted ordinals in the TSDB codec. With ordinal range encoding, we can bulk-append the entire range for the next ordinal, rather than reading and appending one document at a time. * Fix serialization assymetry (writeOptional vs plain read), fixed VerifierTett * Relax (again) the check in blockLoader in RangeFieldMapper, since it seem to fail in practice * [CI] Auto commit changes from spotless * AnalyzerTest fix * Fix APM tracer to respect recording status of spans (#137015) Co-authored-by: Jack Shirazi <jack.shirazi@elastic.co> * Mute org.elasticsearch.xpack.esql.plan.logical.CommandLicenseTests testLicenseCheck #137168 * Catch-and-rethrow TooComplexToDeterminizeException within ESQL (#137024) --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * [ES|QL] Non-Correlated Subquery in FROM command (#135744) * non-correlated subquery in from command * [docs] Update category field to use keyword type in full text tutorial (#137169) closes elastic/docs-content#3329 * [Inference API] Rename E5 model/inference endpoint in EIS to JinaAI counterpart (#137028) * ESQL: Add a bit of javadoc to preanalysis methods (#137099) Explain where minimum transport version is determined for a given query. * Adds ml-cpp release notes (#137178) * Remove mutes pointing to closed ESQL flaky test issues (#137174) * rest-api-spec: Add missing options to enum types (#136640) * rest-api-spec: Add missing options to enum types * Allow 0-9 in enum options * Add applies_to frontmatter to mapping-reference docs (#137184) * [docs] Update ESQL command H1 titles for SEO (#137188) * style fix * Some fixes: change the TopNEncoder to the correct one, fix the unsupported test to match the expected results (good ones and not unsupported). * Transport version * [CI] Auto commit changes from spotless * Remove temporary print * Add missing fix in boolean tranport flag in IndexResolver * Serialize "unsupported" DataType when communicating to old clusters * Block date_range on yaml tests for bwc * Fixed missing import after resolving conflicts with main * Fix mistake on merging main with new element types * Revert fix to DataType serialization We should handle this at a higher level, since this code is widely used in serialization tests which fail with this change. * Try a temporary fix for enrichment * [CI] Auto commit changes from spotless * Some small fixes * cat API: added endpoint for Circuit Breakers (#136890) Added CAT Action to display Circuit Breakers stats for all nodes. The API supports pattern matching as a path parameter and the standard query parameters of CAT actions. This change includes spec and yamlRestTest. Addresses #132688 * ESQL: Fix release tests (#137298) New field type isn't serializable outside of snapshot. * Reject invalid `reverse_nested` aggs (#137047) * Mute org.elasticsearch.xpack.esql.plan.physical.ShowExecSerializationTests testConcurrentSerialization #137338 * Mute org.elasticsearch.readiness.ReadinessClusterIT testReadinessDuringRestartsNormalOrder #136955 * [docs] Update changelog summary for semantic_text ELSER on EIS default (#137339) * Fix method visibility in public-callers-finder (#137200) Public-callers-finder did not correctly report if methods are externally visible due to the following issues: - checking if interfaces are exported (see findAccessibility) must be done using the exports of the interface’s module, not the exports of the current class - super classes must be treated the same way as interfaces, currently these are ignored - currently all public methods of a (potentially private, non-exported) class implementing a public, exported interface are considered accessible / visible regardless if part of the interface or not This fixes visibility to be consistent with the JdkApiExtractor tool by implementing both using the same common logic in `AccessibleJdkMethods.loadAccessibleMethods`. Note: this currently includes #137193, which will be merged independently. Relates to ES-13117 * [ML] Adding bulk create functionality to ModelRegistry (#136569) * Adding bulk storage of multiple models * Adding tests * Adding log for duplicate ids * [CI] Auto commit changes from spotless * Removing unused code * Removing constructor * Adding more tests * Adding in logic to delete models when a failure occurs * revert rename changes * formatting * Starting on feedback * Improving tests * Moving most tests to ModelRegistryIT * [CI] Auto commit changes from spotless * Fixing test * Removing duplicate tests * Handling empty list and duplicates * Fixing empty delete --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Remove deprecated constant for default elser inference id (#137329) * Cleaned up the old constant * Cleaned up test * Update docs/changelog/137329.yaml * Update docs/changelog/137329.yaml * Update docs/changelog/137329.yaml * Delete docs/changelog/137329.yaml * ESQL: Handle release of 9.2 in test (#137070) I'd made a mistake in #136327 when writing the test for fetching fields that the release of 9.2.0 revealed. This fixes it and adds one more test that we needed from #136327. * Fixed inconsistency in the isSyntheticSourceEnabled flag (#137297) * Fixed inconsistency in the isSyntheticSourceEnabled flag * Update docs/changelog/137297.yaml * [ML] Disable CPS for Dataframes (#136716) Cross-Project Search and Cross-Cluster Search is indefinitely disabled for Dataframe Analytics. The error message will now display if the syntax would have otherwise resolved to the respective feature. * [Docs] Improve semantic_text updates documentation organization (#137340) * Move script update restrictions section for consistency * Update docs/reference/elasticsearch/mapping-reference/semantic-text.md Co-authored-by: Liam Thompson <leemthompo@gmail.com> * Update docs/reference/elasticsearch/mapping-reference/semantic-text.md Co-authored-by: Liam Thompson <leemthompo@gmail.com> * Update docs/reference/elasticsearch/mapping-reference/semantic-text.md Co-authored-by: Liam Thompson <leemthompo@gmail.com> --------- Co-authored-by: Liam Thompson <leemthompo@gmail.com> * Remove unused field from IndexModule (#137342) * Improving random sampling performance by lazily calling getSamplingConfiguration() (#137223) * [ML] Disable CrossProject for Datafeeds (#136897) Initially, Datafeeds will not support cross-project source indices. We will verify that the IndicesOptions is not trying to resolve a cross-project index expression and throw a cross-project specific error message. * [ES-12998] Invoking gradle continue flag on periodic runs to allow for more information on test failures (#136900) * Add --continue flag to invoke maven when a task failure has been hit so that we can see the outcome of more testing over time. * ES|QL: Improve value loading for match_only_text mapping (#137026) * [Transform] Remove extra reset calls (#137346) ESRestTestCase already calls the `_reset` API, we do not need to do it twice. Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Add chunk_rescorer usage to output of explain and profile for text_similarity_rank_retriever (#137249) * Add chunk_rescorer usage to output of explain for text_similarity_rank_retriever * Update docs/changelog/137249.yaml * Update toString * Add support for profile * Update 137249.yaml * That's what you get for resolving conflicts in the UI, fixed compile failure * [CI] Auto commit changes from spotless * Fix test compilation --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Remove `first` and `last` functions from documentation (#137341) * Remove `first` and `last` functions from documentation * Apply suggestions from code review Co-authored-by: Liam Thompson <leemthompo@gmail.com> * update --------- Co-authored-by: Liam Thompson <leemthompo@gmail.com> * ESQL: Work around concurrent serialization bug (#137350) The bug still exists in the underlying code, but it isn't released so I'm working around it by doing an eager call to `hashCode`. We'll fix the real bug real soon. Closes #137338 * [ES|QL] Add CHUNK function (#134320) * Add new function to chunk strings * Refactor CHUNK function to support multiple values * Default to returning all chunks * [CI] Auto commit changes from spotless * Handle warnings * Loosen export restrictions to try to get compile error working * Remove inference dependencies * Fix compilation errors * Remove more inference deps * Fix compile errors from merge * Fix existing tests * Exclude from CSV tests * Add more tests * Cleanup * [CI] Auto commit changes from spotless * Cleanup * Update docs/changelog/134320.yaml * PR feedback * Remove null field constraint * [CI] Auto commit changes from spotless * PR feedback: Refactor to use an options map * Cleanup * Regenerate docs * Add test on a concatenated field * Add multivalued field test * Don't hardcode strings * [CI] Auto commit changes from spotless * PR feedback --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Add default sort for message.template_id field in logsdb indices (#136571) This PR adds the index setting `index.logsdb.default_sort_on_message_template`. When set, LogsDB indices will add `message.template_id` to the default sort fields (assuming the `message` field is present and of type `pattern_text`). * Transport version * [CI] Update transport version definitions * Specifically allow to_date_range and to_string * Tansport version * Update docs/changelog/133309.yaml * Fixed forbidden toString * Revert enrichment hack * transport version * Fixed accidentally deleted files * [CI] Update transport version definitions * Generated files for to_string / is_null tests * Transport version * [CI] Update transport version definitions * Transport version * Small merge accident fix * Added missing case for BlockTestUtils * Small fix for AllSupportedFieldTest * Transport version * [CI] Update transport version definitions * [CI] Update transport version definitions * Transport version * merge fixes * change date_range to under_construction * Some error tests fixes * More error tests fixes * More merge fixes * [CI] Auto commit changes from spotless * Smale checstyle fix * error tests fixes * More fixes * One more error fix * Fix the date range capability * Update docs/changelog/133309.yaml * bring back accidentally deleted files * changelog type * AllSupportedFieldTest fix for no snapshot build * Transport version * Fix for CSV test to avoid order false errors * Another fix after changing to under construction * merge fixes * [CI] Auto commit changes from spotless * [CI] Update transport version definitions * [CI] Update transport version definitions * [CI] Update transport version definitions * [CI] Update transport version definitions * Fix some date->long text replacements we missed * Removed duplicated test data `date_ranges.csv` * Update transport version after merging main * Nicer string literal * Update transport version after merging main * TO_DATERANGE should be snapshot-only * Fix failing tests after moving TO_DATERANGE to snapshot-only * Remove `useDateRangeWhenNotSupported` since that seems only relevant to partial support across multiple versions * Disabled TO_DATE_RANGE on release builds of testInlineCast * [CI] Auto commit changes from spotless * Update docs/changelog/133309.yaml * transport version * merge fixes * another transport version * Another try in fixing the multi cluster tests * [CI] Update transport version definitions * Fixed compilation error * Fix thread leak in ContextIndexSearcherTests.testMaxClause Add missing executor cleanup in finally block to prevent thread leaks when the test creates a ThreadPoolExecutor. * Add queryContainsIndices utility method to EsqlTestUtils This new method checks if a given ESQL query contains any specified indices, facilitating special handling for queries using indices loaded into multiple clusters. The method is integrated into MultiClusterSpecIT to streamline the logic for determining when to use remote indices, improving code clarity and maintainability. * Fix for the last review fix * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Craig Taverner <craig@amanzi.com> Co-authored-by: Fabrizio Ferri-Benedetti <algernon@fastmail.com> Co-authored-by: David Turner <david.turner@elastic.co> Co-authored-by: elasticsearchmachine <58790826+elasticsearchmachine@users.noreply.github.com> Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co> Co-authored-by: Moritz Mack <mmack@apache.org> Co-authored-by: Jack Shirazi <jack.shirazi@elastic.co> Co-authored-by: Matt <matthew.alp@elastic.co> Co-authored-by: Fang Xing <155562079+fang-xing-esql@users.noreply.github.com> Co-authored-by: Liam Thompson <leemthompo@gmail.com> Co-authored-by: Tim Grein <tim.grein@elastic.co> Co-authored-by: Alexander Spies <alexander.spies@elastic.co> Co-authored-by: kosabogi <105062005+kosabogi@users.noreply.github.com> Co-authored-by: Ievgen Degtiarenko <ievgen.degtiarenko@elastic.co> Co-authored-by: Quentin Pradet <quentin.pradet@elastic.co> Co-authored-by: Matteo Mazzola <matteo.mazzola@elastic.co> Co-authored-by: Nik Everett <nik9000@gmail.com> Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com> Co-authored-by: Mridula <mridula.s@elastic.co> Co-authored-by: Dmitry Kubikov <Kubik42@users.noreply.github.com> Co-authored-by: Pat Whelan <pat.whelan@elastic.co> Co-authored-by: Alan Woodward <romseygeek@apache.org> Co-authored-by: Keith Massey <keith.massey@elastic.co> Co-authored-by: Neil Bhavsar <neil.bhavsar@elastic.co> Co-authored-by: Ioana Tagirta <ioanatia@users.noreply.github.com> Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co> Co-authored-by: Kostas Krikellas <131142368+kkrik-es@users.noreply.github.com> Co-authored-by: Jordan Powers <jordan.powers@elastic.co>

* Basic support for DATE_RANGE field type in ESQL * [CI] Update transport version definitions * Update docs/changelog/133309.yaml * Suppres two more tests * Two more tests fixes * Two more tests * Some more infra for new tests * Some more fixes for tests * And some more tests * And some more tests * Fix capanility check in CSV tests and some more fixes * Block lookup join * Block lookup join * Relaxed the previous transport version check, which was in fact always false - replace it with something that can work for now. * Fixed a typo * Style fixes * More fixes: block lookup join. Delete date_range from unsupported yaml test, it's not unsupported anymore and there is a normal yaml test for it. * Supress RestEsqlIT because complex queries like this are not supported yet * Removed unused field * Bring back deleted yaml tests * Refactor DateRange Block type (and element type) to LongRange * Review fixes * small fixes * Added identity option to TO_DATE_RANGE, plus more tests * [CI] Update transport version definitions * Fix irregular spaces (elastic#137014) * Fix irregular spaces * Update analysis-keyword-repeat-tokenfilter.md * Update search-suggesters.md * Update search-profile.md * Test utility for `POST _features/_reset` (elastic#137133) We call `POST _features/_reset` in various places in the tests, only sometimes asserting that the response is `200 OK`. This commit extracts a utility that makes it harder to miss this assertion. It also adds the `?error_trace` parameter so that on failure the details are visible in the response, and `?master_timeout=-1` to avoid spurious timeouts if CI is running a little slow. * Revert "Test utility for `POST _features/_reset` (elastic#137133)" This reverts commit df67e27. * Mute org.elasticsearch.xpack.esql.qa.multi_node.EsqlSpecIT test {csv-spec:lookup-join-expression.LookupJoinExpressionWithTerm} elastic#137157 * Mute org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT test {csv-spec:lookup-join-expression.LookupJoinExpressionWithTerm} elastic#137157 * Mute org.elasticsearch.xpack.esql.qa.single_node.GenerativeForkIT test {csv-spec:lookup-join-expression.LookupJoinExpressionWithTerm} elastic#137160 * Mute org.elasticsearch.cluster.routing.allocation.decider.WriteLoadConstraintDeciderIT testCanRemainNotPreferredIsIgnoredWhenAllOtherNodesReturnNotPreferred elastic#137162 * Try bulk load sorted ordinals across ranges (elastic#137076) This change implements the TODO for loading sorted ordinals in the TSDB codec. With ordinal range encoding, we can bulk-append the entire range for the next ordinal, rather than reading and appending one document at a time. * Fix serialization assymetry (writeOptional vs plain read), fixed VerifierTett * Relax (again) the check in blockLoader in RangeFieldMapper, since it seem to fail in practice * [CI] Auto commit changes from spotless * AnalyzerTest fix * Fix APM tracer to respect recording status of spans (elastic#137015) Co-authored-by: Jack Shirazi <jack.shirazi@elastic.co> * Mute org.elasticsearch.xpack.esql.plan.logical.CommandLicenseTests testLicenseCheck elastic#137168 * Catch-and-rethrow TooComplexToDeterminizeException within ESQL (elastic#137024) --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * [ES|QL] Non-Correlated Subquery in FROM command (elastic#135744) * non-correlated subquery in from command * [docs] Update category field to use keyword type in full text tutorial (elastic#137169) closes elastic/docs-content#3329 * [Inference API] Rename E5 model/inference endpoint in EIS to JinaAI counterpart (elastic#137028) * ESQL: Add a bit of javadoc to preanalysis methods (elastic#137099) Explain where minimum transport version is determined for a given query. * Adds ml-cpp release notes (elastic#137178) * Remove mutes pointing to closed ESQL flaky test issues (elastic#137174) * rest-api-spec: Add missing options to enum types (elastic#136640) * rest-api-spec: Add missing options to enum types * Allow 0-9 in enum options * Add applies_to frontmatter to mapping-reference docs (elastic#137184) * [docs] Update ESQL command H1 titles for SEO (elastic#137188) * style fix * Some fixes: change the TopNEncoder to the correct one, fix the unsupported test to match the expected results (good ones and not unsupported). * Transport version * [CI] Auto commit changes from spotless * Remove temporary print * Add missing fix in boolean tranport flag in IndexResolver * Serialize "unsupported" DataType when communicating to old clusters * Block date_range on yaml tests for bwc * Fixed missing import after resolving conflicts with main * Fix mistake on merging main with new element types * Revert fix to DataType serialization We should handle this at a higher level, since this code is widely used in serialization tests which fail with this change. * Try a temporary fix for enrichment * [CI] Auto commit changes from spotless * Some small fixes * cat API: added endpoint for Circuit Breakers (elastic#136890) Added CAT Action to display Circuit Breakers stats for all nodes. The API supports pattern matching as a path parameter and the standard query parameters of CAT actions. This change includes spec and yamlRestTest. Addresses elastic#132688 * ESQL: Fix release tests (elastic#137298) New field type isn't serializable outside of snapshot. * Reject invalid `reverse_nested` aggs (elastic#137047) * Mute org.elasticsearch.xpack.esql.plan.physical.ShowExecSerializationTests testConcurrentSerialization elastic#137338 * Mute org.elasticsearch.readiness.ReadinessClusterIT testReadinessDuringRestartsNormalOrder elastic#136955 * [docs] Update changelog summary for semantic_text ELSER on EIS default (elastic#137339) * Fix method visibility in public-callers-finder (elastic#137200) Public-callers-finder did not correctly report if methods are externally visible due to the following issues: - checking if interfaces are exported (see findAccessibility) must be done using the exports of the interface’s module, not the exports of the current class - super classes must be treated the same way as interfaces, currently these are ignored - currently all public methods of a (potentially private, non-exported) class implementing a public, exported interface are considered accessible / visible regardless if part of the interface or not This fixes visibility to be consistent with the JdkApiExtractor tool by implementing both using the same common logic in `AccessibleJdkMethods.loadAccessibleMethods`. Note: this currently includes elastic#137193, which will be merged independently. Relates to ES-13117 * [ML] Adding bulk create functionality to ModelRegistry (elastic#136569) * Adding bulk storage of multiple models * Adding tests * Adding log for duplicate ids * [CI] Auto commit changes from spotless * Removing unused code * Removing constructor * Adding more tests * Adding in logic to delete models when a failure occurs * revert rename changes * formatting * Starting on feedback * Improving tests * Moving most tests to ModelRegistryIT * [CI] Auto commit changes from spotless * Fixing test * Removing duplicate tests * Handling empty list and duplicates * Fixing empty delete --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Remove deprecated constant for default elser inference id (elastic#137329) * Cleaned up the old constant * Cleaned up test * Update docs/changelog/137329.yaml * Update docs/changelog/137329.yaml * Update docs/changelog/137329.yaml * Delete docs/changelog/137329.yaml * ESQL: Handle release of 9.2 in test (elastic#137070) I'd made a mistake in elastic#136327 when writing the test for fetching fields that the release of 9.2.0 revealed. This fixes it and adds one more test that we needed from elastic#136327. * Fixed inconsistency in the isSyntheticSourceEnabled flag (elastic#137297) * Fixed inconsistency in the isSyntheticSourceEnabled flag * Update docs/changelog/137297.yaml * [ML] Disable CPS for Dataframes (elastic#136716) Cross-Project Search and Cross-Cluster Search is indefinitely disabled for Dataframe Analytics. The error message will now display if the syntax would have otherwise resolved to the respective feature. * [Docs] Improve semantic_text updates documentation organization (elastic#137340) * Move script update restrictions section for consistency * Update docs/reference/elasticsearch/mapping-reference/semantic-text.md Co-authored-by: Liam Thompson <leemthompo@gmail.com> * Update docs/reference/elasticsearch/mapping-reference/semantic-text.md Co-authored-by: Liam Thompson <leemthompo@gmail.com> * Update docs/reference/elasticsearch/mapping-reference/semantic-text.md Co-authored-by: Liam Thompson <leemthompo@gmail.com> --------- Co-authored-by: Liam Thompson <leemthompo@gmail.com> * Remove unused field from IndexModule (elastic#137342) * Improving random sampling performance by lazily calling getSamplingConfiguration() (elastic#137223) * [ML] Disable CrossProject for Datafeeds (elastic#136897) Initially, Datafeeds will not support cross-project source indices. We will verify that the IndicesOptions is not trying to resolve a cross-project index expression and throw a cross-project specific error message. * [ES-12998] Invoking gradle continue flag on periodic runs to allow for more information on test failures (elastic#136900) * Add --continue flag to invoke maven when a task failure has been hit so that we can see the outcome of more testing over time. * ES|QL: Improve value loading for match_only_text mapping (elastic#137026) * [Transform] Remove extra reset calls (elastic#137346) ESRestTestCase already calls the `_reset` API, we do not need to do it twice. Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Add chunk_rescorer usage to output of explain and profile for text_similarity_rank_retriever (elastic#137249) * Add chunk_rescorer usage to output of explain for text_similarity_rank_retriever * Update docs/changelog/137249.yaml * Update toString * Add support for profile * Update 137249.yaml * That's what you get for resolving conflicts in the UI, fixed compile failure * [CI] Auto commit changes from spotless * Fix test compilation --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Remove `first` and `last` functions from documentation (elastic#137341) * Remove `first` and `last` functions from documentation * Apply suggestions from code review Co-authored-by: Liam Thompson <leemthompo@gmail.com> * update --------- Co-authored-by: Liam Thompson <leemthompo@gmail.com> * ESQL: Work around concurrent serialization bug (elastic#137350) The bug still exists in the underlying code, but it isn't released so I'm working around it by doing an eager call to `hashCode`. We'll fix the real bug real soon. Closes elastic#137338 * [ES|QL] Add CHUNK function (elastic#134320) * Add new function to chunk strings * Refactor CHUNK function to support multiple values * Default to returning all chunks * [CI] Auto commit changes from spotless * Handle warnings * Loosen export restrictions to try to get compile error working * Remove inference dependencies * Fix compilation errors * Remove more inference deps * Fix compile errors from merge * Fix existing tests * Exclude from CSV tests * Add more tests * Cleanup * [CI] Auto commit changes from spotless * Cleanup * Update docs/changelog/134320.yaml * PR feedback * Remove null field constraint * [CI] Auto commit changes from spotless * PR feedback: Refactor to use an options map * Cleanup * Regenerate docs * Add test on a concatenated field * Add multivalued field test * Don't hardcode strings * [CI] Auto commit changes from spotless * PR feedback --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Add default sort for message.template_id field in logsdb indices (elastic#136571) This PR adds the index setting `index.logsdb.default_sort_on_message_template`. When set, LogsDB indices will add `message.template_id` to the default sort fields (assuming the `message` field is present and of type `pattern_text`). * Transport version * [CI] Update transport version definitions * Specifically allow to_date_range and to_string * Tansport version * Update docs/changelog/133309.yaml * Fixed forbidden toString * Revert enrichment hack * transport version * Fixed accidentally deleted files * [CI] Update transport version definitions * Generated files for to_string / is_null tests * Transport version * [CI] Update transport version definitions * Transport version * Small merge accident fix * Added missing case for BlockTestUtils * Small fix for AllSupportedFieldTest * Transport version * [CI] Update transport version definitions * [CI] Update transport version definitions * Transport version * merge fixes * change date_range to under_construction * Some error tests fixes * More error tests fixes * More merge fixes * [CI] Auto commit changes from spotless * Smale checstyle fix * error tests fixes * More fixes * One more error fix * Fix the date range capability * Update docs/changelog/133309.yaml * bring back accidentally deleted files * changelog type * AllSupportedFieldTest fix for no snapshot build * Transport version * Fix for CSV test to avoid order false errors * Another fix after changing to under construction * merge fixes * [CI] Auto commit changes from spotless * [CI] Update transport version definitions * [CI] Update transport version definitions * [CI] Update transport version definitions * [CI] Update transport version definitions * Fix some date->long text replacements we missed * Removed duplicated test data `date_ranges.csv` * Update transport version after merging main * Nicer string literal * Update transport version after merging main * TO_DATERANGE should be snapshot-only * Fix failing tests after moving TO_DATERANGE to snapshot-only * Remove `useDateRangeWhenNotSupported` since that seems only relevant to partial support across multiple versions * Disabled TO_DATE_RANGE on release builds of testInlineCast * [CI] Auto commit changes from spotless * Update docs/changelog/133309.yaml * transport version * merge fixes * another transport version * Another try in fixing the multi cluster tests * [CI] Update transport version definitions * Fixed compilation error * Fix thread leak in ContextIndexSearcherTests.testMaxClause Add missing executor cleanup in finally block to prevent thread leaks when the test creates a ThreadPoolExecutor. * Add queryContainsIndices utility method to EsqlTestUtils This new method checks if a given ESQL query contains any specified indices, facilitating special handling for queries using indices loaded into multiple clusters. The method is integrated into MultiClusterSpecIT to streamline the logic for determining when to use remote indices, improving code clarity and maintainability. * Fix for the last review fix * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Craig Taverner <craig@amanzi.com> Co-authored-by: Fabrizio Ferri-Benedetti <algernon@fastmail.com> Co-authored-by: David Turner <david.turner@elastic.co> Co-authored-by: elasticsearchmachine <58790826+elasticsearchmachine@users.noreply.github.com> Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co> Co-authored-by: Moritz Mack <mmack@apache.org> Co-authored-by: Jack Shirazi <jack.shirazi@elastic.co> Co-authored-by: Matt <matthew.alp@elastic.co> Co-authored-by: Fang Xing <155562079+fang-xing-esql@users.noreply.github.com> Co-authored-by: Liam Thompson <leemthompo@gmail.com> Co-authored-by: Tim Grein <tim.grein@elastic.co> Co-authored-by: Alexander Spies <alexander.spies@elastic.co> Co-authored-by: kosabogi <105062005+kosabogi@users.noreply.github.com> Co-authored-by: Ievgen Degtiarenko <ievgen.degtiarenko@elastic.co> Co-authored-by: Quentin Pradet <quentin.pradet@elastic.co> Co-authored-by: Matteo Mazzola <matteo.mazzola@elastic.co> Co-authored-by: Nik Everett <nik9000@gmail.com> Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com> Co-authored-by: Mridula <mridula.s@elastic.co> Co-authored-by: Dmitry Kubikov <Kubik42@users.noreply.github.com> Co-authored-by: Pat Whelan <pat.whelan@elastic.co> Co-authored-by: Alan Woodward <romseygeek@apache.org> Co-authored-by: Keith Massey <keith.massey@elastic.co> Co-authored-by: Neil Bhavsar <neil.bhavsar@elastic.co> Co-authored-by: Ioana Tagirta <ioanatia@users.noreply.github.com> Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co> Co-authored-by: Kostas Krikellas <131142368+kkrik-es@users.noreply.github.com> Co-authored-by: Jordan Powers <jordan.powers@elastic.co>

ESQL: Enable new data types with created version

57b63c2

Enables `dense_vector` and `aggregate_metric_double` if all nodes in all of the target clusters support it. Relates to elastic#135193

nik9000 added auto-backport Automatically create backport pull requests when merged :Analytics/ES|QL AKA ESQL v9.2.1 v9.3.0 labels Oct 9, 2025

nik9000 added the test-release Trigger CI checks against release build label Oct 10, 2025

nik9000 added 5 commits October 10, 2025 09:51

Fixup

3b5c4e2

Merge branch 'main' into version_enable

ac6efb9

Merge branch 'main' into version_enable

ad8b9b9

Merge

fb665c1

Rename and fix check

8b22742

nik9000 commented Oct 21, 2025

View reviewed changes

nik9000 marked this pull request as ready for review October 21, 2025 22:57

Merge branch 'main' into version_enable

a7d19df

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Oct 21, 2025

nik9000 requested a review from alex-spies October 21, 2025 22:58

Merge branch 'main' into version_enable

5016b73

nik9000 added the >enhancement label Oct 22, 2025

Update docs/changelog/136327.yaml

8e2d031

alex-spies approved these changes Oct 22, 2025

View reviewed changes

nik9000 added 5 commits October 23, 2025 11:13

Merge branch 'main' into version_enable

47590a1

Comments

0527749

Merge remote-tracking branch 'nik9000/version_enable' into version_en…

e9b20c8

…able

More rename

dd895a1

Merge branch 'main' into version_enable

b86a096

nik9000 merged commit b311383 into elastic:main Oct 23, 2025
35 checks passed

elasticsearchmachine added the backport pending label Oct 23, 2025

nik9000 mentioned this pull request Oct 23, 2025

ESQL: Handle release of 9.2 in test #137070

Merged

nik9000 added a commit that referenced this pull request Oct 24, 2025

ESQL: Enable new data types with created version (#136327) (#137073)

247d9dc

Enables `dense_vector` and `aggregate_metric_double` if all nodes in all of the target clusters support it. Relates to #135193

carlosdelest mentioned this pull request Oct 29, 2025

Unmutes #134639 #137333

Merged

alex-spies mentioned this pull request Nov 3, 2025

ESQL: version-aware planning follow-ups and testing improvements #137534

Closed

15 tasks

phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Nov 6, 2025

Mirror upstream elastic#136327 as single snapshot commit for AI review

4ddbf5f

BASE=440faee03ee04b4ab8dbf2e41afcc3cc6f0a26c0 HEAD=b86a0963774cbc1ccdf27e4fd19e5e312d736e33 Branch=main

phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Nov 7, 2025

Mirror upstream elastic#136327 as single snapshot commit for AI review

3f61837

BASE=440faee03ee04b4ab8dbf2e41afcc3cc6f0a26c0 HEAD=b86a0963774cbc1ccdf27e4fd19e5e312d736e33 Branch=main

	public void onlyOnSnapshot() throws IOException {
	public void onlyOnProductionBuilds() throws IOException {

Conversation

nik9000 commented Oct 9, 2025

Uh oh!

github-actions bot commented Oct 9, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Oct 21, 2025

Uh oh!

elasticsearchmachine commented Oct 22, 2025

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nik9000 commented Oct 23, 2025 • edited by alex-spies Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 23, 2025

💔 Backport failed

Uh oh!

nik9000 commented Oct 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nik9000 commented Oct 23, 2025 •

edited by alex-spies

Loading