[ML] Adding bulk create functionality to ModelRegistry by jonathan-buttner · Pull Request #136569 · elastic/elasticsearch

jonathan-buttner · 2025-10-14T19:59:24Z

This PR adds functionality to the ModelRegistry to store multiple inference endpoints at the same time by using a bulk index operation. This will be useful for when the master handles polling EIS for the authorized preconfigured endpoints so that it can create the new ones in one operation.

This doesn't leverage the ability to create multiple endpoints (beyond the storeModel using it internally). It will be used in a follow up PR.

…ticsearch into ml-eis-auth-master

jonathan-buttner · 2025-10-14T20:08:33Z

...plugin/inference/src/main/java/org/elasticsearch/xpack/inference/registry/ModelRegistry.java

+                    new ElasticsearchStatusException(
+                        "Inference endpoint [{}] already exists",
+                        RestStatus.BAD_REQUEST,
+                        failureItem.failureCause,


I changed this from a ResourceAlreadyExistsException so we could include the cause, but maybe we don't want to include it 🤷‍♂️

jonathan-buttner · 2025-10-14T20:39:59Z

...plugin/inference/src/main/java/org/elasticsearch/xpack/inference/registry/ModelRegistry.java

+
+    private void updateClusterState(List<Model> models, ActionListener<AcknowledgedResponse> listener, TimeValue timeout) {
+        var inferenceIdsSet = models.stream().map(Model::getInferenceEntityId).collect(Collectors.toSet());
+        var storeListener = listener.delegateResponse((delegate, exc) -> {


I switched to this instead of creating an anonymous class.

…uth-master

elasticsearchmachine · 2025-10-17T12:52:21Z

Pinging @elastic/ml-core (Team:ML)

DonalEvans

Not strictly related to this PR, but it seems like a lot of the tests in ModelRegistryTests could/should be moved to ModelRegistryIT since they're integration tests rather than unit tests (at least, as far as I understand those terms to be defined).

...plugin/inference/src/main/java/org/elasticsearch/xpack/inference/registry/ModelRegistry.java

DonalEvans · 2025-10-17T17:32:22Z

...plugin/inference/src/main/java/org/elasticsearch/xpack/inference/registry/ModelRegistry.java

-        };
+
+            var secretsItem = bulkItems[i + 1];
+            var secretsStoreResponse = createModelStoreResponse(secretsItem, docIdToInferenceId);


Is it worth doing some kind of check that the inference ID for the secrets item matches the inference ID for the configuration item? Or is it not possible to lose items from part way through the bulk response, only at the end?

We probably don't need the check for not getting an even number of responses. I was mostly trying to create a nicer error message in the very very unlikely event that it happened. I'll add an assertion that the inference IDs are the same.

...plugin/inference/src/main/java/org/elasticsearch/xpack/inference/registry/ModelRegistry.java

DonalEvans · 2025-10-17T18:07:31Z

...plugin/inference/src/main/java/org/elasticsearch/xpack/inference/registry/ModelRegistry.java

+                return;
+            }
+
+            var failureItem = firstFailureResponse.get();


When creating multiple inference endpoints in one go, it's possible that there may be multiple failures with multiple reasons. Would it be better to report all of the failures in the exception? It could be misleading if we only report the first endpoint that failed to be created when in fact none of them were created. Also, what would be the correct way to handle the case where one of the endpoints wasn't created because it already exists (400 error), but another wasn't created due to some other issue not caused by VersionConflictEngineException (500 error)?

This logic is within the storeModel() method. It only allows one endpoint to be created. The previous behavior to this PR for the storeModel() method was to retrieve the first failure (at most there will be 2 failures, one for each of the bulk items).

Your point about combining the errors is a good idea though. I'd rather create an issue and address that later though if that's ok.

The storeModels() does report each failure/success but it does it using the onResponse() of the listener and returns a list of responses which includes the rest status and an exception if one occurred.

Also, what would be the correct way to handle the case where one of the endpoints wasn't created because it already exists (400 error), but another wasn't created due to some other issue not caused by VersionConflictEngineException (500 error)?

In this situation, VersionConflictEngineException represents the case where the endpoint already exists. The change I made here was to return a raw ElasticsearchStatusException instead of a ResourceExistsException. The reason I chose that was so we could return the actual cause but we don't necessarily need to do that. I figured that might be more informative but maybe it's unnecessarily information for the user.

I believe VersionConflictEngineException could also occur if we were trying to do an update and sequence number we're using is incorrect (some other request occurred before ours). I don't think we need to handle that scenario in this case though, that'd be in the update flow.

Oh, my mistake, I missed that this was the case where there was only one endpoint. Combining the errors doesn't need to be done as part of this PR.

...n/inference/src/test/java/org/elasticsearch/xpack/inference/registry/ModelRegistryTests.java

…uth-master

jonathan-buttner · 2025-10-20T14:16:33Z

Not strictly related to this PR, but it seems like a lot of the tests in ModelRegistryTests could/should be moved to ModelRegistryIT since they're integration tests rather than unit tests (at least, as far as I understand those terms to be defined).

I took a stab at moving over the tests to ModelRegistryIT. I agree that most of them should live in that file now. I left a few that didn't seem to be leveraging elasticsearch and needed package private access to ModelRegistry.

…ticsearch into ml-eis-auth-master

.../internalClusterTest/java/org/elasticsearch/xpack/inference/integration/ModelRegistryIT.java

…uth-master

davidkyle · 2025-10-21T09:12:43Z

...plugin/inference/src/main/java/org/elasticsearch/xpack/inference/registry/ModelRegistry.java

-                    format("Storing inference endpoint [%s] failed, no items were received from the bulk response", inferenceEntityId)
-                );
+                var inferenceEntityIds = String.join(", ", models.stream().map(Model::getInferenceEntityId).toList());
+                logger.warn("Storing inference endpoints [{}] failed, no items were received from the bulk response", inferenceEntityIds);


Check that the bulk request or models list is not empty in the storeModels function and trivially return success if that is the case. Otherwise an empty request would return a 500 error code

Good call 👍

davidkyle · 2025-10-21T09:16:07Z

...plugin/inference/src/main/java/org/elasticsearch/xpack/inference/registry/ModelRegistry.java

-            var inferenceEntityId = model.getInferenceEntityId();
+            var docIdToInferenceId = models.stream()
+                .collect(Collectors.toMap(m -> Model.documentId(m.getInferenceEntityId()), Model::getInferenceEntityId, (id1, id2) -> {
+                    logger.warn("Encountered duplicate inference ids when storing endpoints: [{}]", id1);


To avoid one config overwriting another (or throwing a version conflict exception) the check for duplicate Ids should be performed before indexing. The storeModels function is called automatically by internal code and we want it to be resilient so maybe filter out the duplicates if the id and model config are exactly the same.

…uth-master

davidkyle · 2025-10-22T11:52:27Z

...plugin/inference/src/main/java/org/elasticsearch/xpack/inference/registry/ModelRegistry.java

+
+            var storageResponses = responses.stream().map(StoreResponseWithIndexInfo::modelStoreResponse).toList();
+
+            deleteModels(inferenceIdsToBeRemoved, ActionListener.running(() -> delegate.onResponse(storageResponses)));


Is it possible for inferenceIdsToBeRemoved to be an empty set and in which case can deleteModels be skipped?

Good point, I'll have deleteModels return early if the set is empty.

…uth-master

davidkyle

LGTM

* Adding bulk storage of multiple models * Adding tests * Adding log for duplicate ids * [CI] Auto commit changes from spotless * Removing unused code * Removing constructor * Adding more tests * Adding in logic to delete models when a failure occurs * revert rename changes * formatting * Starting on feedback * Improving tests * Moving most tests to ModelRegistryIT * [CI] Auto commit changes from spotless * Fixing test * Removing duplicate tests * Handling empty list and duplicates * Fixing empty delete --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>

* Basic support for DATE_RANGE field type in ESQL * [CI] Update transport version definitions * Update docs/changelog/133309.yaml * Suppres two more tests * Two more tests fixes * Two more tests * Some more infra for new tests * Some more fixes for tests * And some more tests * And some more tests * Fix capanility check in CSV tests and some more fixes * Block lookup join * Block lookup join * Relaxed the previous transport version check, which was in fact always false - replace it with something that can work for now. * Fixed a typo * Style fixes * More fixes: block lookup join. Delete date_range from unsupported yaml test, it's not unsupported anymore and there is a normal yaml test for it. * Supress RestEsqlIT because complex queries like this are not supported yet * Removed unused field * Bring back deleted yaml tests * Refactor DateRange Block type (and element type) to LongRange * Review fixes * small fixes * Added identity option to TO_DATE_RANGE, plus more tests * [CI] Update transport version definitions * Fix irregular spaces (#137014) * Fix irregular spaces * Update analysis-keyword-repeat-tokenfilter.md * Update search-suggesters.md * Update search-profile.md * Test utility for `POST _features/_reset` (#137133) We call `POST _features/_reset` in various places in the tests, only sometimes asserting that the response is `200 OK`. This commit extracts a utility that makes it harder to miss this assertion. It also adds the `?error_trace` parameter so that on failure the details are visible in the response, and `?master_timeout=-1` to avoid spurious timeouts if CI is running a little slow. * Revert "Test utility for `POST _features/_reset` (#137133)" This reverts commit df67e27. * Mute org.elasticsearch.xpack.esql.qa.multi_node.EsqlSpecIT test {csv-spec:lookup-join-expression.LookupJoinExpressionWithTerm} #137157 * Mute org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT test {csv-spec:lookup-join-expression.LookupJoinExpressionWithTerm} #137157 * Mute org.elasticsearch.xpack.esql.qa.single_node.GenerativeForkIT test {csv-spec:lookup-join-expression.LookupJoinExpressionWithTerm} #137160 * Mute org.elasticsearch.cluster.routing.allocation.decider.WriteLoadConstraintDeciderIT testCanRemainNotPreferredIsIgnoredWhenAllOtherNodesReturnNotPreferred #137162 * Try bulk load sorted ordinals across ranges (#137076) This change implements the TODO for loading sorted ordinals in the TSDB codec. With ordinal range encoding, we can bulk-append the entire range for the next ordinal, rather than reading and appending one document at a time. * Fix serialization assymetry (writeOptional vs plain read), fixed VerifierTett * Relax (again) the check in blockLoader in RangeFieldMapper, since it seem to fail in practice * [CI] Auto commit changes from spotless * AnalyzerTest fix * Fix APM tracer to respect recording status of spans (#137015) Co-authored-by: Jack Shirazi <jack.shirazi@elastic.co> * Mute org.elasticsearch.xpack.esql.plan.logical.CommandLicenseTests testLicenseCheck #137168 * Catch-and-rethrow TooComplexToDeterminizeException within ESQL (#137024) --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * [ES|QL] Non-Correlated Subquery in FROM command (#135744) * non-correlated subquery in from command * [docs] Update category field to use keyword type in full text tutorial (#137169) closes elastic/docs-content#3329 * [Inference API] Rename E5 model/inference endpoint in EIS to JinaAI counterpart (#137028) * ESQL: Add a bit of javadoc to preanalysis methods (#137099) Explain where minimum transport version is determined for a given query. * Adds ml-cpp release notes (#137178) * Remove mutes pointing to closed ESQL flaky test issues (#137174) * rest-api-spec: Add missing options to enum types (#136640) * rest-api-spec: Add missing options to enum types * Allow 0-9 in enum options * Add applies_to frontmatter to mapping-reference docs (#137184) * [docs] Update ESQL command H1 titles for SEO (#137188) * style fix * Some fixes: change the TopNEncoder to the correct one, fix the unsupported test to match the expected results (good ones and not unsupported). * Transport version * [CI] Auto commit changes from spotless * Remove temporary print * Add missing fix in boolean tranport flag in IndexResolver * Serialize "unsupported" DataType when communicating to old clusters * Block date_range on yaml tests for bwc * Fixed missing import after resolving conflicts with main * Fix mistake on merging main with new element types * Revert fix to DataType serialization We should handle this at a higher level, since this code is widely used in serialization tests which fail with this change. * Try a temporary fix for enrichment * [CI] Auto commit changes from spotless * Some small fixes * cat API: added endpoint for Circuit Breakers (#136890) Added CAT Action to display Circuit Breakers stats for all nodes. The API supports pattern matching as a path parameter and the standard query parameters of CAT actions. This change includes spec and yamlRestTest. Addresses #132688 * ESQL: Fix release tests (#137298) New field type isn't serializable outside of snapshot. * Reject invalid `reverse_nested` aggs (#137047) * Mute org.elasticsearch.xpack.esql.plan.physical.ShowExecSerializationTests testConcurrentSerialization #137338 * Mute org.elasticsearch.readiness.ReadinessClusterIT testReadinessDuringRestartsNormalOrder #136955 * [docs] Update changelog summary for semantic_text ELSER on EIS default (#137339) * Fix method visibility in public-callers-finder (#137200) Public-callers-finder did not correctly report if methods are externally visible due to the following issues: - checking if interfaces are exported (see findAccessibility) must be done using the exports of the interface’s module, not the exports of the current class - super classes must be treated the same way as interfaces, currently these are ignored - currently all public methods of a (potentially private, non-exported) class implementing a public, exported interface are considered accessible / visible regardless if part of the interface or not This fixes visibility to be consistent with the JdkApiExtractor tool by implementing both using the same common logic in `AccessibleJdkMethods.loadAccessibleMethods`. Note: this currently includes #137193, which will be merged independently. Relates to ES-13117 * [ML] Adding bulk create functionality to ModelRegistry (#136569) * Adding bulk storage of multiple models * Adding tests * Adding log for duplicate ids * [CI] Auto commit changes from spotless * Removing unused code * Removing constructor * Adding more tests * Adding in logic to delete models when a failure occurs * revert rename changes * formatting * Starting on feedback * Improving tests * Moving most tests to ModelRegistryIT * [CI] Auto commit changes from spotless * Fixing test * Removing duplicate tests * Handling empty list and duplicates * Fixing empty delete --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Remove deprecated constant for default elser inference id (#137329) * Cleaned up the old constant * Cleaned up test * Update docs/changelog/137329.yaml * Update docs/changelog/137329.yaml * Update docs/changelog/137329.yaml * Delete docs/changelog/137329.yaml * ESQL: Handle release of 9.2 in test (#137070) I'd made a mistake in #136327 when writing the test for fetching fields that the release of 9.2.0 revealed. This fixes it and adds one more test that we needed from #136327. * Fixed inconsistency in the isSyntheticSourceEnabled flag (#137297) * Fixed inconsistency in the isSyntheticSourceEnabled flag * Update docs/changelog/137297.yaml * [ML] Disable CPS for Dataframes (#136716) Cross-Project Search and Cross-Cluster Search is indefinitely disabled for Dataframe Analytics. The error message will now display if the syntax would have otherwise resolved to the respective feature. * [Docs] Improve semantic_text updates documentation organization (#137340) * Move script update restrictions section for consistency * Update docs/reference/elasticsearch/mapping-reference/semantic-text.md Co-authored-by: Liam Thompson <leemthompo@gmail.com> * Update docs/reference/elasticsearch/mapping-reference/semantic-text.md Co-authored-by: Liam Thompson <leemthompo@gmail.com> * Update docs/reference/elasticsearch/mapping-reference/semantic-text.md Co-authored-by: Liam Thompson <leemthompo@gmail.com> --------- Co-authored-by: Liam Thompson <leemthompo@gmail.com> * Remove unused field from IndexModule (#137342) * Improving random sampling performance by lazily calling getSamplingConfiguration() (#137223) * [ML] Disable CrossProject for Datafeeds (#136897) Initially, Datafeeds will not support cross-project source indices. We will verify that the IndicesOptions is not trying to resolve a cross-project index expression and throw a cross-project specific error message. * [ES-12998] Invoking gradle continue flag on periodic runs to allow for more information on test failures (#136900) * Add --continue flag to invoke maven when a task failure has been hit so that we can see the outcome of more testing over time. * ES|QL: Improve value loading for match_only_text mapping (#137026) * [Transform] Remove extra reset calls (#137346) ESRestTestCase already calls the `_reset` API, we do not need to do it twice. Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Add chunk_rescorer usage to output of explain and profile for text_similarity_rank_retriever (#137249) * Add chunk_rescorer usage to output of explain for text_similarity_rank_retriever * Update docs/changelog/137249.yaml * Update toString * Add support for profile * Update 137249.yaml * That's what you get for resolving conflicts in the UI, fixed compile failure * [CI] Auto commit changes from spotless * Fix test compilation --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Remove `first` and `last` functions from documentation (#137341) * Remove `first` and `last` functions from documentation * Apply suggestions from code review Co-authored-by: Liam Thompson <leemthompo@gmail.com> * update --------- Co-authored-by: Liam Thompson <leemthompo@gmail.com> * ESQL: Work around concurrent serialization bug (#137350) The bug still exists in the underlying code, but it isn't released so I'm working around it by doing an eager call to `hashCode`. We'll fix the real bug real soon. Closes #137338 * [ES|QL] Add CHUNK function (#134320) * Add new function to chunk strings * Refactor CHUNK function to support multiple values * Default to returning all chunks * [CI] Auto commit changes from spotless * Handle warnings * Loosen export restrictions to try to get compile error working * Remove inference dependencies * Fix compilation errors * Remove more inference deps * Fix compile errors from merge * Fix existing tests * Exclude from CSV tests * Add more tests * Cleanup * [CI] Auto commit changes from spotless * Cleanup * Update docs/changelog/134320.yaml * PR feedback * Remove null field constraint * [CI] Auto commit changes from spotless * PR feedback: Refactor to use an options map * Cleanup * Regenerate docs * Add test on a concatenated field * Add multivalued field test * Don't hardcode strings * [CI] Auto commit changes from spotless * PR feedback --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Add default sort for message.template_id field in logsdb indices (#136571) This PR adds the index setting `index.logsdb.default_sort_on_message_template`. When set, LogsDB indices will add `message.template_id` to the default sort fields (assuming the `message` field is present and of type `pattern_text`). * Transport version * [CI] Update transport version definitions * Specifically allow to_date_range and to_string * Tansport version * Update docs/changelog/133309.yaml * Fixed forbidden toString * Revert enrichment hack * transport version * Fixed accidentally deleted files * [CI] Update transport version definitions * Generated files for to_string / is_null tests * Transport version * [CI] Update transport version definitions * Transport version * Small merge accident fix * Added missing case for BlockTestUtils * Small fix for AllSupportedFieldTest * Transport version * [CI] Update transport version definitions * [CI] Update transport version definitions * Transport version * merge fixes * change date_range to under_construction * Some error tests fixes * More error tests fixes * More merge fixes * [CI] Auto commit changes from spotless * Smale checstyle fix * error tests fixes * More fixes * One more error fix * Fix the date range capability * Update docs/changelog/133309.yaml * bring back accidentally deleted files * changelog type * AllSupportedFieldTest fix for no snapshot build * Transport version * Fix for CSV test to avoid order false errors * Another fix after changing to under construction * merge fixes * [CI] Auto commit changes from spotless * [CI] Update transport version definitions * [CI] Update transport version definitions * [CI] Update transport version definitions * [CI] Update transport version definitions * Fix some date->long text replacements we missed * Removed duplicated test data `date_ranges.csv` * Update transport version after merging main * Nicer string literal * Update transport version after merging main * TO_DATERANGE should be snapshot-only * Fix failing tests after moving TO_DATERANGE to snapshot-only * Remove `useDateRangeWhenNotSupported` since that seems only relevant to partial support across multiple versions * Disabled TO_DATE_RANGE on release builds of testInlineCast * [CI] Auto commit changes from spotless * Update docs/changelog/133309.yaml * transport version * merge fixes * another transport version * Another try in fixing the multi cluster tests * [CI] Update transport version definitions * Fixed compilation error * Fix thread leak in ContextIndexSearcherTests.testMaxClause Add missing executor cleanup in finally block to prevent thread leaks when the test creates a ThreadPoolExecutor. * Add queryContainsIndices utility method to EsqlTestUtils This new method checks if a given ESQL query contains any specified indices, facilitating special handling for queries using indices loaded into multiple clusters. The method is integrated into MultiClusterSpecIT to streamline the logic for determining when to use remote indices, improving code clarity and maintainability. * Fix for the last review fix * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Craig Taverner <craig@amanzi.com> Co-authored-by: Fabrizio Ferri-Benedetti <algernon@fastmail.com> Co-authored-by: David Turner <david.turner@elastic.co> Co-authored-by: elasticsearchmachine <58790826+elasticsearchmachine@users.noreply.github.com> Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co> Co-authored-by: Moritz Mack <mmack@apache.org> Co-authored-by: Jack Shirazi <jack.shirazi@elastic.co> Co-authored-by: Matt <matthew.alp@elastic.co> Co-authored-by: Fang Xing <155562079+fang-xing-esql@users.noreply.github.com> Co-authored-by: Liam Thompson <leemthompo@gmail.com> Co-authored-by: Tim Grein <tim.grein@elastic.co> Co-authored-by: Alexander Spies <alexander.spies@elastic.co> Co-authored-by: kosabogi <105062005+kosabogi@users.noreply.github.com> Co-authored-by: Ievgen Degtiarenko <ievgen.degtiarenko@elastic.co> Co-authored-by: Quentin Pradet <quentin.pradet@elastic.co> Co-authored-by: Matteo Mazzola <matteo.mazzola@elastic.co> Co-authored-by: Nik Everett <nik9000@gmail.com> Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com> Co-authored-by: Mridula <mridula.s@elastic.co> Co-authored-by: Dmitry Kubikov <Kubik42@users.noreply.github.com> Co-authored-by: Pat Whelan <pat.whelan@elastic.co> Co-authored-by: Alan Woodward <romseygeek@apache.org> Co-authored-by: Keith Massey <keith.massey@elastic.co> Co-authored-by: Neil Bhavsar <neil.bhavsar@elastic.co> Co-authored-by: Ioana Tagirta <ioanatia@users.noreply.github.com> Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co> Co-authored-by: Kostas Krikellas <131142368+kkrik-es@users.noreply.github.com> Co-authored-by: Jordan Powers <jordan.powers@elastic.co>

* Basic support for DATE_RANGE field type in ESQL * [CI] Update transport version definitions * Update docs/changelog/133309.yaml * Suppres two more tests * Two more tests fixes * Two more tests * Some more infra for new tests * Some more fixes for tests * And some more tests * And some more tests * Fix capanility check in CSV tests and some more fixes * Block lookup join * Block lookup join * Relaxed the previous transport version check, which was in fact always false - replace it with something that can work for now. * Fixed a typo * Style fixes * More fixes: block lookup join. Delete date_range from unsupported yaml test, it's not unsupported anymore and there is a normal yaml test for it. * Supress RestEsqlIT because complex queries like this are not supported yet * Removed unused field * Bring back deleted yaml tests * Refactor DateRange Block type (and element type) to LongRange * Review fixes * small fixes * Added identity option to TO_DATE_RANGE, plus more tests * [CI] Update transport version definitions * Fix irregular spaces (elastic#137014) * Fix irregular spaces * Update analysis-keyword-repeat-tokenfilter.md * Update search-suggesters.md * Update search-profile.md * Test utility for `POST _features/_reset` (elastic#137133) We call `POST _features/_reset` in various places in the tests, only sometimes asserting that the response is `200 OK`. This commit extracts a utility that makes it harder to miss this assertion. It also adds the `?error_trace` parameter so that on failure the details are visible in the response, and `?master_timeout=-1` to avoid spurious timeouts if CI is running a little slow. * Revert "Test utility for `POST _features/_reset` (elastic#137133)" This reverts commit df67e27. * Mute org.elasticsearch.xpack.esql.qa.multi_node.EsqlSpecIT test {csv-spec:lookup-join-expression.LookupJoinExpressionWithTerm} elastic#137157 * Mute org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT test {csv-spec:lookup-join-expression.LookupJoinExpressionWithTerm} elastic#137157 * Mute org.elasticsearch.xpack.esql.qa.single_node.GenerativeForkIT test {csv-spec:lookup-join-expression.LookupJoinExpressionWithTerm} elastic#137160 * Mute org.elasticsearch.cluster.routing.allocation.decider.WriteLoadConstraintDeciderIT testCanRemainNotPreferredIsIgnoredWhenAllOtherNodesReturnNotPreferred elastic#137162 * Try bulk load sorted ordinals across ranges (elastic#137076) This change implements the TODO for loading sorted ordinals in the TSDB codec. With ordinal range encoding, we can bulk-append the entire range for the next ordinal, rather than reading and appending one document at a time. * Fix serialization assymetry (writeOptional vs plain read), fixed VerifierTett * Relax (again) the check in blockLoader in RangeFieldMapper, since it seem to fail in practice * [CI] Auto commit changes from spotless * AnalyzerTest fix * Fix APM tracer to respect recording status of spans (elastic#137015) Co-authored-by: Jack Shirazi <jack.shirazi@elastic.co> * Mute org.elasticsearch.xpack.esql.plan.logical.CommandLicenseTests testLicenseCheck elastic#137168 * Catch-and-rethrow TooComplexToDeterminizeException within ESQL (elastic#137024) --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * [ES|QL] Non-Correlated Subquery in FROM command (elastic#135744) * non-correlated subquery in from command * [docs] Update category field to use keyword type in full text tutorial (elastic#137169) closes elastic/docs-content#3329 * [Inference API] Rename E5 model/inference endpoint in EIS to JinaAI counterpart (elastic#137028) * ESQL: Add a bit of javadoc to preanalysis methods (elastic#137099) Explain where minimum transport version is determined for a given query. * Adds ml-cpp release notes (elastic#137178) * Remove mutes pointing to closed ESQL flaky test issues (elastic#137174) * rest-api-spec: Add missing options to enum types (elastic#136640) * rest-api-spec: Add missing options to enum types * Allow 0-9 in enum options * Add applies_to frontmatter to mapping-reference docs (elastic#137184) * [docs] Update ESQL command H1 titles for SEO (elastic#137188) * style fix * Some fixes: change the TopNEncoder to the correct one, fix the unsupported test to match the expected results (good ones and not unsupported). * Transport version * [CI] Auto commit changes from spotless * Remove temporary print * Add missing fix in boolean tranport flag in IndexResolver * Serialize "unsupported" DataType when communicating to old clusters * Block date_range on yaml tests for bwc * Fixed missing import after resolving conflicts with main * Fix mistake on merging main with new element types * Revert fix to DataType serialization We should handle this at a higher level, since this code is widely used in serialization tests which fail with this change. * Try a temporary fix for enrichment * [CI] Auto commit changes from spotless * Some small fixes * cat API: added endpoint for Circuit Breakers (elastic#136890) Added CAT Action to display Circuit Breakers stats for all nodes. The API supports pattern matching as a path parameter and the standard query parameters of CAT actions. This change includes spec and yamlRestTest. Addresses elastic#132688 * ESQL: Fix release tests (elastic#137298) New field type isn't serializable outside of snapshot. * Reject invalid `reverse_nested` aggs (elastic#137047) * Mute org.elasticsearch.xpack.esql.plan.physical.ShowExecSerializationTests testConcurrentSerialization elastic#137338 * Mute org.elasticsearch.readiness.ReadinessClusterIT testReadinessDuringRestartsNormalOrder elastic#136955 * [docs] Update changelog summary for semantic_text ELSER on EIS default (elastic#137339) * Fix method visibility in public-callers-finder (elastic#137200) Public-callers-finder did not correctly report if methods are externally visible due to the following issues: - checking if interfaces are exported (see findAccessibility) must be done using the exports of the interface’s module, not the exports of the current class - super classes must be treated the same way as interfaces, currently these are ignored - currently all public methods of a (potentially private, non-exported) class implementing a public, exported interface are considered accessible / visible regardless if part of the interface or not This fixes visibility to be consistent with the JdkApiExtractor tool by implementing both using the same common logic in `AccessibleJdkMethods.loadAccessibleMethods`. Note: this currently includes elastic#137193, which will be merged independently. Relates to ES-13117 * [ML] Adding bulk create functionality to ModelRegistry (elastic#136569) * Adding bulk storage of multiple models * Adding tests * Adding log for duplicate ids * [CI] Auto commit changes from spotless * Removing unused code * Removing constructor * Adding more tests * Adding in logic to delete models when a failure occurs * revert rename changes * formatting * Starting on feedback * Improving tests * Moving most tests to ModelRegistryIT * [CI] Auto commit changes from spotless * Fixing test * Removing duplicate tests * Handling empty list and duplicates * Fixing empty delete --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Remove deprecated constant for default elser inference id (elastic#137329) * Cleaned up the old constant * Cleaned up test * Update docs/changelog/137329.yaml * Update docs/changelog/137329.yaml * Update docs/changelog/137329.yaml * Delete docs/changelog/137329.yaml * ESQL: Handle release of 9.2 in test (elastic#137070) I'd made a mistake in elastic#136327 when writing the test for fetching fields that the release of 9.2.0 revealed. This fixes it and adds one more test that we needed from elastic#136327. * Fixed inconsistency in the isSyntheticSourceEnabled flag (elastic#137297) * Fixed inconsistency in the isSyntheticSourceEnabled flag * Update docs/changelog/137297.yaml * [ML] Disable CPS for Dataframes (elastic#136716) Cross-Project Search and Cross-Cluster Search is indefinitely disabled for Dataframe Analytics. The error message will now display if the syntax would have otherwise resolved to the respective feature. * [Docs] Improve semantic_text updates documentation organization (elastic#137340) * Move script update restrictions section for consistency * Update docs/reference/elasticsearch/mapping-reference/semantic-text.md Co-authored-by: Liam Thompson <leemthompo@gmail.com> * Update docs/reference/elasticsearch/mapping-reference/semantic-text.md Co-authored-by: Liam Thompson <leemthompo@gmail.com> * Update docs/reference/elasticsearch/mapping-reference/semantic-text.md Co-authored-by: Liam Thompson <leemthompo@gmail.com> --------- Co-authored-by: Liam Thompson <leemthompo@gmail.com> * Remove unused field from IndexModule (elastic#137342) * Improving random sampling performance by lazily calling getSamplingConfiguration() (elastic#137223) * [ML] Disable CrossProject for Datafeeds (elastic#136897) Initially, Datafeeds will not support cross-project source indices. We will verify that the IndicesOptions is not trying to resolve a cross-project index expression and throw a cross-project specific error message. * [ES-12998] Invoking gradle continue flag on periodic runs to allow for more information on test failures (elastic#136900) * Add --continue flag to invoke maven when a task failure has been hit so that we can see the outcome of more testing over time. * ES|QL: Improve value loading for match_only_text mapping (elastic#137026) * [Transform] Remove extra reset calls (elastic#137346) ESRestTestCase already calls the `_reset` API, we do not need to do it twice. Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Add chunk_rescorer usage to output of explain and profile for text_similarity_rank_retriever (elastic#137249) * Add chunk_rescorer usage to output of explain for text_similarity_rank_retriever * Update docs/changelog/137249.yaml * Update toString * Add support for profile * Update 137249.yaml * That's what you get for resolving conflicts in the UI, fixed compile failure * [CI] Auto commit changes from spotless * Fix test compilation --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Remove `first` and `last` functions from documentation (elastic#137341) * Remove `first` and `last` functions from documentation * Apply suggestions from code review Co-authored-by: Liam Thompson <leemthompo@gmail.com> * update --------- Co-authored-by: Liam Thompson <leemthompo@gmail.com> * ESQL: Work around concurrent serialization bug (elastic#137350) The bug still exists in the underlying code, but it isn't released so I'm working around it by doing an eager call to `hashCode`. We'll fix the real bug real soon. Closes elastic#137338 * [ES|QL] Add CHUNK function (elastic#134320) * Add new function to chunk strings * Refactor CHUNK function to support multiple values * Default to returning all chunks * [CI] Auto commit changes from spotless * Handle warnings * Loosen export restrictions to try to get compile error working * Remove inference dependencies * Fix compilation errors * Remove more inference deps * Fix compile errors from merge * Fix existing tests * Exclude from CSV tests * Add more tests * Cleanup * [CI] Auto commit changes from spotless * Cleanup * Update docs/changelog/134320.yaml * PR feedback * Remove null field constraint * [CI] Auto commit changes from spotless * PR feedback: Refactor to use an options map * Cleanup * Regenerate docs * Add test on a concatenated field * Add multivalued field test * Don't hardcode strings * [CI] Auto commit changes from spotless * PR feedback --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> * Add default sort for message.template_id field in logsdb indices (elastic#136571) This PR adds the index setting `index.logsdb.default_sort_on_message_template`. When set, LogsDB indices will add `message.template_id` to the default sort fields (assuming the `message` field is present and of type `pattern_text`). * Transport version * [CI] Update transport version definitions * Specifically allow to_date_range and to_string * Tansport version * Update docs/changelog/133309.yaml * Fixed forbidden toString * Revert enrichment hack * transport version * Fixed accidentally deleted files * [CI] Update transport version definitions * Generated files for to_string / is_null tests * Transport version * [CI] Update transport version definitions * Transport version * Small merge accident fix * Added missing case for BlockTestUtils * Small fix for AllSupportedFieldTest * Transport version * [CI] Update transport version definitions * [CI] Update transport version definitions * Transport version * merge fixes * change date_range to under_construction * Some error tests fixes * More error tests fixes * More merge fixes * [CI] Auto commit changes from spotless * Smale checstyle fix * error tests fixes * More fixes * One more error fix * Fix the date range capability * Update docs/changelog/133309.yaml * bring back accidentally deleted files * changelog type * AllSupportedFieldTest fix for no snapshot build * Transport version * Fix for CSV test to avoid order false errors * Another fix after changing to under construction * merge fixes * [CI] Auto commit changes from spotless * [CI] Update transport version definitions * [CI] Update transport version definitions * [CI] Update transport version definitions * [CI] Update transport version definitions * Fix some date->long text replacements we missed * Removed duplicated test data `date_ranges.csv` * Update transport version after merging main * Nicer string literal * Update transport version after merging main * TO_DATERANGE should be snapshot-only * Fix failing tests after moving TO_DATERANGE to snapshot-only * Remove `useDateRangeWhenNotSupported` since that seems only relevant to partial support across multiple versions * Disabled TO_DATE_RANGE on release builds of testInlineCast * [CI] Auto commit changes from spotless * Update docs/changelog/133309.yaml * transport version * merge fixes * another transport version * Another try in fixing the multi cluster tests * [CI] Update transport version definitions * Fixed compilation error * Fix thread leak in ContextIndexSearcherTests.testMaxClause Add missing executor cleanup in finally block to prevent thread leaks when the test creates a ThreadPoolExecutor. * Add queryContainsIndices utility method to EsqlTestUtils This new method checks if a given ESQL query contains any specified indices, facilitating special handling for queries using indices loaded into multiple clusters. The method is integrated into MultiClusterSpecIT to streamline the logic for determining when to use remote indices, improving code clarity and maintainability. * Fix for the last review fix * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co> Co-authored-by: Craig Taverner <craig@amanzi.com> Co-authored-by: Fabrizio Ferri-Benedetti <algernon@fastmail.com> Co-authored-by: David Turner <david.turner@elastic.co> Co-authored-by: elasticsearchmachine <58790826+elasticsearchmachine@users.noreply.github.com> Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co> Co-authored-by: Moritz Mack <mmack@apache.org> Co-authored-by: Jack Shirazi <jack.shirazi@elastic.co> Co-authored-by: Matt <matthew.alp@elastic.co> Co-authored-by: Fang Xing <155562079+fang-xing-esql@users.noreply.github.com> Co-authored-by: Liam Thompson <leemthompo@gmail.com> Co-authored-by: Tim Grein <tim.grein@elastic.co> Co-authored-by: Alexander Spies <alexander.spies@elastic.co> Co-authored-by: kosabogi <105062005+kosabogi@users.noreply.github.com> Co-authored-by: Ievgen Degtiarenko <ievgen.degtiarenko@elastic.co> Co-authored-by: Quentin Pradet <quentin.pradet@elastic.co> Co-authored-by: Matteo Mazzola <matteo.mazzola@elastic.co> Co-authored-by: Nik Everett <nik9000@gmail.com> Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com> Co-authored-by: Mridula <mridula.s@elastic.co> Co-authored-by: Dmitry Kubikov <Kubik42@users.noreply.github.com> Co-authored-by: Pat Whelan <pat.whelan@elastic.co> Co-authored-by: Alan Woodward <romseygeek@apache.org> Co-authored-by: Keith Massey <keith.massey@elastic.co> Co-authored-by: Neil Bhavsar <neil.bhavsar@elastic.co> Co-authored-by: Ioana Tagirta <ioanatia@users.noreply.github.com> Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co> Co-authored-by: Kostas Krikellas <131142368+kkrik-es@users.noreply.github.com> Co-authored-by: Jordan Powers <jordan.powers@elastic.co>

jonathan-buttner added 3 commits October 10, 2025 17:26

Adding bulk storage of multiple models

bef0079

Adding tests

feb96ba

Adding log for duplicate ids

5f36d45

jonathan-buttner added >non-issue :ml Machine learning Team:ML Meta label for the ML team v9.3.0 labels Oct 14, 2025

elasticsearchmachine and others added 3 commits October 14, 2025 20:06

[CI] Auto commit changes from spotless

ebb6476

Removing unused code

c30df2b

Merge branch 'ml-eis-auth-master' of github.com:jonathan-buttner/elas…

5d4382e

…ticsearch into ml-eis-auth-master

jonathan-buttner commented Oct 14, 2025

View reviewed changes

jonathan-buttner and others added 4 commits October 14, 2025 16:50

Removing constructor

4ea87d1

Merge branch 'main' into ml-eis-auth-master

651876d

Adding more tests

155c366

Merge branch 'main' of github.com:elastic/elasticsearch into ml-eis-a…

44ddd92

…uth-master

jonathan-buttner mentioned this pull request Oct 16, 2025

[ML] Transition EIS auth polling to persistent task on a single node #136713

Merged

jonathan-buttner added 4 commits October 16, 2025 15:38

Merge branch 'main' of github.com:elastic/elasticsearch into ml-eis-a…

dbe11ce

…uth-master

Adding in logic to delete models when a failure occurs

b7f41f8

revert rename changes

f4d9f2c

formatting

cd8c832

jonathan-buttner requested a review from DonalEvans October 17, 2025 12:51

jonathan-buttner marked this pull request as ready for review October 17, 2025 12:51

jonathan-buttner requested a review from davidkyle October 17, 2025 12:52

DonalEvans reviewed Oct 17, 2025

View reviewed changes

jonathan-buttner added 4 commits October 17, 2025 17:05

Starting on feedback

b6cebf5

Merge branch 'main' of github.com:elastic/elasticsearch into ml-eis-a…

1130e82

…uth-master

Improving tests

5441da4

Moving most tests to ModelRegistryIT

58f9a75

jonathan-buttner requested a review from DonalEvans October 20, 2025 14:15

elasticsearchmachine and others added 3 commits October 20, 2025 14:20

[CI] Auto commit changes from spotless

1854e50

Fixing test

a8db6cf

Merge branch 'ml-eis-auth-master' of github.com:jonathan-buttner/elas…

9640639

…ticsearch into ml-eis-auth-master

DonalEvans reviewed Oct 20, 2025

View reviewed changes

jonathan-buttner added 2 commits October 20, 2025 15:30

Removing duplicate tests

426fcbe

Merge branch 'main' of github.com:elastic/elasticsearch into ml-eis-a…

4e9a320

…uth-master

jonathan-buttner requested a review from DonalEvans October 20, 2025 19:31

DonalEvans approved these changes Oct 20, 2025

View reviewed changes

davidkyle reviewed Oct 21, 2025

View reviewed changes

jonathan-buttner added 2 commits October 21, 2025 10:58

Handling empty list and duplicates

65e008f

Merge branch 'main' of github.com:elastic/elasticsearch into ml-eis-a…

d477b34

…uth-master

jonathan-buttner requested a review from davidkyle October 21, 2025 15:00

davidkyle reviewed Oct 22, 2025

View reviewed changes

jonathan-buttner and others added 4 commits October 22, 2025 10:57

Fixing empty delete

7cea6c1

Merge branch 'main' of github.com:elastic/elasticsearch into ml-eis-a…

96dbd62

…uth-master

Merge branch 'main' of github.com:elastic/elasticsearch into ml-eis-a…

da24021

…uth-master

Merge branch 'main' into ml-eis-auth-master

221dc34

davidkyle approved these changes Oct 29, 2025

View reviewed changes

Merge branch 'main' into ml-eis-auth-master

a994357

jonathan-buttner enabled auto-merge (squash) October 29, 2025 12:51

jonathan-buttner merged commit 22fe5be into elastic:main Oct 29, 2025
34 checks passed


		var storageResponses = responses.stream().map(StoreResponseWithIndexInfo::modelStoreResponse).toList();

		deleteModels(inferenceIdsToBeRemoved, ActionListener.running(() -> delegate.onResponse(storageResponses)));

Conversation

jonathan-buttner commented Oct 14, 2025

Uh oh!

jonathan-buttner Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Oct 17, 2025

Uh oh!

DonalEvans left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jonathan-buttner commented Oct 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidkyle left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jonathan-buttner Oct 14, 2025 •

edited

Loading