Automatically map floats as dense vector by kderusso · Pull Request #98512 · elastic/elasticsearch

kderusso · 2023-08-15T20:38:08Z

Resolves #97532
Resolves #98512

Automatically maps float arrays between size of 128 and the max dense vector size to dense_vector type with a default similarity of cosine.

kderusso · 2023-08-16T14:55:17Z

@elasticmachine test this please

elasticsearchmachine · 2023-08-22T13:09:45Z

Hi @kderusso, I've created a changelog YAML for you.

elasticsearchmachine · 2023-08-22T13:12:54Z

Pinging @elastic/es-search (Team:Search)

elasticsearchmachine · 2023-08-22T13:12:55Z

Pinging @elastic/ent-search-eng (Team:Enterprise Search)

demjened

LGTM with 1 question

server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java

saikatsarkar056 · 2023-08-22T16:16:14Z

From this issue: #97532

It's been suggested to automatically map float arrays as dense vectors when there are more than 100 values

Where do we check if the dense vector has more than 100 values? 🤔

benwtrent

I think adding my suggested tests will show some edges (especially around dynamic_templates).

...yamlRestTest/resources/rest-api-spec/test/search.vectors/60_dense_vector_dynamic_mapping.yml

kderusso · 2023-08-23T18:02:10Z

@elasticmachine test this please

kderusso · 2023-08-23T19:51:02Z

@elasticmachine test this please

benwtrent

I really like all the additional tests. Covers us well.

...yamlRestTest/resources/rest-api-spec/test/search.vectors/60_dense_vector_dynamic_mapping.yml

server/src/main/java/org/elasticsearch/index/mapper/DocumentParserContext.java

server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java

...yamlRestTest/resources/rest-api-spec/test/search.vectors/60_dense_vector_dynamic_mapping.yml

server/src/main/java/org/elasticsearch/index/mapper/DocumentParserContext.java

server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java

carlosdelest

Great work! 👏

docs/reference/mapping/types/dense-vector.asciidoc

...yamlRestTest/resources/rest-api-spec/test/search.vectors/60_dense_vector_dynamic_mapping.yml

server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java

server/src/test/java/org/elasticsearch/index/mapper/DynamicMappingTests.java

benwtrent

Great work on finding the issue with the dynamic mapping and dims!

Two more comments. This PR is looking good.

server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java

server/src/main/java/org/elasticsearch/index/mapper/DocumentParserContext.java

server/src/main/java/org/elasticsearch/index/mapper/DocumentParser.java

server/src/main/java/org/elasticsearch/index/mapper/DocumentParserContext.java

benwtrent

I think this is good. We got some great test coverage and it all seems ✅ .

Once all the tests are green and full-bwc-test run, I think this is good.

…101967) After #98512 we incorrectly attempt to map an array of any single value type to dense_vector. Instead, we should validate that ALL mappers are numeric and that ALL of them are `float`. closes: #101965

…1965 (elastic#101967) After elastic#98512 we incorrectly attempt to map an array of any single value type to dense_vector. Instead, we should validate that ALL mappers are numeric and that ALL of them are `float`. closes: elastic#101965

…1965 (#101967) (#101975) * Fix incorrect dynamic mapping for non-numeric-value arrays #101965 (#101967) After #98512 we incorrectly attempt to map an array of any single value type to dense_vector. Instead, we should validate that ALL mappers are numeric and that ALL of them are `float`. closes: #101965 * Update rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/60_dense_vector_dynamic_mapping.yml

kderusso added :EnterpriseSearch/Application Enterprise Search Team:Enterprise Search Meta label for Enterprise Search team v8.11.0 labels Aug 15, 2023

kderusso force-pushed the kderusso/dense-vector-add-default-mappings branch 4 times, most recently from b99e0aa to 30348f5 Compare August 18, 2023 19:42

kderusso changed the title ~~WIP: Automatically map floats as dense vector~~ Automatically map floats as dense vector Aug 21, 2023

kderusso added the >feature label Aug 22, 2023

kderusso mentioned this pull request Aug 22, 2023

Add default dynamic mappings for dense_vector #97532

Closed

kderusso marked this pull request as ready for review August 22, 2023 13:12

kderusso added :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Meta label for search team labels Aug 22, 2023

kderusso requested review from a team and jimczi August 22, 2023 13:12

demjened approved these changes Aug 22, 2023

View reviewed changes

server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java Outdated Show resolved Hide resolved

saikatsarkar056 reviewed Aug 22, 2023

View reviewed changes

server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java Outdated Show resolved Hide resolved

benwtrent self-requested a review August 22, 2023 16:29

benwtrent reviewed Aug 22, 2023

View reviewed changes

...yamlRestTest/resources/rest-api-spec/test/search.vectors/60_dense_vector_dynamic_mapping.yml Outdated Show resolved Hide resolved

kderusso requested a review from benwtrent August 23, 2023 17:39

benwtrent reviewed Aug 24, 2023

View reviewed changes

kderusso mentioned this pull request Aug 28, 2023

Support setting the number of dimensions dynamically on dense_vector #97530

Closed

kderusso linked an issue Aug 28, 2023 that may be closed by this pull request

Support setting the number of dimensions dynamically on dense_vector #97530

Closed

kderusso added 4 commits August 31, 2023 16:30

Cleanup

b814adb

Spotless

2b42034

Cleanup

e0fa759

Committing test authored by @benwtrent

1cccc2d

kderusso force-pushed the kderusso/dense-vector-add-default-mappings branch from 587f052 to 1cccc2d Compare August 31, 2023 20:30

carlosdelest approved these changes Sep 1, 2023

View reviewed changes

kderusso added 4 commits September 6, 2023 09:24

Committing working subobject before cleanup

676d343

Cleanup

ad0be33

Spotless

71918af

Fix typos in yaml test

8934f8e

benwtrent reviewed Sep 6, 2023

View reviewed changes

server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java Outdated Show resolved Hide resolved

server/src/main/java/org/elasticsearch/index/mapper/DocumentParserContext.java Outdated Show resolved Hide resolved

PR feedback

84676ad

kderusso requested a review from benwtrent September 6, 2023 18:16

benwtrent reviewed Sep 6, 2023

View reviewed changes

server/src/main/java/org/elasticsearch/index/mapper/DocumentParser.java Outdated Show resolved Hide resolved

benwtrent reviewed Sep 6, 2023

View reviewed changes

server/src/main/java/org/elasticsearch/index/mapper/DocumentParserContext.java Outdated Show resolved Hide resolved

PR feedback

49f4577

kderusso requested review from benwtrent and carlosdelest September 6, 2023 18:45

benwtrent added the test-full-bwc Trigger full BWC version matrix tests label Sep 6, 2023

benwtrent approved these changes Sep 6, 2023

View reviewed changes

kderusso merged commit 258d0cb into elastic:main Sep 6, 2023

benwtrent mentioned this pull request Nov 9, 2023

Fix incorrect dynamic mapping for non-numeric-value arrays #101965 #101967

Merged

ChrisHegarty mentioned this pull request Nov 21, 2023

Dense Vector without dims property causes NullPointerException when calling _cluster/stats #102416

Closed

kderusso deleted the kderusso/dense-vector-add-default-mappings branch July 8, 2024 17:26

This was referenced Mar 6, 2026

[bug] Dynamic Templates with path_match Ignored by Automatic dense_vector Mapping #143732

Closed

fix: dynamic template vector array is overridden by automatic dense_vector mapping #143733

Merged

Conversation

kderusso commented Aug 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kderusso commented Aug 16, 2023

Uh oh!

elasticsearchmachine commented Aug 22, 2023

Uh oh!

elasticsearchmachine commented Aug 22, 2023

Uh oh!

elasticsearchmachine commented Aug 22, 2023

Uh oh!

demjened left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

saikatsarkar056 commented Aug 22, 2023

Uh oh!

benwtrent left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kderusso commented Aug 23, 2023

Uh oh!

kderusso commented Aug 23, 2023

Uh oh!

benwtrent left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

carlosdelest left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

benwtrent left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

benwtrent left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

kderusso commented Aug 15, 2023 •

edited

Loading