Skip to content

sparse_vector field support #168186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
carlosdelest merged 10 commits into from
Oct 11, 2023
Merged

sparse_vector field support #168186

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
1b87c3c
Add sparse_vector field support
carlosdelest Oct 5, 2023
ae9df2f
ML pipelines create sparse_vector fields instead of rank_features
carlosdelest Oct 5, 2023
d8086e8
dense_vector icon looks better on a sparse_vector field type
carlosdelest Oct 6, 2023
316ca69
Fix test snapshot
carlosdelest Oct 6, 2023
9a4c78f
Merge branch 'main' into carlosdelest/sparse_vector_field_support
carlosdelest Oct 6, 2023
51bf42f
Merge branch 'main' into carlosdelest/sparse_vector_field_support
carlosdelest Oct 6, 2023
995e2de
Merge branch 'main' into carlosdelest/sparse_vector_field_support
carlosdelest Oct 6, 2023
0a29bfe
Add sparse vector to mappings editor
carlosdelest Oct 10, 2023
3b9cb51
Merge remote-tracking branch 'carlosdelest/carlosdelest/sparse_vector…
carlosdelest Oct 10, 2023
829bf71
Merge branch 'main' into carlosdelest/sparse_vector_field_support
carlosdelest Oct 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/kbn-field-utils/src/utils/field_types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ export enum KNOWN_FIELD_TYPES {
RANK_FEATURES = 'rank_features',
POINT = 'point',
SHAPE = 'shape',
SPARSE_VECTOR = 'sparse_vector',
STRING = 'string',
TEXT = 'text',
VERSION = 'version',
Expand Down
4 changes: 4 additions & 0 deletions packages/kbn-field-utils/src/utils/get_field_type_description.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,10 @@ export function getFieldTypeDescription(type?: string) {
return i18n.translate('fieldUtils.fieldNameDescription.shapeField', {
defaultMessage: 'Arbitrary cartesian geometries.',
});
case KNOWN_FIELD_TYPES.SPARSE_VECTOR:
return i18n.translate('fieldUtils.fieldNameDescription.sparseVectorField', {
defaultMessage: 'Records sparse vectors of float values.',
});
case KNOWN_FIELD_TYPES.STRING:
return i18n.translate('fieldUtils.fieldNameDescription.stringField', {
defaultMessage: 'Full text such as the body of an email or a product description.',
Expand Down
4 changes: 4 additions & 0 deletions packages/kbn-field-utils/src/utils/get_field_type_name.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,10 @@ export function getFieldTypeName(type?: string) {
return i18n.translate('fieldUtils.fieldNameIcons.shapeFieldAriaLabel', {
defaultMessage: 'Shape',
});
case KNOWN_FIELD_TYPES.SPARSE_VECTOR:
return i18n.translate('fieldUtils.fieldNameIcons.sparseVectorFieldAriaLabel', {
defaultMessage: 'Sparse vector',
});
case KNOWN_FIELD_TYPES.STRING:
return i18n.translate('fieldUtils.fieldNameIcons.stringFieldAriaLabel', {
defaultMessage: 'String',
Expand Down
10 changes: 10 additions & 0 deletions packages/kbn-react-field/src/field_icon/__snapshots__/field_icon.test.tsx.snap
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions packages/kbn-react-field/src/field_icon/field_icon.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ export interface FieldIconProps extends Omit<EuiTokenProps, 'iconType'> {
| '_source'
| 'point'
| 'shape'
| 'sparse_vector'
| 'string'
| string
| 'nested'
Expand Down Expand Up @@ -69,6 +70,7 @@ export const typeToEuiIconMap: Partial<Record<string, EuiTokenProps>> = {
_source: { iconType: 'editorCodeBlock', color: 'gray' },
point: { iconType: 'tokenShape' }, // there is no separate icon for `point` yet
shape: { iconType: 'tokenShape' },
sparse_vector: { iconType: 'tokenDenseVector' },
Copy link
Copy Markdown
Member

@kderusso kderusso Oct 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question here, do we want to use tokenDenseVector or should we create a new token for sparse vector?

Copy link
Copy Markdown
Contributor

@jughosta jughosta Oct 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andreadelrio do we have an icon we can use or add to eui?

Copy link
Copy Markdown
Member Author

@carlosdelest carlosdelest Oct 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Andrea is already pinged in elastic/eui#7257 (comment). 😉 My plan is to open a separate PR when that's ready.

string: { iconType: 'tokenString' },
text: { iconType: 'tokenString' },
keyword: { iconType: 'tokenKeyword' },
Expand Down
4 changes: 2 additions & 2 deletions ...ise_search/server/lib/indices/pipelines/ml_inference/update_ml_inference_mappings.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ describe('updateMlInferenceMappings', () => {
input_one_expanded: {
properties: {
predicted_value: {
type: 'rank_features',
type: 'sparse_vector',
},
model_id: {
type: 'keyword',
Expand All @@ -54,7 +54,7 @@ describe('updateMlInferenceMappings', () => {
input_two_expanded: {
properties: {
predicted_value: {
type: 'rank_features',
type: 'sparse_vector',
},
model_id: {
type: 'keyword',
Expand Down
4 changes: 2 additions & 2 deletions ...terprise_search/server/lib/indices/pipelines/ml_inference/update_ml_inference_mappings.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ const generateTextExpansionMappingProperties = (sourceFields: string[], targetFi
type: 'keyword',
},
predicted_value: {
type: 'rank_features',
type: 'sparse_vector',
},
},
},
Expand All @@ -97,7 +97,7 @@ const formDefaultElserMappingProps = (sourceFields: string[]) => {
type: 'keyword',
},
predicted_value: {
type: 'rank_features',
type: 'sparse_vector',
},
},
},
Expand Down
18 changes: 18 additions & 0 deletions ...agement/public/application/components/mappings_editor/constants/data_types_definition.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -851,6 +851,23 @@ export const TYPE_DEFINITION: { [key in DataType]: DataTypeDefinition } = {
</p>
),
},
sparse_vector: {
Copy link
Copy Markdown
Contributor

@ElenaStoeva ElenaStoeva Oct 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @carlosdelest, just a question: why are we adding these changes to the Index management plugin?
Did you mean to add support for the sparse_vector field type in the Mappings editor?

Screenshot 2023-10-06 at 17 52 47

Copy link
Copy Markdown
Member Author

@carlosdelest carlosdelest Oct 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ElenaStoeva , thanks for pointing that out - I totally missed it.

I've made the changes here.

image

Copy link
Copy Markdown
Contributor

@ElenaStoeva ElenaStoeva Oct 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for clarifying! Tested locally and LGTM.

label: i18n.translate('xpack.idxMgmt.mappingsEditor.dataType.sparseVectorDescription', {
defaultMessage: 'Sparse vector',
}),
value: 'sparse_vector',
documentation: {
main: 'sparse-vector.html',
},
description: () => (
<p>
<FormattedMessage
id="xpack.idxMgmt.mappingsEditor.dataType.sparseVectorLongDescription"
defaultMessage="Sparse vector fields index features and weights, useful for document scoring."
/>
</p>
),
},
point: {
label: i18n.translate('xpack.idxMgmt.mappingsEditor.dataType.pointDescription', {
defaultMessage: 'Point',
Expand Down Expand Up @@ -958,6 +975,7 @@ export const MAIN_TYPES: MainType[] = [
'rank_features',
'search_as_you_type',
'shape',
'sparse_vector',
'text',
'token_count',
'histogram',
Expand Down
1 change: 1 addition & 0 deletions ...s/index_management/public/application/components/mappings_editor/types/document_fields.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ export type MainType =
| 'rank_features'
| 'shape'
| 'search_as_you_type'
| 'sparse_vector'
| 'date'
| 'date_nanos'
| 'geo_point'
Expand Down