Skip to content

Modify numeric FieldMappers to enable SkipList #18066

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Modify numeric FieldMappers to enable SkipList #18066

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Pass index settings to system ingest processor factories. ([#18708](https://github.com/opensearch-project/OpenSearch/pull/18708))
- Include named queries from rescore contexts in matched_queries array ([#18697](https://github.com/opensearch-project/OpenSearch/pull/18697))
- Add the configurable limit on rule cardinality ([#18663](https://github.com/opensearch-project/OpenSearch/pull/18663))
- Modify numeric FieldMappers to enable SkipList ([#18066](https://github.com/opensearch-project/OpenSearch/pull/18066))

### Changed
- Update Subject interface to use CheckedRunnable ([#18570](https://github.com/opensearch-project/OpenSearch/issues/18570))
Expand Down
4 changes: 2 additions & 2 deletions ...test/java/org/opensearch/search/aggregations/matrix/stats/MatrixStatsAggregatorTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,10 +110,10 @@ public void testTwoFields() throws Exception {
for (int docId = 0; docId < numDocs; docId++) {
Document document = new Document();
fieldAValues[docId] = randomDouble();
document.add(new SortedNumericDocValuesField(fieldA, NumericUtils.doubleToSortableLong(fieldAValues[docId])));
document.add(SortedNumericDocValuesField.indexedField(fieldA, NumericUtils.doubleToSortableLong(fieldAValues[docId])));
Copy link
Contributor

@bharath-techie bharath-techie May 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Lets not enable by default ? One option is to introduce a new parameter for skipper similar to docValues / index / stored_fields to the user. [ Of course docValues must be true for skipper to be true ]

  2. Also we must see if we need to enable in experimental mode for now until search takes advantage of this data structure.

  3. Do we have any benchmarks in terms of storage if we enable for all fields like this ? We can try atleast in few OSB workloads with and without this change.


fieldBValues[docId] = randomDouble();
document.add(new SortedNumericDocValuesField(fieldB, NumericUtils.doubleToSortableLong(fieldBValues[docId])));
document.add(SortedNumericDocValuesField.indexedField(fieldB, NumericUtils.doubleToSortableLong(fieldBValues[docId])));
indexWriter.addDocument(document);
}

Expand Down
6 changes: 3 additions & 3 deletions ...es/mapper-extras/src/test/java/org/opensearch/index/mapper/ScaledFloatFieldTypeTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,9 +186,9 @@ public void testFieldData() throws IOException {
Directory dir = newDirectory();
IndexWriter w = new IndexWriter(dir, new IndexWriterConfig(null));
Document doc = new Document();
doc.add(new SortedNumericDocValuesField("scaled_float1", 10));
doc.add(new SortedNumericDocValuesField("scaled_float2", 5));
doc.add(new SortedNumericDocValuesField("scaled_float2", 12));
doc.add(SortedNumericDocValuesField.indexedField("scaled_float1", 10));
doc.add(SortedNumericDocValuesField.indexedField("scaled_float2", 5));
doc.add(SortedNumericDocValuesField.indexedField("scaled_float2", 12));
w.addDocument(doc);
try (DirectoryReader reader = DirectoryReader.open(w)) {
// single-valued
Expand Down
4 changes: 2 additions & 2 deletions ...-join/src/test/java/org/opensearch/join/aggregations/ChildrenToParentAggregatorTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ private static List<Field> createParentDocument(String id, int value) {
new StringField(IdFieldMapper.NAME, Uid.encodeId(id), Field.Store.NO),
new StringField("join_field", PARENT_TYPE, Field.Store.NO),
createJoinField(PARENT_TYPE, id),
new SortedNumericDocValuesField("number", value)
SortedNumericDocValuesField.indexedField("number", value)
);
}

Expand All @@ -283,7 +283,7 @@ private static List<Field> createChildDocument(String childId, String parentId,
new StringField(IdFieldMapper.NAME, Uid.encodeId(childId), Field.Store.NO),
new StringField("join_field", CHILD_TYPE, Field.Store.NO),
createJoinField(PARENT_TYPE, parentId),
new SortedNumericDocValuesField("subNumber", value)
SortedNumericDocValuesField.indexedField("subNumber", value)
);
}

Expand Down
2 changes: 1 addition & 1 deletion ...-join/src/test/java/org/opensearch/join/aggregations/ParentToChildrenAggregatorTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ private static List<Field> createChildDocument(String childId, String parentId,
new StringField(IdFieldMapper.NAME, Uid.encodeId(childId), Field.Store.NO),
new StringField("join_field", CHILD_TYPE, Field.Store.NO),
createJoinField(PARENT_TYPE, parentId),
new SortedNumericDocValuesField("number", value)
SortedNumericDocValuesField.indexedField("number", value)
);
}

Expand Down
2 changes: 1 addition & 1 deletion server/src/main/java/org/opensearch/index/mapper/DateFieldMapper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -810,7 +810,7 @@ protected void parseCreateField(ParseContext context) throws IOException {
context.doc().add(new LongPoint(fieldType().name(), timestamp));
}
if (hasDocValues) {
context.doc().add(new SortedNumericDocValuesField(fieldType().name(), timestamp));
context.doc().add(SortedNumericDocValuesField.indexedField(fieldType().name(), timestamp));
} else if (store || indexed) {
createFieldNamesField(context);
}
Expand Down
12 changes: 6 additions & 6 deletions server/src/main/java/org/opensearch/index/mapper/NumberFieldMapper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -411,7 +411,7 @@ public List<Field> createFields(String name, Number value, boolean indexed, bool
fields.add(new HalfFloatPoint(name, value.floatValue()));
}
if (docValued) {
fields.add(new SortedNumericDocValuesField(name, HalfFloatPoint.halfFloatToSortableShort(value.floatValue())));
fields.add(SortedNumericDocValuesField.indexedField(name, HalfFloatPoint.halfFloatToSortableShort(value.floatValue())));
}
if (stored) {
fields.add(new StoredField(name, value.floatValue()));
Expand Down Expand Up @@ -587,7 +587,7 @@ public List<Field> createFields(String name, Number value, boolean indexed, bool
fields.add(new FloatPoint(name, value.floatValue()));
}
if (docValued) {
fields.add(new SortedNumericDocValuesField(name, NumericUtils.floatToSortableInt(value.floatValue())));
fields.add(SortedNumericDocValuesField.indexedField(name, NumericUtils.floatToSortableInt(value.floatValue())));
}
if (stored) {
fields.add(new StoredField(name, value.floatValue()));
Expand Down Expand Up @@ -739,7 +739,7 @@ public List<Field> createFields(String name, Number value, boolean indexed, bool
fields.add(new DoublePoint(name, value.doubleValue()));
}
if (docValued) {
fields.add(new SortedNumericDocValuesField(name, NumericUtils.doubleToSortableLong(value.doubleValue())));
fields.add(SortedNumericDocValuesField.indexedField(name, NumericUtils.doubleToSortableLong(value.doubleValue())));
}
if (stored) {
fields.add(new StoredField(name, value.doubleValue()));
Expand Down Expand Up @@ -1108,7 +1108,7 @@ public List<Field> createFields(String name, Number value, boolean indexed, bool
fields.add(new IntPoint(name, value.intValue()));
}
if (docValued) {
fields.add(new SortedNumericDocValuesField(name, value.intValue()));
fields.add(SortedNumericDocValuesField.indexedField(name, value.intValue()));
}
if (stored) {
fields.add(new StoredField(name, value.intValue()));
Expand Down Expand Up @@ -1250,7 +1250,7 @@ public List<Field> createFields(String name, Number value, boolean indexed, bool
fields.add(new LongPoint(name, value.longValue()));
}
if (docValued) {
fields.add(new SortedNumericDocValuesField(name, value.longValue()));
fields.add(SortedNumericDocValuesField.indexedField(name, value.longValue()));
}
if (stored) {
fields.add(new StoredField(name, value.longValue()));
Expand Down Expand Up @@ -1384,7 +1384,7 @@ public List<Field> createFields(String name, Number value, boolean indexed, bool
}

if (docValued) {
fields.add(new SortedNumericDocValuesField(name, v.longValue()));
fields.add(SortedNumericDocValuesField.indexedField(name, v.longValue()));
}

if (stored) {
Expand Down
38 changes: 19 additions & 19 deletions ...g/opensearch/index/codec/composite912/datacube/startree/StarTreeDocValuesFormatTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,29 +57,29 @@ public void testStarTreeDocValues() throws IOException {
conf.setMergePolicy(newLogMergePolicy());
RandomIndexWriter iw = new RandomIndexWriter(random(), directory, conf);
Document doc = new Document();
doc.add(new SortedNumericDocValuesField("unsignedLongDimension", 10));
doc.add(new SortedNumericDocValuesField("sndv", 1));
doc.add(new SortedNumericDocValuesField("dv1", 1));
doc.add(new SortedNumericDocValuesField("field1", -1));
doc.add(SortedNumericDocValuesField.indexedField("unsignedLongDimension", 10));
doc.add(SortedNumericDocValuesField.indexedField("sndv", 1));
doc.add(SortedNumericDocValuesField.indexedField("dv1", 1));
doc.add(SortedNumericDocValuesField.indexedField("field1", -1));
iw.addDocument(doc);
doc = new Document();
doc.add(new SortedNumericDocValuesField("unsignedLongDimension", 10));
doc.add(new SortedNumericDocValuesField("sndv", 1));
doc.add(new SortedNumericDocValuesField("dv1", 1));
doc.add(new SortedNumericDocValuesField("field1", -1));
doc.add(SortedNumericDocValuesField.indexedField("unsignedLongDimension", 10));
doc.add(SortedNumericDocValuesField.indexedField("sndv", 1));
doc.add(SortedNumericDocValuesField.indexedField("dv1", 1));
doc.add(SortedNumericDocValuesField.indexedField("field1", -1));
iw.addDocument(doc);
doc = new Document();
iw.forceMerge(1);
doc.add(new SortedNumericDocValuesField("unsignedLongDimension", -20));
doc.add(new SortedNumericDocValuesField("sndv", 2));
doc.add(new SortedNumericDocValuesField("dv1", 2));
doc.add(new SortedNumericDocValuesField("field1", -2));
doc.add(SortedNumericDocValuesField.indexedField("unsignedLongDimension", -20));
doc.add(SortedNumericDocValuesField.indexedField("sndv", 2));
doc.add(SortedNumericDocValuesField.indexedField("dv1", 2));
doc.add(SortedNumericDocValuesField.indexedField("field1", -2));
iw.addDocument(doc);
doc = new Document();
doc.add(new SortedNumericDocValuesField("unsignedLongDimension", -20));
doc.add(new SortedNumericDocValuesField("sndv", 2));
doc.add(new SortedNumericDocValuesField("dv1", 2));
doc.add(new SortedNumericDocValuesField("field1", -2));
doc.add(SortedNumericDocValuesField.indexedField("unsignedLongDimension", -20));
doc.add(SortedNumericDocValuesField.indexedField("sndv", 2));
doc.add(SortedNumericDocValuesField.indexedField("dv1", 2));
doc.add(SortedNumericDocValuesField.indexedField("field1", -2));
iw.addDocument(doc);
iw.forceMerge(1);
iw.close();
Expand Down Expand Up @@ -169,14 +169,14 @@ public void testStarTreeDocValuesWithDeletions() throws IOException {
Document doc = new Document();
doc.add(new StringField("_id", id, Field.Store.YES));
int fieldValue = random().nextInt(5) + 1;
doc.add(new SortedNumericDocValuesField("field", fieldValue));
doc.add(SortedNumericDocValuesField.indexedField("field", fieldValue));

int sndvValue = random().nextInt(3);

doc.add(new SortedNumericDocValuesField("sndv", sndvValue));
doc.add(SortedNumericDocValuesField.indexedField("sndv", sndvValue));
int dvValue = random().nextInt(3);

doc.add(new SortedNumericDocValuesField("dv", dvValue));
doc.add(SortedNumericDocValuesField.indexedField("dv", dvValue));
map.put(sndvValue + "-" + dvValue, fieldValue + map.getOrDefault(sndvValue + "-" + dvValue, 0));
doc.add(new NumericDocValuesField("field-ndv", fieldValue));

Expand Down
Loading
Loading