Store flattened field data in binary doc values #140246
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jordan-powers
added
>feature
:StorageEngine/Mapping
Collaborator
|
Hi @jordan-powers, I've created a changelog YAML for you. |
…s-2' into flattened-field-binary-doc-values-2
jordan-powers
commented
Jan 7, 2026
server/src/main/java/org/elasticsearch/index/fielddata/MultiValuedSortedBinaryDocValues.java
Outdated
Show resolved
Hide resolved
jordan-powers
commented
Jan 7, 2026
server/src/test/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldMapperTests.java
Outdated
Show resolved
Hide resolved
Collaborator
|
Hi @jordan-powers, I've updated the changelog YAML for you. |
Collaborator
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
martijnvg
reviewed
Jan 9, 2026
Member
martijnvg
left a comment
martijnvg
left a comment
There was a problem hiding this comment.
Thanks @jordan-powers this looks good.
Before this gets merged, maybe first open a pr that adds a rolling upgrade test for the flattened field type? (Similar to TextRollingUpgradeIT?)
server/src/main/java/org/elasticsearch/index/mapper/MultiValuedBinaryDocValuesField.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/search/aggregations/support/ValuesSourceConfigTests.java
Show resolved
Hide resolved
jordan-powers
added a commit
to jordan-powers/elasticsearch
that referenced
this pull request
Jan 10, 2026
This PR adds a test to compare the synthetic source produced by flattened fields using the TSDB codec against flattened fields not using that codec. The test generates 32 random documents, indexes them into two indices (one using the codec, the other not using the codec), then retrieves the documents from the two indices and compares them. This is a test for elastic#140246, since once that PR is merged, flattened fields using the TSDB codec will use binary doc values while flattened fields using the default codec will continue to use sorted set doc values.
jordan-powers
added a commit
that referenced
this pull request
Jan 12, 2026
This PR adds a test to compare the synthetic source produced by flattened fields using the TSDB codec against flattened fields not using that codec. The test generates 32 random documents, indexes them into two indices (one using the codec, the other not using the codec), then retrieves the documents from the two indices and compares them. This is a test for #140246, since once that PR is merged, flattened fields using the TSDB codec will use binary doc values while flattened fields using the default codec will continue to use sorted set doc values.
Contributor
Author
|
Opened #140611 to add a rolling upgrade test for flattened fields. |
martijnvg
approved these changes
Jan 15, 2026
| * This class wraps the field data that is built directly on the keyed flattened field, | ||
| * and filters out values whose prefix doesn't match the requested key. | ||
| */ | ||
| public class BinaryKeyedFlattenedLeafFieldData implements LeafFieldData { |
| } | ||
| } | ||
|
|
||
| public static class BinaryKeyedFlattenedFieldData implements IndexFieldData<LeafFieldData> { |
jordan-powers
added a commit
to jordan-powers/elasticsearch
that referenced
this pull request
Jan 20, 2026
As of elastic#140246, flattened fields might use either binary or sorted set doc values, and the associated synthetic field loader can handle either format. This patch updates the name and javadoc of that field loader to document this behavior.
jordan-powers
added a commit
that referenced
this pull request
Jan 21, 2026
As of #140246, flattened fields might use either binary or sorted set doc values, and the associated synthetic field loader can handle either format. This patch updates the name and javadoc of that field loader to document this behavior.
spinscale
pushed a commit
to spinscale/elasticsearch
that referenced
this pull request
Jan 21, 2026
…c#140489) This PR adds a test to compare the synthetic source produced by flattened fields using the TSDB codec against flattened fields not using that codec. The test generates 32 random documents, indexes them into two indices (one using the codec, the other not using the codec), then retrieves the documents from the two indices and compares them. This is a test for elastic#140246, since once that PR is merged, flattened fields using the TSDB codec will use binary doc values while flattened fields using the default codec will continue to use sorted set doc values.
spinscale
pushed a commit
to spinscale/elasticsearch
that referenced
this pull request
Jan 21, 2026
This PR updates the FlattenedFieldMapper to use binary doc values instead of sorted set doc values
spinscale
pushed a commit
to spinscale/elasticsearch
that referenced
this pull request
Jan 21, 2026
As of elastic#140246, flattened fields might use either binary or sorted set doc values, and the associated synthetic field loader can handle either format. This patch updates the name and javadoc of that field loader to document this behavior.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR updates the FlattenedFieldMapper to use binary doc values instead of sorted set doc values.