Expose byte offsets on XContentParser via getCurrentLocation() by quackaplop · Pull Request #143501 · elastic/elasticsearch

quackaplop · 2026-03-03T16:42:46Z

Summary

Adds byte-offset awareness to XContentParser, enabling callers to determine the exact byte range of any token or sub-structure in the underlying stream. This is a prerequisite for zero-copy byte slicing in JSON_EXTRACT (PR #142375), where extracting an object/array result can become a direct Arrays.copyOfRange() instead of walking every token via copyCurrentStructure().

Changes

New API surface:

XContentParser.getCurrentLocation() — returns the parser's current byte position (just past the last consumed byte), complementing the existing getTokenLocation() (start of current token). Together they define the byte range [tokenLocation, currentLocation) for slicing.
XContentLocation record gains a byteOffset field (third component). The existing 2-arg constructor defaults it to -1 (not available). A new UNDEFINED constant (0, 0, -1) is added for parsers with no underlying byte stream (e.g., MapXContentParser).

Implementation across parser hierarchy:

JsonXContentParser — passes through Jackson's getByteOffset() from both currentTokenLocation() and currentLocation()
FilterXContentParser — delegates getCurrentLocation() to the wrapped parser (all 11+ decorators inherit this)
DotExpandingXContentParser — returns cached location for synthetic dot-expanded tokens, delegates for original content
CompletionFieldMapper.MultiFieldParser — returns fixed locationOffset (no real positions for rewritten metadata)
MapXContentParser — returns XContentLocation.UNDEFINED (no underlying stream)
ParameterizableYamlXContentParser — delegates to inner parser

Byte slicing feasibility by format:

Format	Byte offsets?	Slicing safe?	Notes
JSON	Yes	Yes	Sliced sub-structure is valid standalone JSON
CBOR	Yes	Yes	Self-contained data items
SMILE	Yes	No	Back-references for repeated field names
YAML	No (`-1`)	No	Whitespace-sensitive, anchors/aliases

Tests

XContentLocationTests — record constructors, UNKNOWN vs UNDEFINED, equality including byteOffset
JsonXContentParserByteOffsetTests — 15 tests covering token offsets, object/array/scalar/boolean/null slicing, multi-byte UTF-8, surrogate pairs (4-byte U+1F389), empty containers, multi-line JSON
XContentParserTests — cross-format tests (YAML returns -1, CBOR/Smile have offsets), FilterXContentParser delegation
MapXContentParserTests — both location methods return UNDEFINED
DotExpandingXContentParserTests — getCurrentLocation() mirrors getTokenLocation() pattern (synthetic vs original tokens)
CompletionFieldMapperTests — getCurrentLocation() assertions alongside existing getTokenLocation()
GeoPointFieldMapperTests — getCurrentLocation() delegation verified

Closes #142873

elasticsearchmachine · 2026-03-03T16:43:12Z

Pinging @elastic/es-core-infra (Team:Core/Infra)

quackaplop · 2026-03-03T17:37:18Z

buildkite test this

smalyshev · 2026-03-05T00:20:57Z

Nit: the description says "returns XContentLocation.UNDEFINED" but it looks like it's actually XContentLocation.INVALID?

smalyshev · 2026-03-05T00:26:04Z

libs/x-content/src/main/java/org/elasticsearch/xcontent/XContentLocation.java

+     * Sentinel for parsers that have no underlying stream (e.g. {@code MapXContentParser}).
+     * Line and column are zero (outside the valid 1-based range), byte offset is {@code -1}.
+     */
+    public static final XContentLocation INVALID = new XContentLocation(0, 0, -1L);


Not sure I understand the reason why there's both UNKNOWN and INVALID and what's the difference between them. Maybe add some comment when each is supposed to be used?

Honestly, this is a bit of BCW mess. Maybe I should just remove INVALID - it is really just used in tests. Will do this.

libs/x-content/src/main/java/org/elasticsearch/xcontent/XContentLocation.java

smalyshev

lgtm, added some nitpicks

Add getCurrentLocation() to XContentParser, returning the parser's cursor position (how far it has consumed into the stream). Together with getTokenLocation() (token start), this enables zero-copy byte-range slicing of JSON values. - XContentLocation: add byteOffset field, UNKNOWN and UNDEFINED constants - JsonXContentParser: pass through Jackson's byte offset - FilterXContentParser: delegate getCurrentLocation() - DotExpandingXContentParser: return cached location for synthetic tokens - CompletionFieldMapper.MultiFieldParser: return fixed location - MapXContentParser: return UNDEFINED (no underlying stream) - ParameterizableYamlXContentParser: delegate getCurrentLocation()

End-to-end tests that navigate to values by key, nested path, and array index using standard streaming token walking, then extract via byte-offset slicing and verify against copyCurrentStructure(). Covers object/array/scalar extraction, mixed object+array paths, and pretty-printed JSON.

- Remove XContentLocation.INVALID constant; MapXContentParser returns inline new XContentLocation(0, 0) as it did before this PR. - Add hasValidLineNumber(), hasValidColumnNumber(), hasValidByteOffset() so callers don't need to know the magic sentinel values.

…ic#143501) * Expose byte offsets on XContentParser via getCurrentLocation() Add getCurrentLocation() to XContentParser, returning the parser's cursor position (how far it has consumed into the stream). Together with getTokenLocation() (token start), this enables zero-copy byte-range slicing of JSON values. - XContentLocation: add byteOffset field, UNKNOWN and UNDEFINED constants - JsonXContentParser: pass through Jackson's byte offset - FilterXContentParser: delegate getCurrentLocation() - DotExpandingXContentParser: return cached location for synthetic tokens - CompletionFieldMapper.MultiFieldParser: return fixed location - MapXContentParser: return UNDEFINED (no underlying stream) - ParameterizableYamlXContentParser: delegate getCurrentLocation() * Add changelog for elastic#143501 * Add streaming JSON navigation + byte-slicing tests End-to-end tests that navigate to values by key, nested path, and array index using standard streaming token walking, then extract via byte-offset slicing and verify against copyCurrentStructure(). Covers object/array/scalar extraction, mixed object+array paths, and pretty-printed JSON. * Rename XContentLocation.UNDEFINED to INVALID * Address review feedback: remove INVALID, add validity methods - Remove XContentLocation.INVALID constant; MapXContentParser returns inline new XContentLocation(0, 0) as it did before this PR. - Add hasValidLineNumber(), hasValidColumnNumber(), hasValidByteOffset() so callers don't need to know the magic sentinel values.

…traction Replace copyCurrentStructure() re-serialization with zero-copy byte slicing for JSON input. When the extracted value is an object, array, or number, slice bytes directly from the input buffer using XContentLocation.byteOffset() offsets (exposed in elastic#143501). Also refactors navigation from recursive descent to iterative loop, confining raw byte access to the extraction point. Adds JMH benchmarks for JSON_EXTRACT through the full eval pipeline.

…ic#143501) * Expose byte offsets on XContentParser via getCurrentLocation() Add getCurrentLocation() to XContentParser, returning the parser's cursor position (how far it has consumed into the stream). Together with getTokenLocation() (token start), this enables zero-copy byte-range slicing of JSON values. - XContentLocation: add byteOffset field, UNKNOWN and UNDEFINED constants - JsonXContentParser: pass through Jackson's byte offset - FilterXContentParser: delegate getCurrentLocation() - DotExpandingXContentParser: return cached location for synthetic tokens - CompletionFieldMapper.MultiFieldParser: return fixed location - MapXContentParser: return UNDEFINED (no underlying stream) - ParameterizableYamlXContentParser: delegate getCurrentLocation() * Add changelog for elastic#143501 * Add streaming JSON navigation + byte-slicing tests End-to-end tests that navigate to values by key, nested path, and array index using standard streaming token walking, then extract via byte-offset slicing and verify against copyCurrentStructure(). Covers object/array/scalar extraction, mixed object+array paths, and pretty-printed JSON. * Rename XContentLocation.UNDEFINED to INVALID * Address review feedback: remove INVALID, add validity methods - Remove XContentLocation.INVALID constant; MapXContentParser returns inline new XContentLocation(0, 0) as it did before this PR. - Add hasValidLineNumber(), hasValidColumnNumber(), hasValidByteOffset() so callers don't need to know the magic sentinel values.

…traction (#143702) * JSON_EXTRACT: zero-copy byte slicing for object, array, and number extraction Replace copyCurrentStructure() re-serialization with zero-copy byte slicing for JSON input. When the extracted value is an object, array, or number, slice bytes directly from the input buffer using XContentLocation.byteOffset() offsets (exposed in #143501). Also refactors navigation from recursive descent to iterative loop, confining raw byte access to the extraction point. Adds JMH benchmarks for JSON_EXTRACT through the full eval pipeline. * Add changelog for #143702 * [CI] Auto commit changes from spotless * Clean up navigation helpers to avoid threading unused parameters Navigation methods now only position the parser — they no longer carry builder, segments, depth, rawBytes, or rawOffset. * Use full variable names instead of abbreviations --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>

…traction (elastic#143702) * JSON_EXTRACT: zero-copy byte slicing for object, array, and number extraction Replace copyCurrentStructure() re-serialization with zero-copy byte slicing for JSON input. When the extracted value is an object, array, or number, slice bytes directly from the input buffer using XContentLocation.byteOffset() offsets (exposed in elastic#143501). Also refactors navigation from recursive descent to iterative loop, confining raw byte access to the extraction point. Adds JMH benchmarks for JSON_EXTRACT through the full eval pipeline. * Add changelog for elastic#143702 * [CI] Auto commit changes from spotless * Clean up navigation helpers to avoid threading unused parameters Navigation methods now only position the parser — they no longer carry builder, segments, depth, rawBytes, or rawOffset. * Use full variable names instead of abbreviations --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>

quackaplop requested a review from a team as a code owner March 3, 2026 16:42

quackaplop added >enhancement :Core/Infra/Core Core issues without another label Team:Core/Infra Meta label for core/infra team labels Mar 3, 2026

elasticsearchmachine added the v9.4.0 label Mar 3, 2026

quackaplop added a commit to quackaplop/elasticsearch that referenced this pull request Mar 3, 2026

Add changelog for elastic#143501

8be3cf4

quackaplop marked this pull request as draft March 3, 2026 18:18

quackaplop added a commit to quackaplop/elasticsearch that referenced this pull request Mar 4, 2026

Add changelog for elastic#143501

98ccc72

quackaplop force-pushed the feature/xcontent-location-offsets branch 2 times, most recently from 78ad2d7 to ef29423 Compare March 4, 2026 14:12

quackaplop added a commit to quackaplop/elasticsearch that referenced this pull request Mar 4, 2026

Add changelog for elastic#143501

9deb3ba

quackaplop marked this pull request as ready for review March 4, 2026 14:13

quackaplop requested review from Mikep86 and smalyshev March 4, 2026 22:57

smalyshev reviewed Mar 5, 2026

View reviewed changes

libs/x-content/src/main/java/org/elasticsearch/xcontent/XContentLocation.java Show resolved Hide resolved

smalyshev approved these changes Mar 5, 2026

View reviewed changes

quackaplop enabled auto-merge (squash) March 5, 2026 11:04

elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Mar 5, 2026

quackaplop added 5 commits March 5, 2026 12:25

Add changelog for elastic#143501

634d64f

Rename XContentLocation.UNDEFINED to INVALID

f481c7a

quackaplop force-pushed the feature/xcontent-location-offsets branch from 913b1c5 to 38501c9 Compare March 5, 2026 12:26

quackaplop merged commit 71cf379 into elastic:main Mar 5, 2026
34 checks passed

quackaplop mentioned this pull request Mar 5, 2026

JSON_EXTRACT: zero-copy byte slicing for object, array, and number extraction #143702

Merged

prwhelan mentioned this pull request Mar 6, 2026

[ML] Wait for cluster state in test #143767

Merged

prwhelan mentioned this pull request Mar 9, 2026

[Transform] Disable PIT for CPS #143876

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose byte offsets on XContentParser via getCurrentLocation()#143501

Expose byte offsets on XContentParser via getCurrentLocation()#143501
quackaplop merged 5 commits intoelastic:mainfrom
quackaplop:feature/xcontent-location-offsets

quackaplop commented Mar 3, 2026

Uh oh!

elasticsearchmachine commented Mar 3, 2026

Uh oh!

quackaplop commented Mar 3, 2026

Uh oh!

smalyshev commented Mar 5, 2026

Uh oh!

smalyshev Mar 5, 2026

Uh oh!

quackaplop Mar 5, 2026

Uh oh!

Uh oh!

smalyshev left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

quackaplop commented Mar 3, 2026

Summary

Changes

Tests

Uh oh!

elasticsearchmachine commented Mar 3, 2026

Uh oh!

quackaplop commented Mar 3, 2026

Uh oh!

smalyshev commented Mar 5, 2026

Uh oh!

smalyshev Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

quackaplop Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

smalyshev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants