fix(mapper): handle VOID entries in _ignored_source with FLS by salvatore-campagna · Pull Request #141506 · elastic/elasticsearch

salvatore-campagna · 2026-01-29T11:07:52Z

Problem

When Field Level Security (FLS) is active on an index using synthetic source, any field with a copy_to target causes search requests to fail with:

[-1:12] Unexpected Break (0xFF) token in definite length (-1) Object at [Source: (byte[])[12 bytes]; byte offset: #12]

The failure path is:

During indexing, copy_to target fields are recorded in _ignored_source as VOID entries (XContentDataHelper.voidValue()): markers indicating the field exists but its data is stored elsewhere (in the copy_to source field).
When a query runs with FLS enabled, FieldSubsetReader iterates over _ignored_source stored fields and calls IgnoredSourceFieldMapper.decodeAsMap for each entry to decide which fields to keep or strip based on FLS access permissions.
decodeAsMap builds a CBOR object containing the field name but writes no value for VOID entries (XContentDataHelper.decodeAndWrite is a no-op). When XContentHelper.convertToMap tries to parse this malformed CBOR, the Jackson CBOR parser encounters an unexpected Break token (0xFF) and throws.

Fix

The core fix is in nameValueToMapped(), which both encoding paths share. It now checks XContentDataHelper.isDataPresent(nameValue.value()) before attempting to parse, returning null for VOID entries instead of building malformed CBOR.

Both encoding paths call nameValueToMapped() through their respective decodeAsMap methods, so both are protected by the same check. The only difference is how the null propagates:

Coalesced path: CoalescedIgnoredSourceEncoding.decodeAsMap iterates over entries and skips null results, filtering VOID entries out of the returned list.
Legacy path: LegacyIgnoredSourceEncoding.decodeAsMap returns null directly, and filterValue propagates the null to FieldSubsetReader, which drops the entry.

In both cases, VOID entries carry no field data, so omitting them is correct.

Tests

testCoalescedDecodeAsMapReturnsNullForVoidEntry: unit test verifying CoalescedIgnoredSourceEncoding.decodeAsMap filters out VOID entries, returning an empty list.
testLegacyDecodeAsMapReturnsNullForVoidEntry: unit test verifying LegacyIgnoredSourceEncoding.decodeAsMap returns null for a VOID entry.
testSyntheticSourceWithCopyToAndFLSCoalesced: integration test exercising FieldSubsetReader with copy_to, synthetic source, and FLS using the coalesced encoding format.
testSyntheticSourceWithCopyToAndFLSLegacy: same integration test but using the legacy encoding format (pre-IGNORED_SOURCE_COALESCED_ENTRIES_WITH_FF).
YAML REST tests: two end-to-end scenarios: one with API key FLS + synthetic source + copy_to, and one covering the skip_ignored_source_read workaround.

./gradlew :server:test --tests "org.elasticsearch.index.mapper.IgnoredSourceFieldMapperTests.testCoalescedDecodeAsMapReturnsNullForVoidEntry"
./gradlew :server:test --tests "org.elasticsearch.index.mapper.IgnoredSourceFieldMapperTests.testLegacyDecodeAsMapReturnsNullForVoidEntry"
./gradlew :x-pack:plugin:core:test --tests "org.elasticsearch.xpack.core.security.authz.accesscontrol.FieldSubsetReaderTests.testSyntheticSourceWithCopyToAndFLS*"
./gradlew :x-pack:plugin:yamlRestTest -Dtests.method="test {p0=security/authz_api_keys/30_field_level_security_synthetic_source/*copy_to*}"

When Field Level Security (FLS) processes _ignored_source entries from documents with synthetic source enabled, it calls decodeAsMap() to parse and potentially filter field values. However, copy_to targets are recorded in _ignored_source as VOID entries marking the field as existing but stored elsewhere (the copy_to source field). Previously, decodeAsMap() would attempt to parse these VOID entries, building a CBOR object with a field name but no value. When convertToMap tried to parse this malformed CBOR, the Jackson parser encountered an unexpected Break token (0xFF) and threw. The core fix is in nameValueToMapped(), which both encoding paths share. It now checks isDataPresent() before attempting to parse, returning null for VOID entries instead of building malformed CBOR. Both encoding paths call nameValueToMapped() through their respective decodeAsMap methods, so both are protected by the same check. The only difference is how the null propagates: 1. Coalesced path: decodeAsMap iterates over entries and skips null results, filtering VOID entries out of the returned list. 2. Legacy path: decodeAsMap returns null directly, and filterValue propagates the null to FieldSubsetReader, which drops the entry. In both cases, VOID entries carry no field data, so omitting them is correct.

romseygeek

One small suggesetion, LGTM otherwise.

romseygeek · 2026-02-02T11:59:40Z

server/src/main/java/org/elasticsearch/index/mapper/IgnoredSourceFieldMapper.java

    }

    private static MappedNameValue nameValueToMapped(NameValue nameValue) throws IOException {
+        if (XContentDataHelper.isDataPresent(nameValue.value()) == false) {


Can this just be nameValue.hasValue() == false?

…per.isDataPresent()

…#141506) * fix(mapper): handle VOID entries in _ignored_source with FLS When Field Level Security (FLS) processes _ignored_source entries from documents with synthetic source enabled, it calls decodeAsMap() to parse and potentially filter field values. However, copy_to targets are recorded in _ignored_source as VOID entries marking the field as existing but stored elsewhere (the copy_to source field). Previously, decodeAsMap() would attempt to parse these VOID entries, building a CBOR object with a field name but no value. When convertToMap tried to parse this malformed CBOR, the Jackson parser encountered an unexpected Break token (0xFF) and threw. The core fix is in nameValueToMapped(), which both encoding paths share. It now checks isDataPresent() before attempting to parse, returning null for VOID entries instead of building malformed CBOR. Both encoding paths call nameValueToMapped() through their respective decodeAsMap methods, so both are protected by the same check. The only difference is how the null propagates: 1. Coalesced path: decodeAsMap iterates over entries and skips null results, filtering VOID entries out of the returned list. 2. Legacy path: decodeAsMap returns null directly, and filterValue propagates the null to FieldSubsetReader, which drops the entry. In both cases, VOID entries carry no field data, so omitting them is correct.

elasticsearchmachine · 2026-02-03T09:37:58Z

💔 Backport failed

Status	Branch	Result
✅	9.3
❌	8.19	Commit could not be cherrypicked due to conflicts
✅	9.2

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 141506

elasticsearchmachine · 2026-02-03T09:39:02Z

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 141506

…#141506) * fix(mapper): handle VOID entries in _ignored_source with FLS When Field Level Security (FLS) processes _ignored_source entries from documents with synthetic source enabled, it calls decodeAsMap() to parse and potentially filter field values. However, copy_to targets are recorded in _ignored_source as VOID entries marking the field as existing but stored elsewhere (the copy_to source field). Previously, decodeAsMap() would attempt to parse these VOID entries, building a CBOR object with a field name but no value. When convertToMap tried to parse this malformed CBOR, the Jackson parser encountered an unexpected Break token (0xFF) and threw. The core fix is in nameValueToMapped(), which both encoding paths share. It now checks isDataPresent() before attempting to parse, returning null for VOID entries instead of building malformed CBOR. Both encoding paths call nameValueToMapped() through their respective decodeAsMap methods, so both are protected by the same check. The only difference is how the null propagates: 1. Coalesced path: decodeAsMap iterates over entries and skips null results, filtering VOID entries out of the returned list. 2. Legacy path: decodeAsMap returns null directly, and filterValue propagates the null to FieldSubsetReader, which drops the entry. In both cases, VOID entries carry no field data, so omitting them is correct.

#141703) * fix(mapper): handle VOID entries in _ignored_source with FLS When Field Level Security (FLS) processes _ignored_source entries from documents with synthetic source enabled, it calls decodeAsMap() to parse and potentially filter field values. However, copy_to targets are recorded in _ignored_source as VOID entries marking the field as existing but stored elsewhere (the copy_to source field). Previously, decodeAsMap() would attempt to parse these VOID entries, building a CBOR object with a field name but no value. When convertToMap tried to parse this malformed CBOR, the Jackson parser encountered an unexpected Break token (0xFF) and threw. The core fix is in nameValueToMapped(), which both encoding paths share. It now checks isDataPresent() before attempting to parse, returning null for VOID entries instead of building malformed CBOR. Both encoding paths call nameValueToMapped() through their respective decodeAsMap methods, so both are protected by the same check. The only difference is how the null propagates: 1. Coalesced path: decodeAsMap iterates over entries and skips null results, filtering VOID entries out of the returned list. 2. Legacy path: decodeAsMap returns null directly, and filterValue propagates the null to FieldSubsetReader, which drops the entry. In both cases, VOID entries carry no field data, so omitting them is correct.

#141705) * fix(mapper): handle VOID entries in _ignored_source with FLS When Field Level Security (FLS) processes _ignored_source entries from documents with synthetic source enabled, it calls decodeAsMap() to parse and potentially filter field values. However, copy_to targets are recorded in _ignored_source as VOID entries marking the field as existing but stored elsewhere (the copy_to source field). Previously, decodeAsMap() would attempt to parse these VOID entries, building a CBOR object with a field name but no value. When convertToMap tried to parse this malformed CBOR, the Jackson parser encountered an unexpected Break token (0xFF) and threw. The core fix is in nameValueToMapped(), which both encoding paths share. It now checks isDataPresent() before attempting to parse, returning null for VOID entries instead of building malformed CBOR. Both encoding paths call nameValueToMapped() through their respective decodeAsMap methods, so both are protected by the same check. The only difference is how the null propagates: 1. Coalesced path: decodeAsMap iterates over entries and skips null results, filtering VOID entries out of the returned list. 2. Legacy path: decodeAsMap returns null directly, and filterValue propagates the null to FieldSubsetReader, which drops the entry. In both cases, VOID entries carry no field data, so omitting them is correct.

…#141506) * fix(mapper): handle VOID entries in _ignored_source with FLS When Field Level Security (FLS) processes _ignored_source entries from documents with synthetic source enabled, it calls decodeAsMap() to parse and potentially filter field values. However, copy_to targets are recorded in _ignored_source as VOID entries marking the field as existing but stored elsewhere (the copy_to source field). Previously, decodeAsMap() would attempt to parse these VOID entries, building a CBOR object with a field name but no value. When convertToMap tried to parse this malformed CBOR, the Jackson parser encountered an unexpected Break token (0xFF) and threw. The core fix is in nameValueToMapped(), which both encoding paths share. It now checks isDataPresent() before attempting to parse, returning null for VOID entries instead of building malformed CBOR. Both encoding paths call nameValueToMapped() through their respective decodeAsMap methods, so both are protected by the same check. The only difference is how the null propagates: 1. Coalesced path: decodeAsMap iterates over entries and skips null results, filtering VOID entries out of the returned list. 2. Legacy path: decodeAsMap returns null directly, and filterValue propagates the null to FieldSubsetReader, which drops the entry. In both cases, VOID entries carry no field data, so omitting them is correct.

…141506) (#141716) When Field Level Security (FLS) processes _ignored_source entries from documents with synthetic source enabled, it calls decodeAsMap() to parse and potentially filter field values. However, copy_to targets are recorded in _ignored_source as VOID entries marking the field as existing but stored elsewhere (the copy_to source field). Previously, decodeAsMap() would attempt to parse these VOID entries, building a CBOR object with a field name but no value. When convertToMap tried to parse this malformed CBOR, the Jackson parser encountered an unexpected Break token (0xFF) and threw. The core fix is in nameValueToMapped(), which both encoding paths share. It now checks isDataPresent() before attempting to parse, returning null for VOID entries instead of building malformed CBOR. Both encoding paths call nameValueToMapped() through their respective decodeAsMap methods, so both are protected by the same check. The only difference is how the null propagates: 1. Coalesced path: decodeAsMap iterates over entries and skips null results, filtering VOID entries out of the returned list. 2. Legacy path: decodeAsMap returns null directly, and filterValue propagates the null to FieldSubsetReader, which drops the entry. In both cases, VOID entries carry no field data, so omitting them is correct. Backport note: On 8.19, FieldSubsetReader.java requires an explicit null check for the return value of decodeAsMap(). On main, this class was refactored with an IgnoredSourceFormat parameter that handles nulls through a different code path, so the null check is not needed there.

…#141506) Backport of elastic#141506 to 9.1. When Field Level Security is active on an index using synthetic source, fields with copy_to targets are recorded in _ignored_source as VOID entries. FieldSubsetReader attempted to parse these entries as CBOR, causing search failures with an unexpected Break token (0xFF). The fix checks hasValue() in decodeAsMap before parsing and returns null for VOID entries. FieldSubsetReader now skips null entries gracefully. Also unmutes the test from elastic#142341.

salvatore-campagna self-assigned this Jan 29, 2026

elasticsearchmachine added the v9.4.0 label Jan 29, 2026

delete file

b69a569

salvatore-campagna added backport pending backport auto-backport Automatically create backport pull requests when merged v8.19.11 v9.2.5 v9.3.1 labels Jan 29, 2026

salvatore-campagna marked this pull request as ready for review January 29, 2026 11:17

Merge branch 'main' into fix/fls-void-entries-exception-synthetic-source

f75e201

salvatore-campagna added the :StorageEngine/Mapping The storage related side of mappings label Jan 29, 2026

salvatore-campagna added 4 commits January 29, 2026 16:56

Merge branch 'main' into fix/fls-void-entries-exception-synthetic-source

ba4aaaf

Merge branch 'main' into fix/fls-void-entries-exception-synthetic-source

d3e9dbc

Merge branch 'main' into fix/fls-void-entries-exception-synthetic-source

09f9ffa

Merge branch 'main' into fix/fls-void-entries-exception-synthetic-source

304be11

romseygeek approved these changes Feb 2, 2026

View reviewed changes

salvatore-campagna and others added 5 commits February 2, 2026 18:03

refactor(mapper): use nameValue.hasValue() instead of XContentDataHel…

0f1b923

…per.isDataPresent()

Merge branch 'main' into fix/fls-void-entries-exception-synthetic-source

44eeece

Merge branch 'main' into fix/fls-void-entries-exception-synthetic-source

71f6bb7

Merge branch 'main' into fix/fls-void-entries-exception-synthetic-source

edc1344

Merge branch 'main' into fix/fls-void-entries-exception-synthetic-source

525b914

salvatore-campagna merged commit fea892d into elastic:main Feb 3, 2026
34 of 35 checks passed

salvatore-campagna mentioned this pull request Feb 3, 2026

[9.3] fix(mapper): handle VOID entries in _ignored_source with FLS (#141506) #141703

Merged

salvatore-campagna mentioned this pull request Feb 3, 2026

[9.2] fix(mapper): handle VOID entries in _ignored_source with FLS (#141506) #141705

Merged

salvatore-campagna mentioned this pull request Feb 3, 2026

[8.19] fix(mapper): handle VOID entries in _ignored_source with FLS (#141506) #141716

Merged

salvatore-campagna mentioned this pull request Mar 23, 2026

[9.1] fix(mapper): handle VOID entries in _ignored_source with FLS (#141506) #144765

Closed

salvatore-campagna mentioned this pull request Mar 23, 2026

[CI] XPackRestIT test {p0=security/authz_api_keys/30_field_level_security_synthetic_source/Fields with copy_to, field level security and synthetic source} failing #142341

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(mapper): handle VOID entries in _ignored_source with FLS#141506

fix(mapper): handle VOID entries in _ignored_source with FLS#141506
salvatore-campagna merged 12 commits intoelastic:mainfrom
salvatore-campagna:fix/fls-void-entries-exception-synthetic-source

salvatore-campagna commented Jan 29, 2026

Uh oh!

romseygeek left a comment

Uh oh!

romseygeek Feb 2, 2026

Uh oh!

Uh oh!

elasticsearchmachine commented Feb 3, 2026

Uh oh!

elasticsearchmachine commented Feb 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

salvatore-campagna commented Jan 29, 2026

Problem

Fix

Tests

Uh oh!

romseygeek left a comment

Choose a reason for hiding this comment

Uh oh!

romseygeek Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Feb 3, 2026

💔 Backport failed

Uh oh!

elasticsearchmachine commented Feb 3, 2026

💔 Backport failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants