Skip to content

Don't serialize endpoint metadata in semantic text mappings#145021

Merged
Mikep86 merged 12 commits intoelastic:mainfrom
Mikep86:semantic-text_dont-serialize-endpoint-metadata
Mar 30, 2026
Merged

Don't serialize endpoint metadata in semantic text mappings#145021
Mikep86 merged 12 commits intoelastic:mainfrom
Mikep86:semantic-text_dont-serialize-endpoint-metadata

Conversation

@Mikep86
Copy link
Copy Markdown
Contributor

@Mikep86 Mikep86 commented Mar 26, 2026

In #141393, MinimalServiceSettings was updated to add endpoint metadata. This change introduced a bug (see #144969) where endpoint metadata is serialized in semantic_text index mappings. Endpoint metadata is only necessary when selecting the default inference service to use; once the inference service is selected it is no longer useful and can be discarded.

This PR fixes that by implementing filtered XContent for MinimalServiceSettings that excludes endpoint metadata. An alternative approach considered was to make a new class, MappingServiceSettings, that models all params in MinimalServiceSettings except for endpoint metadata. We did not choose to go with that for a couple reasons:

  • Due to this bug, there are some users out there with semantic_text index mappings on disk with endpoint metadata. We need to know how to parse those or the mappings will be invalid, breaking Elasticsearch. See testMappingWithEndpointMetadata for a test validating this case with the chosen approach.
  • There are places where we rely on the semantic_text model settings being MinimalServiceSettings, such as when updating the inference ID. Using a new class like MappingServiceSettings breaks this logic, and would require more significant changes to resolve.

Nothing precludes us from refactoring at a later time to introduce a new class like MappingServiceSettings. This PR aims to fix the bug with the fewest and simplest changes possible so that it can be resolved quickly and without side-effects.

Fixes #144969

@Mikep86 Mikep86 requested review from a team, jimczi, jonathan-buttner and tballison March 26, 2026 17:50
@Mikep86 Mikep86 added >bug :Search Foundations/Mapping Index mappings, including merging and defining field types v9.4.0 labels Mar 26, 2026
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-search-foundations (Team:Search Foundations)

@elasticsearchmachine elasticsearchmachine added the Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch label Mar 26, 2026
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Hi @Mikep86, I've created a changelog YAML for you.

@Override
public ToXContentObject getFilteredXContentObject() {
return this::toXContent;
return (b, p) -> toXContent(b, p, false);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally in the future we can remove MinimalServiceSettings from implementing ServiceSettings. Maybe when we get around to doing that we can rename the method then. The other option we have is to leverage the Params instead which I think is the typical way to handle this.

@@ -329,7 +329,11 @@ public Builder(
() -> null,
(n, c, o) -> SemanticTextField.parseModelSettingsFromMap(o),
mapper -> ((SemanticTextFieldType) mapper.fieldType()).modelSettings,
XContentBuilder::field,
(b, n, v) -> {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we wanted to use Params here instead, I think we'd do something like this:

(b, n, v) -> {
  if (v != null) {
    b.field(MODEL_SETTINGS_FIELD, v, new ToXContent.MapParams(Map.of("exclude_metadata", "true")));
  }
}

@Mikep86
Copy link
Copy Markdown
Contributor Author

Mikep86 commented Mar 30, 2026

@elasticmachine update branch

@Mikep86 Mikep86 merged commit 2f73ff7 into elastic:main Mar 30, 2026
37 checks passed
mouhc1ne pushed a commit to shmuelhanoch/elasticsearch that referenced this pull request Mar 31, 2026
ncordon pushed a commit to ncordon/elasticsearch that referenced this pull request Apr 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

>bug :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EndpointMetadata is serialized in an index mapping

5 participants