move createGetterSetter to plugins/kibana_utils#49069
Merged
Merged
Conversation
alexwizp
added
v7.6.0
v8.0.0
Feature:NP Migration
release_note:skip
Contributor
💚 Build Succeeded
|
alexwizp
force-pushed
the
createGetterSetter
branch
from
f4fe33a to
ac962f3
Compare
Contributor
💚 Build Succeeded |
alexwizp
added a commit
to alexwizp/kibana
that referenced
this pull request
Oct 24, 2019
alexwizp
added a commit
that referenced
this pull request
Oct 24, 2019
Contributor
|
We might need to backport this to 7.5 |
patrykkopycinski
pushed a commit
to patrykkopycinski/kibana
that referenced
this pull request
May 6, 2026
Apmats
added a commit
to Apmats/kibana
that referenced
this pull request
May 12, 2026
Adds the @ menu / typeahead query path on top of the schema PR. The
autocomplete surface is unified through `discovery_labels` — title and
type are no longer special-cased.
## Schema
discovery_labels: nested { value, kind }
- `value.autocomplete` is a `text` field with a custom edge-ngram analyzer
(NOT `search_as_you_type` — see WHY below).
- The indexer auto-prepends `{value: chunk.title, kind: 'title'}` and
`{value: chunk.type, kind: 'type'}` to every record, plus any entries
the producer provides (taglines, nicknames, categories, etc.). Type
writers don't change their SmlChunk shape.
Drops `title_autocomplete` (top-level SAYT field) and the `.autocomplete`
subfield from `type`. Title's and type's autocomplete reach now goes
through their auto-prepended discovery_labels entries; `type` stays as
plain keyword for exact filtering.
## Server / query
POST /internal/agent_context_layer/sml/_autocomplete — dedicated route
for the @ menu, separate from /sml/_search.
buildSmlAutocompleteQuery: a single nested query against
discovery_labels.value.autocomplete with `match` + `operator: and` (every
typed token must match). `inner_hits` returns the matched entries with
their `kind` and an ES `unified`-highlighter snippet wrapping the
matched word(s) in <em>...</em>.
Response: { id, type, title, origin_id, matched_discovery_labels? }.
No content, no description, no score — order is sufficient; UI gets
exactly what it needs to render a typeahead row with badges.
filterResultsByPermissions is generic over the result type and shared
with the search path.
## Why custom edge-ngram, not search_as_you_type
SAYT (`type: 'search_as_you_type'`) is the obvious first choice for an
autocomplete field. It does not produce useful highlights for prefix
queries in a nested context. Confirmed in two ways:
1. Empirical: tested unified, plain, fvh highlighters; tested
`require_field_match: false`, `highlight_query: prefix`, term_vector
with offsets, top-level vs inner_hits highlight — none returned a
snippet for nested + SAYT + bool_prefix on prefix-only matches.
2. Documented: this is the bug behind elastic/elasticsearch#53744
("search_as_you_type, prefix queries and broken highlighting"),
open since 2020-03, explicitly low-priority per the ES Search team
lead at the time. Root cause documented in elastic#49069 ("Highlight with
index_prefixes"): SAYT's `_index_prefix` subfield indexes
position-spanning ngrams with offsets that span from the prefix
start to the end of the entire field, which the highlighter can't
wrap per-word. The ES-community workaround (see
elastic/elasticsearch#53744 comment by peterbe) is exactly what
this PR does: use a regular text field with a custom analyzer.
The custom analyzer: standard tokenizer + lowercase + edge_ngram filter
(1–20) at index time; standard + lowercase at search time. Per-token
edge_ngrams preserve each word's offsets, so the highlighter wraps the
matched word correctly — verified live for queries like "git" →
"<em>github</em>", "github c" → "<em>GitHub</em> <em>Connector</em>",
"prod logs ana" → "<em>production</em> <em>logs</em> <em>analytics</em>".
Trade-offs vs SAYT:
- Smaller inverted index (one field, not four).
- Comparable query latency (posting-list lookup per token).
- Loses SAYT's _2gram/_3gram shingle subfields used for phrase-prefix
scoring. BM25 on edge_ngram tokens is good enough for typeahead
ranking.
## kbn-storage-adapter change
Adds an optional `analysis` field to `IndexStorageSettings`, threaded
through to the index template's `settings.analysis` block. Required so
that consumers (like SML here) can declare custom analyzers without
bypassing the storage adapter's bootstrap flow. Backwards-compatible —
field is optional. Ignored on serverless ES (no index-level settings).
Owners: @elastic/observability-ui, @elastic/kibana-core.
## Tests
- Service test: query shape (match + operator: and + nested + inner_hits
+ highlight), inner_hits mapping into matched_discovery_labels,
highlight passthrough.
- Route test: response shape, no permissions/spaces/content leak.
- FTR (real-ES): /sml/_autocomplete against an indexed record with
title / type / tagline discovery_labels; asserts matched_discovery_labels,
kind, and that the highlight wraps the matched word.
- 121/121 jest in agent_context_layer; 14/14 jest in kbn-storage-adapter.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apmats
added a commit
to Apmats/kibana
that referenced
this pull request
May 13, 2026
Adds the @ menu / typeahead query path on top of the schema PR. The
autocomplete surface is unified through `discovery_labels` — title and
type are no longer special-cased.
discovery_labels: nested { value, kind }
- `value.autocomplete` is a `text` field with a custom edge-ngram analyzer
(NOT `search_as_you_type` — see WHY below).
- The indexer auto-prepends `{value: chunk.title, kind: 'title'}` and
`{value: chunk.type, kind: 'type'}` to every record, plus any entries
the producer provides (taglines, nicknames, categories, etc.). Type
writers don't change their SmlChunk shape.
Drops `title_autocomplete` (top-level SAYT field) and the `.autocomplete`
subfield from `type`. Title's and type's autocomplete reach now goes
through their auto-prepended discovery_labels entries; `type` stays as
plain keyword for exact filtering.
POST /internal/agent_context_layer/sml/_autocomplete — dedicated route
for the @ menu, separate from /sml/_search.
buildSmlAutocompleteQuery: a single nested query against
discovery_labels.value.autocomplete with `match` + `operator: and` (every
typed token must match). `inner_hits` returns the matched entries with
their `kind` and an ES `unified`-highlighter snippet wrapping the
matched word(s) in <em>...</em>.
Response: { id, type, title, origin_id, matched_discovery_labels? }.
No content, no description, no score — order is sufficient; UI gets
exactly what it needs to render a typeahead row with badges.
filterResultsByPermissions is generic over the result type and shared
with the search path.
SAYT (`type: 'search_as_you_type'`) is the obvious first choice for an
autocomplete field. It does not produce useful highlights for prefix
queries in a nested context. Confirmed in two ways:
1. Empirical: tested unified, plain, fvh highlighters; tested
`require_field_match: false`, `highlight_query: prefix`, term_vector
with offsets, top-level vs inner_hits highlight — none returned a
snippet for nested + SAYT + bool_prefix on prefix-only matches.
2. Documented: this is the bug behind elastic/elasticsearch#53744
("search_as_you_type, prefix queries and broken highlighting"),
open since 2020-03, explicitly low-priority per the ES Search team
lead at the time. Root cause documented in elastic#49069 ("Highlight with
index_prefixes"): SAYT's `_index_prefix` subfield indexes
position-spanning ngrams with offsets that span from the prefix
start to the end of the entire field, which the highlighter can't
wrap per-word. The ES-community workaround (see
elastic/elasticsearch#53744 comment by peterbe) is exactly what
this PR does: use a regular text field with a custom analyzer.
The custom analyzer: standard tokenizer + lowercase + edge_ngram filter
(1–20) at index time; standard + lowercase at search time. Per-token
edge_ngrams preserve each word's offsets, so the highlighter wraps the
matched word correctly — verified live for queries like "git" →
"<em>github</em>", "github c" → "<em>GitHub</em> <em>Connector</em>",
"prod logs ana" → "<em>production</em> <em>logs</em> <em>analytics</em>".
Trade-offs vs SAYT:
- Smaller inverted index (one field, not four).
- Comparable query latency (posting-list lookup per token).
- Loses SAYT's _2gram/_3gram shingle subfields used for phrase-prefix
scoring. BM25 on edge_ngram tokens is good enough for typeahead
ranking.
Adds an optional `analysis` field to `IndexStorageSettings`, threaded
through to the index template's `settings.analysis` block. Required so
that consumers (like SML here) can declare custom analyzers without
bypassing the storage adapter's bootstrap flow. Backwards-compatible —
field is optional. Ignored on serverless ES (no index-level settings).
Owners: @elastic/observability-ui, @elastic/kibana-core.
- Service test: query shape (match + operator: and + nested + inner_hits
+ highlight), inner_hits mapping into matched_discovery_labels,
highlight passthrough.
- Route test: response shape, no permissions/spaces/content leak.
- FTR (real-ES): /sml/_autocomplete against an indexed record with
title / type / tagline discovery_labels; asserts matched_discovery_labels,
kind, and that the highlight wraps the matched word.
- 121/121 jest in agent_context_layer; 14/14 jest in kbn-storage-adapter.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
move createGetterSetter to plugins/kibana_utils
Checklist
Use
strikethroughsto remove checklist items you don't feel are applicable to this PR.For maintainers