[One Discover] [Logs UX]Add support to disable highlighting for certain filters we deem internal

[achyutjhunjhunwala]

Summary

closes #234215

Implements selective filter highlighting to prevent context/internal filters from being highlighted in search results while still using them for document retrieval.

Problem

When using SavedSearchComponent with filters (e.g., filtering logs to service.name: synth-node-0), all filter matches are highlighted in yellow. This creates visual noise when filters are used for context rather than active search criteria.

For example, in APM Service Logs:

• Service name filter is used to scope the view but shouldn't be highlighted
• User search filters should be highlighted to help locate matches

Previously, there was no way to distinguish between these two types of filters.

Solution

This PR uses Elasticsearch's highlight_query parameter to control which filters trigger highlighting. The implementation adds a skipHighlight metadata field to filters.

• Consumers mark filters with skipHighlight: true when creating them:

• SearchSource builds two queries:

1. Main query - Uses all filters for document retrieval
2. Highlight query - Excludes filters with skipHighlight: true

Example ES request:

{
  "query": {
    "bool": {
      "filter": [
        { "term": { "service.name": "synth.node" } },
        { "term": { "host.name": "instance" } }
      ]
    }
  },
  "highlight": {
    "fields": { "*": {} },
    "highlight_query": {
      "bool": {
        "filter": [
          { "term": { "host.name": "instance" } }
        ]
      }
    }
  }
}

Result: Documents are filtered by both conditions, but only "instance" is highlighted.

Performance Impact

The highlight_query parameter is processed by Elasticsearch on already-retrieved documents, so there's no additional data fetching or query execution overhead.

Screenshots

Before (APM)

After

Before (Host View)

After

After (Hosts Logs inside Flyout)

Before (Discover flyout for traces)

After

Logic Implemented for ?

• APM Logs Page
• All Host View
• Host View inside Flyout
• Discover Flyout for Traces

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[dej611]

Code review only. Left minor nit comment.

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[jennypavlova]

Thanks for the highlighting changes! I tested on the host view, and if the user makes a typo, it crashes the whole page (The refresh button is not helping, and nothing works except leaving the app

hosts_logs_search.mov

In this case, I would expect a validation around the search field or similar to what we have in the metadata tab.

Can you please take a look?

Also, on the single host view, the page is not crashing, but I can only search using a string log. level: "info" won't work there (and if the string is "Error," for example, I see mixhighlighting - which I guess is expected)

host_logs_search.mov

[achyutjhunjhunwala]

Can you please take a look?

@jennypavlova Good catch, i too saw that issue and fixed it in the flyout but completely forgot to replicate the change to the main hosts view

[achyutjhunjhunwala]

@crespocarlos @jennypavlova Thank you both for the detailed review. I have fixed most of them and added comments where required

[crespocarlos]

LGTM. CR only. Thanks for the adjustments

[lukasolson]

I'm not a huge fan of including the skipHighlight in the filter meta, simply because this is more of a UX representation of the filter, and there's no way in the UX to configure skipHighlight. This means you could get into an odd situation where you have filters showing up and they're not highlighting as expected, and there's no indication how to fix this.

I did notice @davismcphee suggested here trying to use component props, and I'm curious if that approach was attempted, and if so, what was challenging or didn't work as expected. I think what would make more sense to me is if we added an additional highlightQuery prop to the SavedSearchComponent which allowed passing in DSL for the highlight query (or even the entire highlight configuration) and then used something like searchSource.setField('highlight', highlightProps) in the same place we're setting query and filters.

[achyutjhunjhunwala]

@lukasolson Thank you for the detailed review. I actually started with props and then moved to meta based approach because it was much cleaner and simpler against the prop based approach.

"You could get into an odd situation where you have filters showing up and they're not highlighting as expected, and there's no indication how to fix this."

isn't it more prone in the prop based approach rather than the meta based ?

The filter is

• Optional, meaning false by default, meaning every filter would get highlighted by default on the Platform side
• On the Consumer side, they simple need to add this flag to a filter they deem internal/hidden

Aspect	Current: skipHighlight Metadata	Proposed: highlightQuery Prop	Result
User Visibility	Metadata invisible in UI	Prop invisible in UI	SAME
User Control	No UI to configure	No UI to configure	SAME
"Why isn't this filter highlighting?"	Applicable	Applicable
Code at consumer	simple	has to manage to props, easy to mix the filter causing undesired results
Responsibility Separation	SearchSource handles filtering logic	Consumer must handle filtering, now they only set a meta
--

I agree on the part unlike other props negate or disabled which are UI controllable, skipHighlight is not, but i don't find it as a strong reason to not add.

[lukasolson]

In general I think we should avoid altering the filter metadata for a few reasons. It's stored directly in the URL and inside saved objects (saved searches, saved queries, dashboards, etc.). Any change to the serialized filter.meta shape is, functionally, a storage-schema change that triggers migrations, bwc handling, export/import & validation considerations. Also, search source isn't the only consumer of filters, and we would need to check other consumers to make sure they properly handle this new skipHighlight parameter (e.g. kbn-es-query).

At its core, Filter.meta is really about filter pill behavior & identity. Using it to steer unrelated features (like highlight behavior) blurs separation of concerns and spreads app-specific semantics into global persisted state.

I would really like to push for adding something directly to SearchSource instead. Would it be acceptable to have something to the effect of searchSource.setHiddenFilters or searchSource.setNonHighlightingFilters instead? I can probably throw together a patch against your PR to give a better idea of how this might work if that's helpful. In this way we wouldn't be changing the external contract of Filter objects.

[achyutjhunjhunwala]

@lukasolson Oh wow, in my hindsight i complete missed the bigger picture that indeed its a storage schema and could have cascading effects. I am on SDH duty this week and then Spacetime next week. So i will try what you suggested, and ping you once i am done. If its still something not ideal, i will ask you for the patch 👍🏼

[iblancof]

Great to see this added!

Do you think it would be possible to include this other component in the scope as well?

If not, no worries!
I’ll take a look at how you implemented it, and once your PR is merged, I can create a follow-up to update it.

[achyutjhunjhunwala]

@iblancof thank you pointing this out. I was not aware about this place. I will add it as part of my PR itself

[achyutjhunjhunwala]

@iblancof I have added the support for the Traces Flyout in Discover, check the last commit here. Also updated the description with a screenshot

@lukasolson I have changed the logic and removed the skipHighlight meta filter and moved the logic to props.

PR is ready for review again

[lukasolson]

Thanks for making these changes, this feels better! I just left some minor comments below about not typing nonHighlightingFilters as a function.

[achyutjhunjhunwala]

@lukasolson Fixed the remaining ones here - ab173d5

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 7f5cb01

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
apm	2.8MB	2.8MB	+124.0B
fleet	2.1MB	2.1MB	+66.0B
infra	1.1MB	1.1MB	+1.2KB
logsShared	198.2KB	198.8KB	+529.0B
unifiedDocViewer	274.0KB	274.0KB	+15.0B
total			+1.9KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
data	433.2KB	433.9KB	+678.0B
logsShared	90.0KB	90.6KB	+585.0B
total			+1.2KB

Unknown metric groups

API count

id	before	after	diff
@kbn/saved-search-component	16	17	+1
data	3208	3210	+2
total			+3

History

• 💔 Build #358094 failed ecfa6d7
• 💚 Build #357659 succeeded 451d4d7
• 💛 Build #353642 was flaky adea8ef
• 💛 Build #353537 was flaky 77d2b7e
• 💔 Build #353183 failed eefddd3

cc @achyutjhunjhunwala