[Synthetics] Honor searchExcludedDataTiers advanced setting in SyntheticsEsClient

[shahzad31]

Summary

Addresses #273086 (fix #1 — the single choke point).

Synthetics issued every Elasticsearch query against the broad synthetics-* wildcard with only ignore_unavailable: true — no data-tier filter. As a result, queries fanned out to all backing indices across hot/warm/frozen tiers. On clusters with long frozen retention (e.g. hot: 3d / warm: 4d / frozen: 365d) this caused high latency (slow screenshots and history) and contributed to cluster-wide search thread pool rejections, because frozen searchable-snapshot shards are slow and deliberately throttled.

This PR makes SyntheticsEsClient honor the existing observability:searchExcludedDataTiers advanced setting — the same mechanism already used by APM and Infrastructure. When configured, a must_not { terms: { _tier: [...] } } filter is injected into every search / msearch / count, so excluded tiers (e.g. data_frozen) are skipped.

What changed

• SyntheticsEsClient reads observability:searchExcludedDataTiers (cached per instance) and wraps each query via a new exported applyExcludedDataTiersToQuery helper, reusing the shared excludeTiersQuery util from @kbn/observability-utils-common.
• The client now also accepts a plain IUiSettingsClient (uiSettingsClient option) in addition to the route-handler uiSettings context, so alerting rule executors can resolve the setting.
• Wired the status and TLS rule executors (and the inspect-TLS route) to pass their uiSettingsClient, so alerting queries are covered too.
• Updated the advanced-setting description to list Synthetics among the affected apps.

Behavior / risk

• The setting defaults to [], so behavior is unchanged unless an operator opts in — no functional change for existing deployments.
• All Synthetics UI queries flow through SyntheticsEsClient via the route wrapper (which already passes uiSettings), so screenshots, history, latest-ping, and overview queries are all covered with no call-site changes.

Follow-ups (tracked in #273086, not in this PR)

• Add @timestamp lower bounds to the time-less queries (screenshots/steps/network events, latest-ping hooks, get_last_successful_check).
• Drop the residual 30-day default in fetchMonitorRecentPings.

Testing

• node scripts/jest x-pack/solutions/observability/plugins/synthetics/server/lib.test.ts — added unit tests for search/msearch/count exclusion and the applyExcludedDataTiersToQuery helper.
• Status + TLS rule executor jest suites pass unchanged.
• Manual: set Stack Management → Advanced Settings → Excluded data tiers from search to data_frozen and confirm Synthetics search requests include the must_not _tier filter (visible via the inspector / observability:enableInspectEsQueries).

Checklist

• Unit tests updated
• Type-check passes for the synthetics project
• ESLint + i18n checks pass

Release note

Synthetics now respects the observability:searchExcludedDataTiers advanced setting, allowing operators to exclude slow data tiers (such as data_frozen) from Synthetics searches to reduce latency and search thread pool pressure.

[infra-vault-gh-plugin-prod]

Pinging @elastic/actionable-obs-team (Team:actionable-obs)

[mgiota]

LGTM! Nicely scoped fix — follows the existing APM/Infra pattern for searchExcludedDataTiers, defaults to no behavior change ([]), and centralizing it in SyntheticsEsClient means every UI route and the wired-up rule executors are covered without touching call sites. Tests cover the new search/msearch/count paths plus the helper.

Thanks for tracking the @timestamp lower bounds and the fetchMonitorRecentPings default as follow-ups in #273086.

[mgiota]

nit: search has both the tiers configured and no tiers excluded cases in the new describe('excluded data tiers') block, but msearch and count only cover the "configured" path.

The pre-existing tests for those methods do exercise the unmodified-request path, but only via the uiSettingsClient short-circuit in getExcludedDataTiers() (no client provided).

What's not covered for msearch/count is the case where the client is provided and returns [] — i.e. the falsy branch of the if (excludedDataTiers.length) guard inside each method. Worth two small symmetric tests for completeness. Pure nit, non-blocking.

[mgiota]

it('does not modify msearch requests when no tiers are excluded', async () => {
  esClient.msearch.mockResolvedValueOnce({
    body: { responses: [{}] },
  } as unknown as MsearchResponse);
  const { client, uiSettings } = createClientWithExcludedTiers([]);

  await client.msearch([{ query: matchAll }]);

  expect(uiSettings.get).toHaveBeenCalledWith('observability:searchExcludedDataTiers');
  expect(esClient.msearch).toHaveBeenCalledWith(
    {
      searches: [
        { index: 'synthetics-*', ignore_unavailable: true },
        { query: matchAll },
      ],
    },
    { meta: true }
  );
});

it('does not modify the count query when no tiers are excluded', async () => {
  const { client, uiSettings } = createClientWithExcludedTiers([]);

  await client.count({ query: matchAll });

  expect(uiSettings.get).toHaveBeenCalledWith('observability:searchExcludedDataTiers');
  expect(esClient.count).toHaveBeenCalledWith(
    { index: 'synthetics-*', ignore_unavailable: true, query: matchAll },
    { meta: true, context: { loggingOptions: { loggerName: 'synthetics' } } }
  );
});

[shahzad31]

Good call — added the two symmetric tests (msearch + count with the client provided but the setting returning [], so the falsy branch of the excludedDataTiers.length guard is covered) in 198f655.

[kibanamachine]

💚 Build Succeeded

• Buildkite Build
• Commit: fe7f884
• Build duration: 50 mins

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
synthetics	1352	1355	+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
synthetics	1.3MB	1.3MB	+487.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
observability	102.6KB	102.7KB	+84.0B

History

• 💚 Build #455188 succeeded d11d7e8
• 💔 Build #455157 failed 89656dc
• 💛 Build #454864 was flaky 198f655
• 💛 Build #454697 was flaky a81bfc7
• 💔 Build #454694 failed b76e637

[kibanamachine]

Starting backport for target branches: 8.19, 9.3, 9.4

https://github.com/elastic/kibana/actions/runs/27790679644

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts
❌	9.3	Backport failed because of merge conflicts
❌	9.4	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 273418

Questions ?

Please refer to the Backport tool documentation

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 273418 locally
cc: @shahzad31

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 273418 locally
cc: @shahzad31

[kibanamachine]

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @shahzad31

[kibanamachine]

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @shahzad31