Skip to content

[Synthetics] Fix monitor health API for monitors in non-default spaces#270540

Merged
shahzad31 merged 5 commits into
elastic:mainfrom
shahzad31:fix/synthetics-health-cross-space
May 22, 2026
Merged

[Synthetics] Fix monitor health API for monitors in non-default spaces#270540
shahzad31 merged 5 commits into
elastic:mainfrom
shahzad31:fix/synthetics-health-cross-space

Conversation

@shahzad31
Copy link
Copy Markdown
Contributor

@shahzad31 shahzad31 commented May 22, 2026

Release note

When using Kibana Spaces, the Synthetics monitor health endpoint could incorrectly report monitors as unhealthy — showing errors such as "missing location", "missing agent policy", or "missing package policy" — even when everything was properly configured. This happened because the health check was only looking for monitors, private locations, and Fleet policies in the current space, missing resources that existed in other spaces.

These issues are now fixed: the health check correctly resolves monitors, private locations, package policies, and agent policies across all relevant spaces, giving an accurate health status regardless of how resources are distributed across your Kibana Spaces.

Summary

Closes #270477.

POST /internal/synthetics/monitors/_health returned wrong results when monitors lived outside the request's space — missing_package_policy errors when called from the monitor's space, and 404s when called from default.

Two independent space-scoping bugs:

  1. Package policy lookup ignored space. getExistingPackagePoliciesMap called Fleet's packagePolicyService.getByIDs with createInternalRepository(), which is scoped to the default namespace. Package policies created for monitors in another space were therefore invisible.
  2. Monitor saved-object lookup was space-scoped. MonitorConfigRepository.get used the request-scoped saved-objects client, restricted to the request's space. Calling _health from default for a monitor that lives elsewhere returned a 404.

What changed

  • PackagePolicyService.getByIds — accepts a new optional additionalSpaceIds, so the wrapper's per-space scoped-client fan-out can broaden beyond [spaceId, default]. Existing callers keep their old behavior.
  • MonitorConfigRepository.getAcrossSpaces(id, namespaces, soClient?) — new method that resolves a monitor across an arbitrary list of spaces. Uses the multi-space type's per-object namespaces array in one bulkGet entry, plus one entry per namespace for the namespaceType: 'single' legacy type. Accepts an injected soClient so the health API can pass createInternalRepository() and bypass the request's space restriction.
  • MonitorIntegrationHealthApi:
    • Computes allSpaces = { requestSpace, ...allSpacesWithMonitors } once, up-front.
    • fetchMonitors calls monitorConfigRepository.getAcrossSpaces with the internal repository → fixes bug Change of links to styles and js #2.
    • getExistingPackagePoliciesMap uses the PackagePolicyService wrapper with additionalSpaceIds → fixes bug links to styles and js #1.

Test plan

  • node scripts/jest on the three affected suites — 77/77 passing (includes new cross-space coverage and a new getAcrossSpaces test block).
  • node scripts/type_check --project x-pack/solutions/observability/plugins/synthetics/tsconfig.json — clean.
  • node scripts/eslint on the changed files — clean.
  • Manual: create a monitor with a private location in a non-default space, then call POST /internal/synthetics/monitors/_health both from that space and from default. Verify each call reports the monitor accurately instead of missing_package_policy / 404.

Related

Made with Cursor

`POST /internal/synthetics/monitors/_health` returned wrong results when
monitors lived outside the request's space:

- Package policy lookup used `createInternalRepository()` scoped to the
  default namespace, so policies created for monitors in another space
  were invisible — surfaced as `missing_package_policy` even when the
  Fleet policy existed.
- Monitor saved-object lookup used the request-scoped SO client, so
  calling `_health` from `default` for monitors that live in another
  space returned 404s.

Fixes:

- Route the package-policy lookup through `PackagePolicyService.getByIds`
  with `additionalSpaceIds = allSpacesWithMonitors`, so namespace-scoped
  SO clients are built per space and results deduplicated.
- Add `MonitorConfigRepository.getAcrossSpaces`, which uses an internal
  SO repository with per-object `namespaces` (multi-space type) and a
  one-entry-per-space lookup for the legacy single-namespace type.

Closes elastic#270477

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions github-actions Bot added the author:actionable-obs PRs authored by the actionable obs team label May 22, 2026
@shahzad31 shahzad31 marked this pull request as ready for review May 22, 2026 08:22
@shahzad31 shahzad31 requested a review from a team as a code owner May 22, 2026 08:22
// per space and de-duplicates the results, so package policies created
// for monitors in any space are visible regardless of the caller's space.
const packagePolicyService = new PackagePolicyService(this.server);
const additionalSpaceIds = [...allSpaces].filter((space) => space !== this.spaceId);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really needed? spaceId is added to the set in x-pack/solutions/observability/plugins/synthetics/server/synthetics_service/private_location/package_policy_service.ts:72 anyway

@miguelmartin-elastic
Copy link
Copy Markdown
Contributor

The bugs described in the issue are fixed. But a third one showed up now, look at the prompt below

  1. In custom space:
image
  1. In default space:
image
In monitor_integration_health_api.ts, getExistingAgentPoliciesMap calls [agentPolicyService.getByIds(internalSoClient, ...)](vscode-file://vscode-app/Applications/Visual%20Studio%20Code.app/Contents/Resources/app/out/vs/code/electron-browser/workbench/workbench.html) where internalSoClient comes from createInternalRepository(). That repository is scoped to the default namespace, so it cannot find agent policies that have [space_ids: ['some-custom-space']](vscode-file://vscode-app/Applications/Visual%20Studio%20Code.app/Contents/Resources/app/out/vs/code/electron-browser/workbench/workbench.html).

As a result, when a monitor is assigned to a private location whose agent policy lives exclusively in a non-default space, the health API returns missing_agent_policy even though the policy exists and is healthy.

This is the same class of bug as the package-policy lookup (#1) — the fix for that one (getExistingPackagePoliciesMap) should serve as a reference for how to approach this.

Steps to reproduce:

Create an agent policy with [space_ids: ['space-two']](vscode-file://vscode-app/Applications/Visual%20Studio%20Code.app/Contents/Resources/app/out/vs/code/electron-browser/workbench/workbench.html) (Fleet API, POST kbn:/s/space-two/api/fleet/agent_policies).
Create a private location backed by that policy in [space-two](vscode-file://vscode-app/Applications/Visual%20Studio%20Code.app/Contents/Resources/app/out/vs/code/electron-browser/workbench/workbench.html).
Create a monitor in [space-two](vscode-file://vscode-app/Applications/Visual%20Studio%20Code.app/Contents/Resources/app/out/vs/code/electron-browser/workbench/workbench.html) assigned to that location.
Call POST /s/space-two/internal/synthetics/monitors/_health with that monitor's ID.
Observe privateLocations[0].status === "missing_agent_policy".

@kibanamachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

@shahzad31 shahzad31 merged commit 4e2f2e6 into elastic:main May 22, 2026
31 checks passed
@shahzad31 shahzad31 deleted the fix/synthetics-health-cross-space branch May 22, 2026 12:46
@kibanamachine
Copy link
Copy Markdown
Contributor

Starting backport for target branches: 9.4

https://github.com/elastic/kibana/actions/runs/26288622513

@kibanamachine
Copy link
Copy Markdown
Contributor

💔 All backports failed

Status Branch Result
9.4 Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 270540

Questions ?

Please refer to the Backport tool documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author:actionable-obs PRs authored by the actionable obs team backport:version Backport to applied version labels release_note:fix v9.4.0 v9.5.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Synthetics monitor health API fails for monitors in non-default spaces

3 participants