[Fleet] Create API that queries remote kibana sync status API by output ID by criamico · Pull Request #217799 · elastic/kibana

criamico · 2025-04-10T09:32:09Z

Part II of #217025

Summary

Create API that queries remote kibana sync status API by output ID.

From the main cluster we call the remote kibana (simply using node-fetch) and query the endpoint added in #216178; this way the main cluster can have the status of the synced integrations on the remote cluster.

Testing

Note that dev_docs now have a guide to setup locally the remote clusters: https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md

Follow the testing steps from this PR
Install some integrations on cluster A (main) and wait 5 minutes to get SyncIntegrationsTask running
Verify that cluster B (remote) has the same integrations installed. From dev tools, run

GET kbn:/api/fleet/remote_synced_integrations/status

Go on dev tools on cluster A and run the new endpoint - remote_id is the id of the remote output configured on cluster A:

GET kbn:/api/fleet/remote_synced_integrations/<remote_id>/remote_status

The response should be the same as above

Screenshot

On Remote cluster (Cluster B):

On main cluster (Cluster A):

Checklist

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

…ut ID

elasticmachine · 2025-04-10T13:57:37Z

Pinging @elastic/fleet (Team:Fleet)

…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/dashboards --include-path /api/saved_objects/_import --include-path /api/saved_objects/_export --update'

juliaElastic · 2025-04-10T15:15:24Z

+        authz: {
+          requiredPrivileges: [
+            FLEET_API_PRIVILEGES.SETTINGS.READ,
+            FLEET_API_PRIVILEGES.INTEGRATIONS.READ,


I think the flyout has to be adapted with the correct minimal privileges:

kibana/x-pack/platform/plugins/shared/fleet/public/applications/fleet/sections/settings/components/edit_output_flyout/output_form_remote_es.tsx

Lines 391 to 404 in a616a40

{

"name": "integration_sync_api_key",

"role_descriptors": {

"integration_writer": {

"cluster": [],

"indices":[],

"applications": [{

"application": "kibana-.kibana",

"privileges": ["feature_fleet.read", "feature_fleetv2.read"],

"resources": ["*"]

}]

}

}

}`}

We should test that the API works correctly with a user role having only minimal privileges

I created a role with very minimal privileges - "dev tools" it's there because I need it for testing purposes

Logged in with an user that has that role and verified that API runs correctly.

criamico · 2025-04-11T09:13:25Z


 export const REMOTE_SYNCED_INTEGRATIONS_API_ROUTES = {
  STATUS_PATTERN: `${API_ROOT}/remote_synced_integrations/status`,
+  INFO_PATTERN: `${API_ROOT}/remote_synced_integrations/{outputId}/remote_status`,


I changed the path to remote_status to clarify that is a different status than the above. We might need to fetch the global status remotely (i.e. without outputId) and in that case we could add another endpoint like this:

api/fleet/remote_synced_integrations/remote_status

…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/dashboards --include-path /api/saved_objects/_import --include-path /api/saved_objects/_export --update'

juliaElastic

LGTM

criamico · 2025-04-11T13:03:33Z

@elasticmachine merge upstream

criamico · 2025-04-14T07:15:29Z

@elasticmachine merge upstream

criamico · 2025-04-14T09:34:42Z

@elasticmachine merge upstream

criamico · 2025-04-14T12:09:31Z

@elasticmachine merge upstream

jillguyonnet · 2025-04-14T12:26:08Z

Hey @criamico I'm testing this afresh again and I'm seeing the following when running GET kbn:/api/fleet/remote_synced_integrations/status on the remote cluster:

{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "Failed output validation: [response body.integrations.0.install_status]: definition for this key is missing"
}

This seems to be fixed by adding the install_status field here and here.

jillguyonnet · 2025-04-14T12:49:01Z

+
+    return response.ok({ body: res });
+  } catch (error) {
+    throw error;


Maybe a naive question: what is the benefit of catching and re-throwing these errors?

More generally, I was a little surprised to see thrown errors in kibana e.g. when the output doesn't exist (as opposed to a regular 404 response).

juliaElastic · 2025-04-14T13:16:15Z

Hey @criamico I'm testing this afresh again and I'm seeing the following when running GET kbn:/api/fleet/remote_synced_integrations/status on the remote cluster:
{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "Failed output validation: [response body.integrations.0.install_status]: definition for this key is missing"
}
This seems to be fixed by adding the install_status field here and here.

I think we don't need to return install_status in this API, we can just replace ... with expanding the needed fields here:

kibana/x-pack/platform/plugins/shared/fleet/server/tasks/sync_integrations/compare_synced_integrations.ts

Line 134 in c10cd62

...ccrIntegration,

criamico · 2025-04-14T14:58:44Z

@elasticmachine merge upstream

criamico · 2025-04-15T07:55:26Z

@elasticmachine merge upstream

jillguyonnet · 2025-04-15T09:49:04Z

Edit: a little late, sorry. Let me know your thoughts on the below.

Hmm failed queries look different now.

Example 1: incorrect remote Kibana URL

Response to GET kbn:/api/fleet/remote_synced_integrations/<outputId>/remote_status:

Request failed to get to the server (status code: 500)

Kibana logs:

[2025-04-15T11:39:01.405+02:00][INFO ][plugins.fleet] Fetching http://localhost:5701/jills/api/fleet/remote_synced_integrations/status
[2025-04-15T11:39:01.413+02:00][ERROR][plugins.fleet] FetchError: invalid json response body at http://localhost:5701/jills/api/fleet/remote_synced_integrations/status reason: Unexpected token 'N', "Not Found" is not valid JSON
[2025-04-15T11:39:01.413+02:00][ERROR][plugins.fleet] FetchError: invalid json response body at http://localhost:5701/jills/api/fleet/remote_synced_integrations/status reason: Unexpected token 'N', "Not Found" is not valid JSON
... (long stack trace)

Example 2: incorrect output id

Response to GET kbn:/api/fleet/remote_synced_integrations/foo/remote_status:

Request failed to get to the server (status code: 404)

Kibana logs:

[2025-04-15T11:45:37.939+02:00][ERROR][plugins.fleet] Error: Saved object [ingest-outputs/b84ed8ed-a7b1-502f-83f6-90132e68adef] not found

It looks like the added condition didn't work as expected? I don't want to block this PR, so I would say the previous version is more helpful. We shouldn't risk throwing 500s if the setup is incorrect.

elasticmachine · 2025-04-15T09:49:19Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: db8709d

Failed CI Steps

Fleet Cypress Tests #1

Test Failures

[job] [logs] Fleet Cypress Tests #1 / View agents list Agent status filter should filter on healthy and unhealthy
[job] [logs] Fleet Cypress Tests #1 / View agents list Bulk actions should allow to bulk upgrade agents and cancel that upgrade

Metrics [docs]

✅ unchanged

History

💚 Build #292843 succeeded d789da4
💔 Build #292645 failed ccecb69
💔 Build #292522 failed 029b210
💛 Build #292312 was flaky aac6631
💛 Build #292249 was flaky 372743e
💔 Build #292012 failed 4ab0406

cc @criamico

criamico · 2025-04-15T09:52:33Z

@jillguyonnet the PR got merged, at this point this can be addressed in the next piece of work (probably the UI work at this point)

[Fleet] Create API that queries remote kibana sync status API by outp…

342ba35

…ut ID

criamico self-assigned this Apr 10, 2025

criamico mentioned this pull request Apr 10, 2025

[Fleet] Call remote kibana sync integrations status API from main kibana #217025

Closed

5 tasks

Add unit tests

34fcd67

criamico added Team:Fleet Team label for Observability Data Collection Fleet team release_note:skip Skip the PR/issue when compiling release notes v9.1.0 backport:skip This PR does not require backporting labels Apr 10, 2025

Add unit test for output not found case

bdc889e

criamico marked this pull request as ready for review April 10, 2025 13:57

criamico requested a review from a team as a code owner April 10, 2025 13:57

juliaElastic reviewed Apr 10, 2025

View reviewed changes

Comment thread .../platform/plugins/shared/fleet/server/tasks/sync_integrations/compare_synced_integrations.ts Outdated

[CI] Auto-commit changed files from 'make api-docs'

4ab0406

juliaElastic reviewed Apr 10, 2025

View reviewed changes

criamico added 2 commits April 11, 2025 10:17

Move function to separate file

ffec996

Change api path from remote to remote_status

370a4bb

criamico commented Apr 11, 2025

View reviewed changes

juliaElastic reviewed Apr 11, 2025

View reviewed changes

Comment thread x-pack/platform/plugins/shared/fleet/server/tasks/sync_integrations/get_remote_status.ts Outdated

juliaElastic reviewed Apr 11, 2025

View reviewed changes

Comment thread x-pack/platform/plugins/shared/fleet/server/tasks/sync_integrations/get_remote_status.ts Outdated

kibanamachine and others added 2 commits April 11, 2025 09:45

[CI] Auto-commit changed files from 'make api-docs'

8bcff5d

Update error message

372743e

juliaElastic approved these changes Apr 11, 2025

View reviewed changes

Improve error handling in case of fetch errors

f4df25a

Merge branch 'main' into 217025_remote_status_by_outputId

aac6631

Merge branch 'main' into 217025_remote_status_by_outputId

029b210

Merge branch 'main' into 217025_remote_status_by_outputId

ccecb69

Merge branch 'main' into 217025_remote_status_by_outputId

5a1301b

jillguyonnet reviewed Apr 14, 2025

View reviewed changes

Comment thread x-pack/platform/plugins/shared/fleet/server/tasks/sync_integrations/get_remote_status.ts Outdated

Improve error handling and fix return type

de4b396

elasticmachine and others added 2 commits April 14, 2025 16:58

Merge branch 'main' into 217025_remote_status_by_outputId

d789da4

Improve not found error

82c31ab

Merge branch 'main' into 217025_remote_status_by_outputId

db8709d

criamico enabled auto-merge (squash) April 15, 2025 08:57

criamico merged commit 89d6dab into elastic:main Apr 15, 2025
9 checks passed

criamico deleted the 217025_remote_status_by_outputId branch April 15, 2025 09:57

This was referenced Apr 15, 2025

[Fleet] Update fleet/remote_synced_integrations/status API to return package uninstall errors #217683

Closed

[Fleet] Sync integration status UI #218389

Merged

	{
	"name": "integration_sync_api_key",
	"role_descriptors": {
	"integration_writer": {
	"cluster": [],
	"indices":[],
	"applications": [{
	"application": "kibana-.kibana",
	"privileges": ["feature_fleet.read", "feature_fleetv2.read"],
	"resources": ["*"]
	}]
	}
	}
	}`}

Conversation

criamico commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Screenshot

Checklist

Uh oh!

elasticmachine commented Apr 10, 2025

Uh oh!

Uh oh!

juliaElastic Apr 10, 2025

Choose a reason for hiding this comment

Uh oh!

criamico Apr 11, 2025

Choose a reason for hiding this comment

Uh oh!

criamico Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

juliaElastic left a comment

Choose a reason for hiding this comment

Uh oh!

criamico commented Apr 11, 2025

Uh oh!

criamico commented Apr 14, 2025

Uh oh!

criamico commented Apr 14, 2025

Uh oh!

criamico commented Apr 14, 2025

Uh oh!

jillguyonnet commented Apr 14, 2025

Uh oh!

jillguyonnet Apr 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

juliaElastic commented Apr 14, 2025

Uh oh!

criamico commented Apr 14, 2025

Uh oh!

criamico commented Apr 15, 2025

Uh oh!

Uh oh!

jillguyonnet commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example 1: incorrect remote Kibana URL

Example 2: incorrect output id

Uh oh!

elasticmachine commented Apr 15, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

Uh oh!

criamico commented Apr 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

criamico commented Apr 10, 2025 •

edited

Loading

criamico Apr 11, 2025 •

edited

Loading

jillguyonnet commented Apr 15, 2025 •

edited

Loading