[Fleet] Create API to report status of integrations synchronization by criamico · Pull Request #216178 · elastic/kibana

criamico · 2025-03-27T13:37:06Z

Closes #192363

Summary

Add endpoint that compares integrations installed on remote cluster with integrations in ccr index fleet-synced-integrations-ccr-<outputId>. Feature flag: enableSyncIntegrationsOnRemote

Use the ccr info api to check that the ccr index is active
Compare the content of the two indices and report the sync status for each integration:

GET kbn:/api/fleet/remote_synced_integrations/status

{
  "integrations": [
    {
      "package_name": "akamai",
      "package_version": "2.28.0",
      "updated_at": "2025-03-27T10:29:52.485Z",
      "sync_status": true
    },
    {
      "package_name": "auth0",
      "package_version": "1.21.0",
      "updated_at": "2025-03-26T12:06:26.268Z",
      "sync_status": false,
      "error": "Installation status: not_installed" 
    },
]

Check that the integrations have custom assets and report their sync status (follow up of [Fleet] install custom assets on remote cluster #215504)

Testing

Setup local env with the guide added in dev_docs (preview)

Install some integrations on local cluster, wait that they are synced on remote
From remote cluster dev tools, run

GET kbn:/api/fleet/remote_synced_integrations/status

To verify that custom assets are synced choose an integration, for instance system
From the package policy select a var, advanced options and add a custom mapping and a custom pipeline. In my example I used system

Run the endpoint again and you should see the status of custom assets too:

{
  "integrations": [
    {
      "package_name": "akamai",
      "package_version": "2.28.0",
      "updated_at": "2025-03-27T10:29:52.485Z",
      "sync_status": "completed"
    },
    {
      "package_name": "elastic_agent",
      "package_version": "2.2.0",
      "updated_at": "2025-03-26T14:06:29.216Z",
      "sync_status": "completed"
    },
    {
      "package_name": "synthetics",
      "package_version": "1.4.1",
      "updated_at": "2025-03-26T14:06:31.909Z",
      "sync_status": "completed"
    },
    {
      "package_name": "system",
      "package_version": "1.67.3",
      "updated_at": "2025-03-28T10:08:00.602Z",
      "sync_status": "completed"
    }
  ],
  "custom_assets": {
    "component_template:logs-system.auth@custom": {
      "name": "logs-system.auth@custom",
      "type": "component_template",
      "package_name": "system",
      "package_version": "1.67.3",
      "sync_status": "completed"
    },
    "ingest_pipeline:logs-system.auth@custom": {
      "name": "logs-system.auth@custom",
      "type": "ingest_pipeline",
      "package_name": "system",
      "package_version": "1.67.3",
      "sync_status": "completed"
    }
  }
}

Checklist

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

criamico · 2025-03-27T13:37:31Z

@elasticmachine merge upstream

juliaElastic · 2025-03-28T08:39:43Z

+    if (!installedIntegrations) {
+      return {
+        items: [],
+        error: `No integrations installed on remote`


it's not necessarily an error if the main cluster doesn't have any integrations either

criamico · 2025-04-01T09:22:36Z

+
+  try {
+    const installedPipelines = await getPipeline(esClient, abortController);
+    const installedComponentTemplates = await getComponentTemplate(esClient, abortController);


I went for the solution of fetching all the pipelines and component templates and keep them on memory, instead of doing a call for each one in the loop below. @juliaElastic do you think it could become a performance issue?

We should only compare pipelines and component templates that match '*@custom', like it's done in custom_assets.ts. The other assets are already installed when the package is installed, no need to compare them.

juliaElastic · 2025-04-01T09:37:54Z

+    Object.entries(ccrCustomAssets).forEach(([ccrCustomName, ccrCustomAsset]) => {
+      if (ccrCustomAsset.type === 'ingest_pipeline') {
+        const installedAsset = installedPipelines[ccrCustomAsset?.name];
+        if (isEqual(installedAsset?.processors, ccrCustomAsset?.pipeline?.processors)) {


pipelines have an optional version which we can use to compare like here

kibana/x-pack/platform/plugins/shared/fleet/server/tasks/sync_integrations/custom_assets.ts

Line 241 in b438a71

(existingPipeline.version && existingPipeline.version < customAsset.pipeline.version) ||

we should probably have a common logic to compare here and custom_assets.ts

criamico · 2025-04-01T15:33:28Z

@elasticmachine merge upstream

elasticmachine · 2025-04-01T16:05:36Z

Pinging @elastic/fleet (Team:Fleet)

criamico · 2025-04-03T20:03:27Z

@juliaElastic based on the new requirements in #217025 we should be able to query by output_id:

Create API that queries remote kibana sync status API by output ID (to be used by the UI to show status)
e.g. /api/fleet/remote_synced_integrations/{output_id}/status -> https://{remote_kibana}/api/fleet/remote_synced_integrations/status

In a previous commit it was already by output_id. Do you think we'll need to keep the general status? Otherwise I'll change it directly in this PR.

criamico · 2025-04-04T06:26:40Z

@elasticmachine merge upstream

…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/dashboards --update'

juliaElastic · 2025-04-04T07:15:10Z

@juliaElastic based on the new requirements in #217025 we should be able to query by output_id:

Create API that queries remote kibana sync status API by output ID (to be used by the UI to show status)
e.g. /api/fleet/remote_synced_integrations/{output_id}/status -> https://{remote_kibana}/api/fleet/remote_synced_integrations/status

In a previous commit it was already by output_id. Do you think we'll need to keep the general status? Otherwise I'll change it directly in this PR.

We can keep as is in the current pr, as it collects the status in the remote cluster. The new API by output_id will only call the remote API using the remote output kibana url and API key.

criamico · 2025-04-04T12:48:42Z

@elasticmachine merge upstream

juliaElastic · 2025-04-07T07:23:39Z

+    return { info: res.follower_indices[0] };
+  } catch (err) {
+    if (err?.body?.error?.type === 'index_not_found_exception')
+      throw new IndexNotFoundError(`Index not found`);


should we return the error message instead of throwing an error?

It's handled here: https://github.com/elastic/kibana/pull/216178/files#diff-f8de6a6d308d65b9d61c400a2fdbebe1078e67eaf036537c028476260be254f5R312-R315

I left the throw block and handled outside because we might need this function elsewhere, this is a basic utility for the ccr case.

okay, then shouldn't we return the error in the catch there? https://github.com/elastic/kibana/pull/216178/files#diff-f8de6a6d308d65b9d61c400a2fdbebe1078e67eaf036537c028476260be254f5R324

Fixed, I added some unit tests to cover these cases

juliaElastic · 2025-04-07T07:32:00Z

+      };
+    } else if (
+      installedPipeline?.version &&
+      installedPipeline.version < ccrCustomAsset.pipeline.version


the version comparison should be done before the equality check, so we can skip the equality check if the version is not equal

juliaElastic · 2025-04-07T07:44:35Z

+    if (ccrCustomAsset.is_deleted === true && installedPipeline) {
+      return {
+        ...result,
+        sync_status: 'failed' as SyncStatus.FAILED,


should this be synchronizing (the deletion might not have happened yet) unless we know there was an error deleting?

juliaElastic · 2025-04-07T07:45:23Z

+    if (ccrCustomAsset.is_deleted === true && installedCompTemplate) {
+      return {
+        ...result,
+        sync_status: 'failed' as SyncStatus.FAILED,


same here, should these be synchronizing unless we know there was an error deleting?

criamico · 2025-04-07T10:17:47Z

@elasticmachine merge upstream

juliaElastic · 2025-04-07T10:35:06Z

@@ -255,6 +250,11 @@ const compareCustomAssets = ({
        sync_status: 'failed' as SyncStatus.FAILED,


this should be synchronizing too

juliaElastic · 2025-04-07T13:05:19Z

+        return {
+          ...ccrIntegration,
+          sync_status: 'failed' as SyncStatus.FAILED,
+          error: `Installation status: ${localIntegrationSO?.attributes.install_status}`,


can we return the error message if install_status: install_failed from latest_install_failed_attempts or latest_executed_state?

kibana/x-pack/platform/plugins/shared/fleet/common/types/models/epm.ts

Line 674 in 0c5fa84

latest_install_failed_attempts?: InstallFailedAttempt[];

juliaElastic

LGTM, thanks for the updates

elasticmachine · 2025-04-07T17:13:32Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: e51b00c

Failed CI Steps

Jest Tests #7

Test Failures

[job] [logs] Jest Tests #7 / Templates renders empty templates correctly

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fleet`	1199	1201	+2

History

💚 Build #290662 succeeded f884e10
💚 Build #290276 succeeded 317ed18
💔 Build #290206 failed 2b4cb2e
💔 Build #290141 failed 4519bd6
💔 Build #290074 failed 0907204
💛 Build #289957 was flaky 8129bf4

cc @criamico

…ut ID (#217799) Part II of #217025 ## Summary Create API that queries remote kibana sync status API by output ID. From the main cluster we call the remote kibana (simply using node-fetch) and query the endpoint added in #216178; this way the main cluster can have the status of the synced integrations on the remote cluster. ### Testing Note that dev_docs now have a guide to setup locally the remote clusters: https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md - Follow the testing steps from [this PR](#216178) - Install some integrations on cluster A (main) and wait 5 minutes to get `SyncIntegrationsTask` running - Verify that cluster B (remote) has the same integrations installed. From dev tools, run ``` GET kbn:/api/fleet/remote_synced_integrations/status ``` - Go on dev tools on cluster A and run the new endpoint - `remote_id` is the id of the remote output configured on cluster A: ``` GET kbn:/api/fleet/remote_synced_integrations/<remote_id>/remote_status ``` The response should be the same as above ### Screenshot On Remote cluster (Cluster B): <img width="1183" alt="Screenshot 2025-04-10 at 15 40 46" src="https://github.com/user-attachments/assets/60ea1c1e-9ccf-4bcf-8637-bc4079483e61" /> On main cluster (Cluster A): <img width="1690" alt="Screenshot 2025-04-11 at 11 10 30" src="https://github.com/user-attachments/assets/e72fd729-3486-41b0-9194-487233415a75" /> ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

criamico added 4 commits March 19, 2025 09:59

[Fleet] Create API to report status of integrations synchronization

bdc0100

use follower info api

ce1de43

Merge main

fe5c7f1

Get working endpoint

d7fbe56

criamico added the Team:Fleet Team label for Observability Data Collection Fleet team label Mar 27, 2025

criamico self-assigned this Mar 27, 2025

elasticmachine and others added 2 commits March 27, 2025 15:37

Merge branch 'main' into 192363_integrations_sync_status

6e1f6b8

Add top level error

525d0d5

juliaElastic reviewed Mar 28, 2025

View reviewed changes

Comment thread .../platform/plugins/shared/fleet/server/tasks/sync_integrations/sync_integrations_on_remote.ts Outdated

juliaElastic reviewed Mar 28, 2025

View reviewed changes

Comment thread .../platform/plugins/shared/fleet/server/tasks/sync_integrations/sync_integrations_on_remote.ts Outdated

juliaElastic reviewed Mar 28, 2025

View reviewed changes

Comment thread x-pack/platform/plugins/shared/fleet/server/routes/remote_synced_integrations/index.ts Outdated

criamico added 2 commits March 28, 2025 10:57

Address code review comments

84e10d6

Update dev docs

796b101

juliaElastic reviewed Mar 31, 2025

View reviewed changes

Comment thread x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md Outdated

Add comparison for custom assets

b438a71

criamico commented Apr 1, 2025

View reviewed changes

Comment thread x-pack/platform/plugins/shared/fleet/server/tasks/sync_integrations/custom_assets.ts Outdated

criamico commented Apr 1, 2025

View reviewed changes

juliaElastic reviewed Apr 1, 2025

View reviewed changes

criamico changed the title ~~192363 integrations sync status~~ [Fleet] Create API to report status of integrations synchronization Apr 1, 2025

Address comments and add unit tests

f022ee1

Merge branch 'main' into 192363_integrations_sync_status

682a307

criamico added v9.1.0 release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting labels Apr 1, 2025

criamico marked this pull request as ready for review April 1, 2025 16:05

criamico requested a review from a team as a code owner April 1, 2025 16:05

elasticmachine and others added 3 commits April 4, 2025 07:26

Merge branch 'main' into 192363_integrations_sync_status

6e10aa5

[CI] Auto-commit changed files from 'make api-docs'

4519bd6

criamico added 2 commits April 4, 2025 11:02

Fix unit test

c186486

Update docs and diagram

2b4cb2e

Merge branch 'main' into 192363_integrations_sync_status

317ed18

juliaElastic reviewed Apr 7, 2025

View reviewed changes

Address code review comments pt.3, add more tests

fb1eb39

Merge branch 'main' into 192363_integrations_sync_status

f884e10

juliaElastic reviewed Apr 7, 2025

View reviewed changes

Replace failed state with synchronizing when versions do not match

fb34535

juliaElastic reviewed Apr 7, 2025

View reviewed changes

Add failure reason from latest_install_failed_attempts

fc6a8c8

juliaElastic approved these changes Apr 7, 2025

View reviewed changes

juliaElastic reviewed Apr 7, 2025

View reviewed changes

Comment thread .../platform/plugins/shared/fleet/dev_docs/diagrams/remote_clusters/remote_clusters_diagram.png Outdated

Update diagram

e51b00c

criamico enabled auto-merge (squash) April 7, 2025 16:07

criamico merged commit ab6f7c6 into elastic:main Apr 7, 2025

criamico mentioned this pull request Apr 10, 2025

[Fleet] Create API that queries remote kibana sync status API by output ID #217799

Merged

2 tasks

lcawl mentioned this pull request Jun 10, 2025

[Internal]: Add context and improve tag for Fleet remote synced integration APIs elastic/docs-content#1691

Closed

		@@ -255,6 +250,11 @@ const compareCustomAssets = ({
		sync_status: 'failed' as SyncStatus.FAILED,

Conversation

criamico commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Checklist

Uh oh!

criamico commented Mar 27, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

criamico Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliaElastic Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

criamico commented Apr 1, 2025

Uh oh!

elasticmachine commented Apr 1, 2025

Uh oh!

criamico commented Apr 3, 2025

Uh oh!

criamico commented Apr 4, 2025

Uh oh!

juliaElastic commented Apr 4, 2025

Uh oh!

criamico commented Apr 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

criamico commented Apr 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliaElastic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticmachine commented Apr 7, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

criamico commented Mar 27, 2025 •

edited

Loading

criamico Apr 1, 2025 •

edited

Loading

juliaElastic Apr 1, 2025 •

edited

Loading