Skip to content

[Fleet] install custom assets on remote cluster#215504

Merged
juliaElastic merged 3 commits into
elastic:mainfrom
juliaElastic:install-custom-assets
Mar 25, 2025
Merged

[Fleet] install custom assets on remote cluster#215504
juliaElastic merged 3 commits into
elastic:mainfrom
juliaElastic:install-custom-assets

Conversation

@juliaElastic
Copy link
Copy Markdown
Contributor

@juliaElastic juliaElastic commented Mar 21, 2025

Summary

Closes #206244

Install custom assets on the remote cluster

To test locally:

  • enable feature flag enableSyncIntegrationsOnRemote
  • Create ccr index with some custom ingest pipelines and component templates
PUT fleet-synced-integrations-ccr-remote1/_doc/fleet-synced-integrations
{
  "id": "fleet-synced-integrations",
          "remote_es_hosts": [
            {
              "name": "Preconfiged remote output",
              "hosts": [
                "http://192.168.64.1:9200"
              ],
              "sync_integrations": true
            }
          ],
          "integrations": [
            {
              "package_name": "elastic_agent",
              "package_version": "2.2.0",
              "updated_at": "2025-03-24T09:36:12.127Z"
            },
            {
              "package_name": "endpoint",
              "package_version": "9.0.0",
              "updated_at": "2025-03-24T09:35:58.257Z"
            },
            {
              "package_name": "fleet_server",
              "package_version": "1.6.0",
              "updated_at": "2025-03-24T09:29:04.081Z"
            },
            {
              "package_name": "system",
              "package_version": "1.67.3",
              "updated_at": "2025-03-24T09:36:11.981Z"
            }
          ],
          "custom_assets": {
            "component_template:logs-system.auth@custom": {
              "type": "component_template",
              "name": "logs-system.auth@custom",
              "package_name": "system",
              "package_version": "1.67.3",
              "is_deleted": false,
              "template": {
                "mappings": {
                  "properties": {
                    "field1": {
                      "type": "text"
                    }
                  }
                }
              }
            },
            "ingest_pipeline:logs-system.auth@custom": {
              "type": "ingest_pipeline",
              "name": "logs-system.auth@custom",
              "package_name": "system",
              "package_version": "1.67.3",
              "is_deleted": false,
              "pipeline": {
                "processors": [
                  {
                    "set": {
                      "field": "test_field",
                      "value": "value"
                    }
                  }
                ]
              }
            }
          }
        }
  • Wait 5m for the sync task to run
  • Verify that the custom assets are created
image image

Checklist

@juliaElastic juliaElastic added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting labels Mar 21, 2025
@juliaElastic juliaElastic self-assigned this Mar 21, 2025
@juliaElastic juliaElastic changed the title [Fleet] install clustom assets [Fleet] install custom assets on remote cluster Mar 21, 2025
@juliaElastic juliaElastic marked this pull request as ready for review March 24, 2025 11:59
@juliaElastic juliaElastic requested a review from a team as a code owner March 24, 2025 11:59
@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

cc @juliaElastic

@nchaulet nchaulet self-requested a review March 25, 2025 12:42
Copy link
Copy Markdown
Member

@nchaulet nchaulet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code LGTM 🚀

@botelastic botelastic Bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Mar 25, 2025
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/fleet (Team:Fleet)

@juliaElastic juliaElastic merged commit 46da28a into elastic:main Mar 25, 2025
cqliu1 pushed a commit to cqliu1/kibana that referenced this pull request Mar 31, 2025
## Summary

Closes elastic#206244

Install custom assets on the remote cluster

To test locally:
- enable feature flag `enableSyncIntegrationsOnRemote`
- Create ccr index with some custom ingest pipelines and component
templates
```
PUT fleet-synced-integrations-ccr-remote1/_doc/fleet-synced-integrations
{
  "id": "fleet-synced-integrations",
          "remote_es_hosts": [
            {
              "name": "Preconfiged remote output",
              "hosts": [
                "http://192.168.64.1:9200"
              ],
              "sync_integrations": true
            }
          ],
          "integrations": [
            {
              "package_name": "elastic_agent",
              "package_version": "2.2.0",
              "updated_at": "2025-03-24T09:36:12.127Z"
            },
            {
              "package_name": "endpoint",
              "package_version": "9.0.0",
              "updated_at": "2025-03-24T09:35:58.257Z"
            },
            {
              "package_name": "fleet_server",
              "package_version": "1.6.0",
              "updated_at": "2025-03-24T09:29:04.081Z"
            },
            {
              "package_name": "system",
              "package_version": "1.67.3",
              "updated_at": "2025-03-24T09:36:11.981Z"
            }
          ],
          "custom_assets": {
            "component_template:logs-system.auth@custom": {
              "type": "component_template",
              "name": "logs-system.auth@custom",
              "package_name": "system",
              "package_version": "1.67.3",
              "is_deleted": false,
              "template": {
                "mappings": {
                  "properties": {
                    "field1": {
                      "type": "text"
                    }
                  }
                }
              }
            },
            "ingest_pipeline:logs-system.auth@custom": {
              "type": "ingest_pipeline",
              "name": "logs-system.auth@custom",
              "package_name": "system",
              "package_version": "1.67.3",
              "is_deleted": false,
              "pipeline": {
                "processors": [
                  {
                    "set": {
                      "field": "test_field",
                      "value": "value"
                    }
                  }
                ]
              }
            }
          }
        }
```
- Wait 5m for the sync task to run
- Verify that the custom assets are created

<img width="1772" alt="image"
src="https://github.com/user-attachments/assets/8d51e725-5646-4c21-b96d-bc0cbf9c2910"
/>
<img width="1772" alt="image"
src="https://github.com/user-attachments/assets/9346c819-a926-4e0a-a1fe-3172a59b7b59"
/>


### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
criamico added a commit that referenced this pull request Apr 7, 2025
…216178)

Closes #192363

## Summary
Add endpoint that compares integrations installed on remote cluster with
integrations in ccr index `fleet-synced-integrations-ccr-<outputId>`.
Feature flag: `enableSyncIntegrationsOnRemote`

- Use the [ccr info
api](https://www.elastic.co/guide/en/elasticsearch/reference/current/ccr-get-follow-info.html)
to check that the ccr index is active
- Compare the content of the two indices and report the sync status for
each integration:

```
GET kbn:/api/fleet/remote_synced_integrations/status

{
  "integrations": [
    {
      "package_name": "akamai",
      "package_version": "2.28.0",
      "updated_at": "2025-03-27T10:29:52.485Z",
      "sync_status": true
    },
    {
      "package_name": "auth0",
      "package_version": "1.21.0",
      "updated_at": "2025-03-26T12:06:26.268Z",
      "sync_status": false,
      "error": "Installation status: not_installed" 
    },
]
```
- Check that the integrations have custom assets and report their sync
status (follow up of #215504)

### Testing 
Setup local env with the guide added in dev_docs
([preview](https://github.com/elastic/kibana/blob/b438a719ee34d0dfb19a7a6cbbafc616936581bb/x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md))

- Install some integrations on local cluster, wait that they are synced
on remote
- From remote cluster dev tools, run

```
GET kbn:/api/fleet/remote_synced_integrations/status
```

- To verify that custom assets are synced choose an integration, for
instance `system`
- From the package policy select a var, advanced options and add a
custom mapping and a custom pipeline. In my example I used `system`
<img width="1195" alt="Screenshot 2025-04-01 at 11 18 40"
src="https://github.com/user-attachments/assets/c339d6f0-4404-4e9b-b489-7b37977f201b"
/>

- Run the endpoint again and you should see the status of custom assets
too:

```
{
  "integrations": [
    {
      "package_name": "akamai",
      "package_version": "2.28.0",
      "updated_at": "2025-03-27T10:29:52.485Z",
      "sync_status": "completed"
    },
    {
      "package_name": "elastic_agent",
      "package_version": "2.2.0",
      "updated_at": "2025-03-26T14:06:29.216Z",
      "sync_status": "completed"
    },
    {
      "package_name": "synthetics",
      "package_version": "1.4.1",
      "updated_at": "2025-03-26T14:06:31.909Z",
      "sync_status": "completed"
    },
    {
      "package_name": "system",
      "package_version": "1.67.3",
      "updated_at": "2025-03-28T10:08:00.602Z",
      "sync_status": "completed"
    }
  ],
  "custom_assets": {
    "component_template:logs-system.auth@custom": {
      "name": "logs-system.auth@custom",
      "type": "component_template",
      "package_name": "system",
      "package_version": "1.67.3",
      "sync_status": "completed"
    },
    "ingest_pipeline:logs-system.auth@custom": {
      "name": "logs-system.auth@custom",
      "type": "ingest_pipeline",
      "package_name": "system",
      "package_version": "1.67.3",
      "sync_status": "completed"
    }
  }
}
```



### Checklist
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v9.1.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Fleet] Synchronize custom assets to remote clusters

4 participants