Introduce Kibana task to deploy agentless connectors for 9.0 by artem-shelkovnikov · Pull Request #203973 · elastic/kibana

artem-shelkovnikov · 2024-12-12T09:21:21Z

Closes https://github.com/elastic/search-team/issues/8508

Closes https://github.com/elastic/search-team/issues/8465

Summary

This PR adds a background task for search_connectors plugin. This task checks connector records and agentless package policies and sees if new connector was added/old was deleted, and then adds/deletes package policies for these connectors.

Scenario 1: a new connector was added by a user/API call

User creates an Elastic-managed connector:

Screen.Recording.2024-12-25.at.12.59.14.mov

When the user is done, a package policy is created by this background task:

Screen.Recording.2024-12-25.at.13.00.14.mov

Scenario 2: a connector was deleted by a user/API call

User deletes an Elastic-managed connector:

Screen.Recording.2024-12-25.at.13.21.13.mov

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
Flaky Test Runner was used on any tests changed
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

x-pack/plugins/search_connectors/server/plugin.ts

artem-shelkovnikov · 2024-12-25T12:27:48Z

x-pack/platform/plugins/shared/fleet/server/plugin.ts

This was done due to the fact that I needed a create method that depends on a lot of other private/internal methods.

I had to either make the methods public + add here; or I could pass the service itself. Potentially there might be other way, but I'm not familiar enough with Kibana development yet to know, please tell me if there's a better way :)

elasticmachine · 2024-12-25T12:27:53Z

Pinging @elastic/fleet (Team:Fleet)

artem-shelkovnikov · 2024-12-25T12:29:01Z

x-pack/platform/plugins/shared/fleet/server/routes/agent_policy/handlers.ts

Side-effect of removing the usage of AgentPolicyServiceInterface: interface had getByIDs and implementation has getByIds. I chose the latter to stay, but it's easy to rename implementation to getByIDs. This was mostly done to avoid pinging other code owners that might have used the interface method name.

artem-shelkovnikov · 2024-12-25T12:38:41Z

x-pack/plugins/search_connectors/server/services/index.ts

For some reason this doesn't work - I never get a policy that has supports_agentless field.

artem-shelkovnikov · 2024-12-25T12:57:37Z

x-pack/plugins/search_connectors/server/services/index.ts

Using our regular NATIVE_CONNECTOR_DEFINITIONS as a source of truth for connectors that we support. I could theoretically instead list integrations that are branched off connectors-py instead, is it possible/better?

artem-shelkovnikov · 2024-12-25T13:06:10Z

x-pack/plugins/search_connectors/server/task.ts

@elastic/fleet - what's the minimal interval with which we could query fleet package policies (we narrow them with a kuery that only returns our package elastic_connectors?

Can we do 10 seconds? 30 seconds?

If we only query for a certain package, there shouldn't be too many results, so it shouldn't be a problem with scale. I think using 30s sounds fine too, 10s might be too frequent.

artem-shelkovnikov · 2024-12-25T13:06:39Z

x-pack/plugins/search_connectors/server/task.ts

Do we even need to retry, since we run pretty often?

juliaElastic

Fleet changes LGTM

jedrazb

Great stuff! The changes in the search_connectors plugin LGTM. I have a couple of minor comments regarding naming and one question about hardcoding the package version in the task manager logic.

I’ll defer reviewing the changes in the fleet plugin to the fleet team. EDIT: I see they just approved 🚀

x-pack/solutions/search/plugins/search_connectors/server/task.ts

x-pack/test/plugin_api_integration/test_suites/task_manager/check_registered_task_types.ts

jedrazb · 2025-01-02T11:19:16Z

x-pack/solutions/search/plugins/search_connectors/server/services/index.ts

+
+const connectorsInputName = 'connectors-py';
+const pkgName = 'elastic_connectors';
+const pkgVersion = '0.0.4';


do we need this version hardcoded here? The current (latest) version in the integration registry should be def tracked somewhere by fleet, can we look it up in the package registry dynamically?

Context, 0.0.4 is already outdated

Maybe code edited in this PR will help? https://github.com/elastic/kibana/pull/192081/files here I was able to access package info and adjust permissions dynamically

x-pack/solutions/search/plugins/search_connectors/server/task.ts

Co-authored-by: Jedr Blaszyk <jedrazb@gmail.com>

jedrazb

🚢

mikecote · 2025-01-03T13:58:04Z

x-pack/solutions/search/plugins/search_connectors/server/task.ts

+      const taskInstance = await taskManager.ensureScheduled({
+        id: AGENTLESS_CONNECTOR_DEPLOYMENTS_SYNC_TASK_ID,
+        taskType: AGENTLESS_CONNECTOR_DEPLOYMENTS_SYNC_TASK_TYPE,
+        schedule: SCHEDULE,


Taking a quick look here from Response Ops. I was reading the PR description and was wondering if we need to have this task run every 30s indefinitely or if it would be possible to make it event based so it runs after a user creates or deletes a connector? Or perhaps a combo of the two but the schedule runs less frequently?

was thinking the same ...

For now this seemed to us like the best way to move forward:

The task runs and checks if any agentless policies need to be created for our connector records. Connector records can be created in multiple ways:

User creates a connector via UI

Connector is created automatically by already running agentless connector deployment

User creates a connector via API/CLI

Scenario #1 can be done with an event triggered by Kibana UI easily. Scenario #2 does not need this logic. Scenario #3 really needs this task - our CLI doesn't have access to Task Manager + our API is hosted in Elasticsearch, and Elasticsearch also has no way to affect this task run time.

This way we've taken current approach with polling every 30 seconds (a minute should be fine too), plus the task itself queries reasonably small amount of data, I believe, for it hopefully not to be too problematic.

The GenAI connectors have a similar sort of constraint, where something in Kibana wants to know when connectors get created / updated / deleted. Added in #189027

That PR originally contained some connector logic for the new "hooks", but we extracted that and restructured into a stand-alone PR: #194081 , rather than ship the two pieces together.

So, in theory case 3 can be handled this way.

Looking at those PRs, I'm also wondering if you need to handle the case of connectors being updated / deleted ...

I've skimmed through. the change but don't understand how it handles case 3 - we have customer calling Elasticsearch API directly, Kibana is not involved in this.

So we cannot have hooks attached to this call, all we can do is poll the content of a couple indices to see if changes were made. Am I missing some detail in the mentioned PR that works around this limitation?

Connector update is not important for us, but deletion is also handled in this PR

Oh, these aren't alerting connectors? These are "search" connectors? If so, you're correct, completely different "connector" framework I was talking about (I was talking about the alerting connectors).

pmuellr

ResponseOps code LGTM, left a few comments

pmuellr · 2025-01-07T20:22:23Z

x-pack/solutions/search/plugins/search_connectors/server/task.ts

+const AGENTLESS_CONNECTOR_DEPLOYMENTS_SYNC_TASK_ID = 'search:agentless-connectors-manager-task';
+const AGENTLESS_CONNECTOR_DEPLOYMENTS_SYNC_TASK_TYPE = 'search:agentless-connectors-manager';
+
+const SCHEDULE = { interval: '30s' };


Setting this to the largest value you are willing to live with, will be helpful to Kibana's task throughput :-)

I believe a comment in the PR indicated it could be set to "1m" which would cut down the executions by 50% (useful!)

I'll change to "1m" indeed, it should not hurt us much, and we can iterate on this number later if it's gonna be too much/too little!

pmuellr · 2025-01-07T20:34:04Z

x-pack/solutions/search/plugins/search_connectors/server/task.ts

+          };
+        }
+      },
+      cancel: async () => {


Note that if you want to have the cancel actually stop the task from running, you'll have to do a bit more. This function is invoked when TM decides the task needs to be cancelled (running longer than it's time limit). The basic idea is you set a local indicating you've been cancelled, and then can check that in the run() method. Example here:

kibana/x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/entity_store/task/field_retention_enrichment_task.ts

Lines 250 to 279 in 0e13d86

const createTaskRunnerFactory =

({

logger,

telemetry,

executeEnrichPolicy,

getStoreSize,

}: {

logger: Logger;

telemetry: AnalyticsServiceSetup;

executeEnrichPolicy: ExecuteEnrichPolicy;

getStoreSize: GetStoreSize;

}) =>

({ taskInstance }: { taskInstance: ConcreteTaskInstance }) => {

let cancelled = false;

const isCancelled = () => cancelled;

return {

run: async () =>

runTask({

executeEnrichPolicy,

getStoreSize,

isCancelled,

logger,

taskInstance,

telemetry,

}),

cancel: async () => {

cancelled = true;

},

};

};

- note that this code doesn't actually seem to use the isCancelled() local function they created - I think it did at one point, must have been removed in another PR ...

Thanks, I see now - I copied original code from some other place.

The task we have is supposed to be very fast (obviously, depending on response times from Elasticsearch). At best it's 2 calls to Elasticsearch, at worst it's probably does 10s of calls (realistically, I expect 2 calls 99.99% of the time, and occasionally 4 calls).

IMO adding true cancellation is not gonna add a lot here, but I'm not sure yet. I will merge what is there for now and keep it in mind for when we need it :)

elasticmachine · 2025-01-09T16:02:30Z

💔 Build Failed

Buildkite Build
Commit: 9717116

Failed CI Steps

Test Failures

[job] [logs] Fleet Cypress Tests #4 / Add Integration - Automatic Import should create an integration
[job] [logs] Fleet Cypress Tests #4 / Add Integration - Automatic Import should create an integration
[job] [logs] Fleet Cypress Tests #2 / When the user has All permissions for Integrations and All permissions for actions Create Assistant is not accessible but upload is accessible
[job] [logs] Fleet Cypress Tests #2 / When the user has All permissions for Integrations and All permissions for actions Create Assistant is not accessible but upload is accessible

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	1309	1319	+10

Unknown metric groups

API count

id	before	after	diff
`fleet`	1436	1446	+10

History

💔 Build #265577 failed 189ee5b
💔 Build #265533 failed 3ec5393
💛 Build #264291 was flaky 75b22f7
💔 Build #264260 failed 0f4db80

…6606) ## Summary This PR makes it so that the Agentless Kibana task implemented in #203973 properly handles soft-deleted connectors. This helps with the situation when an integration policy has been created for an agentless connector but a connector record has not yet been created by an agentless host. With current Kibana task implementation it could lead to the Policy being deleted. With this change, only policies that refer to soft-deleted connectors will be cleaned up. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…#203973) ## Closes elastic/search-team#8508 ## Closes elastic/search-team#8465 ## Summary This PR adds a background task for search_connectors plugin. This task checks connector records and agentless package policies and sees if new connector was added/old was deleted, and then adds/deletes package policies for these connectors. Scenario 1: a new connector was added by a user/API call User creates an Elastic-managed connector: https://github.com/user-attachments/assets/38296e48-b281-4b2b-9750-ab0a47334b55 When the user is done, a package policy is created by this background task: https://github.com/user-attachments/assets/12dbc33f-32bf-472d-b854-64588fc1e5b1 Scenario 2: a connector was deleted by a user/API call User deletes an Elastic-managed connector: https://github.com/user-attachments/assets/5997897e-fb9d-4199-8045-abe163264976 ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Jedr Blaszyk <jedrazb@gmail.com>

…stic#206606) ## Summary This PR makes it so that the Agentless Kibana task implemented in elastic#203973 properly handles soft-deleted connectors. This helps with the situation when an integration policy has been created for an agentless connector but a connector record has not yet been created by an agentless host. With current Kibana task implementation it could lead to the Policy being deleted. With this change, only policies that refer to soft-deleted connectors will be cleaned up. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

seanstory reviewed Dec 12, 2024

View reviewed changes

x-pack/plugins/search_connectors/server/plugin.ts Outdated Show resolved Hide resolved

artem-shelkovnikov force-pushed the artem/add-agentless-connectors-task branch 2 times, most recently from 0500e73 to 7456f1b Compare December 25, 2024 11:37

artem-shelkovnikov changed the title ~~WIP~~ Introduce Kibana task to deploy agentless connectors for 9.0 Dec 25, 2024

artem-shelkovnikov added the release_note:skip Skip the PR/issue when compiling release notes label Dec 25, 2024

artem-shelkovnikov marked this pull request as ready for review December 25, 2024 12:26

artem-shelkovnikov requested review from a team as code owners December 25, 2024 12:26

artem-shelkovnikov commented Dec 25, 2024

View reviewed changes

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Dec 25, 2024

artem-shelkovnikov commented Dec 25, 2024

View reviewed changes

artem-shelkovnikov added backport:skip This PR does not require backporting Team:Search labels Dec 25, 2024

artem-shelkovnikov commented Dec 25, 2024

View reviewed changes

x-pack/plugins/search_connectors/server/task.ts Outdated

Copy link

Member Author

artem-shelkovnikov Dec 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we even need to retry, since we run pretty often?

artem-shelkovnikov requested a review from a team as a code owner December 27, 2024 19:08

artem-shelkovnikov added 10 commits December 30, 2024 12:49

WIP

4927e05

WIP 2

5716a4e

More changes, fun fun

45278cb

More WIP + tests

1aedf35

Revert artifacts/task.ts

e791be2

Minor tweaks

8c27d06

More tests + refactoring

8f996be

Make methods private again

65a50eb

Humanise the creation time

3c294fe

Clean up a comment

3ec5dce

artem-shelkovnikov added 2 commits December 30, 2024 20:06

Merge branch 'main' into artem/add-agentless-connectors-task

c59f0cb

Rebase fix

e6e96b8

juliaElastic approved these changes Jan 2, 2025

View reviewed changes

jedrazb reviewed Jan 2, 2025

View reviewed changes

artem-shelkovnikov and others added 4 commits January 2, 2025 17:25

Apply suggestions from code review

f53d034

Co-authored-by: Jedr Blaszyk <jedrazb@gmail.com>

[CI] Auto-commit changed files from 'node scripts/notice'

898020b

Improvements here and there

0d83497

Merge branch 'main' into artem/add-agentless-connectors-task

b96ec2e

jedrazb approved these changes Jan 3, 2025

View reviewed changes

mikecote reviewed Jan 3, 2025

View reviewed changes

artem-shelkovnikov requested review from mikecote and pmuellr January 6, 2025 09:15

artem-shelkovnikov added 2 commits January 6, 2025 10:25

Merge branch 'main' into artem/add-agentless-connectors-task

0f4db80

Merge branch 'main' into artem/add-agentless-connectors-task

75b22f7

pmuellr approved these changes Jan 7, 2025

View reviewed changes

artem-shelkovnikov added 4 commits January 9, 2025 11:27

Update interval to 1m

89b18b8

Merge branch 'main' into artem/add-agentless-connectors-task

3ec5393

Merge branch 'main' into artem/add-agentless-connectors-task

189ee5b

Merge branch 'main' into artem/add-agentless-connectors-task

9717116

Merge branch 'main' into artem/add-agentless-connectors-task

543af73

artem-shelkovnikov enabled auto-merge (squash) January 10, 2025 09:54

artem-shelkovnikov merged commit c88d519 into main Jan 10, 2025

artem-shelkovnikov deleted the artem/add-agentless-connectors-task branch January 10, 2025 11:22

kibanamachine added the v9.0.0 label Jan 10, 2025

artem-shelkovnikov mentioned this pull request Jan 14, 2025

Make Agentless Connectors task handle connector.deleted properly #206606

Merged

7 tasks

	const createTaskRunnerFactory =
	({
	logger,
	telemetry,
	executeEnrichPolicy,
	getStoreSize,
	}: {
	logger: Logger;
	telemetry: AnalyticsServiceSetup;
	executeEnrichPolicy: ExecuteEnrichPolicy;
	getStoreSize: GetStoreSize;
	}) =>
	({ taskInstance }: { taskInstance: ConcreteTaskInstance }) => {
	let cancelled = false;
	const isCancelled = () => cancelled;
	return {
	run: async () =>
	runTask({
	executeEnrichPolicy,
	getStoreSize,
	isCancelled,
	logger,
	taskInstance,
	telemetry,
	}),
	cancel: async () => {
	cancelled = true;
	},
	};
	};

Conversation

artem-shelkovnikov commented Dec 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Closes https://github.com/elastic/search-team/issues/8508

Closes https://github.com/elastic/search-team/issues/8465

Summary

Checklist

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Dec 25, 2024

Uh oh!

artem-shelkovnikov Dec 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliaElastic left a comment

Choose a reason for hiding this comment

Uh oh!

jedrazb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jedrazb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

artem-shelkovnikov Jan 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pmuellr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jan 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

Public APIs missing comments

API count

History

artem-shelkovnikov commented Dec 12, 2024 •

edited

Loading

artem-shelkovnikov Dec 25, 2024 •

edited

Loading

jedrazb left a comment •

edited

Loading

artem-shelkovnikov Jan 7, 2025 •

edited

Loading

elasticmachine commented Jan 9, 2025 •

edited

Loading