Use atomic agent and package policy API in cloud connector task

[MichelLosier]

Summary

Resolves: https://github.com/elastic/ingest-dev/issues/5821

• Adds to AgentPolicyService a createWithPackagePolicies method based on the implementation of the /internal/fleet/agent_and_package_policies handler (link)
  • This allows it to be available from our plugin server API to be used outside user request contexts such as this background task.
  • This repeats the same set of operations in the original handler without pass through of other handlers.
  • I didn't go for code re-use here because extracting the core functionality from this flow to be re-used both in this method and in the original handler would also have involved extracting and recomposing auth and other operations from the dependent handlers into agent_and_package_policies and would have added too much complexity.

Steps

To reproduce issue on main:

1. Prepare the Kibana dev environment to appear as a cloud env with agentless enabled:

xpack.cloud.serverless.project_id: anything
xpack.cloud.id: local
xpack.cloud.serverless.project_type: observability
aiAssistantManagementSelection.preferredAIAssistantType: observability
xpack.fleet.agentless.enabled: true
xpack.fleet.agentless.api.url: 'https://127.0.0.1:8443'
xpack.fleet.agentless.api.tls.certificate: './config/certs/tls.crt'
xpack.fleet.agentless.api.tls.key: './config/certs/tls.key'
xpack.fleet.agentless.api.tls.ca: './config/certs/ca.crt'

## And logging to observe issue and solution

logging:
  loggers:
    - name: plugins.fleet
      appenders: [console]
      level: debug
    - name: plugins.contentConnectors
      appenders: [console]
      level: debug

2. Go to Stack Management > Content Connectors > Create Connector
3. Select a Content Connector like Github and click "Generate configuration"
4. The contentConnector task should try to start creating an agentless agent policy deployment from the connector configuration.
5. This will fail since local kibana is not setup with an agentless api endpoint, when the task runs again though you may see:

Error creating an agentless deployment for connector 8PNpV5gB13G-58ghrG35: Unable to create integration. Agentless policy for github connector: 8PNpV5gB13G-58ghrG35 already exist. Please rename the integration name 8PNpV5gB13G-58ghrG35.

On this branch:
Run through the same steps above, but you should not get stuck on name collisions with existing policies. It should just keep failing on not being able to reach the agentless api.

If you want to observe a successful creation, you can mock the agentless api response by commenting out the agentless api request in and returning

    const response = {
      status: 'SUCCESS',
      error: null,
    };

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
• Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

• See some risk examples
• ...

[MichelLosier]

Callout: I removed this conditional here, as it seems to be if we are creating an agentless agent policy, but fail to actually deploy it we should delete the policy so its not orphaned. I can't think of a case where we would want this to still stick around.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[MichelLosier]

Worth mentioning, when packagePolicyService.delete is called in the clean up and runPostDeleteExternalCallbacks is ran we do get this gnarly error:

Attempt to create index [.logs-endpoint.actions-default] failed with: illegal_argument_exception
        Root causes:
                illegal_argument_exception: no matching index template found for data stream [.logs-endpoint.actions-default]
ResponseActionsClientError: Attempt to create index [.logs-endpoint.actions-default] failed with: illegal_argument_exception
        Root causes:
                illegal_argument_exception: no matching index template found for data stream [.logs-endpoint.actions-default]
    at ensure_action_requests_index_is_configured.ts:30:13
    at processTicksAndRejections (node:internal/process/task_queues:105:5)
    at ensureActionRequestsIndexIsConfigured (ensure_action_requests_index_is_configured.ts:27:3)
    at updateDeletedPolicyResponseActions (update_deleted_policy_response_actions.ts:33:3)
    at async Promise.all (index 0)
    at fleet_integration.ts:498:5
    at PackagePolicyClientImpl.runPostDeleteExternalCallbacks (package_policy.ts:2333:11)
    at PackagePolicyClientImpl.delete (package_policy.ts:2016:7)
    at AgentPolicyService.delete (agent_policy.ts:1547:9)
    at createAgentPolicyWithPackages (agent_policy_create.ts:225:7)
    at AgentPolicyService.createWithPackagePolicies (agent_policy.ts:539:28)
    at AgentlessConnectorsInfraService.deployConnector (index.ts:239:25)
    at Object.run (task.ts:80:15)
    at TaskManagerRunner.run (task_runner.ts:404:22)

I'm not sure what causes this, and whether if its just something missing in my local setup

Update:

This is due to the .logs-endpoint.actions-default endpoint data_stream not being setup in any successful form prior to deletion. If I separately install Elastic Defend on an agent, this error clears up on these deletions

[juliaElastic]

LGTM

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: de78a2b

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
fleet	1459	1465	+6

Unknown metric groups

API count

id	before	after	diff
fleet	1591	1597	+6

History

• 💚 Build #325726 succeeded e53dead
• 💚 Build #325364 succeeded 88cdf2a
• 💔 Build #325354 failed fc9c066

cc @MichelLosier

[yansavitski]

nitpick: would be good to see a return type here Promise<AgentPolicyServiceInterface['createWithPackagePolicies']>

[yansavitski]

LGTM