[Security Solution] add defend insights elastic assistant tool

[joeypoon]

Summary

Adds the new Defend Insights Elastic Assistant tool. This assistant tool provides Elastic Defend configuration insights. For this initial PR, only incompatible antivirus detection is supported. Telemetry is collected for success and error events.

For incompatible antivirus detection, Defend Insights will review the last 200 file events for the given endpoint and output suspected antiviruses. Improvements such as customizable event count and date range will come in the future.

This PR does not include any UI, that will come in a separate PR. 3 internal APIs for interacting with Defend Insights are provided here:

• POST /defend_insights for creating a new Defend Insight
• GET /defend_insights/{id} for getting a Defend Insight
• GET /defend_insights for getting multiple Defend Insights
  • available optional query params:
    • size - default 10
    • ids
    • connector_id
    • type - incompatible_antivirus
    • status - running, completed, failed, canceled
    • endpoint_ids

This initial implementation does not include the LangGraph/output chunking upgrades seen in Attack Discovery due to time constraints. We'll look to make this upgrade in a future PR.

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[elasticmachine]

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

[paul-tavares]

Left some feedback that I would I would like to hear back on. Note that I focused only on files under security solution.

Also - why is there no feature flag for this code? With out one it just be available in serverless with the next release after it is merged to main - is that ok?

[joeypoon]

Left some feedback that I would I would like to hear back on. Note that I focused only on files under security solution.

Thanks for the review @paul-tavares. Responded directly on each one. Will update code for all of the ones I reacted with 👍.

Also - why is there no feature flag for this code? With out one it just be available in serverless with the next release after it is merged to main - is that ok?

That's an oversight by me. It's technically an internal API but we should still add a feature flag. Will update.

[andrew-goldstein]

Thanks @joeypoon!
LGTM 🚀

[tomsonpl]

Hey @joeypoon 👋 so exciting to see the cool stuff here!

However, given this PR has +++4.5k lines, it's very difficult to actually get a solid understanding and therefore do a review.
Do you think it's possible to merge this functionality in smaller chunks?
edit:
Or maybe not this particular PR since it's been open for a while (sorry for a delay in reviewing), but at least for the future ones?

[joeypoon]

Hey @joeypoon 👋 so exciting to see the cool stuff here!

However, given this PR has +++4.5k lines, it's very difficult to actually get a solid understanding and therefore do a review. Do you think it's possible to merge this functionality in smaller chunks? edit: Or maybe not this particular PR since it's been open for a while (sorry for a delay in reviewing), but at least for the future ones?

I agree with you in concept but unfortunately this one is really hard to split up. In fact, this is already split up and there will be quite a few more PRs following this 😅. Happy to walk through it with you if it helps but breaking this one up any smaller was unfortunately not feasible.

[paul-tavares]

Thanks for adding this. I searched around and I don't actually see it being used... Where is it checked?

[joeypoon]

Each of the 3 API handlers check for it like this. We don't actually use it in security_solution plugin yet but will for following PRs.

[paul-tavares]

@joeypoon - Thats strange (to me). So somehow this elastic_assistant plugin has access to our internal features? The code you pointed to is a bit hard to follow. Do you know how where it is actually is checking the feature flag is enabled/disabled?

[joeypoon]

We're checking for it at the beginning of all 3 new API handlers (in elastic_assistant) and that definition for those is actually here. The definition in x-pack/plugins/security_solution/common/experimental_features.ts isn't actually being used right now and is just prep for next PRs to have less conflicts.

[paul-tavares]

yeah. ok. I'm just a bit confused how the FF from security solution will be used in another plugin.

[joeypoon]

yeah. ok. I'm just a bit confused how the FF from security solution will be used in another plugin.

It's not, it will be used within the security solution plugin. It's only introduced here since it's the first PR being merged for the overall feature.

[paul-tavares]

Thanks for the changes. I left a question regarding the feature flag, but am approving and you can just answer it when you can

[tomsonpl]

LGTM 👍 Left a few questions, but just out of curiosity. Thanks!

[tomsonpl]

Just a thought:
Since _source: true is included in the request, wouldn’t every document already have a _source field?
I’m wondering if this filter is still necessary in that case.

[tomsonpl]

Ah, now I see - you’re ensuring _source is always present. Just to clarify, does it actually go missing sometimes, or is this primarily to satisfy TypeScript?

[tomsonpl]

I’m not sure if using the bang (!) operator in these places is the best approach. Wouldn’t it be clearer and safer to define the response type more explicitly instead?

[joeypoon]

SearchResponse is defined at ES level and both _id and _source are defined as optional. To your point though, we can do a null check instead.

[tomsonpl]

Why not just:

users: insightSchema.users ?? [];

?

[joeypoon]

Honestly, I copied this over from attack discovery and just changed the necessary keys 😅. My impression though is that we want to ensure we don't potentially leak additional fields? and maybe also just because new objects.

[kibanamachine]

Starting backport for target branches: 7.17, 8.15, 8.16, 8.18, 8.x

https://github.com/elastic/kibana/actions/runs/11910742167

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0059617

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
integrationAssistant	466	471	+5
securitySolution	6213	6218	+5
total			+10

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/elastic-assistant-common	369	405	+36

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	13.4MB	13.4MB	+97.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	86.1KB	86.1KB	+18.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/elastic-assistant-common	400	442	+42

ESLint disabled in files

id	before	after	diff
elasticAssistant	0	5	+5

ESLint disabled line counts

id	before	after	diff
elasticAssistant	32	34	+2

Total ESLint disabled count

id	before	after	diff
elasticAssistant	32	39	+7

History

• 💚 Build #251855 succeeded eb9ab8319a6e3940d509ca2d26356ba75ba572a6
• 💔 Build #251785 failed 6ba79a6bfb78730cc56953194c667c62c7fc9ece
• 💚 Build #250868 succeeded 86812fa1957bed1e3e135f7c231f13a2a56b1765
• 💔 Build #250736 failed 9e0a42fdc4d078d450d157b8d62031f28b3ba9ee
• 💛 Build #248060 was flaky e67cdf4e9487a9a1ab7bb73a960788228ae5fa8e
• 💔 Build #247986 failed 68d1c85ca3543cfd1ec2c40390d1fde7ebbbfad6

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	7.17	Backport failed because of merge conflicts
❌	8.15	Backport failed because of merge conflicts
❌	8.16	Backport failed because of merge conflicts
❌	8.18	The branch "8.18" does not exist
❌	8.x	Backport failed because of merge conflicts You might need to backport the following PRs to 8.x: - [Cloud Security] Fixed an issue with Host.name Alerts contextual flyout (#200626) - Surface Kibana security route deprecations in Upgrade Assistant (#199656) - [Infra] Use callback for logger.trace calls (#199805) - [Security Solution][Bidirectional Integrations Banner][Crowdstrike][SentinelOne] Banner for bidirectional integrations (#200625) - [Fleet] Use metering API in serverless (#200063) - Migrate /test/apm_api_integration/tests/suggestions to be deployment agnostic api tests (#200556) - [Inventory] Remove open in Discover button (#200574) - [Entity Analytics] API changes for right placement of deleting the old component template (#199734)

Manual backport

To create the backport manually run:

node scripts/backport --pr 198676

Questions ?

Please refer to the Backport tool documentation

[joeypoon]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[kibanamachine]

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.