[EDR Workflows][Artifact transfer 3] Import API validator

[gergoabraham]

prerequisites:

• [EDR Workflows][Artifact transfer 1] Export Endpoint artifacts #247967
• [EDR Workflows][Artifact transfer 2] Import UI #247976

follow-up:

• [EDR Workflows][Artifact transfer 4] UI fine tuning #257983

closes #250470

Summary

Note

Hidden behind feature flag (as part of the Endpoint exception move effort):

xpack.securitySolution.enableExperimental:
 - endpointExceptionsMovedUnderManagement

This PR allows importing all Endpoint artifacts! Lot's of changes, here's a summary written by Claude, plus some additional information:

This PR updates the Endpoint artifact import validator registered for the POST /api/exception_lists/_import API. The changes are gated behind the endpointExceptionsMovedUnderManagement feature flag.

What's changed

Import validation overhaul (getExceptionsPreImportHandler):

• Replaces the old simple "block all non-endpoint-exception artifacts" logic with full per-artifact-type validation support
• When the feature flag is enabled, dispatches validation to the appropriate artifact-specific validator (TrustedAppValidator, BlocklistValidator, EventFilterValidator, HostIsolationExceptionsValidator, TrustedDeviceValidator, EndpointExceptionsValidator)
• Supports importing only one artifact list type per import call (returns 400 if multiple types are mixed)
• Handles overwrite semantics by deleting visible items before import (respecting user permissions), then disabling the Lists API's own overwrite to avoid full-list deletion

Space-aware validation:

• Each imported item is validated against the user's space permissions: non-global artifact management users can only import items owned by the current space; users with global artifact management privilege can import items from other spaces, provided those items are visible in the current space
• Invalid owner space IDs are rejected (this means error); invalid policy IDs are stripped and noted in a comment on the item

On error handling:

• if there's a general problem (like no user access), the request is rejected.
• if there's a per-item problem, the request succeeds, but it'll contain the errors per item, which errors we can later show to the user

Data enrichment on import:

• Adds the imported_artifact tag to all imported items
• Adds an audit comment to each item with the original author and creation date
• Adds the ownerSpaceId tag to items that lack one (backward compatibility for pre-space-awareness exports)

ExceptionsListPreImportServerExtension type change:

• The extension point's data type changed from PromiseFromStreams → { data: PromiseFromStreams; overwrite: boolean }, allowing the extension to inspect and override the overwrite flag

ExceptionListClient.bulkDeleteExceptionListItems:

• New public method for bulk-deleting exception list items (replaces the previous slow per-item sequential deletion via asyncForEach)

ExceptionItemImportError class:

• New error class implementing BulkErrorErrorSchema, used to surface per-item validation errors in the import response (instead of aborting the entire import)

Tests:

• Comprehensive new FTR integration test suite (artifact_import.ts) covering privilege checks, space-awareness scenarios, invalid data handling, overwrite behavior, comment/tag enrichment, and backward compatibility

Testing

To ease testing, I added some ugly details (58fcfb3) to the success toast, but this is not something that will stay.

Follow-up PR possibilities

There are some things we probably also need to add, but I'd like to keep them out of this PR. I can think of these:

• improve UI feedback to user on errors (the toast doesn't look ideal)
• do not allow any artifact type be imported on any artifact pages and rule exceptions page. this is now allowed, since the API endpoint is the same. (we could fix this e.g. by adding a new query parameter to the import API, or by adding a new endpoint artifact import API that calls the lists import service)

Follow-up PR is already in-progress:

• [EDR Workflows][Artifact transfer 4] UI fine tuning #257983

Screenshots

Some error messages

Response:

{
    "message": "EndpointArtifactError: Importing multiple Endpoint artifact exception list types at the same time is not supported",
    "status_code": 400
}

☝️ probably won't be like this, it's just to visualize the response for this PR. will be improved in a follow-up pr

Response:

{
    "errors": [
        {
            "error": {
                "status_code": 403,
                "message": "EndpointArtifactError: Importing artifacts with invalid owner space IDs is not allowed. The following space ID is invalid or unaccessible by current user: nope"
            },
            "list_id": "endpoint_host_isolation_exceptions",
            "item_id": "5cc4585e-edbb-445f-a0d6-a6a67bb29d04"
        },
        {
            "error": {
                "status_code": 403,
                "message": "EndpointArtifactError: Endpoint authorization failure. Importing artifacts that are not visible in the current space is not allowed"
            },
            "list_id": "endpoint_host_isolation_exceptions",
            "item_id": "4c791b4a-5e87-4eb3-9713-1c80f6c98b80"
        }
    ],
    "success": false,
    "success_count": 1,
    "success_exception_lists": true,
    "success_count_exception_lists": 0,
    "success_exception_list_items": false,
    "success_count_exception_list_items": 1
}

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
• Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

• See some risk examples
• ...

[rylnd]

Detection Engine changes (shared Lists code) LGTM.

[rylnd]

While this isn't worse than the code it's replacing: is there a blocker to adding a bulkGet to allow this?

[gergoabraham]

yeah, that's good question. I don't know about any blocker - I think this was the point where I intentionally lost momentum, and instead of going down the rabbit hole focused on getting this functionality ready

[paul-tavares]

Hi @gergo - I did some code review, but did not finish it - got up to the base validator. I may not have enought context or knownledge of this feature, so some of my comments may not actually be applicable. A few things that raised concern for me:

• when overwrite is true, we seem to delete items before even attempting to import new ones. This could lead to a bad state of the customer's env. if the actual import fails.
• I'm not clear why we are validating space owner tags for imports as well as whether the spaces are valid in the current environment. This seems to suggest that a user will no longer be able to export exceptions from one environemnt/space and import it into another - without having to manipulate the export manually. I would have invisioned that the space owner id tag would be ignored and all imported items would be owned by the space where the import is happening

I may reach out to you tomorrow to disucss this further.

[paul-tavares]

can you add a return Type to this function

[gergoabraham]

f16665b

[paul-tavares]

can you add a return Type here

[gergoabraham]

f16665b

[paul-tavares]

Careful with .info() loggers. Maybe move this to .debug()? Same comment for the `logger.info() on line 295 below

[gergoabraham]

5a64b4f

[paul-tavares]

strange - I'm still seeing logger.info() although you referenced a commit where you seem to have moved them to .debug() 🤔

[gergoabraham]

🤦
because in the next commit i changed it back to info 😅 probably when testing how to delete more than 1k artifacts

thanks, Paul : )
067fc20

[paul-tavares]

This is allot of looping through the data. If the imports are capped at low numbers, then it should be ok... but perhpas consider looping through the data items less often by making the necessary adjustment from one loop instead.

[gergoabraham]

e1ba1ff

[gergoabraham]

hey @paul-tavares, thanks a lot for your feedback!

when overwrite is true, we seem to delete items before even attempting to import new ones. This could lead to a bad state of the customer's env. if the actual import fails.

yep, that's unfortunately a valid scenario. i guess moving the deletion into a 'post-import' handler won't help as well, since we can still have the problem of successfully importing new artifacts, and then not deleting the old ones in case of any problem.

looked into the existing import, and it's also cannot be considered atomic, so i think we should be okay with this. the existing import also deletes the existing items first (called from here), and only then creates the new ones.

I'm not clear why we are validating space owner tags for imports as well as whether the spaces are valid in the current environment. This seems to suggest that a user will no longer be able to export exceptions from one environemnt/space and import it into another - without having to manipulate the export manually. I would have invisioned that the space owner id tag would be ignored and all imported items would be owned by the space where the import is happening

i see your point. validating owner space IDs was a request from product side, you can see the questions/answers in the issue linked in the development section.
afaik space IDs are based on their names, so if you have the same space names when migrating to a new environment, that should work perfectly. copying exceptions to another space, well... that use case is indeed not supported with this behavior. cc @roxana-gheorghe

[paul-tavares]

Thanks @gergoabraham for the above response. Seems as if my concerns has already come up with discussion with PM. I will complete the code review here soon.

[paul-tavares]

Ok. I completed the code review - no other significant feedback from me.

[gergoabraham]

Thanks @gergoabraham for the above response. Seems as if my concerns has already come up with discussion with PM. I will complete the code review here soon.

thank you @paul-tavares for your feedback, i always appreciate it! 🙌

[paul-tavares]

Thanks for the changes.

[paul-tavares]

(optional) For large amounts of data, it might be best to handle the deletes from within executeFunctionOnStream() and use an async queue to process them (ex. using QueueProcessor), rather than to collect all IDs and potentially send off a very large bulk delete.

[paul-tavares]

strange - I'm still seeing logger.info() although you referenced a commit where you seem to have moved them to .debug() 🤔

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 067fc20

Failed CI Steps

• FTR Configs #177
• Serverless Detection Engine - Security Solution Cypress Tests #7

Test Failures

• [job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #7 / Detection ES|QL rules - Rule Creation ES|QL investigation fields shows custom ES|QL field in investigation fields autocomplete and saves it in rule shows custom ES|QL field in investigation fields autocomplete and saves it in rule
• [job] [logs] Serverless Detection Engine - Security Solution Cypress Tests #7 / Detection ES|QL rules - Rule Creation ES|QL query validation allows non-aggregating ES|QL query without metadata operator (auto-injected at execution) allows non-aggregating ES|QL query without metadata operator (auto-injected at execution)
• [job] [logs] FTR Configs #177 / discover - group 2 feature controls discover feature controls security global discover read-only privileges "before all" hook for "shows discover navlink"

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/securitysolution-io-ts-list-types	521	522	+1
lists	98	99	+1
total			+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	11.4MB	11.4MB	+307.0B

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
lists	52	53	+1

Unknown metric groups

API count

id	before	after	diff
@kbn/securitysolution-io-ts-list-types	535	536	+1
lists	227	229	+2
total			+3

ESLint disabled line counts

id	before	after	diff
lists	31	30	-1

Total ESLint disabled count

id	before	after	diff
lists	32	31	-1

History

• 💛 Build #411549 was flaky bdaca74
• 💛 Build #409689 was flaky 89fbdeb
• 💔 Build #409596 failed ba330c4
• 💛 Build #409054 was flaky 36e8ea2
• 💔 Build #407764 failed 6458bbc
• 💔 Build #406302 failed 7542077

cc @gergoabraham