[SIEM][Detection Engine] Speeds up value list imports by enabling streaming of files.

[FrankHassanabad]

Summary

• Changes the value list imports to use a streaming in model
• Adds a custom light hand spun multi-part parser for the incoming text
• Adds a buffer pause and resume which continues to buffer the incoming data if an async event such as creating a list from the attachment file needs to happen but does not emit the lines until the resume continues.
• Adds a data slicing if the buffer becomes larger than the maximum so that if we begin buffering too quickly within memory we don't blow up the limit of Elastic Search.
• Adds unit tests

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[FrankHassanabad]

@elasticmachine merge upstream

[FrankHassanabad]

@elasticmachine merge upstream

[FrankHassanabad]

@elasticmachine merge upstream

[rylnd]

I got a walkthrough of the streaming logic from @FrankHassanabad, and the code looks good!

I tested this against #67068, and it decreased import time of a 1.2kB file from 52s to 444ms. So yeah, this is an improvement to say the least 😉 .

[rylnd]

I was gonna say we should leave the comment, but the typescript error generated if one removes this method should be straightforward 👍

[yctercero]

LGTM. Saw Ryland pulled down to test, so was just looking at the code. This is really awesome!

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 53d1013

Build metrics

✅ unchanged

History

• 💚 Build #59767 succeeded 880c1c5
• 💔 Build #59644 failed 9d9fd61
• 💔 Build #59634 failed f66c152
• 💚 Build #59605 succeeded a4df0a7
• 💚 Build #59446 succeeded 3f026e1

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)