[8.15] [Automatic Import] Adding support for larger samples in ECS graph (#190426)#191314
Merged
P1llus merged 2 commits intoelastic:8.15from Aug 27, 2024
Merged
Conversation
…astic#190426) ## Summary This PR prepares the ECS Mapping graph to support larger samples by chunking and running certain parts of the graph concurrently side by side and merging the results rather than trying to use one large context. More details below, but in general there is only a slight modification to the actual code, most of the lines are related to moving code around to new files and updated tests. There are also some minor tweaks to the ECS graph code in general, below is the related changes: 1. Moved some code out of graph.ts to make it a bit smaller (moved model* functions to a new model.ts, moved state to its own file. 2. Added chunkSize as a optional input to the graph (default to 10 fields with an actual string value per chunk). Just to allow it to be overwritten if necessary later. 3. Renamed the `samples` state to `prefixedSamples` and `formattedSamples` to `combinedSamples` as it got really confusing at some point when debugging. I also updated the function argument names that used them to the new names to better understand which sample type they are using. 4. Renamed `modifySamples` to `prefixSamples` to clarify what it actually modifies 5. Moved `mapping`, `invalid`, `duplicate`, `missing` and `validate` nodes to its own subgraph. The `combinedSamples` state is now set when invoking the subgraph, the value will be its related `chunk`, so it only needs to work on this smaller subset of data. 6. The `currentMapping` state is now only used by the sub graph, once all the subgraphs has finished, the will post their own results to `finalMapping` state. This state uses a reducer function, that combines the existing state with the new, so all results from the X subgraphs running will be merged into the same resulting object as before this PR. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit 8e66a3e)
2 tasks
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Member
|
@elasticmachine merge upstream |
💚 Build Succeeded
Metrics [docs]Unknown metric groupsESLint disabled in files
Total ESLint disabled count
History
To update your PR or re-run it, just comment with: cc @P1llus |
bhapas
approved these changes
Aug 27, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto8.15:Questions ?
Please refer to the Backport tool documentation