[9.0] [Security Solution] Reduce the _review rule upgrade endpoint response size (#211045)#212922
Merged
kibanamachine merged 1 commit intoelastic:9.0from Mar 3, 2025
Merged
Conversation
… size (elastic#211045) **Resolves: elastic#208361 **Resolves: elastic#210544 ## Summary This PR introduces significant memory consumption improvements to the prebuilt rule endpoints, ensuring users won't encounter OOM errors on memory-limited Kibana instances. Memory consumption testing results provided in elastic#211045 (comment). ## Details This PR implements a number of memory usage optimizations to the prebuilt rule endpoints with the final goal reducing chances of getting OOM errors. The changes are extensive and require thorough testing before merging. The changes are described by the following bullets - The most significant change is the addition of pagination to the `upgrade/_review` endpoint. This endpoint was known for causing OOM errors due to its large and ever-growing response size. With pagination, it now returns upgrade information for no more than 20-100 rules at a time, significantly reducing its memory footprint. - New backend methods, such as `ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced. These methods return rule IDs with their corresponding installed versions, allowing to build a map of outdated rules without loading all available rules into memory. Previously, all installed rules, along with their base and target versions, were fetched unconditionally before filtering for updates. - The `stats` data structure of the review endpoint has been deprecated (it can be safely removed after one Serverless release cycle). Since the endpoint now returns paginated results, building stats is no longer feasible due to the limited rule set size fetched on the server side. As the side effect it required removing related Cypress tests asserting `Update All` disabled when rules can't be updated. - All changes to the endpoints are backward-compatible. All previously required returned structures still present in response. All newly added structures are optional. - Upgradeable rule tags are now returned from the prebuilt rule status endpoint. - The frontend logic has been updated to move sorting and filtering of prebuilt rules from the client side to the server side. - The `upgrade/_perform` endpoint has been rewritten to use lightweight rule version information rather than full rules to determine upgradeable rules. Additionally, upgrades are now performed in batches of up to 100 rules, further reducing memory usage. - A dry run option has been added to the upgrade perform endpoint. This is needed for the "Update all" rules scenario to determine if any rules contain conflicts and display a confirmation modal to the user. - An option to skip conflicting rules has been added to the upgrade endpoint when called with the `ALL_RULES` mode. - The `install/_review` endpoint's memory consumption has been optimized by avoiding loading all rules into memory to determine available rules for installation. Redundant fetching of all base versions has also been removed, as they do not participate in the calculation. --------- Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co> (cherry picked from commit c4a016e)
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto9.0:Questions ?
Please refer to the Backport tool documentation