[Security Solution] Reduce the _review rule upgrade endpoint response size#211045
Merged
xcrzx merged 10 commits intoelastic:mainfrom Mar 3, 2025
xcrzx:review-rework
Merged
[Security Solution] Reduce the _review rule upgrade endpoint response size#211045xcrzx merged 10 commits intoelastic:mainfrom xcrzx:review-rework
xcrzx merged 10 commits intoelastic:mainfrom
xcrzx:review-rework
Conversation
Contributor
|
PR Cloud deployment started at: https://buildkite.com/elastic/kibana-deploy-cloud-from-pr/builds/51 |
xcrzx
added
performance
v9.0.0
Team:Detections and Resp
Contributor
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
Contributor
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
Contributor
|
Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management) |
Contributor
|
Starting backport for target branches: 8.17, 8.18, 8.x, 9.0 https://github.com/elastic/kibana/actions/runs/13632673583 |
4 tasks
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Mar 3, 2025
… size (elastic#211045) **Resolves: elastic#208361 **Resolves: elastic#210544 ## Summary This PR introduces significant memory consumption improvements to the prebuilt rule endpoints, ensuring users won't encounter OOM errors on memory-limited Kibana instances. Memory consumption testing results provided in elastic#211045 (comment). ## Details This PR implements a number of memory usage optimizations to the prebuilt rule endpoints with the final goal reducing chances of getting OOM errors. The changes are extensive and require thorough testing before merging. The changes are described by the following bullets - The most significant change is the addition of pagination to the `upgrade/_review` endpoint. This endpoint was known for causing OOM errors due to its large and ever-growing response size. With pagination, it now returns upgrade information for no more than 20-100 rules at a time, significantly reducing its memory footprint. - New backend methods, such as `ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced. These methods return rule IDs with their corresponding installed versions, allowing to build a map of outdated rules without loading all available rules into memory. Previously, all installed rules, along with their base and target versions, were fetched unconditionally before filtering for updates. - The `stats` data structure of the review endpoint has been deprecated (it can be safely removed after one Serverless release cycle). Since the endpoint now returns paginated results, building stats is no longer feasible due to the limited rule set size fetched on the server side. As the side effect it required removing related Cypress tests asserting `Update All` disabled when rules can't be updated. - All changes to the endpoints are backward-compatible. All previously required returned structures still present in response. All newly added structures are optional. - Upgradeable rule tags are now returned from the prebuilt rule status endpoint. - The frontend logic has been updated to move sorting and filtering of prebuilt rules from the client side to the server side. - The `upgrade/_perform` endpoint has been rewritten to use lightweight rule version information rather than full rules to determine upgradeable rules. Additionally, upgrades are now performed in batches of up to 100 rules, further reducing memory usage. - A dry run option has been added to the upgrade perform endpoint. This is needed for the "Update all" rules scenario to determine if any rules contain conflicts and display a confirmation modal to the user. - An option to skip conflicting rules has been added to the upgrade endpoint when called with the `ALL_RULES` mode. - The `install/_review` endpoint's memory consumption has been optimized by avoiding loading all rules into memory to determine available rules for installation. Redundant fetching of all base versions has also been removed, as they do not participate in the calculation. --------- Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co> (cherry picked from commit c4a016e)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Mar 3, 2025
… size (elastic#211045) **Resolves: elastic#208361 **Resolves: elastic#210544 ## Summary This PR introduces significant memory consumption improvements to the prebuilt rule endpoints, ensuring users won't encounter OOM errors on memory-limited Kibana instances. Memory consumption testing results provided in elastic#211045 (comment). ## Details This PR implements a number of memory usage optimizations to the prebuilt rule endpoints with the final goal reducing chances of getting OOM errors. The changes are extensive and require thorough testing before merging. The changes are described by the following bullets - The most significant change is the addition of pagination to the `upgrade/_review` endpoint. This endpoint was known for causing OOM errors due to its large and ever-growing response size. With pagination, it now returns upgrade information for no more than 20-100 rules at a time, significantly reducing its memory footprint. - New backend methods, such as `ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced. These methods return rule IDs with their corresponding installed versions, allowing to build a map of outdated rules without loading all available rules into memory. Previously, all installed rules, along with their base and target versions, were fetched unconditionally before filtering for updates. - The `stats` data structure of the review endpoint has been deprecated (it can be safely removed after one Serverless release cycle). Since the endpoint now returns paginated results, building stats is no longer feasible due to the limited rule set size fetched on the server side. As the side effect it required removing related Cypress tests asserting `Update All` disabled when rules can't be updated. - All changes to the endpoints are backward-compatible. All previously required returned structures still present in response. All newly added structures are optional. - Upgradeable rule tags are now returned from the prebuilt rule status endpoint. - The frontend logic has been updated to move sorting and filtering of prebuilt rules from the client side to the server side. - The `upgrade/_perform` endpoint has been rewritten to use lightweight rule version information rather than full rules to determine upgradeable rules. Additionally, upgrades are now performed in batches of up to 100 rules, further reducing memory usage. - A dry run option has been added to the upgrade perform endpoint. This is needed for the "Update all" rules scenario to determine if any rules contain conflicts and display a confirmation modal to the user. - An option to skip conflicting rules has been added to the upgrade endpoint when called with the `ALL_RULES` mode. - The `install/_review` endpoint's memory consumption has been optimized by avoiding loading all rules into memory to determine available rules for installation. Redundant fetching of all base versions has also been removed, as they do not participate in the calculation. --------- Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co> (cherry picked from commit c4a016e)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Mar 3, 2025
… size (elastic#211045) **Resolves: elastic#208361 **Resolves: elastic#210544 ## Summary This PR introduces significant memory consumption improvements to the prebuilt rule endpoints, ensuring users won't encounter OOM errors on memory-limited Kibana instances. Memory consumption testing results provided in elastic#211045 (comment). ## Details This PR implements a number of memory usage optimizations to the prebuilt rule endpoints with the final goal reducing chances of getting OOM errors. The changes are extensive and require thorough testing before merging. The changes are described by the following bullets - The most significant change is the addition of pagination to the `upgrade/_review` endpoint. This endpoint was known for causing OOM errors due to its large and ever-growing response size. With pagination, it now returns upgrade information for no more than 20-100 rules at a time, significantly reducing its memory footprint. - New backend methods, such as `ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced. These methods return rule IDs with their corresponding installed versions, allowing to build a map of outdated rules without loading all available rules into memory. Previously, all installed rules, along with their base and target versions, were fetched unconditionally before filtering for updates. - The `stats` data structure of the review endpoint has been deprecated (it can be safely removed after one Serverless release cycle). Since the endpoint now returns paginated results, building stats is no longer feasible due to the limited rule set size fetched on the server side. As the side effect it required removing related Cypress tests asserting `Update All` disabled when rules can't be updated. - All changes to the endpoints are backward-compatible. All previously required returned structures still present in response. All newly added structures are optional. - Upgradeable rule tags are now returned from the prebuilt rule status endpoint. - The frontend logic has been updated to move sorting and filtering of prebuilt rules from the client side to the server side. - The `upgrade/_perform` endpoint has been rewritten to use lightweight rule version information rather than full rules to determine upgradeable rules. Additionally, upgrades are now performed in batches of up to 100 rules, further reducing memory usage. - A dry run option has been added to the upgrade perform endpoint. This is needed for the "Update all" rules scenario to determine if any rules contain conflicts and display a confirmation modal to the user. - An option to skip conflicting rules has been added to the upgrade endpoint when called with the `ALL_RULES` mode. - The `install/_review` endpoint's memory consumption has been optimized by avoiding loading all rules into memory to determine available rules for installation. Redundant fetching of all base versions has also been removed, as they do not participate in the calculation. --------- Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co> (cherry picked from commit c4a016e)
Contributor
💔 Some backports could not be created
Note: Successful backport PRs will be merged automatically after passing CI. Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Mar 3, 2025
…esponse size (#211045) (#212920) # Backport This will backport the following commits from `main` to `8.18`: - [[Security Solution] Reduce the _review rule upgrade endpoint response size (#211045)](#211045) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Dmitrii Shevchenko","email":"dmitrii.shevchenko@elastic.co"},"sourceCommit":{"committedDate":"2025-03-03T14:03:07Z","message":"[Security Solution] Reduce the _review rule upgrade endpoint response size (#211045)\n\n**Resolves: https://github.com/elastic/kibana/issues/208361**\n**Resolves: https://github.com/elastic/kibana/issues/210544**\n\n## Summary\n\nThis PR introduces significant memory consumption improvements to the\nprebuilt rule endpoints, ensuring users won't encounter OOM errors on\nmemory-limited Kibana instances.\n\nMemory consumption testing results provided in\nhttps://github.com//pull/211045#issuecomment-2689854328.\n\n## Details\n\nThis PR implements a number of memory usage optimizations to the\nprebuilt rule endpoints with the final goal reducing chances of getting\nOOM errors. The changes are extensive and require thorough testing\nbefore merging.\n\nThe changes are described by the following bullets\n\n- The most significant change is the addition of pagination to the\n`upgrade/_review` endpoint. This endpoint was known for causing OOM\nerrors due to its large and ever-growing response size. With pagination,\nit now returns upgrade information for no more than 20-100 rules at a\ntime, significantly reducing its memory footprint.\n- New backend methods, such as\n`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.\nThese methods return rule IDs with their corresponding installed\nversions, allowing to build a map of outdated rules without loading all\navailable rules into memory. Previously, all installed rules, along with\ntheir base and target versions, were fetched unconditionally before\nfiltering for updates.\n- The `stats` data structure of the review endpoint has been deprecated\n(it can be safely removed after one Serverless release cycle). Since the\nendpoint now returns paginated results, building stats is no longer\nfeasible due to the limited rule set size fetched on the server side. As\nthe side effect it required removing related Cypress tests asserting\n`Update All` disabled when rules can't be updated.\n- All changes to the endpoints are backward-compatible. All previously\nrequired returned structures still present in response. All newly added\nstructures are optional.\n- Upgradeable rule tags are now returned from the prebuilt rule status\nendpoint.\n- The frontend logic has been updated to move sorting and filtering of\nprebuilt rules from the client side to the server side.\n- The `upgrade/_perform` endpoint has been rewritten to use lightweight\nrule version information rather than full rules to determine upgradeable\nrules. Additionally, upgrades are now performed in batches of up to 100\nrules, further reducing memory usage.\n- A dry run option has been added to the upgrade perform endpoint. This\nis needed for the \"Update all\" rules scenario to determine if any rules\ncontain conflicts and display a confirmation modal to the user.\n- An option to skip conflicting rules has been added to the upgrade\nendpoint when called with the `ALL_RULES` mode.\n- The `install/_review` endpoint's memory consumption has been optimized\nby avoiding loading all rules into memory to determine available rules\nfor installation. Redundant fetching of all base versions has also been\nremoved, as they do not participate in the calculation.\n\n---------\n\nCo-authored-by: Maxim Palenov <maxim.palenov@elastic.co>","sha":"c4a016eda30ae8f224fdd485a634dc6773898e31","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","performance","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0","v9.1.0","v8.19.0","v8.17.3"],"title":"[Security Solution] Reduce the _review rule upgrade endpoint response size","number":211045,"url":"https://github.com/elastic/kibana/pull/211045","mergeCommit":{"message":"[Security Solution] Reduce the _review rule upgrade endpoint response size (#211045)\n\n**Resolves: https://github.com/elastic/kibana/issues/208361**\n**Resolves: https://github.com/elastic/kibana/issues/210544**\n\n## Summary\n\nThis PR introduces significant memory consumption improvements to the\nprebuilt rule endpoints, ensuring users won't encounter OOM errors on\nmemory-limited Kibana instances.\n\nMemory consumption testing results provided in\nhttps://github.com//pull/211045#issuecomment-2689854328.\n\n## Details\n\nThis PR implements a number of memory usage optimizations to the\nprebuilt rule endpoints with the final goal reducing chances of getting\nOOM errors. The changes are extensive and require thorough testing\nbefore merging.\n\nThe changes are described by the following bullets\n\n- The most significant change is the addition of pagination to the\n`upgrade/_review` endpoint. This endpoint was known for causing OOM\nerrors due to its large and ever-growing response size. With pagination,\nit now returns upgrade information for no more than 20-100 rules at a\ntime, significantly reducing its memory footprint.\n- New backend methods, such as\n`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.\nThese methods return rule IDs with their corresponding installed\nversions, allowing to build a map of outdated rules without loading all\navailable rules into memory. Previously, all installed rules, along with\ntheir base and target versions, were fetched unconditionally before\nfiltering for updates.\n- The `stats` data structure of the review endpoint has been deprecated\n(it can be safely removed after one Serverless release cycle). Since the\nendpoint now returns paginated results, building stats is no longer\nfeasible due to the limited rule set size fetched on the server side. As\nthe side effect it required removing related Cypress tests asserting\n`Update All` disabled when rules can't be updated.\n- All changes to the endpoints are backward-compatible. All previously\nrequired returned structures still present in response. All newly added\nstructures are optional.\n- Upgradeable rule tags are now returned from the prebuilt rule status\nendpoint.\n- The frontend logic has been updated to move sorting and filtering of\nprebuilt rules from the client side to the server side.\n- The `upgrade/_perform` endpoint has been rewritten to use lightweight\nrule version information rather than full rules to determine upgradeable\nrules. Additionally, upgrades are now performed in batches of up to 100\nrules, further reducing memory usage.\n- A dry run option has been added to the upgrade perform endpoint. This\nis needed for the \"Update all\" rules scenario to determine if any rules\ncontain conflicts and display a confirmation modal to the user.\n- An option to skip conflicting rules has been added to the upgrade\nendpoint when called with the `ALL_RULES` mode.\n- The `install/_review` endpoint's memory consumption has been optimized\nby avoiding loading all rules into memory to determine available rules\nfor installation. Redundant fetching of all base versions has also been\nremoved, as they do not participate in the calculation.\n\n---------\n\nCo-authored-by: Maxim Palenov <maxim.palenov@elastic.co>","sha":"c4a016eda30ae8f224fdd485a634dc6773898e31"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x","8.17"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/211045","number":211045,"mergeCommit":{"message":"[Security Solution] Reduce the _review rule upgrade endpoint response size (#211045)\n\n**Resolves: https://github.com/elastic/kibana/issues/208361**\n**Resolves: https://github.com/elastic/kibana/issues/210544**\n\n## Summary\n\nThis PR introduces significant memory consumption improvements to the\nprebuilt rule endpoints, ensuring users won't encounter OOM errors on\nmemory-limited Kibana instances.\n\nMemory consumption testing results provided in\nhttps://github.com//pull/211045#issuecomment-2689854328.\n\n## Details\n\nThis PR implements a number of memory usage optimizations to the\nprebuilt rule endpoints with the final goal reducing chances of getting\nOOM errors. The changes are extensive and require thorough testing\nbefore merging.\n\nThe changes are described by the following bullets\n\n- The most significant change is the addition of pagination to the\n`upgrade/_review` endpoint. This endpoint was known for causing OOM\nerrors due to its large and ever-growing response size. With pagination,\nit now returns upgrade information for no more than 20-100 rules at a\ntime, significantly reducing its memory footprint.\n- New backend methods, such as\n`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.\nThese methods return rule IDs with their corresponding installed\nversions, allowing to build a map of outdated rules without loading all\navailable rules into memory. Previously, all installed rules, along with\ntheir base and target versions, were fetched unconditionally before\nfiltering for updates.\n- The `stats` data structure of the review endpoint has been deprecated\n(it can be safely removed after one Serverless release cycle). Since the\nendpoint now returns paginated results, building stats is no longer\nfeasible due to the limited rule set size fetched on the server side. As\nthe side effect it required removing related Cypress tests asserting\n`Update All` disabled when rules can't be updated.\n- All changes to the endpoints are backward-compatible. All previously\nrequired returned structures still present in response. All newly added\nstructures are optional.\n- Upgradeable rule tags are now returned from the prebuilt rule status\nendpoint.\n- The frontend logic has been updated to move sorting and filtering of\nprebuilt rules from the client side to the server side.\n- The `upgrade/_perform` endpoint has been rewritten to use lightweight\nrule version information rather than full rules to determine upgradeable\nrules. Additionally, upgrades are now performed in batches of up to 100\nrules, further reducing memory usage.\n- A dry run option has been added to the upgrade perform endpoint. This\nis needed for the \"Update all\" rules scenario to determine if any rules\ncontain conflicts and display a confirmation modal to the user.\n- An option to skip conflicting rules has been added to the upgrade\nendpoint when called with the `ALL_RULES` mode.\n- The `install/_review` endpoint's memory consumption has been optimized\nby avoiding loading all rules into memory to determine available rules\nfor installation. Redundant fetching of all base versions has also been\nremoved, as they do not participate in the calculation.\n\n---------\n\nCo-authored-by: Maxim Palenov <maxim.palenov@elastic.co>","sha":"c4a016eda30ae8f224fdd485a634dc6773898e31"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.17","label":"v8.17.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Dmitrii Shevchenko <dmitrii.shevchenko@elastic.co>
kibanamachine
added a commit
that referenced
this pull request
Mar 3, 2025
…sponse size (#211045) (#212922) # Backport This will backport the following commits from `main` to `9.0`: - [[Security Solution] Reduce the _review rule upgrade endpoint response size (#211045)](#211045) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Dmitrii Shevchenko","email":"dmitrii.shevchenko@elastic.co"},"sourceCommit":{"committedDate":"2025-03-03T14:03:07Z","message":"[Security Solution] Reduce the _review rule upgrade endpoint response size (#211045)\n\n**Resolves: https://github.com/elastic/kibana/issues/208361**\n**Resolves: https://github.com/elastic/kibana/issues/210544**\n\n## Summary\n\nThis PR introduces significant memory consumption improvements to the\nprebuilt rule endpoints, ensuring users won't encounter OOM errors on\nmemory-limited Kibana instances.\n\nMemory consumption testing results provided in\nhttps://github.com//pull/211045#issuecomment-2689854328.\n\n## Details\n\nThis PR implements a number of memory usage optimizations to the\nprebuilt rule endpoints with the final goal reducing chances of getting\nOOM errors. The changes are extensive and require thorough testing\nbefore merging.\n\nThe changes are described by the following bullets\n\n- The most significant change is the addition of pagination to the\n`upgrade/_review` endpoint. This endpoint was known for causing OOM\nerrors due to its large and ever-growing response size. With pagination,\nit now returns upgrade information for no more than 20-100 rules at a\ntime, significantly reducing its memory footprint.\n- New backend methods, such as\n`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.\nThese methods return rule IDs with their corresponding installed\nversions, allowing to build a map of outdated rules without loading all\navailable rules into memory. Previously, all installed rules, along with\ntheir base and target versions, were fetched unconditionally before\nfiltering for updates.\n- The `stats` data structure of the review endpoint has been deprecated\n(it can be safely removed after one Serverless release cycle). Since the\nendpoint now returns paginated results, building stats is no longer\nfeasible due to the limited rule set size fetched on the server side. As\nthe side effect it required removing related Cypress tests asserting\n`Update All` disabled when rules can't be updated.\n- All changes to the endpoints are backward-compatible. All previously\nrequired returned structures still present in response. All newly added\nstructures are optional.\n- Upgradeable rule tags are now returned from the prebuilt rule status\nendpoint.\n- The frontend logic has been updated to move sorting and filtering of\nprebuilt rules from the client side to the server side.\n- The `upgrade/_perform` endpoint has been rewritten to use lightweight\nrule version information rather than full rules to determine upgradeable\nrules. Additionally, upgrades are now performed in batches of up to 100\nrules, further reducing memory usage.\n- A dry run option has been added to the upgrade perform endpoint. This\nis needed for the \"Update all\" rules scenario to determine if any rules\ncontain conflicts and display a confirmation modal to the user.\n- An option to skip conflicting rules has been added to the upgrade\nendpoint when called with the `ALL_RULES` mode.\n- The `install/_review` endpoint's memory consumption has been optimized\nby avoiding loading all rules into memory to determine available rules\nfor installation. Redundant fetching of all base versions has also been\nremoved, as they do not participate in the calculation.\n\n---------\n\nCo-authored-by: Maxim Palenov <maxim.palenov@elastic.co>","sha":"c4a016eda30ae8f224fdd485a634dc6773898e31","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","performance","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0","v9.1.0","v8.19.0","v8.17.3"],"title":"[Security Solution] Reduce the _review rule upgrade endpoint response size","number":211045,"url":"https://github.com/elastic/kibana/pull/211045","mergeCommit":{"message":"[Security Solution] Reduce the _review rule upgrade endpoint response size (#211045)\n\n**Resolves: https://github.com/elastic/kibana/issues/208361**\n**Resolves: https://github.com/elastic/kibana/issues/210544**\n\n## Summary\n\nThis PR introduces significant memory consumption improvements to the\nprebuilt rule endpoints, ensuring users won't encounter OOM errors on\nmemory-limited Kibana instances.\n\nMemory consumption testing results provided in\nhttps://github.com//pull/211045#issuecomment-2689854328.\n\n## Details\n\nThis PR implements a number of memory usage optimizations to the\nprebuilt rule endpoints with the final goal reducing chances of getting\nOOM errors. The changes are extensive and require thorough testing\nbefore merging.\n\nThe changes are described by the following bullets\n\n- The most significant change is the addition of pagination to the\n`upgrade/_review` endpoint. This endpoint was known for causing OOM\nerrors due to its large and ever-growing response size. With pagination,\nit now returns upgrade information for no more than 20-100 rules at a\ntime, significantly reducing its memory footprint.\n- New backend methods, such as\n`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.\nThese methods return rule IDs with their corresponding installed\nversions, allowing to build a map of outdated rules without loading all\navailable rules into memory. Previously, all installed rules, along with\ntheir base and target versions, were fetched unconditionally before\nfiltering for updates.\n- The `stats` data structure of the review endpoint has been deprecated\n(it can be safely removed after one Serverless release cycle). Since the\nendpoint now returns paginated results, building stats is no longer\nfeasible due to the limited rule set size fetched on the server side. As\nthe side effect it required removing related Cypress tests asserting\n`Update All` disabled when rules can't be updated.\n- All changes to the endpoints are backward-compatible. All previously\nrequired returned structures still present in response. All newly added\nstructures are optional.\n- Upgradeable rule tags are now returned from the prebuilt rule status\nendpoint.\n- The frontend logic has been updated to move sorting and filtering of\nprebuilt rules from the client side to the server side.\n- The `upgrade/_perform` endpoint has been rewritten to use lightweight\nrule version information rather than full rules to determine upgradeable\nrules. Additionally, upgrades are now performed in batches of up to 100\nrules, further reducing memory usage.\n- A dry run option has been added to the upgrade perform endpoint. This\nis needed for the \"Update all\" rules scenario to determine if any rules\ncontain conflicts and display a confirmation modal to the user.\n- An option to skip conflicting rules has been added to the upgrade\nendpoint when called with the `ALL_RULES` mode.\n- The `install/_review` endpoint's memory consumption has been optimized\nby avoiding loading all rules into memory to determine available rules\nfor installation. Redundant fetching of all base versions has also been\nremoved, as they do not participate in the calculation.\n\n---------\n\nCo-authored-by: Maxim Palenov <maxim.palenov@elastic.co>","sha":"c4a016eda30ae8f224fdd485a634dc6773898e31"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x","8.17"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/211045","number":211045,"mergeCommit":{"message":"[Security Solution] Reduce the _review rule upgrade endpoint response size (#211045)\n\n**Resolves: https://github.com/elastic/kibana/issues/208361**\n**Resolves: https://github.com/elastic/kibana/issues/210544**\n\n## Summary\n\nThis PR introduces significant memory consumption improvements to the\nprebuilt rule endpoints, ensuring users won't encounter OOM errors on\nmemory-limited Kibana instances.\n\nMemory consumption testing results provided in\nhttps://github.com//pull/211045#issuecomment-2689854328.\n\n## Details\n\nThis PR implements a number of memory usage optimizations to the\nprebuilt rule endpoints with the final goal reducing chances of getting\nOOM errors. The changes are extensive and require thorough testing\nbefore merging.\n\nThe changes are described by the following bullets\n\n- The most significant change is the addition of pagination to the\n`upgrade/_review` endpoint. This endpoint was known for causing OOM\nerrors due to its large and ever-growing response size. With pagination,\nit now returns upgrade information for no more than 20-100 rules at a\ntime, significantly reducing its memory footprint.\n- New backend methods, such as\n`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.\nThese methods return rule IDs with their corresponding installed\nversions, allowing to build a map of outdated rules without loading all\navailable rules into memory. Previously, all installed rules, along with\ntheir base and target versions, were fetched unconditionally before\nfiltering for updates.\n- The `stats` data structure of the review endpoint has been deprecated\n(it can be safely removed after one Serverless release cycle). Since the\nendpoint now returns paginated results, building stats is no longer\nfeasible due to the limited rule set size fetched on the server side. As\nthe side effect it required removing related Cypress tests asserting\n`Update All` disabled when rules can't be updated.\n- All changes to the endpoints are backward-compatible. All previously\nrequired returned structures still present in response. All newly added\nstructures are optional.\n- Upgradeable rule tags are now returned from the prebuilt rule status\nendpoint.\n- The frontend logic has been updated to move sorting and filtering of\nprebuilt rules from the client side to the server side.\n- The `upgrade/_perform` endpoint has been rewritten to use lightweight\nrule version information rather than full rules to determine upgradeable\nrules. Additionally, upgrades are now performed in batches of up to 100\nrules, further reducing memory usage.\n- A dry run option has been added to the upgrade perform endpoint. This\nis needed for the \"Update all\" rules scenario to determine if any rules\ncontain conflicts and display a confirmation modal to the user.\n- An option to skip conflicting rules has been added to the upgrade\nendpoint when called with the `ALL_RULES` mode.\n- The `install/_review` endpoint's memory consumption has been optimized\nby avoiding loading all rules into memory to determine available rules\nfor installation. Redundant fetching of all base versions has also been\nremoved, as they do not participate in the calculation.\n\n---------\n\nCo-authored-by: Maxim Palenov <maxim.palenov@elastic.co>","sha":"c4a016eda30ae8f224fdd485a634dc6773898e31"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.17","label":"v8.17.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Dmitrii Shevchenko <dmitrii.shevchenko@elastic.co>
kibanamachine
added a commit
that referenced
this pull request
Mar 3, 2025
…sponse size (#211045) (#212921) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution] Reduce the _review rule upgrade endpoint response size (#211045)](#211045) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Dmitrii Shevchenko","email":"dmitrii.shevchenko@elastic.co"},"sourceCommit":{"committedDate":"2025-03-03T14:03:07Z","message":"[Security Solution] Reduce the _review rule upgrade endpoint response size (#211045)\n\n**Resolves: https://github.com/elastic/kibana/issues/208361**\n**Resolves: https://github.com/elastic/kibana/issues/210544**\n\n## Summary\n\nThis PR introduces significant memory consumption improvements to the\nprebuilt rule endpoints, ensuring users won't encounter OOM errors on\nmemory-limited Kibana instances.\n\nMemory consumption testing results provided in\nhttps://github.com//pull/211045#issuecomment-2689854328.\n\n## Details\n\nThis PR implements a number of memory usage optimizations to the\nprebuilt rule endpoints with the final goal reducing chances of getting\nOOM errors. The changes are extensive and require thorough testing\nbefore merging.\n\nThe changes are described by the following bullets\n\n- The most significant change is the addition of pagination to the\n`upgrade/_review` endpoint. This endpoint was known for causing OOM\nerrors due to its large and ever-growing response size. With pagination,\nit now returns upgrade information for no more than 20-100 rules at a\ntime, significantly reducing its memory footprint.\n- New backend methods, such as\n`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.\nThese methods return rule IDs with their corresponding installed\nversions, allowing to build a map of outdated rules without loading all\navailable rules into memory. Previously, all installed rules, along with\ntheir base and target versions, were fetched unconditionally before\nfiltering for updates.\n- The `stats` data structure of the review endpoint has been deprecated\n(it can be safely removed after one Serverless release cycle). Since the\nendpoint now returns paginated results, building stats is no longer\nfeasible due to the limited rule set size fetched on the server side. As\nthe side effect it required removing related Cypress tests asserting\n`Update All` disabled when rules can't be updated.\n- All changes to the endpoints are backward-compatible. All previously\nrequired returned structures still present in response. All newly added\nstructures are optional.\n- Upgradeable rule tags are now returned from the prebuilt rule status\nendpoint.\n- The frontend logic has been updated to move sorting and filtering of\nprebuilt rules from the client side to the server side.\n- The `upgrade/_perform` endpoint has been rewritten to use lightweight\nrule version information rather than full rules to determine upgradeable\nrules. Additionally, upgrades are now performed in batches of up to 100\nrules, further reducing memory usage.\n- A dry run option has been added to the upgrade perform endpoint. This\nis needed for the \"Update all\" rules scenario to determine if any rules\ncontain conflicts and display a confirmation modal to the user.\n- An option to skip conflicting rules has been added to the upgrade\nendpoint when called with the `ALL_RULES` mode.\n- The `install/_review` endpoint's memory consumption has been optimized\nby avoiding loading all rules into memory to determine available rules\nfor installation. Redundant fetching of all base versions has also been\nremoved, as they do not participate in the calculation.\n\n---------\n\nCo-authored-by: Maxim Palenov <maxim.palenov@elastic.co>","sha":"c4a016eda30ae8f224fdd485a634dc6773898e31","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","performance","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0","v9.1.0","v8.19.0","v8.17.3"],"title":"[Security Solution] Reduce the _review rule upgrade endpoint response size","number":211045,"url":"https://github.com/elastic/kibana/pull/211045","mergeCommit":{"message":"[Security Solution] Reduce the _review rule upgrade endpoint response size (#211045)\n\n**Resolves: https://github.com/elastic/kibana/issues/208361**\n**Resolves: https://github.com/elastic/kibana/issues/210544**\n\n## Summary\n\nThis PR introduces significant memory consumption improvements to the\nprebuilt rule endpoints, ensuring users won't encounter OOM errors on\nmemory-limited Kibana instances.\n\nMemory consumption testing results provided in\nhttps://github.com//pull/211045#issuecomment-2689854328.\n\n## Details\n\nThis PR implements a number of memory usage optimizations to the\nprebuilt rule endpoints with the final goal reducing chances of getting\nOOM errors. The changes are extensive and require thorough testing\nbefore merging.\n\nThe changes are described by the following bullets\n\n- The most significant change is the addition of pagination to the\n`upgrade/_review` endpoint. This endpoint was known for causing OOM\nerrors due to its large and ever-growing response size. With pagination,\nit now returns upgrade information for no more than 20-100 rules at a\ntime, significantly reducing its memory footprint.\n- New backend methods, such as\n`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.\nThese methods return rule IDs with their corresponding installed\nversions, allowing to build a map of outdated rules without loading all\navailable rules into memory. Previously, all installed rules, along with\ntheir base and target versions, were fetched unconditionally before\nfiltering for updates.\n- The `stats` data structure of the review endpoint has been deprecated\n(it can be safely removed after one Serverless release cycle). Since the\nendpoint now returns paginated results, building stats is no longer\nfeasible due to the limited rule set size fetched on the server side. As\nthe side effect it required removing related Cypress tests asserting\n`Update All` disabled when rules can't be updated.\n- All changes to the endpoints are backward-compatible. All previously\nrequired returned structures still present in response. All newly added\nstructures are optional.\n- Upgradeable rule tags are now returned from the prebuilt rule status\nendpoint.\n- The frontend logic has been updated to move sorting and filtering of\nprebuilt rules from the client side to the server side.\n- The `upgrade/_perform` endpoint has been rewritten to use lightweight\nrule version information rather than full rules to determine upgradeable\nrules. Additionally, upgrades are now performed in batches of up to 100\nrules, further reducing memory usage.\n- A dry run option has been added to the upgrade perform endpoint. This\nis needed for the \"Update all\" rules scenario to determine if any rules\ncontain conflicts and display a confirmation modal to the user.\n- An option to skip conflicting rules has been added to the upgrade\nendpoint when called with the `ALL_RULES` mode.\n- The `install/_review` endpoint's memory consumption has been optimized\nby avoiding loading all rules into memory to determine available rules\nfor installation. Redundant fetching of all base versions has also been\nremoved, as they do not participate in the calculation.\n\n---------\n\nCo-authored-by: Maxim Palenov <maxim.palenov@elastic.co>","sha":"c4a016eda30ae8f224fdd485a634dc6773898e31"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x","8.17"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/211045","number":211045,"mergeCommit":{"message":"[Security Solution] Reduce the _review rule upgrade endpoint response size (#211045)\n\n**Resolves: https://github.com/elastic/kibana/issues/208361**\n**Resolves: https://github.com/elastic/kibana/issues/210544**\n\n## Summary\n\nThis PR introduces significant memory consumption improvements to the\nprebuilt rule endpoints, ensuring users won't encounter OOM errors on\nmemory-limited Kibana instances.\n\nMemory consumption testing results provided in\nhttps://github.com//pull/211045#issuecomment-2689854328.\n\n## Details\n\nThis PR implements a number of memory usage optimizations to the\nprebuilt rule endpoints with the final goal reducing chances of getting\nOOM errors. The changes are extensive and require thorough testing\nbefore merging.\n\nThe changes are described by the following bullets\n\n- The most significant change is the addition of pagination to the\n`upgrade/_review` endpoint. This endpoint was known for causing OOM\nerrors due to its large and ever-growing response size. With pagination,\nit now returns upgrade information for no more than 20-100 rules at a\ntime, significantly reducing its memory footprint.\n- New backend methods, such as\n`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.\nThese methods return rule IDs with their corresponding installed\nversions, allowing to build a map of outdated rules without loading all\navailable rules into memory. Previously, all installed rules, along with\ntheir base and target versions, were fetched unconditionally before\nfiltering for updates.\n- The `stats` data structure of the review endpoint has been deprecated\n(it can be safely removed after one Serverless release cycle). Since the\nendpoint now returns paginated results, building stats is no longer\nfeasible due to the limited rule set size fetched on the server side. As\nthe side effect it required removing related Cypress tests asserting\n`Update All` disabled when rules can't be updated.\n- All changes to the endpoints are backward-compatible. All previously\nrequired returned structures still present in response. All newly added\nstructures are optional.\n- Upgradeable rule tags are now returned from the prebuilt rule status\nendpoint.\n- The frontend logic has been updated to move sorting and filtering of\nprebuilt rules from the client side to the server side.\n- The `upgrade/_perform` endpoint has been rewritten to use lightweight\nrule version information rather than full rules to determine upgradeable\nrules. Additionally, upgrades are now performed in batches of up to 100\nrules, further reducing memory usage.\n- A dry run option has been added to the upgrade perform endpoint. This\nis needed for the \"Update all\" rules scenario to determine if any rules\ncontain conflicts and display a confirmation modal to the user.\n- An option to skip conflicting rules has been added to the upgrade\nendpoint when called with the `ALL_RULES` mode.\n- The `install/_review` endpoint's memory consumption has been optimized\nby avoiding loading all rules into memory to determine available rules\nfor installation. Redundant fetching of all base versions has also been\nremoved, as they do not participate in the calculation.\n\n---------\n\nCo-authored-by: Maxim Palenov <maxim.palenov@elastic.co>","sha":"c4a016eda30ae8f224fdd485a634dc6773898e31"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.17","label":"v8.17.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Dmitrii Shevchenko <dmitrii.shevchenko@elastic.co>
SoniaSanzV
pushed a commit
to SoniaSanzV/kibana
that referenced
this pull request
Mar 4, 2025
…sponse size (elastic#211045) (elastic#212921) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution] Reduce the _review rule upgrade endpoint response size (elastic#211045)](elastic#211045) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Dmitrii Shevchenko","email":"dmitrii.shevchenko@elastic.co"},"sourceCommit":{"committedDate":"2025-03-03T14:03:07Z","message":"[Security Solution] Reduce the _review rule upgrade endpoint response size (elastic#211045)\n\n**Resolves: https://github.com/elastic/kibana/issues/208361**\n**Resolves: https://github.com/elastic/kibana/issues/210544**\n\n## Summary\n\nThis PR introduces significant memory consumption improvements to the\nprebuilt rule endpoints, ensuring users won't encounter OOM errors on\nmemory-limited Kibana instances.\n\nMemory consumption testing results provided in\nhttps://github.com/elastic/pull/211045#issuecomment-2689854328.\n\n## Details\n\nThis PR implements a number of memory usage optimizations to the\nprebuilt rule endpoints with the final goal reducing chances of getting\nOOM errors. The changes are extensive and require thorough testing\nbefore merging.\n\nThe changes are described by the following bullets\n\n- The most significant change is the addition of pagination to the\n`upgrade/_review` endpoint. This endpoint was known for causing OOM\nerrors due to its large and ever-growing response size. With pagination,\nit now returns upgrade information for no more than 20-100 rules at a\ntime, significantly reducing its memory footprint.\n- New backend methods, such as\n`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.\nThese methods return rule IDs with their corresponding installed\nversions, allowing to build a map of outdated rules without loading all\navailable rules into memory. Previously, all installed rules, along with\ntheir base and target versions, were fetched unconditionally before\nfiltering for updates.\n- The `stats` data structure of the review endpoint has been deprecated\n(it can be safely removed after one Serverless release cycle). Since the\nendpoint now returns paginated results, building stats is no longer\nfeasible due to the limited rule set size fetched on the server side. As\nthe side effect it required removing related Cypress tests asserting\n`Update All` disabled when rules can't be updated.\n- All changes to the endpoints are backward-compatible. All previously\nrequired returned structures still present in response. All newly added\nstructures are optional.\n- Upgradeable rule tags are now returned from the prebuilt rule status\nendpoint.\n- The frontend logic has been updated to move sorting and filtering of\nprebuilt rules from the client side to the server side.\n- The `upgrade/_perform` endpoint has been rewritten to use lightweight\nrule version information rather than full rules to determine upgradeable\nrules. Additionally, upgrades are now performed in batches of up to 100\nrules, further reducing memory usage.\n- A dry run option has been added to the upgrade perform endpoint. This\nis needed for the \"Update all\" rules scenario to determine if any rules\ncontain conflicts and display a confirmation modal to the user.\n- An option to skip conflicting rules has been added to the upgrade\nendpoint when called with the `ALL_RULES` mode.\n- The `install/_review` endpoint's memory consumption has been optimized\nby avoiding loading all rules into memory to determine available rules\nfor installation. Redundant fetching of all base versions has also been\nremoved, as they do not participate in the calculation.\n\n---------\n\nCo-authored-by: Maxim Palenov <maxim.palenov@elastic.co>","sha":"c4a016eda30ae8f224fdd485a634dc6773898e31","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","performance","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0","v9.1.0","v8.19.0","v8.17.3"],"title":"[Security Solution] Reduce the _review rule upgrade endpoint response size","number":211045,"url":"https://github.com/elastic/kibana/pull/211045","mergeCommit":{"message":"[Security Solution] Reduce the _review rule upgrade endpoint response size (elastic#211045)\n\n**Resolves: https://github.com/elastic/kibana/issues/208361**\n**Resolves: https://github.com/elastic/kibana/issues/210544**\n\n## Summary\n\nThis PR introduces significant memory consumption improvements to the\nprebuilt rule endpoints, ensuring users won't encounter OOM errors on\nmemory-limited Kibana instances.\n\nMemory consumption testing results provided in\nhttps://github.com/elastic/pull/211045#issuecomment-2689854328.\n\n## Details\n\nThis PR implements a number of memory usage optimizations to the\nprebuilt rule endpoints with the final goal reducing chances of getting\nOOM errors. The changes are extensive and require thorough testing\nbefore merging.\n\nThe changes are described by the following bullets\n\n- The most significant change is the addition of pagination to the\n`upgrade/_review` endpoint. This endpoint was known for causing OOM\nerrors due to its large and ever-growing response size. With pagination,\nit now returns upgrade information for no more than 20-100 rules at a\ntime, significantly reducing its memory footprint.\n- New backend methods, such as\n`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.\nThese methods return rule IDs with their corresponding installed\nversions, allowing to build a map of outdated rules without loading all\navailable rules into memory. Previously, all installed rules, along with\ntheir base and target versions, were fetched unconditionally before\nfiltering for updates.\n- The `stats` data structure of the review endpoint has been deprecated\n(it can be safely removed after one Serverless release cycle). Since the\nendpoint now returns paginated results, building stats is no longer\nfeasible due to the limited rule set size fetched on the server side. As\nthe side effect it required removing related Cypress tests asserting\n`Update All` disabled when rules can't be updated.\n- All changes to the endpoints are backward-compatible. All previously\nrequired returned structures still present in response. All newly added\nstructures are optional.\n- Upgradeable rule tags are now returned from the prebuilt rule status\nendpoint.\n- The frontend logic has been updated to move sorting and filtering of\nprebuilt rules from the client side to the server side.\n- The `upgrade/_perform` endpoint has been rewritten to use lightweight\nrule version information rather than full rules to determine upgradeable\nrules. Additionally, upgrades are now performed in batches of up to 100\nrules, further reducing memory usage.\n- A dry run option has been added to the upgrade perform endpoint. This\nis needed for the \"Update all\" rules scenario to determine if any rules\ncontain conflicts and display a confirmation modal to the user.\n- An option to skip conflicting rules has been added to the upgrade\nendpoint when called with the `ALL_RULES` mode.\n- The `install/_review` endpoint's memory consumption has been optimized\nby avoiding loading all rules into memory to determine available rules\nfor installation. Redundant fetching of all base versions has also been\nremoved, as they do not participate in the calculation.\n\n---------\n\nCo-authored-by: Maxim Palenov <maxim.palenov@elastic.co>","sha":"c4a016eda30ae8f224fdd485a634dc6773898e31"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x","8.17"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/211045","number":211045,"mergeCommit":{"message":"[Security Solution] Reduce the _review rule upgrade endpoint response size (elastic#211045)\n\n**Resolves: https://github.com/elastic/kibana/issues/208361**\n**Resolves: https://github.com/elastic/kibana/issues/210544**\n\n## Summary\n\nThis PR introduces significant memory consumption improvements to the\nprebuilt rule endpoints, ensuring users won't encounter OOM errors on\nmemory-limited Kibana instances.\n\nMemory consumption testing results provided in\nhttps://github.com/elastic/pull/211045#issuecomment-2689854328.\n\n## Details\n\nThis PR implements a number of memory usage optimizations to the\nprebuilt rule endpoints with the final goal reducing chances of getting\nOOM errors. The changes are extensive and require thorough testing\nbefore merging.\n\nThe changes are described by the following bullets\n\n- The most significant change is the addition of pagination to the\n`upgrade/_review` endpoint. This endpoint was known for causing OOM\nerrors due to its large and ever-growing response size. With pagination,\nit now returns upgrade information for no more than 20-100 rules at a\ntime, significantly reducing its memory footprint.\n- New backend methods, such as\n`ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced.\nThese methods return rule IDs with their corresponding installed\nversions, allowing to build a map of outdated rules without loading all\navailable rules into memory. Previously, all installed rules, along with\ntheir base and target versions, were fetched unconditionally before\nfiltering for updates.\n- The `stats` data structure of the review endpoint has been deprecated\n(it can be safely removed after one Serverless release cycle). Since the\nendpoint now returns paginated results, building stats is no longer\nfeasible due to the limited rule set size fetched on the server side. As\nthe side effect it required removing related Cypress tests asserting\n`Update All` disabled when rules can't be updated.\n- All changes to the endpoints are backward-compatible. All previously\nrequired returned structures still present in response. All newly added\nstructures are optional.\n- Upgradeable rule tags are now returned from the prebuilt rule status\nendpoint.\n- The frontend logic has been updated to move sorting and filtering of\nprebuilt rules from the client side to the server side.\n- The `upgrade/_perform` endpoint has been rewritten to use lightweight\nrule version information rather than full rules to determine upgradeable\nrules. Additionally, upgrades are now performed in batches of up to 100\nrules, further reducing memory usage.\n- A dry run option has been added to the upgrade perform endpoint. This\nis needed for the \"Update all\" rules scenario to determine if any rules\ncontain conflicts and display a confirmation modal to the user.\n- An option to skip conflicting rules has been added to the upgrade\nendpoint when called with the `ALL_RULES` mode.\n- The `install/_review` endpoint's memory consumption has been optimized\nby avoiding loading all rules into memory to determine available rules\nfor installation. Redundant fetching of all base versions has also been\nremoved, as they do not participate in the calculation.\n\n---------\n\nCo-authored-by: Maxim Palenov <maxim.palenov@elastic.co>","sha":"c4a016eda30ae8f224fdd485a634dc6773898e31"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.17","label":"v8.17.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Dmitrii Shevchenko <dmitrii.shevchenko@elastic.co>
banderror
added
release_note:skip
xcrzx
pushed a commit
that referenced
this pull request
Mar 14, 2025
**This is a follow-up to:#211045 ## Summary This PR removes inefficiencies in prebuilt rule installation memory consumption. ### Before In the worst-case scenario: 1. All currently installed prebuilt rules were fully loaded into memory. 2. All latest rule versions from the rule packages were fully loaded into memory. 3. All base rule versions were pulled into memory, even though they were not used. 4. The algorithm then checked which rules could be installed and installed them all at once. ### After 1. Pull only aggregated information about installed rules (only `rule_id` and `versions`). 2. Pull the same lightweight aggregated info about the latest rules in the package. 3. Using the collected `rule_id`s, fetch rule assets and install them in small batches of up to 100 rules.
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Mar 14, 2025
**This is a follow-up to:elastic#211045 ## Summary This PR removes inefficiencies in prebuilt rule installation memory consumption. ### Before In the worst-case scenario: 1. All currently installed prebuilt rules were fully loaded into memory. 2. All latest rule versions from the rule packages were fully loaded into memory. 3. All base rule versions were pulled into memory, even though they were not used. 4. The algorithm then checked which rules could be installed and installed them all at once. ### After 1. Pull only aggregated information about installed rules (only `rule_id` and `versions`). 2. Pull the same lightweight aggregated info about the latest rules in the package. 3. Using the collected `rule_id`s, fetch rule assets and install them in small batches of up to 100 rules. (cherry picked from commit 6d9fc21)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Mar 14, 2025
**This is a follow-up to:elastic#211045 ## Summary This PR removes inefficiencies in prebuilt rule installation memory consumption. ### Before In the worst-case scenario: 1. All currently installed prebuilt rules were fully loaded into memory. 2. All latest rule versions from the rule packages were fully loaded into memory. 3. All base rule versions were pulled into memory, even though they were not used. 4. The algorithm then checked which rules could be installed and installed them all at once. ### After 1. Pull only aggregated information about installed rules (only `rule_id` and `versions`). 2. Pull the same lightweight aggregated info about the latest rules in the package. 3. Using the collected `rule_id`s, fetch rule assets and install them in small batches of up to 100 rules. (cherry picked from commit 6d9fc21)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Mar 14, 2025
**This is a follow-up to:elastic#211045 ## Summary This PR removes inefficiencies in prebuilt rule installation memory consumption. ### Before In the worst-case scenario: 1. All currently installed prebuilt rules were fully loaded into memory. 2. All latest rule versions from the rule packages were fully loaded into memory. 3. All base rule versions were pulled into memory, even though they were not used. 4. The algorithm then checked which rules could be installed and installed them all at once. ### After 1. Pull only aggregated information about installed rules (only `rule_id` and `versions`). 2. Pull the same lightweight aggregated info about the latest rules in the package. 3. Using the collected `rule_id`s, fetch rule assets and install them in small batches of up to 100 rules. (cherry picked from commit 6d9fc21)
clintandrewhall
pushed a commit
to clintandrewhall/kibana
that referenced
this pull request
Mar 20, 2025
**This is a follow-up to:elastic#211045 ## Summary This PR removes inefficiencies in prebuilt rule installation memory consumption. ### Before In the worst-case scenario: 1. All currently installed prebuilt rules were fully loaded into memory. 2. All latest rule versions from the rule packages were fully loaded into memory. 3. All base rule versions were pulled into memory, even though they were not used. 4. The algorithm then checked which rules could be installed and installed them all at once. ### After 1. Pull only aggregated information about installed rules (only `rule_id` and `versions`). 2. Pull the same lightweight aggregated info about the latest rules in the package. 3. Using the collected `rule_id`s, fetch rule assets and install them in small batches of up to 100 rules.
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Mar 22, 2025
… size (elastic#211045) **Resolves: elastic#208361 **Resolves: elastic#210544 ## Summary This PR introduces significant memory consumption improvements to the prebuilt rule endpoints, ensuring users won't encounter OOM errors on memory-limited Kibana instances. Memory consumption testing results provided in elastic#211045 (comment). ## Details This PR implements a number of memory usage optimizations to the prebuilt rule endpoints with the final goal reducing chances of getting OOM errors. The changes are extensive and require thorough testing before merging. The changes are described by the following bullets - The most significant change is the addition of pagination to the `upgrade/_review` endpoint. This endpoint was known for causing OOM errors due to its large and ever-growing response size. With pagination, it now returns upgrade information for no more than 20-100 rules at a time, significantly reducing its memory footprint. - New backend methods, such as `ruleObjectsClient.fetchInstalledRuleVersions`, have been introduced. These methods return rule IDs with their corresponding installed versions, allowing to build a map of outdated rules without loading all available rules into memory. Previously, all installed rules, along with their base and target versions, were fetched unconditionally before filtering for updates. - The `stats` data structure of the review endpoint has been deprecated (it can be safely removed after one Serverless release cycle). Since the endpoint now returns paginated results, building stats is no longer feasible due to the limited rule set size fetched on the server side. As the side effect it required removing related Cypress tests asserting `Update All` disabled when rules can't be updated. - All changes to the endpoints are backward-compatible. All previously required returned structures still present in response. All newly added structures are optional. - Upgradeable rule tags are now returned from the prebuilt rule status endpoint. - The frontend logic has been updated to move sorting and filtering of prebuilt rules from the client side to the server side. - The `upgrade/_perform` endpoint has been rewritten to use lightweight rule version information rather than full rules to determine upgradeable rules. Additionally, upgrades are now performed in batches of up to 100 rules, further reducing memory usage. - A dry run option has been added to the upgrade perform endpoint. This is needed for the "Update all" rules scenario to determine if any rules contain conflicts and display a confirmation modal to the user. - An option to skip conflicting rules has been added to the upgrade endpoint when called with the `ALL_RULES` mode. - The `install/_review` endpoint's memory consumption has been optimized by avoiding loading all rules into memory to determine available rules for installation. Redundant fetching of all base versions has also been removed, as they do not participate in the calculation. --------- Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Mar 22, 2025
**This is a follow-up to:elastic#211045 ## Summary This PR removes inefficiencies in prebuilt rule installation memory consumption. ### Before In the worst-case scenario: 1. All currently installed prebuilt rules were fully loaded into memory. 2. All latest rule versions from the rule packages were fully loaded into memory. 3. All base rule versions were pulled into memory, even though they were not used. 4. The algorithm then checked which rules could be installed and installed them all at once. ### After 1. Pull only aggregated information about installed rules (only `rule_id` and `versions`). 2. Pull the same lightweight aggregated info about the latest rules in the package. 3. Using the collected `rule_id`s, fetch rule assets and install them in small batches of up to 100 rules.
35 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves: #208361
Resolves: #210544
Summary
This PR introduces significant memory consumption improvements to the prebuilt rule endpoints, ensuring users won't encounter OOM errors on memory-limited Kibana instances.
Memory consumption testing results provided in #211045 (comment).
Details
This PR implements a number of memory usage optimizations to the prebuilt rule endpoints with the final goal reducing chances of getting OOM errors. The changes are extensive and require thorough testing before merging.
The changes are described by the following bullets
upgrade/_reviewendpoint. This endpoint was known for causing OOM errors due to its large and ever-growing response size. With pagination, it now returns upgrade information for no more than 20-100 rules at a time, significantly reducing its memory footprint.ruleObjectsClient.fetchInstalledRuleVersions, have been introduced. These methods return rule IDs with their corresponding installed versions, allowing to build a map of outdated rules without loading all available rules into memory. Previously, all installed rules, along with their base and target versions, were fetched unconditionally before filtering for updates.statsdata structure of the review endpoint has been deprecated (it can be safely removed after one Serverless release cycle). Since the endpoint now returns paginated results, building stats is no longer feasible due to the limited rule set size fetched on the server side. As the side effect it required removing related Cypress tests assertingUpdate Alldisabled when rules can't be updated.upgrade/_performendpoint has been rewritten to use lightweight rule version information rather than full rules to determine upgradeable rules. Additionally, upgrades are now performed in batches of up to 100 rules, further reducing memory usage.ALL_RULESmode.install/_reviewendpoint's memory consumption has been optimized by avoiding loading all rules into memory to determine available rules for installation. Redundant fetching of all base versions has also been removed, as they do not participate in the calculation.