[Cloud Security] Update Findings page runtime fields required for third party data compatibility#198635
Merged
maxcold merged 8 commits intoelastic:mainfrom Nov 1, 2024
Merged
Conversation
opauloh
added
release_note:skip
Contributor
|
Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security) |
Contributor
💚 Build Succeeded
Metrics [docs]Async chunks
History
|
maxcold
approved these changes
Nov 1, 2024
Contributor
|
Starting backport for target branches: 8.16, 8.x https://github.com/elastic/kibana/actions/runs/11627623117 |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Nov 1, 2024
…rd party data compatibility (elastic#198635) ## Summary This PR enhances the solution proposed by [this PR](elastic#195702), by picking only the fields that are currently not mapped by the current Third Party integrations, this fixes performance degradation identified during the QA cycle of 8.16.0. Fixes: - elastic/security-team#11034 ### Misconfiguration Runtime fields - **rule.benchmark.rule_number**: runtime mapping kept because this field is missing on `security_solution-aws.misconfiguration_latest-v1` causing filtering out data when sorting by **Rule Number** column on the Misconfigurations Data Table. - **rule.section**: runtime mapping kept because this field is missing on `security_solution-aws.misconfiguration_latest-v1` causing filtering out data when sorting by **Framework Section** column on the Misconfigurations Data Table. - **resource.sub_type**: runtime mapping kept because this field is missing on `security_solution-aws.misconfiguration_latest-v1` causing filtering out data when sorting by **Resource Type** column on the Misconfigurations Data Table. - **orchestrator.cluster.name**: runtime mapping kept because this field is missing on `security_solution-wiz.misconfiguration_latest-v1` causing filtering out data when grouping by **Kubernetes Cluster** column on the Misconfigurations page. - **cloud.account.name**: runtime mapping kept because this field is missing on `security_solution-aws.misconfiguration_latest-v1` causing filtering out data when grouping by **Kubernetes Cluster** column on the Misconfigurations page. ### Vulnerability Runtime Fields: - **observer.vendor**: runtime mapping added because this field is mapped as `text` on `security_solution-wiz.vulnerability_latest-v1` causing filtering out when sorting by the **Vendor** column on the Vulnerability Data Table - **cloud.provider**: runtime mapping added because this field is mapped as `text` on `security_solution-wiz.vulnerability_latest-v1` causing filtering out when grouping by **Cloud Account** on the Vulnerability page. (This field is needed in order to retrieve the Cloud Provider name and icon) ## Screenshot - Left: After the changes / Right: Current https://github.com/user-attachments/assets/2cbdd8b7-131c-42e4-a881-632f8cd3854b https://github.com/user-attachments/assets/4372feb6-4c01-4047-a90a-d6728f9400fe https://github.com/user-attachments/assets/b9e32514-f2ee-4e4d-ba5f-ea3e20d4d0b2 (cherry picked from commit 7a98aa1)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Nov 1, 2024
…rd party data compatibility (elastic#198635) ## Summary This PR enhances the solution proposed by [this PR](elastic#195702), by picking only the fields that are currently not mapped by the current Third Party integrations, this fixes performance degradation identified during the QA cycle of 8.16.0. Fixes: - elastic/security-team#11034 ### Misconfiguration Runtime fields - **rule.benchmark.rule_number**: runtime mapping kept because this field is missing on `security_solution-aws.misconfiguration_latest-v1` causing filtering out data when sorting by **Rule Number** column on the Misconfigurations Data Table. - **rule.section**: runtime mapping kept because this field is missing on `security_solution-aws.misconfiguration_latest-v1` causing filtering out data when sorting by **Framework Section** column on the Misconfigurations Data Table. - **resource.sub_type**: runtime mapping kept because this field is missing on `security_solution-aws.misconfiguration_latest-v1` causing filtering out data when sorting by **Resource Type** column on the Misconfigurations Data Table. - **orchestrator.cluster.name**: runtime mapping kept because this field is missing on `security_solution-wiz.misconfiguration_latest-v1` causing filtering out data when grouping by **Kubernetes Cluster** column on the Misconfigurations page. - **cloud.account.name**: runtime mapping kept because this field is missing on `security_solution-aws.misconfiguration_latest-v1` causing filtering out data when grouping by **Kubernetes Cluster** column on the Misconfigurations page. ### Vulnerability Runtime Fields: - **observer.vendor**: runtime mapping added because this field is mapped as `text` on `security_solution-wiz.vulnerability_latest-v1` causing filtering out when sorting by the **Vendor** column on the Vulnerability Data Table - **cloud.provider**: runtime mapping added because this field is mapped as `text` on `security_solution-wiz.vulnerability_latest-v1` causing filtering out when grouping by **Cloud Account** on the Vulnerability page. (This field is needed in order to retrieve the Cloud Provider name and icon) ## Screenshot - Left: After the changes / Right: Current https://github.com/user-attachments/assets/2cbdd8b7-131c-42e4-a881-632f8cd3854b https://github.com/user-attachments/assets/4372feb6-4c01-4047-a90a-d6728f9400fe https://github.com/user-attachments/assets/b9e32514-f2ee-4e4d-ba5f-ea3e20d4d0b2 (cherry picked from commit 7a98aa1)
Contributor
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
1 task
kibanamachine
added a commit
that referenced
this pull request
Nov 1, 2024
…or third party data compatibility (#198635) (#198649) # Backport This will backport the following commits from `main` to `8.x`: - [[Cloud Security] Update Findings page runtime fields required for third party data compatibility (#198635)](#198635) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Paulo Silva","email":"paulo.henrique@elastic.co"},"sourceCommit":{"committedDate":"2024-11-01T09:59:09Z","message":"[Cloud Security] Update Findings page runtime fields required for third party data compatibility (#198635)\n\n## Summary\r\n\r\nThis PR enhances the solution proposed by [this\r\nPR](#195702), by picking only the\r\nfields that are currently not mapped by the current Third Party\r\nintegrations, this fixes performance degradation identified during the\r\nQA cycle of 8.16.0.\r\n\r\nFixes:\r\n- https://github.com/elastic/security-team/issues/11034\r\n\r\n### Misconfiguration Runtime fields\r\n\r\n- **rule.benchmark.rule_number**: runtime mapping kept because this\r\nfield is missing on `security_solution-aws.misconfiguration_latest-v1`\r\ncausing filtering out data when sorting by **Rule Number** column on the\r\nMisconfigurations Data Table.\r\n- **rule.section**: runtime mapping kept because this field is missing\r\non `security_solution-aws.misconfiguration_latest-v1` causing filtering\r\nout data when sorting by **Framework Section** column on the\r\nMisconfigurations Data Table.\r\n- **resource.sub_type**: runtime mapping kept because this field is\r\nmissing on `security_solution-aws.misconfiguration_latest-v1` causing\r\nfiltering out data when sorting by **Resource Type** column on the\r\nMisconfigurations Data Table.\r\n- **orchestrator.cluster.name**: runtime mapping kept because this field\r\nis missing on `security_solution-wiz.misconfiguration_latest-v1` causing\r\nfiltering out data when grouping by **Kubernetes Cluster** column on the\r\nMisconfigurations page.\r\n- **cloud.account.name**: runtime mapping kept because this field is\r\nmissing on `security_solution-aws.misconfiguration_latest-v1` causing\r\nfiltering out data when grouping by **Kubernetes Cluster** column on the\r\nMisconfigurations page.\r\n\r\n\r\n### Vulnerability Runtime Fields:\r\n\r\n- **observer.vendor**: runtime mapping added because this field is\r\nmapped as `text` on `security_solution-wiz.vulnerability_latest-v1`\r\ncausing filtering out when sorting by the **Vendor** column on the\r\nVulnerability Data Table\r\n- **cloud.provider**: runtime mapping added because this field is mapped\r\nas `text` on `security_solution-wiz.vulnerability_latest-v1` causing\r\nfiltering out when grouping by **Cloud Account** on the Vulnerability\r\npage. (This field is needed in order to retrieve the Cloud Provider name\r\nand icon)\r\n\r\n\r\n## Screenshot - Left: After the changes / Right: Current\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/2cbdd8b7-131c-42e4-a881-632f8cd3854b\r\n\r\n\r\nhttps://github.com/user-attachments/assets/4372feb6-4c01-4047-a90a-d6728f9400fe\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/b9e32514-f2ee-4e4d-ba5f-ea3e20d4d0b2","sha":"7a98aa176d6dcb3b850b5b9ae2dcd48e7c7ec0cb","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Cloud Security","backport:prev-minor","ci:build-cloud-image","v8.16.0","backport:version","v8.17.0"],"title":"[Cloud Security] Update Findings page runtime fields required for third party data compatibility","number":198635,"url":"https://github.com/elastic/kibana/pull/198635","mergeCommit":{"message":"[Cloud Security] Update Findings page runtime fields required for third party data compatibility (#198635)\n\n## Summary\r\n\r\nThis PR enhances the solution proposed by [this\r\nPR](#195702), by picking only the\r\nfields that are currently not mapped by the current Third Party\r\nintegrations, this fixes performance degradation identified during the\r\nQA cycle of 8.16.0.\r\n\r\nFixes:\r\n- https://github.com/elastic/security-team/issues/11034\r\n\r\n### Misconfiguration Runtime fields\r\n\r\n- **rule.benchmark.rule_number**: runtime mapping kept because this\r\nfield is missing on `security_solution-aws.misconfiguration_latest-v1`\r\ncausing filtering out data when sorting by **Rule Number** column on the\r\nMisconfigurations Data Table.\r\n- **rule.section**: runtime mapping kept because this field is missing\r\non `security_solution-aws.misconfiguration_latest-v1` causing filtering\r\nout data when sorting by **Framework Section** column on the\r\nMisconfigurations Data Table.\r\n- **resource.sub_type**: runtime mapping kept because this field is\r\nmissing on `security_solution-aws.misconfiguration_latest-v1` causing\r\nfiltering out data when sorting by **Resource Type** column on the\r\nMisconfigurations Data Table.\r\n- **orchestrator.cluster.name**: runtime mapping kept because this field\r\nis missing on `security_solution-wiz.misconfiguration_latest-v1` causing\r\nfiltering out data when grouping by **Kubernetes Cluster** column on the\r\nMisconfigurations page.\r\n- **cloud.account.name**: runtime mapping kept because this field is\r\nmissing on `security_solution-aws.misconfiguration_latest-v1` causing\r\nfiltering out data when grouping by **Kubernetes Cluster** column on the\r\nMisconfigurations page.\r\n\r\n\r\n### Vulnerability Runtime Fields:\r\n\r\n- **observer.vendor**: runtime mapping added because this field is\r\nmapped as `text` on `security_solution-wiz.vulnerability_latest-v1`\r\ncausing filtering out when sorting by the **Vendor** column on the\r\nVulnerability Data Table\r\n- **cloud.provider**: runtime mapping added because this field is mapped\r\nas `text` on `security_solution-wiz.vulnerability_latest-v1` causing\r\nfiltering out when grouping by **Cloud Account** on the Vulnerability\r\npage. (This field is needed in order to retrieve the Cloud Provider name\r\nand icon)\r\n\r\n\r\n## Screenshot - Left: After the changes / Right: Current\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/2cbdd8b7-131c-42e4-a881-632f8cd3854b\r\n\r\n\r\nhttps://github.com/user-attachments/assets/4372feb6-4c01-4047-a90a-d6728f9400fe\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/b9e32514-f2ee-4e4d-ba5f-ea3e20d4d0b2","sha":"7a98aa176d6dcb3b850b5b9ae2dcd48e7c7ec0cb"}},"sourceBranch":"main","suggestedTargetBranches":["8.16","8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/198635","number":198635,"mergeCommit":{"message":"[Cloud Security] Update Findings page runtime fields required for third party data compatibility (#198635)\n\n## Summary\r\n\r\nThis PR enhances the solution proposed by [this\r\nPR](#195702), by picking only the\r\nfields that are currently not mapped by the current Third Party\r\nintegrations, this fixes performance degradation identified during the\r\nQA cycle of 8.16.0.\r\n\r\nFixes:\r\n- https://github.com/elastic/security-team/issues/11034\r\n\r\n### Misconfiguration Runtime fields\r\n\r\n- **rule.benchmark.rule_number**: runtime mapping kept because this\r\nfield is missing on `security_solution-aws.misconfiguration_latest-v1`\r\ncausing filtering out data when sorting by **Rule Number** column on the\r\nMisconfigurations Data Table.\r\n- **rule.section**: runtime mapping kept because this field is missing\r\non `security_solution-aws.misconfiguration_latest-v1` causing filtering\r\nout data when sorting by **Framework Section** column on the\r\nMisconfigurations Data Table.\r\n- **resource.sub_type**: runtime mapping kept because this field is\r\nmissing on `security_solution-aws.misconfiguration_latest-v1` causing\r\nfiltering out data when sorting by **Resource Type** column on the\r\nMisconfigurations Data Table.\r\n- **orchestrator.cluster.name**: runtime mapping kept because this field\r\nis missing on `security_solution-wiz.misconfiguration_latest-v1` causing\r\nfiltering out data when grouping by **Kubernetes Cluster** column on the\r\nMisconfigurations page.\r\n- **cloud.account.name**: runtime mapping kept because this field is\r\nmissing on `security_solution-aws.misconfiguration_latest-v1` causing\r\nfiltering out data when grouping by **Kubernetes Cluster** column on the\r\nMisconfigurations page.\r\n\r\n\r\n### Vulnerability Runtime Fields:\r\n\r\n- **observer.vendor**: runtime mapping added because this field is\r\nmapped as `text` on `security_solution-wiz.vulnerability_latest-v1`\r\ncausing filtering out when sorting by the **Vendor** column on the\r\nVulnerability Data Table\r\n- **cloud.provider**: runtime mapping added because this field is mapped\r\nas `text` on `security_solution-wiz.vulnerability_latest-v1` causing\r\nfiltering out when grouping by **Cloud Account** on the Vulnerability\r\npage. (This field is needed in order to retrieve the Cloud Provider name\r\nand icon)\r\n\r\n\r\n## Screenshot - Left: After the changes / Right: Current\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/2cbdd8b7-131c-42e4-a881-632f8cd3854b\r\n\r\n\r\nhttps://github.com/user-attachments/assets/4372feb6-4c01-4047-a90a-d6728f9400fe\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/b9e32514-f2ee-4e4d-ba5f-ea3e20d4d0b2","sha":"7a98aa176d6dcb3b850b5b9ae2dcd48e7c7ec0cb"}},{"branch":"8.16","label":"v8.16.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Paulo Silva <paulo.henrique@elastic.co>
kibanamachine
added a commit
that referenced
this pull request
Nov 1, 2024
…for third party data compatibility (#198635) (#198648) # Backport This will backport the following commits from `main` to `8.16`: - [[Cloud Security] Update Findings page runtime fields required for third party data compatibility (#198635)](#198635) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Paulo Silva","email":"paulo.henrique@elastic.co"},"sourceCommit":{"committedDate":"2024-11-01T09:59:09Z","message":"[Cloud Security] Update Findings page runtime fields required for third party data compatibility (#198635)\n\n## Summary\r\n\r\nThis PR enhances the solution proposed by [this\r\nPR](#195702), by picking only the\r\nfields that are currently not mapped by the current Third Party\r\nintegrations, this fixes performance degradation identified during the\r\nQA cycle of 8.16.0.\r\n\r\nFixes:\r\n- https://github.com/elastic/security-team/issues/11034\r\n\r\n### Misconfiguration Runtime fields\r\n\r\n- **rule.benchmark.rule_number**: runtime mapping kept because this\r\nfield is missing on `security_solution-aws.misconfiguration_latest-v1`\r\ncausing filtering out data when sorting by **Rule Number** column on the\r\nMisconfigurations Data Table.\r\n- **rule.section**: runtime mapping kept because this field is missing\r\non `security_solution-aws.misconfiguration_latest-v1` causing filtering\r\nout data when sorting by **Framework Section** column on the\r\nMisconfigurations Data Table.\r\n- **resource.sub_type**: runtime mapping kept because this field is\r\nmissing on `security_solution-aws.misconfiguration_latest-v1` causing\r\nfiltering out data when sorting by **Resource Type** column on the\r\nMisconfigurations Data Table.\r\n- **orchestrator.cluster.name**: runtime mapping kept because this field\r\nis missing on `security_solution-wiz.misconfiguration_latest-v1` causing\r\nfiltering out data when grouping by **Kubernetes Cluster** column on the\r\nMisconfigurations page.\r\n- **cloud.account.name**: runtime mapping kept because this field is\r\nmissing on `security_solution-aws.misconfiguration_latest-v1` causing\r\nfiltering out data when grouping by **Kubernetes Cluster** column on the\r\nMisconfigurations page.\r\n\r\n\r\n### Vulnerability Runtime Fields:\r\n\r\n- **observer.vendor**: runtime mapping added because this field is\r\nmapped as `text` on `security_solution-wiz.vulnerability_latest-v1`\r\ncausing filtering out when sorting by the **Vendor** column on the\r\nVulnerability Data Table\r\n- **cloud.provider**: runtime mapping added because this field is mapped\r\nas `text` on `security_solution-wiz.vulnerability_latest-v1` causing\r\nfiltering out when grouping by **Cloud Account** on the Vulnerability\r\npage. (This field is needed in order to retrieve the Cloud Provider name\r\nand icon)\r\n\r\n\r\n## Screenshot - Left: After the changes / Right: Current\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/2cbdd8b7-131c-42e4-a881-632f8cd3854b\r\n\r\n\r\nhttps://github.com/user-attachments/assets/4372feb6-4c01-4047-a90a-d6728f9400fe\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/b9e32514-f2ee-4e4d-ba5f-ea3e20d4d0b2","sha":"7a98aa176d6dcb3b850b5b9ae2dcd48e7c7ec0cb","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Cloud Security","backport:prev-minor","ci:build-cloud-image","v8.16.0","backport:version","v8.17.0"],"title":"[Cloud Security] Update Findings page runtime fields required for third party data compatibility","number":198635,"url":"https://github.com/elastic/kibana/pull/198635","mergeCommit":{"message":"[Cloud Security] Update Findings page runtime fields required for third party data compatibility (#198635)\n\n## Summary\r\n\r\nThis PR enhances the solution proposed by [this\r\nPR](#195702), by picking only the\r\nfields that are currently not mapped by the current Third Party\r\nintegrations, this fixes performance degradation identified during the\r\nQA cycle of 8.16.0.\r\n\r\nFixes:\r\n- https://github.com/elastic/security-team/issues/11034\r\n\r\n### Misconfiguration Runtime fields\r\n\r\n- **rule.benchmark.rule_number**: runtime mapping kept because this\r\nfield is missing on `security_solution-aws.misconfiguration_latest-v1`\r\ncausing filtering out data when sorting by **Rule Number** column on the\r\nMisconfigurations Data Table.\r\n- **rule.section**: runtime mapping kept because this field is missing\r\non `security_solution-aws.misconfiguration_latest-v1` causing filtering\r\nout data when sorting by **Framework Section** column on the\r\nMisconfigurations Data Table.\r\n- **resource.sub_type**: runtime mapping kept because this field is\r\nmissing on `security_solution-aws.misconfiguration_latest-v1` causing\r\nfiltering out data when sorting by **Resource Type** column on the\r\nMisconfigurations Data Table.\r\n- **orchestrator.cluster.name**: runtime mapping kept because this field\r\nis missing on `security_solution-wiz.misconfiguration_latest-v1` causing\r\nfiltering out data when grouping by **Kubernetes Cluster** column on the\r\nMisconfigurations page.\r\n- **cloud.account.name**: runtime mapping kept because this field is\r\nmissing on `security_solution-aws.misconfiguration_latest-v1` causing\r\nfiltering out data when grouping by **Kubernetes Cluster** column on the\r\nMisconfigurations page.\r\n\r\n\r\n### Vulnerability Runtime Fields:\r\n\r\n- **observer.vendor**: runtime mapping added because this field is\r\nmapped as `text` on `security_solution-wiz.vulnerability_latest-v1`\r\ncausing filtering out when sorting by the **Vendor** column on the\r\nVulnerability Data Table\r\n- **cloud.provider**: runtime mapping added because this field is mapped\r\nas `text` on `security_solution-wiz.vulnerability_latest-v1` causing\r\nfiltering out when grouping by **Cloud Account** on the Vulnerability\r\npage. (This field is needed in order to retrieve the Cloud Provider name\r\nand icon)\r\n\r\n\r\n## Screenshot - Left: After the changes / Right: Current\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/2cbdd8b7-131c-42e4-a881-632f8cd3854b\r\n\r\n\r\nhttps://github.com/user-attachments/assets/4372feb6-4c01-4047-a90a-d6728f9400fe\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/b9e32514-f2ee-4e4d-ba5f-ea3e20d4d0b2","sha":"7a98aa176d6dcb3b850b5b9ae2dcd48e7c7ec0cb"}},"sourceBranch":"main","suggestedTargetBranches":["8.16","8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/198635","number":198635,"mergeCommit":{"message":"[Cloud Security] Update Findings page runtime fields required for third party data compatibility (#198635)\n\n## Summary\r\n\r\nThis PR enhances the solution proposed by [this\r\nPR](#195702), by picking only the\r\nfields that are currently not mapped by the current Third Party\r\nintegrations, this fixes performance degradation identified during the\r\nQA cycle of 8.16.0.\r\n\r\nFixes:\r\n- https://github.com/elastic/security-team/issues/11034\r\n\r\n### Misconfiguration Runtime fields\r\n\r\n- **rule.benchmark.rule_number**: runtime mapping kept because this\r\nfield is missing on `security_solution-aws.misconfiguration_latest-v1`\r\ncausing filtering out data when sorting by **Rule Number** column on the\r\nMisconfigurations Data Table.\r\n- **rule.section**: runtime mapping kept because this field is missing\r\non `security_solution-aws.misconfiguration_latest-v1` causing filtering\r\nout data when sorting by **Framework Section** column on the\r\nMisconfigurations Data Table.\r\n- **resource.sub_type**: runtime mapping kept because this field is\r\nmissing on `security_solution-aws.misconfiguration_latest-v1` causing\r\nfiltering out data when sorting by **Resource Type** column on the\r\nMisconfigurations Data Table.\r\n- **orchestrator.cluster.name**: runtime mapping kept because this field\r\nis missing on `security_solution-wiz.misconfiguration_latest-v1` causing\r\nfiltering out data when grouping by **Kubernetes Cluster** column on the\r\nMisconfigurations page.\r\n- **cloud.account.name**: runtime mapping kept because this field is\r\nmissing on `security_solution-aws.misconfiguration_latest-v1` causing\r\nfiltering out data when grouping by **Kubernetes Cluster** column on the\r\nMisconfigurations page.\r\n\r\n\r\n### Vulnerability Runtime Fields:\r\n\r\n- **observer.vendor**: runtime mapping added because this field is\r\nmapped as `text` on `security_solution-wiz.vulnerability_latest-v1`\r\ncausing filtering out when sorting by the **Vendor** column on the\r\nVulnerability Data Table\r\n- **cloud.provider**: runtime mapping added because this field is mapped\r\nas `text` on `security_solution-wiz.vulnerability_latest-v1` causing\r\nfiltering out when grouping by **Cloud Account** on the Vulnerability\r\npage. (This field is needed in order to retrieve the Cloud Provider name\r\nand icon)\r\n\r\n\r\n## Screenshot - Left: After the changes / Right: Current\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/2cbdd8b7-131c-42e4-a881-632f8cd3854b\r\n\r\n\r\nhttps://github.com/user-attachments/assets/4372feb6-4c01-4047-a90a-d6728f9400fe\r\n\r\n\r\n\r\nhttps://github.com/user-attachments/assets/b9e32514-f2ee-4e4d-ba5f-ea3e20d4d0b2","sha":"7a98aa176d6dcb3b850b5b9ae2dcd48e7c7ec0cb"}},{"branch":"8.16","label":"v8.16.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Paulo Silva <paulo.henrique@elastic.co>
nreese
pushed a commit
to nreese/kibana
that referenced
this pull request
Nov 1, 2024
…rd party data compatibility (elastic#198635) ## Summary This PR enhances the solution proposed by [this PR](elastic#195702), by picking only the fields that are currently not mapped by the current Third Party integrations, this fixes performance degradation identified during the QA cycle of 8.16.0. Fixes: - elastic/security-team#11034 ### Misconfiguration Runtime fields - **rule.benchmark.rule_number**: runtime mapping kept because this field is missing on `security_solution-aws.misconfiguration_latest-v1` causing filtering out data when sorting by **Rule Number** column on the Misconfigurations Data Table. - **rule.section**: runtime mapping kept because this field is missing on `security_solution-aws.misconfiguration_latest-v1` causing filtering out data when sorting by **Framework Section** column on the Misconfigurations Data Table. - **resource.sub_type**: runtime mapping kept because this field is missing on `security_solution-aws.misconfiguration_latest-v1` causing filtering out data when sorting by **Resource Type** column on the Misconfigurations Data Table. - **orchestrator.cluster.name**: runtime mapping kept because this field is missing on `security_solution-wiz.misconfiguration_latest-v1` causing filtering out data when grouping by **Kubernetes Cluster** column on the Misconfigurations page. - **cloud.account.name**: runtime mapping kept because this field is missing on `security_solution-aws.misconfiguration_latest-v1` causing filtering out data when grouping by **Kubernetes Cluster** column on the Misconfigurations page. ### Vulnerability Runtime Fields: - **observer.vendor**: runtime mapping added because this field is mapped as `text` on `security_solution-wiz.vulnerability_latest-v1` causing filtering out when sorting by the **Vendor** column on the Vulnerability Data Table - **cloud.provider**: runtime mapping added because this field is mapped as `text` on `security_solution-wiz.vulnerability_latest-v1` causing filtering out when grouping by **Cloud Account** on the Vulnerability page. (This field is needed in order to retrieve the Cloud Provider name and icon) ## Screenshot - Left: After the changes / Right: Current https://github.com/user-attachments/assets/2cbdd8b7-131c-42e4-a881-632f8cd3854b https://github.com/user-attachments/assets/4372feb6-4c01-4047-a90a-d6728f9400fe https://github.com/user-attachments/assets/b9e32514-f2ee-4e4d-ba5f-ea3e20d4d0b2
8 tasks
maxcold
added a commit
that referenced
this pull request
Feb 27, 2026
## Summary Removes client-side runtime mappings from CDR (Cloud Detection & Response) misconfigurations and vulnerabilities data grid queries. These runtime mappings were a workaround ([#198635](#198635)) for missing `keyword` mappings in 3rd-party integration indexes (e.g. Wiz), which caused sorting and grouping to fail. **Why they're no longer needed:** The `ecs@mappings` component template — which includes a catch-all `all_strings_to_keywords` dynamic mapping rule — is now applied to all Fleet transform destination index templates ([#223878](#223878)). This means all string fields in CDR latest indexes are automatically mapped as `keyword`, making the runtime mappings redundant. **What changed:** - Removed 4 runtime mapping constants from `constants.ts` - Removed `getRuntimeMappingsFromSort()` from data table query builders (misconfigs + vulns) - Removed `getRuntimeMappingsByGroupField()` from grouping query builders (misconfigs + vulns) - Added `unmapped_type: 'keyword'` to sort fields to handle indexes where a field may not be mapped - Fixed painless sort scripts to handle missing fields with `doc.containsKey()` checks (previously would throw null pointer errors for 3P documents missing native CSP fields) Closes #198651 ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks - **Low risk**: Sorting by fields missing in some indexes (e.g. `rule.benchmark.rule_number` for Wiz) now relies on ES `unmapped_type` instead of runtime mappings. Verified locally that documents are included correctly with missing-field docs appearing at the end of sort results.
qn895
pushed a commit
to qn895/kibana
that referenced
this pull request
Mar 11, 2026
## Summary Removes client-side runtime mappings from CDR (Cloud Detection & Response) misconfigurations and vulnerabilities data grid queries. These runtime mappings were a workaround ([elastic#198635](elastic#198635)) for missing `keyword` mappings in 3rd-party integration indexes (e.g. Wiz), which caused sorting and grouping to fail. **Why they're no longer needed:** The `ecs@mappings` component template — which includes a catch-all `all_strings_to_keywords` dynamic mapping rule — is now applied to all Fleet transform destination index templates ([elastic#223878](elastic#223878)). This means all string fields in CDR latest indexes are automatically mapped as `keyword`, making the runtime mappings redundant. **What changed:** - Removed 4 runtime mapping constants from `constants.ts` - Removed `getRuntimeMappingsFromSort()` from data table query builders (misconfigs + vulns) - Removed `getRuntimeMappingsByGroupField()` from grouping query builders (misconfigs + vulns) - Added `unmapped_type: 'keyword'` to sort fields to handle indexes where a field may not be mapped - Fixed painless sort scripts to handle missing fields with `doc.containsKey()` checks (previously would throw null pointer errors for 3P documents missing native CSP fields) Closes elastic#198651 ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks - **Low risk**: Sorting by fields missing in some indexes (e.g. `rule.benchmark.rule_number` for Wiz) now relies on ES `unmapped_type` instead of runtime mappings. Verified locally that documents are included correctly with missing-field docs appearing at the end of sort results.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR enhances the solution proposed by this PR, by picking only the fields that are currently not mapped by the current Third Party integrations, this fixes performance degradation identified during the QA cycle of 8.16.0.
Fixes:
Misconfiguration Runtime fields
security_solution-aws.misconfiguration_latest-v1causing filtering out data when sorting by Rule Number column on the Misconfigurations Data Table.security_solution-aws.misconfiguration_latest-v1causing filtering out data when sorting by Framework Section column on the Misconfigurations Data Table.security_solution-aws.misconfiguration_latest-v1causing filtering out data when sorting by Resource Type column on the Misconfigurations Data Table.security_solution-wiz.misconfiguration_latest-v1causing filtering out data when grouping by Kubernetes Cluster column on the Misconfigurations page.security_solution-aws.misconfiguration_latest-v1causing filtering out data when grouping by Kubernetes Cluster column on the Misconfigurations page.Vulnerability Runtime Fields:
textonsecurity_solution-wiz.vulnerability_latest-v1causing filtering out when sorting by the Vendor column on the Vulnerability Data Tabletextonsecurity_solution-wiz.vulnerability_latest-v1causing filtering out when grouping by Cloud Account on the Vulnerability page. (This field is needed in order to retrieve the Cloud Provider name and icon)Screenshot - Left: After the changes / Right: Current
misconfigurations.mov
vulnerabilities-1.mov
vulnerabilities-2.mov