Remove runtime mappings from CDR data grid queries#254742
Merged
Conversation
Runtime mappings were a workaround for missing keyword mappings in 3P integration indexes. Now that ecs@mappings component template is applied to all Fleet transform destination indexes, all string fields are automatically mapped as keyword, making runtime mappings redundant. Replace them with unmapped_type: 'keyword' on sort fields and null-safe painless scripts to handle indexes where fields may not exist. Closes #198651
maxcold
added
release_note:skip
Contributor
|
Pinging @elastic/contextual-security-apps (Team:Cloud Security) |
maxcold
commented
Feb 25, 2026
| // and an empty string for descending so missing values also sort last. | ||
| // Note: Painless double-quoted strings only support \\ and \" escapes, | ||
| // so we embed the actual U+FFFF character rather than a \uffff escape sequence. | ||
| const missingFallback = direction === 'asc' ? '\uffff' : ''; |
Contributor
Author
There was a problem hiding this comment.
I tested this approach with data from generator, seems to be working fine, though look a bit weird
Extract sort logic to common/utils/findings_sort.ts so both the frontend and integration tests can share the same Painless sort scripts. Add FTR integration test that validates scripts compile and execute against real ES, including missing-values-last behavior.
Contributor
|
tested locally, LGTM, there some issue with the types in the CI. |
alexreal1314
approved these changes
Feb 26, 2026
The `direction` parameter was typed as `string`, causing `order: direction` to be incompatible with ES client's `SortOrder` type when the return value is passed to `es.search()`.
EsIndexDataProvider.destroyIndex() now resolves aliases to their concrete backing indices before calling indices.delete, since ES 9.x rejects deletion by alias name.
… creation Use waitForPluginInitialized + deleteAll/addBulk pattern matching other tests in the same directory, leveraging the integration's index mappings via the alias rather than creating a custom index.
The test validates Painless sort scripts against ES, not an API route.
Contributor
💚 Build Succeeded
Metrics [docs]Module Count
Async chunks
Page load bundle
Unknown metric groupsESLint disabled line counts
Total ESLint disabled count
History
|
qn895
pushed a commit
to qn895/kibana
that referenced
this pull request
Mar 11, 2026
## Summary Removes client-side runtime mappings from CDR (Cloud Detection & Response) misconfigurations and vulnerabilities data grid queries. These runtime mappings were a workaround ([elastic#198635](elastic#198635)) for missing `keyword` mappings in 3rd-party integration indexes (e.g. Wiz), which caused sorting and grouping to fail. **Why they're no longer needed:** The `ecs@mappings` component template — which includes a catch-all `all_strings_to_keywords` dynamic mapping rule — is now applied to all Fleet transform destination index templates ([elastic#223878](elastic#223878)). This means all string fields in CDR latest indexes are automatically mapped as `keyword`, making the runtime mappings redundant. **What changed:** - Removed 4 runtime mapping constants from `constants.ts` - Removed `getRuntimeMappingsFromSort()` from data table query builders (misconfigs + vulns) - Removed `getRuntimeMappingsByGroupField()` from grouping query builders (misconfigs + vulns) - Added `unmapped_type: 'keyword'` to sort fields to handle indexes where a field may not be mapped - Fixed painless sort scripts to handle missing fields with `doc.containsKey()` checks (previously would throw null pointer errors for 3P documents missing native CSP fields) Closes elastic#198651 ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks - **Low risk**: Sorting by fields missing in some indexes (e.g. `rule.benchmark.rule_number` for Wiz) now relies on ES `unmapped_type` instead of runtime mappings. Verified locally that documents are included correctly with missing-field docs appearing at the end of sort results.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes client-side runtime mappings from CDR (Cloud Detection & Response) misconfigurations and vulnerabilities data grid queries. These runtime mappings were a workaround (#198635) for missing
keywordmappings in 3rd-party integration indexes (e.g. Wiz), which caused sorting and grouping to fail.Why they're no longer needed: The
ecs@mappingscomponent template — which includes a catch-allall_strings_to_keywordsdynamic mapping rule — is now applied to all Fleet transform destination index templates (#223878). This means all string fields in CDR latest indexes are automatically mapped askeyword, making the runtime mappings redundant.What changed:
constants.tsgetRuntimeMappingsFromSort()from data table query builders (misconfigs + vulns)getRuntimeMappingsByGroupField()from grouping query builders (misconfigs + vulns)unmapped_type: 'keyword'to sort fields to handle indexes where a field may not be mappeddoc.containsKey()checks (previously would throw null pointer errors for 3P documents missing native CSP fields)Closes #198651
Checklist
release_note:breakinglabel should be applied in these situations.release_note:*label is applied per the guidelinesbackport:*labels.Identify risks
rule.benchmark.rule_numberfor Wiz) now relies on ESunmapped_typeinstead of runtime mappings. Verified locally that documents are included correctly with missing-field docs appearing at the end of sort results.