Add ecs@mappings component template to transform destination index template#223878
Merged
Add ecs@mappings component template to transform destination index template#223878
Conversation
maxcold
added
release_note:enhancement
Team:Fleet
maxcold
added
v8.19.0
ci:cloud-deploy
Contributor
|
Cloud deployment initiated, see credentials at: https://buildkite.com/elastic/kibana-deploy-cloud-from-pr/builds/241 |
maxcold
changed the title
Contributor
|
Pinging @elastic/fleet (Team:Fleet) |
Contributor
|
Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security) |
| composed_of: Object.keys(componentTemplates), | ||
| composed_of: [ | ||
| ...Object.keys(componentTemplates), | ||
| STACK_COMPONENT_TEMPLATE_ECS_MAPPINGS, |
Contributor
There was a problem hiding this comment.
To confirm, is it expected that this change impacts all packages with transforms?
Contributor
Author
There was a problem hiding this comment.
yes, we had a few discussions in the related threads:
- https://elastic.slack.com/archives/CJZ9U5G9X/p1749565737961249
- Import ECS dynamic templates into transform destinations elastic-package#1641 (comment)
It's an option to make this change opt-in through the package-spec, but no blockers were raised to make it happen for every transform
Contributor
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]
History
|
Contributor
|
Starting backport for target branches: 8.19 https://github.com/elastic/kibana/actions/runs/15702970941 |
Contributor
💔 All backports failed
Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
kibanamachine
added
the
backport missing
Contributor
|
Friendly reminder: Looks like this PR hasn’t been backported yet. |
Contributor
Author
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
1 task
maxcold
added a commit
that referenced
this pull request
Jun 19, 2025
…ndex template (#223878) (#224546) # Backport This will backport the following commits from `main` to `8.19`: - [Add ecs@mappings component template to transform destination index template (#223878)](#223878) <!--- Backport version: 10.0.0 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Maxim Kholod","email":"maxim.kholod@elastic.co"},"sourceCommit":{"committedDate":"2025-06-17T09:05:10Z","message":"Add ecs@mappings component template to transform destination index template (#223878)\n\n## Summary\n\nAdding `ecs@mappings` to the index template created by fleet for the\ntransform destination index template\n\nCloses:\n- https://github.com/elastic/security-team/issues/10720\n\nRelated discussions:\n- https://elastic.slack.com/archives/CJZ9U5G9X/p1749565737961249\n-\nhttps://github.com/elastic/elastic-package/issues/1641#issue-2098524093\n\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [ ] Any text added follows [EUI's writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\nsentence case text and includes [i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n- [ ]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas added for features that require explanation or tutorials\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [ ] If a plugin configuration key changed, check if it needs to be\nallowlisted in the cloud and added to the [docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n- [ ] This was checked for breaking HTTP API changes, and any breaking\nchanges have been approved by the breaking-change committee. The\n`release_note:breaking` label should be applied in these situations.\n- [ ] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [ ] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"76fdb46be8f3a0419c9288c12acbd9d26fd2aa2a","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","backport missing","Team:Fleet","Team:Cloud Security","ci:cloud-deploy","backport:version","v9.1.0","v8.19.0"],"title":"Add ecs@mappings component template to transform destination index template","number":223878,"url":"https://github.com/elastic/kibana/pull/223878","mergeCommit":{"message":"Add ecs@mappings component template to transform destination index template (#223878)\n\n## Summary\n\nAdding `ecs@mappings` to the index template created by fleet for the\ntransform destination index template\n\nCloses:\n- https://github.com/elastic/security-team/issues/10720\n\nRelated discussions:\n- https://elastic.slack.com/archives/CJZ9U5G9X/p1749565737961249\n-\nhttps://github.com/elastic/elastic-package/issues/1641#issue-2098524093\n\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [ ] Any text added follows [EUI's writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\nsentence case text and includes [i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n- [ ]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas added for features that require explanation or tutorials\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [ ] If a plugin configuration key changed, check if it needs to be\nallowlisted in the cloud and added to the [docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n- [ ] This was checked for breaking HTTP API changes, and any breaking\nchanges have been approved by the breaking-change committee. The\n`release_note:breaking` label should be applied in these situations.\n- [ ] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [ ] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"76fdb46be8f3a0419c9288c12acbd9d26fd2aa2a"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/223878","number":223878,"mergeCommit":{"message":"Add ecs@mappings component template to transform destination index template (#223878)\n\n## Summary\n\nAdding `ecs@mappings` to the index template created by fleet for the\ntransform destination index template\n\nCloses:\n- https://github.com/elastic/security-team/issues/10720\n\nRelated discussions:\n- https://elastic.slack.com/archives/CJZ9U5G9X/p1749565737961249\n-\nhttps://github.com/elastic/elastic-package/issues/1641#issue-2098524093\n\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [ ] Any text added follows [EUI's writing\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\nsentence case text and includes [i18n\nsupport](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)\n- [ ]\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\nwas added for features that require explanation or tutorials\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [ ] If a plugin configuration key changed, check if it needs to be\nallowlisted in the cloud and added to the [docker\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\n- [ ] This was checked for breaking HTTP API changes, and any breaking\nchanges have been approved by the breaking-change committee. The\n`release_note:breaking` label should be applied in these situations.\n- [ ] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [ ] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"76fdb46be8f3a0419c9288c12acbd9d26fd2aa2a"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
kibanamachine
removed
the
backport missing
This was referenced Feb 24, 2026
maxcold
added a commit
that referenced
this pull request
Feb 27, 2026
## Summary Removes client-side runtime mappings from CDR (Cloud Detection & Response) misconfigurations and vulnerabilities data grid queries. These runtime mappings were a workaround ([#198635](#198635)) for missing `keyword` mappings in 3rd-party integration indexes (e.g. Wiz), which caused sorting and grouping to fail. **Why they're no longer needed:** The `ecs@mappings` component template — which includes a catch-all `all_strings_to_keywords` dynamic mapping rule — is now applied to all Fleet transform destination index templates ([#223878](#223878)). This means all string fields in CDR latest indexes are automatically mapped as `keyword`, making the runtime mappings redundant. **What changed:** - Removed 4 runtime mapping constants from `constants.ts` - Removed `getRuntimeMappingsFromSort()` from data table query builders (misconfigs + vulns) - Removed `getRuntimeMappingsByGroupField()` from grouping query builders (misconfigs + vulns) - Added `unmapped_type: 'keyword'` to sort fields to handle indexes where a field may not be mapped - Fixed painless sort scripts to handle missing fields with `doc.containsKey()` checks (previously would throw null pointer errors for 3P documents missing native CSP fields) Closes #198651 ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks - **Low risk**: Sorting by fields missing in some indexes (e.g. `rule.benchmark.rule_number` for Wiz) now relies on ES `unmapped_type` instead of runtime mappings. Verified locally that documents are included correctly with missing-field docs appearing at the end of sort results.
qn895
pushed a commit
to qn895/kibana
that referenced
this pull request
Mar 11, 2026
## Summary Removes client-side runtime mappings from CDR (Cloud Detection & Response) misconfigurations and vulnerabilities data grid queries. These runtime mappings were a workaround ([elastic#198635](elastic#198635)) for missing `keyword` mappings in 3rd-party integration indexes (e.g. Wiz), which caused sorting and grouping to fail. **Why they're no longer needed:** The `ecs@mappings` component template — which includes a catch-all `all_strings_to_keywords` dynamic mapping rule — is now applied to all Fleet transform destination index templates ([elastic#223878](elastic#223878)). This means all string fields in CDR latest indexes are automatically mapped as `keyword`, making the runtime mappings redundant. **What changed:** - Removed 4 runtime mapping constants from `constants.ts` - Removed `getRuntimeMappingsFromSort()` from data table query builders (misconfigs + vulns) - Removed `getRuntimeMappingsByGroupField()` from grouping query builders (misconfigs + vulns) - Added `unmapped_type: 'keyword'` to sort fields to handle indexes where a field may not be mapped - Fixed painless sort scripts to handle missing fields with `doc.containsKey()` checks (previously would throw null pointer errors for 3P documents missing native CSP fields) Closes elastic#198651 ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks - **Low risk**: Sorting by fields missing in some indexes (e.g. `rule.benchmark.rule_number` for Wiz) now relies on ES `unmapped_type` instead of runtime mappings. Verified locally that documents are included correctly with missing-field docs appearing at the end of sort results.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adding
ecs@mappingsto the index template created by fleet for the transform destination index templateCloses:
Related discussions:
Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
release_note:breakinglabel should be applied in these situations.release_note:*label is applied per the guidelines