Remove ECS field definitions from CDR transform destinations#17552
Remove ECS field definitions from CDR transform destinations#17552maxcold merged 6 commits intoelastic:mainfrom
Conversation
…ered by ecs@mappings component template Removes redundant ECS field definitions from transform destination index field mappings across 7 integration packages: wiz, cloud_security_posture, qualys_vmdr, tenable_io, rapid7_insightvm, aws, and microsoft_defender_endpoint. These ECS fields are now provided by the ecs@mappings component template added to Fleet transform destination index templates. Only constant_keyword type overrides are preserved in ecs-overridden.yml files where needed.
andrewkroh
added
Integration:microsoft_defender_endpoint
andrewkroh
added
Team:Security-Service Integrations
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
…ransforms Rename ecs.yml to ecs-overridden.yml for transforms that only contain constant_keyword type overrides in aws (awsconfig, awsinspector), aws_securityhub, google_scc, and microsoft_defender_endpoint. Also strip external: ecs from all ecs-overridden.yml files for consistency.
maxcold
force-pushed
the
remove-ecs-mappings-transforms
branch
from
9b843db to
c7d79c2
Compare
💔 Build Failed
Failed CI StepsHistory
|
|
@kcreddy updated |
andrewkroh
added
Integration:aws_securityhub
packages/aws/elasticsearch/transform/latest_cdr_misconfigurations/fields/ecs-overridden.yml
Show resolved
Hide resolved
kcreddy
left a comment
kcreddy
left a comment
There was a problem hiding this comment.
LGTM.
But would like to get @chrisberkhout thoughts as per his earlier comment - elastic/elastic-package#1641 (comment)
🚀 Benchmarks reportTo see the full report comment with |
There was a problem hiding this comment.
This is great!
Very nice to remove the redundant defs and a potential source of divergence between source and destination indexes.
I see the overrides remain, which is good.
The one other case I can think of to have the explicit definitions is when there's an issue with field limit and it's important to ensure that certain fields are definitely present (e.g. for rules) before dynamic mappings fight for what remains. But that should be rare and only a workaround rather than a standard practice.
Not critical to do now, but it would be good to make sure that for each transform the files in elasticsearch/transform/*/fields/*.yml files are identical in name and content to their source data stream's field definition files in data_stream/*/fields/*.yml, except is-transform-source-{false,true}.yml and anything else that doesn't match for a good reason. When definitions exist in only once place, it's good to have them in a uniquely named file. All this will get easier to keep in sync once we start using the elastic-package link function (although it currently has a bug that I'm about to fix).
|
Package aws - 6.3.0 containing this change is available at https://epr.elastic.co/package/aws/6.3.0/ |
|
Package aws_securityhub - 0.2.0 containing this change is available at https://epr.elastic.co/package/aws_securityhub/0.2.0/ |
|
Package cloud_security_posture - 3.3.0-preview07 containing this change is available at https://epr.elastic.co/package/cloud_security_posture/3.3.0-preview07/ |
|
Package google_scc - 2.3.0 containing this change is available at https://epr.elastic.co/package/google_scc/2.3.0/ |
|
Package microsoft_defender_endpoint - 4.4.0 containing this change is available at https://epr.elastic.co/package/microsoft_defender_endpoint/4.4.0/ |
|
Package qualys_vmdr - 6.16.0 containing this change is available at https://epr.elastic.co/package/qualys_vmdr/6.16.0/ |
|
Package rapid7_insightvm - 2.6.0 containing this change is available at https://epr.elastic.co/package/rapid7_insightvm/2.6.0/ |
|
Package tenable_io - 4.9.0 containing this change is available at https://epr.elastic.co/package/tenable_io/4.9.0/ |
|
Package wiz - 4.1.0 containing this change is available at https://epr.elastic.co/package/wiz/4.1.0/ |
6 participants
Proposed commit message
Removes redundant ECS field definitions from transform destination index field mappings across 7 integration packages, now covered by the
ecs@mappingscomponent template added to Fleet transform destination index templates (elastic/kibana#223878).Packages updated: wiz, cloud_security_posture, qualys_vmdr, tenable_io, rapid7_insightvm, aws, microsoft_defender_endpoint
Changes per package:
ecs.ymlfiles containing ECS field definitions from transform field directoriesecs-overridden.ymlwhereconstant_keywordtype overrides need to be preserved (qualys_vmdr, rapid7_insightvm, aws)Note: The Wiz package also bumps its Kibana version constraint from
~8.16.6 || ~8.17.4 || ^8.18.0 || ^9.0.0to^8.19.0 || ^9.1.0sinceecs@mappingswas introduced in 8.19.0/9.1.0.Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Simplest to use https://github.com/elastic/security-documents-generator command
cspto generate some data, build packages from this PR, install them and ingest new data. Everything should work as before. Same for first installing the new version packages and then generating data with the data generatorRelated issues
Screenshots