Fix generation of dynamic mapping for object with specific subfield#204104
Merged
jsoriano merged 4 commits intoelastic:mainfrom Dec 16, 2024
Merged
Fix generation of dynamic mapping for object with specific subfield#204104jsoriano merged 4 commits intoelastic:mainfrom
jsoriano merged 4 commits intoelastic:mainfrom
Conversation
botelastic
bot
added
the
Team:Fleet
Contributor
|
Pinging @elastic/fleet (Team:Fleet) |
jsoriano
force-pushed
the
dynamic-mapping-non-empty-object
branch
from
a481fb2 to
3aa3782
Compare
4 tasks
| - name: labels.count | ||
| type: long | ||
| `; | ||
| const objectFieldWithPropertyReversedMapping = { |
Contributor
There was a problem hiding this comment.
should there be a test with group type?
Member
Author
There was a problem hiding this comment.
The group type cannot have object_type. Only internally in fleet after the deduplication.
Contributor
|
Starting backport for target branches: 8.16, 8.17, 8.x https://github.com/elastic/kibana/actions/runs/12361165154 |
Contributor
💚 Build Succeeded
Metrics [docs]
History
cc @jsoriano |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Dec 16, 2024
…lastic#204104) Fix generation of dynamic mapping for objects that have more specific subfields in separate definitions. This can be reproduced for example with: ``` - name: labels type: object object_type: keyword object_type_mapping_type: '*' - name: labels.count type: long ``` Fleet expands and deduplicates field definitions before generating the mappings, so the definitions above are converted to something like the following: ``` - name: labels type: group object_type: keyword object_type_mapping_type: '*' fields: - name: count type: long ``` Usually fields of type `group` don't have an `object_type`, so this was being ignored, the dynamic mapping was not being generated. This issue was not reproduced if the object field name includes a wildcard, like in `labels.*`, because then the expansion and deduplication resolves to something like this: ``` - name: labels type: group fields: - name: '*' type: object object_type: keyword object_type_mapping_type: '*' - name: count type: long ``` (cherry picked from commit e3877e0)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Dec 16, 2024
…lastic#204104) Fix generation of dynamic mapping for objects that have more specific subfields in separate definitions. This can be reproduced for example with: ``` - name: labels type: object object_type: keyword object_type_mapping_type: '*' - name: labels.count type: long ``` Fleet expands and deduplicates field definitions before generating the mappings, so the definitions above are converted to something like the following: ``` - name: labels type: group object_type: keyword object_type_mapping_type: '*' fields: - name: count type: long ``` Usually fields of type `group` don't have an `object_type`, so this was being ignored, the dynamic mapping was not being generated. This issue was not reproduced if the object field name includes a wildcard, like in `labels.*`, because then the expansion and deduplication resolves to something like this: ``` - name: labels type: group fields: - name: '*' type: object object_type: keyword object_type_mapping_type: '*' - name: count type: long ``` (cherry picked from commit e3877e0)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Dec 16, 2024
…lastic#204104) Fix generation of dynamic mapping for objects that have more specific subfields in separate definitions. This can be reproduced for example with: ``` - name: labels type: object object_type: keyword object_type_mapping_type: '*' - name: labels.count type: long ``` Fleet expands and deduplicates field definitions before generating the mappings, so the definitions above are converted to something like the following: ``` - name: labels type: group object_type: keyword object_type_mapping_type: '*' fields: - name: count type: long ``` Usually fields of type `group` don't have an `object_type`, so this was being ignored, the dynamic mapping was not being generated. This issue was not reproduced if the object field name includes a wildcard, like in `labels.*`, because then the expansion and deduplication resolves to something like this: ``` - name: labels type: group fields: - name: '*' type: object object_type: keyword object_type_mapping_type: '*' - name: count type: long ``` (cherry picked from commit e3877e0)
Contributor
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added
the
backport missing
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
1 similar comment
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
JoseLuisGJ
pushed a commit
to JoseLuisGJ/kibana
that referenced
this pull request
Dec 19, 2024
…lastic#204104) Fix generation of dynamic mapping for objects that have more specific subfields in separate definitions. This can be reproduced for example with: ``` - name: labels type: object object_type: keyword object_type_mapping_type: '*' - name: labels.count type: long ``` Fleet expands and deduplicates field definitions before generating the mappings, so the definitions above are converted to something like the following: ``` - name: labels type: group object_type: keyword object_type_mapping_type: '*' fields: - name: count type: long ``` Usually fields of type `group` don't have an `object_type`, so this was being ignored, the dynamic mapping was not being generated. This issue was not reproduced if the object field name includes a wildcard, like in `labels.*`, because then the expansion and deduplication resolves to something like this: ``` - name: labels type: group fields: - name: '*' type: object object_type: keyword object_type_mapping_type: '*' - name: count type: long ```
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
12 similar comments
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
jsoriano
added a commit
that referenced
this pull request
Jan 8, 2025
…ield (#204104) (#204493) # Backport This will backport the following commits from `main` to `8.x`: - [Fix generation of dynamic mapping for object with specific subfield (#204104)](#204104) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Jaime Soriano Pastor","email":"jaime.soriano@elastic.co"},"sourceCommit":{"committedDate":"2024-12-16T20:58:30Z","message":"Fix generation of dynamic mapping for object with specific subfield (#204104)\n\nFix generation of dynamic mapping for objects that have more specific\r\nsubfields in separate definitions.\r\n\r\nThis can be reproduced for example with:\r\n```\r\n- name: labels\r\n type: object\r\n object_type: keyword\r\n object_type_mapping_type: '*'\r\n- name: labels.count\r\n type: long\r\n```\r\n\r\nFleet expands and deduplicates field definitions before generating the\r\nmappings, so the definitions above are converted to something like the\r\nfollowing:\r\n```\r\n- name: labels\r\n type: group\r\n object_type: keyword\r\n object_type_mapping_type: '*'\r\n fields:\r\n - name: count\r\n type: long\r\n```\r\n\r\nUsually fields of type `group` don't have an `object_type`, so this was\r\nbeing ignored, the dynamic mapping was not being generated.\r\n\r\nThis issue was not reproduced if the object field name includes a\r\nwildcard, like in `labels.*`, because then the expansion and\r\ndeduplication resolves to something like this:\r\n```\r\n- name: labels\r\n type: group\r\n fields:\r\n - name: '*'\r\n type: object\r\n object_type: keyword\r\n object_type_mapping_type: '*'\r\n - name: count\r\n type: long\r\n```","sha":"e3877e053405bae71b5576648cb7c637c4a23f9a","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Fleet","v9.0.0","backport:prev-major"],"title":"Fix generation of dynamic mapping for object with specific subfield","number":204104,"url":"https://github.com/elastic/kibana/pull/204104","mergeCommit":{"message":"Fix generation of dynamic mapping for object with specific subfield (#204104)\n\nFix generation of dynamic mapping for objects that have more specific\r\nsubfields in separate definitions.\r\n\r\nThis can be reproduced for example with:\r\n```\r\n- name: labels\r\n type: object\r\n object_type: keyword\r\n object_type_mapping_type: '*'\r\n- name: labels.count\r\n type: long\r\n```\r\n\r\nFleet expands and deduplicates field definitions before generating the\r\nmappings, so the definitions above are converted to something like the\r\nfollowing:\r\n```\r\n- name: labels\r\n type: group\r\n object_type: keyword\r\n object_type_mapping_type: '*'\r\n fields:\r\n - name: count\r\n type: long\r\n```\r\n\r\nUsually fields of type `group` don't have an `object_type`, so this was\r\nbeing ignored, the dynamic mapping was not being generated.\r\n\r\nThis issue was not reproduced if the object field name includes a\r\nwildcard, like in `labels.*`, because then the expansion and\r\ndeduplication resolves to something like this:\r\n```\r\n- name: labels\r\n type: group\r\n fields:\r\n - name: '*'\r\n type: object\r\n object_type: keyword\r\n object_type_mapping_type: '*'\r\n - name: count\r\n type: long\r\n```","sha":"e3877e053405bae71b5576648cb7c637c4a23f9a"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/204104","number":204104,"mergeCommit":{"message":"Fix generation of dynamic mapping for object with specific subfield (#204104)\n\nFix generation of dynamic mapping for objects that have more specific\r\nsubfields in separate definitions.\r\n\r\nThis can be reproduced for example with:\r\n```\r\n- name: labels\r\n type: object\r\n object_type: keyword\r\n object_type_mapping_type: '*'\r\n- name: labels.count\r\n type: long\r\n```\r\n\r\nFleet expands and deduplicates field definitions before generating the\r\nmappings, so the definitions above are converted to something like the\r\nfollowing:\r\n```\r\n- name: labels\r\n type: group\r\n object_type: keyword\r\n object_type_mapping_type: '*'\r\n fields:\r\n - name: count\r\n type: long\r\n```\r\n\r\nUsually fields of type `group` don't have an `object_type`, so this was\r\nbeing ignored, the dynamic mapping was not being generated.\r\n\r\nThis issue was not reproduced if the object field name includes a\r\nwildcard, like in `labels.*`, because then the expansion and\r\ndeduplication resolves to something like this:\r\n```\r\n- name: labels\r\n type: group\r\n fields:\r\n - name: '*'\r\n type: object\r\n object_type: keyword\r\n object_type_mapping_type: '*'\r\n - name: count\r\n type: long\r\n```","sha":"e3877e053405bae71b5576648cb7c637c4a23f9a"}}]}] BACKPORT--> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
kibanamachine
added
v8.18.0
and removed
backport missing
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix generation of dynamic mapping for objects that have more specific subfields in separate definitions.
This can be reproduced for example with:
Fleet expands and deduplicates field definitions before generating the mappings, so the definitions above are converted to something like the following:
Usually fields of type
groupdon't have anobject_type, so this was being ignored, the dynamic mapping was not being generated.This issue was not reproduced if the object field name includes a wildcard, like in
labels.*, because then the expansion and deduplication resolves to something like this:Checklist
release_note:*label is applied per the guidelinesIdentify risks
Probability: low, we have only found one package (see Add initial support to validate the mappings in system tests elastic-package#2214 (comment)).
Severity: low, in the package where we have found this issue, applying an equivalent workaround (Workaround issue with missing dynamic mapping in tines package integrations#12082) doesn't cause any issue, at least with the testing coverage we have. There are no reasons to think that these additional mappings can cause problems, as they correspond with the intention of the package developer.