[Cases] Revert incremental id and add multifield#230278
Merged
michaelolo24 merged 44 commits intoelastic:mainfrom Sep 29, 2025
Merged
[Cases] Revert incremental id and add multifield#230278michaelolo24 merged 44 commits intoelastic:mainfrom
michaelolo24 merged 44 commits intoelastic:mainfrom
Conversation
2843f8b to
c132a10
Compare
8 tasks
d8402b1 to
bb5de36
Compare
bb5de36 to
1fcc348
Compare
1fcc348 to
062dc19
Compare
56ce0d2 to
df8542d
Compare
6072843 to
9ec7790
Compare
…ic#222874) This adds and enables the case id incrementer service ([design doc](https://docs.google.com/document/d/1DZKTPl7UryYjpjVMNhIYbE82OADVOg93-d02f0ZQtUI/edit?tab=t.0#heading=h.6qjc4qynaeuo)). In order not to stress bulk creation of cases, we're processing incremental ids asynchronously, meaning they will not immediately appear in the UI. The feature is currently disabled by default to allow for testing in additional environments after merging but can be enabled by setting `xpack.cases.incrementalIdService.enabled=true` in `kibana(.dev).yml`. Once the flag is enabled, actually rendering the IDs in the UI is disabled by default (for now) and has to be enabled in the advanced settings (`cases:incrementalIdDisplay:enabled`). Cases can be found by their incremental ID by searching for `#{incremental_case_id}` in the cases table. **Incremental ID in the case detail page** <img width="1506" alt="Screenshot 2025-06-05 at 15 46 42" src="https://github.com/user-attachments/assets/f51ae0cd-a2e8-48f7-a6db-05f9f1285e95" /> **Incremental ID in the cases table** <img width="1240" alt="Screenshot 2025-06-05 at 20 32 32" src="https://github.com/user-attachments/assets/619b3f12-1986-4bc7-b9e8-f7556d0c546c" /> **Searching for case by its incremental ID** <img width="1239" alt="Screenshot 2025-06-05 at 20 33 36" src="https://github.com/user-attachments/assets/771df512-7436-4aa0-88f9-ac3e1e161455" /> <details> <summary>Validation script</summary> Use this script to investigate if there are duplicates or gaps: ```js import * as fs from 'fs'; // Query to get all cases from all namespaces sorted by incremental_id // GET .kibana_alerting_cases/_search?_source_excludes=* // { // "query": { // "exists": { // "field": "cases.incremental_id" // } // }, // "fields": [ // "cases.incremental_id", // "cases.title", // "namespaces" // ], // "from": 0, // "size": 10000, // "sort": [ // { // "cases.incremental_id": { // "order": "asc" // } // } // ] // } // Put those results into `test.json` in the same directory // You might need to add `"search_after": [40007]` in case you want to look at more than 10k cases. // In that case, replace `[40007]` with whatever value the last item has in `"sort": [2102]` // Concatenate hits if needed (10k per file) const cases = [ JSON.parse(fs.readFileSync('./test.json')), // JSON.parse(fs.readFileSync('./test1.json')), // JSON.parse(fs.readFileSync('./test2.json')), // JSON.parse(fs.readFileSync('./test3.json')), // JSON.parse(fs.readFileSync('./test4.json')), ].reduce((allHits, currResult) => { return allHits.concat(currResult.hits.hits); }, []); console.log(`Total amount of cases: ${cases.length}`); // Groups cases but const casesByNamespace = cases.reduce((acc, theCase) => { const id = theCase._id; const space = theCase.fields.namespaces[0]; const incrementalId = theCase.fields['cases.incremental_id'][0]; const title = theCase.fields['cases.title'][0]; const toStore = { id, incrementalId, title }; if (!acc[space]) { acc[space] = new Map(); } // check for duplicates const spaceMap = acc[space]; if (!spaceMap.has(incrementalId)) { acc[space].set(incrementalId, toStore); } else { const storedCase = spaceMap.get(incrementalId); console.error(` ${storedCase.title} and ${toStore.title} have the same incremental id (${incrementalId}) `); } return acc; }, {}); // find gaps in spaces Object.keys(casesByNamespace).forEach((space) => { const spaceHits = casesByNamespace[space]; const gaps = []; spaceHits.forEach(({ incrementalId }, _, map) => { const idBefore = incrementalId - 1; if (incrementalId > 1 && !map.has(idBefore)) { gaps.push(idBefore); } }); console.log(`space:${space} has ${spaceHits.size} cases and ${gaps.length} skipped ids`); gaps.forEach((gap) => console.log(`id #${gap} is not assigned`)); }); ``` </details> - Enable the logger in your `kibana.dev.yml` (optional but helpful) ``` logging.loggers: - name: plugins.cases.incremental_id_task level: debug ``` - Change some of the timings in `x-pack/platform/plugins/shared/cases/server/tasks/incremental_id/incremental_id_task_manager.ts` - Set `timeout: '1m'` - Set `CASES_INCREMENTAL_ID_SYNC_INTERVAL_DEFAULT_MINUTES = 1` - Remove ```runAt: new Date( new Date().getTime() + CASES_INCREMENTAL_ID_SYNC_INTERVAL_DEFAULT_MINUTES * 60 * 1000 ),``` - you can also set the timings to something lower in the seconds e.g. `10s` - Generate a bunch of cases with the generator script `x-pack/platform/plugins/shared/cases/scripts/generate_cases.js`: - `node scripts/generate_cases.js -c 1000 -o securitySolution - Enable `cases:incrementalIdDisplay:enabled` in advanced settings - Wait a couple minutes until the incrementer task ran - Test that the ids show up and that the search works - We ran a large-scale test with ~350k cases in a cloud env and can report the following findings: - The 10min timeout for the incremental id task makes sense. The task was usually finished after around 8-9min (processing 1000 cases at a time) which gives it some buffer even. - While processing the first 50k cases, the service skipped 8 ids and no duplicates have been assigned. This means it skipped `0.016%` ids which is great. - It's unclear when these skips happened though and we investigated the first 50k cases for duplicate ids, just in case, and found no duplicates. - At no point did any of the error logs trigger, meaning the task is running smoothly. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: Michael Olorunnisola <michael.olorunnisola@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
10 tasks
michaelolo24
added a commit
that referenced
this pull request
Sep 23, 2025
### TLDR We need to utilize `text` in place of `keyword` to not break phrase prefix queries, so introducing the field `incremental_id.text` ### Background A new issue appeared where the current search field `title` and `description` are of mapping type text. This typically shouldn't be a problem, but unfortunately when attempting to run a prefixed query with the new `incremental_id.keyword` field in [this pr](#230278) we received the following error: To test, run [this test](https://github.com/elastic/kibana/blob/8ffa408f560b59cde8f045ff440090a05bf7bdbf/x-pack/platform/test/functional_with_es_ssl/apps/cases/group2/list_view.ts#L397) on [this PR](#230278) (after changing all instances in the UI from `incremental_id.text` back to `incremental_id.keyword`: ``` search_phase_execution_exception Root causes: query_shard_exception: failed to create query: Can only use phrase prefix queries on text fields - not on [cases.incremental_id.keyword] which is of type [keyword] ``` After some digging as to exactly why this was happening, the only documentation regarding this we could find was in this comment: #129424 (comment) . We should most likely add this information to the documentation here as well: https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-prefix-query ? For historical reference of mapping changes to the `incremental_id` see: #234054 The primary concern for this change is initializing a mapping change in the serverless environment, but this has been tested in this pr and tested against [this PR](#230278) that also enables the functionality that will populate this field. CI has also been run to make sure all tests pass with the new mapping to avoid any additional surprises [here](#230278 (comment)) For testing: see #230278 (comment) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Sep 24, 2025
## Summary This PR updates the mapping for the `incremental_id` field to be a [mulit-field](https://www.elastic.co/docs/manage-data/data-store/mapping/update-mappings-examples#add-multi-fields-to-an-existing-field). In the initial implementation with just `unsigned_long`, running search when combining this field with non-numeric fields such as `keyword` didn't function due to data type mismatch errors [described here](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/number#_which_type_should_i_use). The main concern for this change is initializing a mapping change in the serverless environment, but this was tested against [this PR](elastic#230278) that also enables the functionality that will populate this field. For testing: see elastic#230278 (comment) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Sep 24, 2025
### TLDR We need to utilize `text` in place of `keyword` to not break phrase prefix queries, so introducing the field `incremental_id.text` ### Background A new issue appeared where the current search field `title` and `description` are of mapping type text. This typically shouldn't be a problem, but unfortunately when attempting to run a prefixed query with the new `incremental_id.keyword` field in [this pr](elastic#230278) we received the following error: To test, run [this test](https://github.com/elastic/kibana/blob/8ffa408f560b59cde8f045ff440090a05bf7bdbf/x-pack/platform/test/functional_with_es_ssl/apps/cases/group2/list_view.ts#L397) on [this PR](elastic#230278) (after changing all instances in the UI from `incremental_id.text` back to `incremental_id.keyword`: ``` search_phase_execution_exception Root causes: query_shard_exception: failed to create query: Can only use phrase prefix queries on text fields - not on [cases.incremental_id.keyword] which is of type [keyword] ``` After some digging as to exactly why this was happening, the only documentation regarding this we could find was in this comment: elastic#129424 (comment) . We should most likely add this information to the documentation here as well: https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-prefix-query ? For historical reference of mapping changes to the `incremental_id` see: elastic#234054 The primary concern for this change is initializing a mapping change in the serverless environment, but this has been tested in this pr and tested against [this PR](elastic#230278) that also enables the functionality that will populate this field. CI has also been run to make sure all tests pass with the new mapping to avoid any additional surprises [here](elastic#230278 (comment)) For testing: see elastic#230278 (comment) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
kc13greiner
approved these changes
Sep 25, 2025
Contributor
kc13greiner
left a comment
There was a problem hiding this comment.
new boolean rendering value LGTM!
jloleysens
approved these changes
Sep 25, 2025
Contributor
There was a problem hiding this comment.
Use of new mapping incremental_id.text (#235290) LGTM
Bamieh
approved these changes
Sep 29, 2025
Contributor
Bamieh
left a comment
There was a problem hiding this comment.
revert and Core changes LGTM
Contributor
💔 Build Failed
Failed CI StepsMetrics [docs]Module Count
Async chunks
Count of Enzyme imports
Page load bundle
Unknown metric groupsESLint disabled line counts
Total ESLint disabled count
History
|
janmonschke
approved these changes
Sep 29, 2025
niros1
pushed a commit
that referenced
this pull request
Sep 30, 2025
## Summary This PR updates the mapping for the `incremental_id` field to be a [mulit-field](https://www.elastic.co/docs/manage-data/data-store/mapping/update-mappings-examples#add-multi-fields-to-an-existing-field). In the initial implementation with just `unsigned_long`, running search when combining this field with non-numeric fields such as `keyword` didn't function due to data type mismatch errors [described here](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/number#_which_type_should_i_use). The main concern for this change is initializing a mapping change in the serverless environment, but this was tested against [this PR](#230278) that also enables the functionality that will populate this field. For testing: see #230278 (comment) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
niros1
pushed a commit
that referenced
this pull request
Sep 30, 2025
### TLDR We need to utilize `text` in place of `keyword` to not break phrase prefix queries, so introducing the field `incremental_id.text` ### Background A new issue appeared where the current search field `title` and `description` are of mapping type text. This typically shouldn't be a problem, but unfortunately when attempting to run a prefixed query with the new `incremental_id.keyword` field in [this pr](#230278) we received the following error: To test, run [this test](https://github.com/elastic/kibana/blob/8ffa408f560b59cde8f045ff440090a05bf7bdbf/x-pack/platform/test/functional_with_es_ssl/apps/cases/group2/list_view.ts#L397) on [this PR](#230278) (after changing all instances in the UI from `incremental_id.text` back to `incremental_id.keyword`: ``` search_phase_execution_exception Root causes: query_shard_exception: failed to create query: Can only use phrase prefix queries on text fields - not on [cases.incremental_id.keyword] which is of type [keyword] ``` After some digging as to exactly why this was happening, the only documentation regarding this we could find was in this comment: #129424 (comment) . We should most likely add this information to the documentation here as well: https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-prefix-query ? For historical reference of mapping changes to the `incremental_id` see: #234054 The primary concern for this change is initializing a mapping change in the serverless environment, but this has been tested in this pr and tested against [this PR](#230278) that also enables the functionality that will populate this field. CI has also been run to make sure all tests pass with the new mapping to avoid any additional surprises [here](#230278 (comment)) For testing: see #230278 (comment) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
niros1
pushed a commit
that referenced
this pull request
Sep 30, 2025
## Summary This is a reversion of the revert of this original #228002 of incremental id work. This original revert was done, due to the search functionality in cases being broken when searching on the new `incremental_id` field. It being of type `unsigned_long` meant that ES would not return when it was included alongside text values due to data type mismatch errors [described here](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/number#_which_type_should_i_use).. Rather than creating a new field, and making the existing `incremental_id` field dead weight, we instead [update it to a multi-field](https://www.elastic.co/docs/manage-data/data-store/mapping/update-mappings-examples#add-multi-fields-to-an-existing-field), so we can search on this value as text alongside other cases values. The update of the mapping was done here: #234054 #### Functional Changes From Reversion 1. Use of `incremental_id.text` in search functionality in place of `incremental_id` 2. There is no longer an advanced setting to hide the feature in the UI as we feel comfortable releasing it as is, and the field is currently only metadata and doesn't introduce any functional changes outside of searching. ## Testing: Feature Flag: `xpack.cases.incrementalId.enabled: true`. Currently disabled and will be enabled after additional testing during FF When running a clean branch, the feature will work correctly. We've verified locally that the mapping update works successfully via the methods described below, but feel free to test as well to confirm our results 👍🏾 ### Serverless 1. Create a security serverless environment via `qaf`. 4. Create cases in the serverless environment 5. Update images to the build image from this pr via the `--kb-docker-images` flag. 6. Wait for the task to run, and verify the numeric ids are applied to the cases 7. Search for cases by then numeric id via `#{incremental_id value}` and also search for `description + {incremental_id value}` and make sure the query successfully returns ### On-Prem 1. Run main locally 2. Create cases in the local environment 3. Switch branches from main to this branch 4. Wait for the task to run, and verify the numeric ids are applied to the cases 5. Search for cases by then numeric id via `#{id number}` and also search for `description + id number` and make sure the query successfully returns ## OUTDATED The below details were provided for manually updating the mapping via dev tools for testing/validation purposes This PR updates the mapping directly. Here are steps to update the mapping manually in a local instance: 0. Remove changes in `packages/kbn-check-mappings-update-cli/current_mappings.json` 1. Create some tests (there could be error toasts because `incremental_id.text` is not yet created 8. Create an `admin` user and assign `system_indices_superuser` and `superuser` role 9. Go to Dev tools, get the current mapping `GET /.kibana_alerting_cases_9.2.0_001/_mapping` 10. Add `text` as multi-field for `incremental_id` 11. Run `POST /.kibana_alerting_cases_9.2.0_001/_update_by_query` to update existing document 12. Go to Cases and observe the case id can be searched individually or in combination of other texts <details> <summary>mapping update query</summary> ``` PUT /.kibana_alerting_cases_9.2.0_001/_mapping { "properties": { "action": { "dynamic": "false", "properties": { "actionTypeId": { "type": "keyword" }, "name": { "type": "text", "fields": { "keyword": { "type": "keyword" } } } } }, "action_task_params": { "dynamic": "false", "properties": { "apiKeyId": { "type": "keyword" } } }, "ad_hoc_run_params": { "dynamic": "false", "properties": { "apiKeyId": { "type": "keyword" }, "createdAt": { "type": "date" }, "end": { "type": "date" }, "rule": { "properties": { "alertTypeId": { "type": "keyword" }, "consumer": { "type": "keyword" } } }, "start": { "type": "date" } } }, "alert": { "dynamic": "false", "properties": { "actions": { "type": "nested", "dynamic": "false", "properties": { "actionRef": { "type": "keyword" }, "actionTypeId": { "type": "keyword" }, "group": { "type": "keyword" } } }, "alertTypeId": { "type": "keyword" }, "artifacts": { "properties": { "investigation_guide": { "properties": { "blob": { "type": "text" } } } } }, "consumer": { "type": "keyword" }, "createdAt": { "type": "date" }, "createdBy": { "type": "keyword" }, "enabled": { "type": "boolean" }, "executionStatus": { "properties": { "error": { "properties": { "message": { "type": "keyword" }, "reason": { "type": "keyword" } } }, "lastDuration": { "type": "long" }, "lastExecutionDate": { "type": "date" }, "numberOfTriggeredActions": { "type": "long" }, "status": { "type": "keyword" }, "warning": { "properties": { "message": { "type": "keyword" }, "reason": { "type": "keyword" } } } } }, "lastRun": { "properties": { "alertsCount": { "properties": { "active": { "type": "float" }, "ignored": { "type": "float" }, "new": { "type": "float" }, "recovered": { "type": "float" } } }, "outcome": { "type": "keyword" }, "outcomeOrder": { "type": "float" } } }, "legacyId": { "type": "keyword" }, "mapped_params": { "properties": { "risk_score": { "type": "float" }, "severity": { "type": "keyword" } } }, "monitoring": { "properties": { "run": { "properties": { "calculated_metrics": { "properties": { "p50": { "type": "long" }, "p95": { "type": "long" }, "p99": { "type": "long" }, "success_ratio": { "type": "float" } } }, "last_run": { "properties": { "metrics": { "properties": { "duration": { "type": "long" }, "gap_duration_s": { "type": "float" }, "total_alerts_created": { "type": "float" }, "total_alerts_detected": { "type": "float" }, "total_indexing_duration_ms": { "type": "long" }, "total_search_duration_ms": { "type": "long" } } }, "timestamp": { "type": "date" } } } } } } }, "muteAll": { "type": "boolean" }, "mutedInstanceIds": { "type": "keyword" }, "name": { "type": "text", "fields": { "keyword": { "type": "keyword", "normalizer": "lowercase" } } }, "notifyWhen": { "type": "keyword" }, "params": { "type": "flattened", "ignore_above": 4096 }, "revision": { "type": "long" }, "running": { "type": "boolean" }, "schedule": { "properties": { "interval": { "type": "keyword" } } }, "scheduledTaskId": { "type": "keyword" }, "snoozeSchedule": { "type": "nested", "properties": { "duration": { "type": "long" }, "id": { "type": "keyword" }, "skipRecurrences": { "type": "date", "format": "strict_date_time" } } }, "tags": { "type": "keyword" }, "throttle": { "type": "keyword" }, "updatedAt": { "type": "date" }, "updatedBy": { "type": "keyword" } } }, "api_key_pending_invalidation": { "properties": { "apiKeyId": { "type": "keyword" }, "createdAt": { "type": "date" } } }, "cases": { "dynamic": "false", "properties": { "assignees": { "properties": { "uid": { "type": "keyword" } } }, "category": { "type": "keyword" }, "closed_at": { "type": "date" }, "closed_by": { "properties": { "email": { "type": "keyword" }, "full_name": { "type": "keyword" }, "profile_uid": { "type": "keyword" }, "username": { "type": "keyword" } } }, "connector": { "properties": { "fields": { "properties": { "key": { "type": "text" }, "value": { "type": "text" } } }, "name": { "type": "text" }, "type": { "type": "keyword" } } }, "created_at": { "type": "date" }, "created_by": { "properties": { "email": { "type": "keyword" }, "full_name": { "type": "keyword" }, "profile_uid": { "type": "keyword" }, "username": { "type": "keyword" } } }, "customFields": { "type": "nested", "properties": { "key": { "type": "keyword" }, "type": { "type": "keyword" }, "value": { "type": "keyword", "fields": { "boolean": { "type": "boolean", "ignore_malformed": true }, "date": { "type": "date", "ignore_malformed": true }, "ip": { "type": "ip", "ignore_malformed": true }, "number": { "type": "long", "ignore_malformed": true }, "string": { "type": "text" } } } } }, "description": { "type": "text" }, "duration": { "type": "unsigned_long" }, "external_service": { "properties": { "connector_name": { "type": "keyword" }, "external_id": { "type": "keyword" }, "external_title": { "type": "text" }, "external_url": { "type": "text" }, "pushed_at": { "type": "date" }, "pushed_by": { "properties": { "email": { "type": "keyword" }, "full_name": { "type": "keyword" }, "profile_uid": { "type": "keyword" }, "username": { "type": "keyword" } } } } }, "incremental_id": { "type": "unsigned_long", "fields": { "raw": { "type": "keyword" } } }, "observables": { "type": "nested", "properties": { "typeKey": { "type": "keyword" }, "value": { "type": "keyword" } } }, "owner": { "type": "keyword" }, "settings": { "properties": { "syncAlerts": { "type": "boolean" } } }, "severity": { "type": "short" }, "status": { "type": "short" }, "tags": { "type": "keyword" }, "title": { "type": "text", "fields": { "keyword": { "type": "keyword" } } }, "total_alerts": { "type": "integer" }, "total_comments": { "type": "integer" }, "updated_at": { "type": "date" }, "updated_by": { "properties": { "email": { "type": "keyword" }, "full_name": { "type": "keyword" }, "profile_uid": { "type": "keyword" }, "username": { "type": "keyword" } } } } }, "cases-comments": { "dynamic": "false", "properties": { "actions": { "properties": { "type": { "type": "keyword" } } }, "alertId": { "type": "keyword" }, "comment": { "type": "text" }, "created_at": { "type": "date" }, "created_by": { "properties": { "username": { "type": "keyword" } } }, "externalReferenceAttachmentTypeId": { "type": "keyword" }, "owner": { "type": "keyword" }, "persistableStateAttachmentTypeId": { "type": "keyword" }, "pushed_at": { "type": "date" }, "type": { "type": "keyword" }, "updated_at": { "type": "date" } } }, "cases-configure": { "dynamic": "false", "properties": { "closure_type": { "type": "keyword" }, "created_at": { "type": "date" }, "owner": { "type": "keyword" } } }, "cases-connector-mappings": { "dynamic": "false", "properties": { "owner": { "type": "keyword" } } }, "cases-incrementing-id": { "dynamic": "false", "properties": { "@timestamp": { "type": "date" }, "last_id": { "type": "keyword" }, "updated_at": { "type": "date" } } }, "cases-rules": { "dynamic": "false", "properties": { "counter": { "type": "unsigned_long" }, "createdAt": { "type": "date" }, "rules": { "properties": { "id": { "type": "keyword" } } }, "updatedAt": { "type": "date" } } }, "cases-telemetry": { "type": "object", "dynamic": "false" }, "cases-user-actions": { "dynamic": "false", "properties": { "action": { "type": "keyword" }, "created_at": { "type": "date" }, "created_by": { "properties": { "username": { "type": "keyword" } } }, "owner": { "type": "keyword" }, "payload": { "dynamic": "false", "properties": { "assignees": { "properties": { "uid": { "type": "keyword" } } }, "comment": { "properties": { "externalReferenceAttachmentTypeId": { "type": "keyword" }, "persistableStateAttachmentTypeId": { "type": "keyword" }, "type": { "type": "keyword" } } }, "connector": { "properties": { "type": { "type": "keyword" } } } } }, "type": { "type": "keyword" } } }, "connector_token": { "dynamic": "false", "properties": { "connectorId": { "type": "keyword" }, "tokenType": { "type": "keyword" } } }, "coreMigrationVersion": { "type": "keyword" }, "created_at": { "type": "date" }, "created_by": { "type": "keyword" }, "maintenance-window": { "dynamic": "false", "properties": { "enabled": { "type": "boolean" }, "events": { "type": "date_range", "format": "epoch_millis||strict_date_optional_time" }, "expirationDate": { "type": "date" }, "title": { "type": "text", "fields": { "keyword": { "type": "keyword" } } }, "updatedAt": { "type": "date" } } }, "managed": { "type": "boolean" }, "namespace": { "type": "keyword" }, "namespaces": { "type": "keyword" }, "originId": { "type": "keyword" }, "references": { "type": "nested", "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" }, "type": { "type": "keyword" } } }, "rules-settings": { "dynamic": "false", "properties": { "flapping": { "type": "object" } } }, "scheduled_report": { "dynamic": "false", "properties": { "createdBy": { "type": "keyword" } } }, "type": { "type": "keyword" }, "typeMigrationVersion": { "type": "version" }, "updated_at": { "type": "date" }, "updated_by": { "type": "keyword" } } } ``` </details> ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: Jan Monschke <jan.monschke@elastic.co> Co-authored-by: Michael Olorunnisola <michael.olorunnisola@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
VladimirFilonov
pushed a commit
to VladimirFilonov/kibana
that referenced
this pull request
Sep 30, 2025
## Summary This is a reversion of the revert of this original elastic#228002 of incremental id work. This original revert was done, due to the search functionality in cases being broken when searching on the new `incremental_id` field. It being of type `unsigned_long` meant that ES would not return when it was included alongside text values due to data type mismatch errors [described here](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/number#_which_type_should_i_use).. Rather than creating a new field, and making the existing `incremental_id` field dead weight, we instead [update it to a multi-field](https://www.elastic.co/docs/manage-data/data-store/mapping/update-mappings-examples#add-multi-fields-to-an-existing-field), so we can search on this value as text alongside other cases values. The update of the mapping was done here: elastic#234054 #### Functional Changes From Reversion 1. Use of `incremental_id.text` in search functionality in place of `incremental_id` 2. There is no longer an advanced setting to hide the feature in the UI as we feel comfortable releasing it as is, and the field is currently only metadata and doesn't introduce any functional changes outside of searching. ## Testing: Feature Flag: `xpack.cases.incrementalId.enabled: true`. Currently disabled and will be enabled after additional testing during FF When running a clean branch, the feature will work correctly. We've verified locally that the mapping update works successfully via the methods described below, but feel free to test as well to confirm our results 👍🏾 ### Serverless 1. Create a security serverless environment via `qaf`. 4. Create cases in the serverless environment 5. Update images to the build image from this pr via the `--kb-docker-images` flag. 6. Wait for the task to run, and verify the numeric ids are applied to the cases 7. Search for cases by then numeric id via `#{incremental_id value}` and also search for `description + {incremental_id value}` and make sure the query successfully returns ### On-Prem 1. Run main locally 2. Create cases in the local environment 3. Switch branches from main to this branch 4. Wait for the task to run, and verify the numeric ids are applied to the cases 5. Search for cases by then numeric id via `#{id number}` and also search for `description + id number` and make sure the query successfully returns ## OUTDATED The below details were provided for manually updating the mapping via dev tools for testing/validation purposes This PR updates the mapping directly. Here are steps to update the mapping manually in a local instance: 0. Remove changes in `packages/kbn-check-mappings-update-cli/current_mappings.json` 1. Create some tests (there could be error toasts because `incremental_id.text` is not yet created 8. Create an `admin` user and assign `system_indices_superuser` and `superuser` role 9. Go to Dev tools, get the current mapping `GET /.kibana_alerting_cases_9.2.0_001/_mapping` 10. Add `text` as multi-field for `incremental_id` 11. Run `POST /.kibana_alerting_cases_9.2.0_001/_update_by_query` to update existing document 12. Go to Cases and observe the case id can be searched individually or in combination of other texts <details> <summary>mapping update query</summary> ``` PUT /.kibana_alerting_cases_9.2.0_001/_mapping { "properties": { "action": { "dynamic": "false", "properties": { "actionTypeId": { "type": "keyword" }, "name": { "type": "text", "fields": { "keyword": { "type": "keyword" } } } } }, "action_task_params": { "dynamic": "false", "properties": { "apiKeyId": { "type": "keyword" } } }, "ad_hoc_run_params": { "dynamic": "false", "properties": { "apiKeyId": { "type": "keyword" }, "createdAt": { "type": "date" }, "end": { "type": "date" }, "rule": { "properties": { "alertTypeId": { "type": "keyword" }, "consumer": { "type": "keyword" } } }, "start": { "type": "date" } } }, "alert": { "dynamic": "false", "properties": { "actions": { "type": "nested", "dynamic": "false", "properties": { "actionRef": { "type": "keyword" }, "actionTypeId": { "type": "keyword" }, "group": { "type": "keyword" } } }, "alertTypeId": { "type": "keyword" }, "artifacts": { "properties": { "investigation_guide": { "properties": { "blob": { "type": "text" } } } } }, "consumer": { "type": "keyword" }, "createdAt": { "type": "date" }, "createdBy": { "type": "keyword" }, "enabled": { "type": "boolean" }, "executionStatus": { "properties": { "error": { "properties": { "message": { "type": "keyword" }, "reason": { "type": "keyword" } } }, "lastDuration": { "type": "long" }, "lastExecutionDate": { "type": "date" }, "numberOfTriggeredActions": { "type": "long" }, "status": { "type": "keyword" }, "warning": { "properties": { "message": { "type": "keyword" }, "reason": { "type": "keyword" } } } } }, "lastRun": { "properties": { "alertsCount": { "properties": { "active": { "type": "float" }, "ignored": { "type": "float" }, "new": { "type": "float" }, "recovered": { "type": "float" } } }, "outcome": { "type": "keyword" }, "outcomeOrder": { "type": "float" } } }, "legacyId": { "type": "keyword" }, "mapped_params": { "properties": { "risk_score": { "type": "float" }, "severity": { "type": "keyword" } } }, "monitoring": { "properties": { "run": { "properties": { "calculated_metrics": { "properties": { "p50": { "type": "long" }, "p95": { "type": "long" }, "p99": { "type": "long" }, "success_ratio": { "type": "float" } } }, "last_run": { "properties": { "metrics": { "properties": { "duration": { "type": "long" }, "gap_duration_s": { "type": "float" }, "total_alerts_created": { "type": "float" }, "total_alerts_detected": { "type": "float" }, "total_indexing_duration_ms": { "type": "long" }, "total_search_duration_ms": { "type": "long" } } }, "timestamp": { "type": "date" } } } } } } }, "muteAll": { "type": "boolean" }, "mutedInstanceIds": { "type": "keyword" }, "name": { "type": "text", "fields": { "keyword": { "type": "keyword", "normalizer": "lowercase" } } }, "notifyWhen": { "type": "keyword" }, "params": { "type": "flattened", "ignore_above": 4096 }, "revision": { "type": "long" }, "running": { "type": "boolean" }, "schedule": { "properties": { "interval": { "type": "keyword" } } }, "scheduledTaskId": { "type": "keyword" }, "snoozeSchedule": { "type": "nested", "properties": { "duration": { "type": "long" }, "id": { "type": "keyword" }, "skipRecurrences": { "type": "date", "format": "strict_date_time" } } }, "tags": { "type": "keyword" }, "throttle": { "type": "keyword" }, "updatedAt": { "type": "date" }, "updatedBy": { "type": "keyword" } } }, "api_key_pending_invalidation": { "properties": { "apiKeyId": { "type": "keyword" }, "createdAt": { "type": "date" } } }, "cases": { "dynamic": "false", "properties": { "assignees": { "properties": { "uid": { "type": "keyword" } } }, "category": { "type": "keyword" }, "closed_at": { "type": "date" }, "closed_by": { "properties": { "email": { "type": "keyword" }, "full_name": { "type": "keyword" }, "profile_uid": { "type": "keyword" }, "username": { "type": "keyword" } } }, "connector": { "properties": { "fields": { "properties": { "key": { "type": "text" }, "value": { "type": "text" } } }, "name": { "type": "text" }, "type": { "type": "keyword" } } }, "created_at": { "type": "date" }, "created_by": { "properties": { "email": { "type": "keyword" }, "full_name": { "type": "keyword" }, "profile_uid": { "type": "keyword" }, "username": { "type": "keyword" } } }, "customFields": { "type": "nested", "properties": { "key": { "type": "keyword" }, "type": { "type": "keyword" }, "value": { "type": "keyword", "fields": { "boolean": { "type": "boolean", "ignore_malformed": true }, "date": { "type": "date", "ignore_malformed": true }, "ip": { "type": "ip", "ignore_malformed": true }, "number": { "type": "long", "ignore_malformed": true }, "string": { "type": "text" } } } } }, "description": { "type": "text" }, "duration": { "type": "unsigned_long" }, "external_service": { "properties": { "connector_name": { "type": "keyword" }, "external_id": { "type": "keyword" }, "external_title": { "type": "text" }, "external_url": { "type": "text" }, "pushed_at": { "type": "date" }, "pushed_by": { "properties": { "email": { "type": "keyword" }, "full_name": { "type": "keyword" }, "profile_uid": { "type": "keyword" }, "username": { "type": "keyword" } } } } }, "incremental_id": { "type": "unsigned_long", "fields": { "raw": { "type": "keyword" } } }, "observables": { "type": "nested", "properties": { "typeKey": { "type": "keyword" }, "value": { "type": "keyword" } } }, "owner": { "type": "keyword" }, "settings": { "properties": { "syncAlerts": { "type": "boolean" } } }, "severity": { "type": "short" }, "status": { "type": "short" }, "tags": { "type": "keyword" }, "title": { "type": "text", "fields": { "keyword": { "type": "keyword" } } }, "total_alerts": { "type": "integer" }, "total_comments": { "type": "integer" }, "updated_at": { "type": "date" }, "updated_by": { "properties": { "email": { "type": "keyword" }, "full_name": { "type": "keyword" }, "profile_uid": { "type": "keyword" }, "username": { "type": "keyword" } } } } }, "cases-comments": { "dynamic": "false", "properties": { "actions": { "properties": { "type": { "type": "keyword" } } }, "alertId": { "type": "keyword" }, "comment": { "type": "text" }, "created_at": { "type": "date" }, "created_by": { "properties": { "username": { "type": "keyword" } } }, "externalReferenceAttachmentTypeId": { "type": "keyword" }, "owner": { "type": "keyword" }, "persistableStateAttachmentTypeId": { "type": "keyword" }, "pushed_at": { "type": "date" }, "type": { "type": "keyword" }, "updated_at": { "type": "date" } } }, "cases-configure": { "dynamic": "false", "properties": { "closure_type": { "type": "keyword" }, "created_at": { "type": "date" }, "owner": { "type": "keyword" } } }, "cases-connector-mappings": { "dynamic": "false", "properties": { "owner": { "type": "keyword" } } }, "cases-incrementing-id": { "dynamic": "false", "properties": { "@timestamp": { "type": "date" }, "last_id": { "type": "keyword" }, "updated_at": { "type": "date" } } }, "cases-rules": { "dynamic": "false", "properties": { "counter": { "type": "unsigned_long" }, "createdAt": { "type": "date" }, "rules": { "properties": { "id": { "type": "keyword" } } }, "updatedAt": { "type": "date" } } }, "cases-telemetry": { "type": "object", "dynamic": "false" }, "cases-user-actions": { "dynamic": "false", "properties": { "action": { "type": "keyword" }, "created_at": { "type": "date" }, "created_by": { "properties": { "username": { "type": "keyword" } } }, "owner": { "type": "keyword" }, "payload": { "dynamic": "false", "properties": { "assignees": { "properties": { "uid": { "type": "keyword" } } }, "comment": { "properties": { "externalReferenceAttachmentTypeId": { "type": "keyword" }, "persistableStateAttachmentTypeId": { "type": "keyword" }, "type": { "type": "keyword" } } }, "connector": { "properties": { "type": { "type": "keyword" } } } } }, "type": { "type": "keyword" } } }, "connector_token": { "dynamic": "false", "properties": { "connectorId": { "type": "keyword" }, "tokenType": { "type": "keyword" } } }, "coreMigrationVersion": { "type": "keyword" }, "created_at": { "type": "date" }, "created_by": { "type": "keyword" }, "maintenance-window": { "dynamic": "false", "properties": { "enabled": { "type": "boolean" }, "events": { "type": "date_range", "format": "epoch_millis||strict_date_optional_time" }, "expirationDate": { "type": "date" }, "title": { "type": "text", "fields": { "keyword": { "type": "keyword" } } }, "updatedAt": { "type": "date" } } }, "managed": { "type": "boolean" }, "namespace": { "type": "keyword" }, "namespaces": { "type": "keyword" }, "originId": { "type": "keyword" }, "references": { "type": "nested", "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" }, "type": { "type": "keyword" } } }, "rules-settings": { "dynamic": "false", "properties": { "flapping": { "type": "object" } } }, "scheduled_report": { "dynamic": "false", "properties": { "createdBy": { "type": "keyword" } } }, "type": { "type": "keyword" }, "typeMigrationVersion": { "type": "version" }, "updated_at": { "type": "date" }, "updated_by": { "type": "keyword" } } } ``` </details> ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: Jan Monschke <jan.monschke@elastic.co> Co-authored-by: Michael Olorunnisola <michael.olorunnisola@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
rylnd
pushed a commit
to rylnd/kibana
that referenced
this pull request
Oct 17, 2025
### TLDR We need to utilize `text` in place of `keyword` to not break phrase prefix queries, so introducing the field `incremental_id.text` ### Background A new issue appeared where the current search field `title` and `description` are of mapping type text. This typically shouldn't be a problem, but unfortunately when attempting to run a prefixed query with the new `incremental_id.keyword` field in [this pr](elastic#230278) we received the following error: To test, run [this test](https://github.com/elastic/kibana/blob/4da9288e4d3f13e24e991f91cc51c3ef9fcf5bf7/x-pack/platform/test/functional_with_es_ssl/apps/cases/group2/list_view.ts#L397) on [this PR](elastic#230278) (after changing all instances in the UI from `incremental_id.text` back to `incremental_id.keyword`: ``` search_phase_execution_exception Root causes: query_shard_exception: failed to create query: Can only use phrase prefix queries on text fields - not on [cases.incremental_id.keyword] which is of type [keyword] ``` After some digging as to exactly why this was happening, the only documentation regarding this we could find was in this comment: elastic#129424 (comment) . We should most likely add this information to the documentation here as well: https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-prefix-query ? For historical reference of mapping changes to the `incremental_id` see: elastic#234054 The primary concern for this change is initializing a mapping change in the serverless environment, but this has been tested in this pr and tested against [this PR](elastic#230278) that also enables the functionality that will populate this field. CI has also been run to make sure all tests pass with the new mapping to avoid any additional surprises [here](elastic#230278 (comment)) For testing: see elastic#230278 (comment) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
rylnd
pushed a commit
to rylnd/kibana
that referenced
this pull request
Oct 17, 2025
## Summary This is a reversion of the revert of this original elastic#228002 of incremental id work. This original revert was done, due to the search functionality in cases being broken when searching on the new `incremental_id` field. It being of type `unsigned_long` meant that ES would not return when it was included alongside text values due to data type mismatch errors [described here](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/number#_which_type_should_i_use).. Rather than creating a new field, and making the existing `incremental_id` field dead weight, we instead [update it to a multi-field](https://www.elastic.co/docs/manage-data/data-store/mapping/update-mappings-examples#add-multi-fields-to-an-existing-field), so we can search on this value as text alongside other cases values. The update of the mapping was done here: elastic#234054 #### Functional Changes From Reversion 1. Use of `incremental_id.text` in search functionality in place of `incremental_id` 2. There is no longer an advanced setting to hide the feature in the UI as we feel comfortable releasing it as is, and the field is currently only metadata and doesn't introduce any functional changes outside of searching. ## Testing: Feature Flag: `xpack.cases.incrementalId.enabled: true`. Currently disabled and will be enabled after additional testing during FF When running a clean branch, the feature will work correctly. We've verified locally that the mapping update works successfully via the methods described below, but feel free to test as well to confirm our results 👍🏾 ### Serverless 1. Create a security serverless environment via `qaf`. 4. Create cases in the serverless environment 5. Update images to the build image from this pr via the `--kb-docker-images` flag. 6. Wait for the task to run, and verify the numeric ids are applied to the cases 7. Search for cases by then numeric id via `#{incremental_id value}` and also search for `description + {incremental_id value}` and make sure the query successfully returns ### On-Prem 1. Run main locally 2. Create cases in the local environment 3. Switch branches from main to this branch 4. Wait for the task to run, and verify the numeric ids are applied to the cases 5. Search for cases by then numeric id via `#{id number}` and also search for `description + id number` and make sure the query successfully returns ## OUTDATED The below details were provided for manually updating the mapping via dev tools for testing/validation purposes This PR updates the mapping directly. Here are steps to update the mapping manually in a local instance: 0. Remove changes in `packages/kbn-check-mappings-update-cli/current_mappings.json` 1. Create some tests (there could be error toasts because `incremental_id.text` is not yet created 8. Create an `admin` user and assign `system_indices_superuser` and `superuser` role 9. Go to Dev tools, get the current mapping `GET /.kibana_alerting_cases_9.2.0_001/_mapping` 10. Add `text` as multi-field for `incremental_id` 11. Run `POST /.kibana_alerting_cases_9.2.0_001/_update_by_query` to update existing document 12. Go to Cases and observe the case id can be searched individually or in combination of other texts <details> <summary>mapping update query</summary> ``` PUT /.kibana_alerting_cases_9.2.0_001/_mapping { "properties": { "action": { "dynamic": "false", "properties": { "actionTypeId": { "type": "keyword" }, "name": { "type": "text", "fields": { "keyword": { "type": "keyword" } } } } }, "action_task_params": { "dynamic": "false", "properties": { "apiKeyId": { "type": "keyword" } } }, "ad_hoc_run_params": { "dynamic": "false", "properties": { "apiKeyId": { "type": "keyword" }, "createdAt": { "type": "date" }, "end": { "type": "date" }, "rule": { "properties": { "alertTypeId": { "type": "keyword" }, "consumer": { "type": "keyword" } } }, "start": { "type": "date" } } }, "alert": { "dynamic": "false", "properties": { "actions": { "type": "nested", "dynamic": "false", "properties": { "actionRef": { "type": "keyword" }, "actionTypeId": { "type": "keyword" }, "group": { "type": "keyword" } } }, "alertTypeId": { "type": "keyword" }, "artifacts": { "properties": { "investigation_guide": { "properties": { "blob": { "type": "text" } } } } }, "consumer": { "type": "keyword" }, "createdAt": { "type": "date" }, "createdBy": { "type": "keyword" }, "enabled": { "type": "boolean" }, "executionStatus": { "properties": { "error": { "properties": { "message": { "type": "keyword" }, "reason": { "type": "keyword" } } }, "lastDuration": { "type": "long" }, "lastExecutionDate": { "type": "date" }, "numberOfTriggeredActions": { "type": "long" }, "status": { "type": "keyword" }, "warning": { "properties": { "message": { "type": "keyword" }, "reason": { "type": "keyword" } } } } }, "lastRun": { "properties": { "alertsCount": { "properties": { "active": { "type": "float" }, "ignored": { "type": "float" }, "new": { "type": "float" }, "recovered": { "type": "float" } } }, "outcome": { "type": "keyword" }, "outcomeOrder": { "type": "float" } } }, "legacyId": { "type": "keyword" }, "mapped_params": { "properties": { "risk_score": { "type": "float" }, "severity": { "type": "keyword" } } }, "monitoring": { "properties": { "run": { "properties": { "calculated_metrics": { "properties": { "p50": { "type": "long" }, "p95": { "type": "long" }, "p99": { "type": "long" }, "success_ratio": { "type": "float" } } }, "last_run": { "properties": { "metrics": { "properties": { "duration": { "type": "long" }, "gap_duration_s": { "type": "float" }, "total_alerts_created": { "type": "float" }, "total_alerts_detected": { "type": "float" }, "total_indexing_duration_ms": { "type": "long" }, "total_search_duration_ms": { "type": "long" } } }, "timestamp": { "type": "date" } } } } } } }, "muteAll": { "type": "boolean" }, "mutedInstanceIds": { "type": "keyword" }, "name": { "type": "text", "fields": { "keyword": { "type": "keyword", "normalizer": "lowercase" } } }, "notifyWhen": { "type": "keyword" }, "params": { "type": "flattened", "ignore_above": 4096 }, "revision": { "type": "long" }, "running": { "type": "boolean" }, "schedule": { "properties": { "interval": { "type": "keyword" } } }, "scheduledTaskId": { "type": "keyword" }, "snoozeSchedule": { "type": "nested", "properties": { "duration": { "type": "long" }, "id": { "type": "keyword" }, "skipRecurrences": { "type": "date", "format": "strict_date_time" } } }, "tags": { "type": "keyword" }, "throttle": { "type": "keyword" }, "updatedAt": { "type": "date" }, "updatedBy": { "type": "keyword" } } }, "api_key_pending_invalidation": { "properties": { "apiKeyId": { "type": "keyword" }, "createdAt": { "type": "date" } } }, "cases": { "dynamic": "false", "properties": { "assignees": { "properties": { "uid": { "type": "keyword" } } }, "category": { "type": "keyword" }, "closed_at": { "type": "date" }, "closed_by": { "properties": { "email": { "type": "keyword" }, "full_name": { "type": "keyword" }, "profile_uid": { "type": "keyword" }, "username": { "type": "keyword" } } }, "connector": { "properties": { "fields": { "properties": { "key": { "type": "text" }, "value": { "type": "text" } } }, "name": { "type": "text" }, "type": { "type": "keyword" } } }, "created_at": { "type": "date" }, "created_by": { "properties": { "email": { "type": "keyword" }, "full_name": { "type": "keyword" }, "profile_uid": { "type": "keyword" }, "username": { "type": "keyword" } } }, "customFields": { "type": "nested", "properties": { "key": { "type": "keyword" }, "type": { "type": "keyword" }, "value": { "type": "keyword", "fields": { "boolean": { "type": "boolean", "ignore_malformed": true }, "date": { "type": "date", "ignore_malformed": true }, "ip": { "type": "ip", "ignore_malformed": true }, "number": { "type": "long", "ignore_malformed": true }, "string": { "type": "text" } } } } }, "description": { "type": "text" }, "duration": { "type": "unsigned_long" }, "external_service": { "properties": { "connector_name": { "type": "keyword" }, "external_id": { "type": "keyword" }, "external_title": { "type": "text" }, "external_url": { "type": "text" }, "pushed_at": { "type": "date" }, "pushed_by": { "properties": { "email": { "type": "keyword" }, "full_name": { "type": "keyword" }, "profile_uid": { "type": "keyword" }, "username": { "type": "keyword" } } } } }, "incremental_id": { "type": "unsigned_long", "fields": { "raw": { "type": "keyword" } } }, "observables": { "type": "nested", "properties": { "typeKey": { "type": "keyword" }, "value": { "type": "keyword" } } }, "owner": { "type": "keyword" }, "settings": { "properties": { "syncAlerts": { "type": "boolean" } } }, "severity": { "type": "short" }, "status": { "type": "short" }, "tags": { "type": "keyword" }, "title": { "type": "text", "fields": { "keyword": { "type": "keyword" } } }, "total_alerts": { "type": "integer" }, "total_comments": { "type": "integer" }, "updated_at": { "type": "date" }, "updated_by": { "properties": { "email": { "type": "keyword" }, "full_name": { "type": "keyword" }, "profile_uid": { "type": "keyword" }, "username": { "type": "keyword" } } } } }, "cases-comments": { "dynamic": "false", "properties": { "actions": { "properties": { "type": { "type": "keyword" } } }, "alertId": { "type": "keyword" }, "comment": { "type": "text" }, "created_at": { "type": "date" }, "created_by": { "properties": { "username": { "type": "keyword" } } }, "externalReferenceAttachmentTypeId": { "type": "keyword" }, "owner": { "type": "keyword" }, "persistableStateAttachmentTypeId": { "type": "keyword" }, "pushed_at": { "type": "date" }, "type": { "type": "keyword" }, "updated_at": { "type": "date" } } }, "cases-configure": { "dynamic": "false", "properties": { "closure_type": { "type": "keyword" }, "created_at": { "type": "date" }, "owner": { "type": "keyword" } } }, "cases-connector-mappings": { "dynamic": "false", "properties": { "owner": { "type": "keyword" } } }, "cases-incrementing-id": { "dynamic": "false", "properties": { "@timestamp": { "type": "date" }, "last_id": { "type": "keyword" }, "updated_at": { "type": "date" } } }, "cases-rules": { "dynamic": "false", "properties": { "counter": { "type": "unsigned_long" }, "createdAt": { "type": "date" }, "rules": { "properties": { "id": { "type": "keyword" } } }, "updatedAt": { "type": "date" } } }, "cases-telemetry": { "type": "object", "dynamic": "false" }, "cases-user-actions": { "dynamic": "false", "properties": { "action": { "type": "keyword" }, "created_at": { "type": "date" }, "created_by": { "properties": { "username": { "type": "keyword" } } }, "owner": { "type": "keyword" }, "payload": { "dynamic": "false", "properties": { "assignees": { "properties": { "uid": { "type": "keyword" } } }, "comment": { "properties": { "externalReferenceAttachmentTypeId": { "type": "keyword" }, "persistableStateAttachmentTypeId": { "type": "keyword" }, "type": { "type": "keyword" } } }, "connector": { "properties": { "type": { "type": "keyword" } } } } }, "type": { "type": "keyword" } } }, "connector_token": { "dynamic": "false", "properties": { "connectorId": { "type": "keyword" }, "tokenType": { "type": "keyword" } } }, "coreMigrationVersion": { "type": "keyword" }, "created_at": { "type": "date" }, "created_by": { "type": "keyword" }, "maintenance-window": { "dynamic": "false", "properties": { "enabled": { "type": "boolean" }, "events": { "type": "date_range", "format": "epoch_millis||strict_date_optional_time" }, "expirationDate": { "type": "date" }, "title": { "type": "text", "fields": { "keyword": { "type": "keyword" } } }, "updatedAt": { "type": "date" } } }, "managed": { "type": "boolean" }, "namespace": { "type": "keyword" }, "namespaces": { "type": "keyword" }, "originId": { "type": "keyword" }, "references": { "type": "nested", "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" }, "type": { "type": "keyword" } } }, "rules-settings": { "dynamic": "false", "properties": { "flapping": { "type": "object" } } }, "scheduled_report": { "dynamic": "false", "properties": { "createdBy": { "type": "keyword" } } }, "type": { "type": "keyword" }, "typeMigrationVersion": { "type": "version" }, "updated_at": { "type": "date" }, "updated_by": { "type": "keyword" } } } ``` </details> ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: Jan Monschke <jan.monschke@elastic.co> Co-authored-by: Michael Olorunnisola <michael.olorunnisola@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is a reversion of the revert of this original #228002 of incremental id work. This original revert was done, due to the search functionality in cases being broken when searching on the new
incremental_idfield. It being of typeunsigned_longmeant that ES would not return when it was included alongside text values due to data type mismatch errors described here.. Rather than creating a new field, and making the existingincremental_idfield dead weight, we instead update it to a multi-field, so we can search on this value as text alongside other cases values. The update of the mapping was done here: #234054Functional Changes From Reversion
incremental_id.textin search functionality in place ofincremental_idTesting:
Feature Flag:
xpack.cases.incrementalId.enabled: true. Currently disabled and will be enabled after additional testing during FFWhen running a clean branch, the feature will work correctly. We've verified locally that the mapping update works successfully via the methods described below, but feel free to test as well to confirm our results 👍🏾
Serverless
qaf.--kb-docker-imagesflag.#{incremental_id value}and also search fordescription + {incremental_id value}and make sure the query successfully returnsOn-Prem
#{id number}and also search fordescription + id numberand make sure the query successfully returnsOUTDATED
The below details were provided for manually updating the mapping via dev tools for testing/validation purposes
This PR updates the mapping directly. Here are steps to update the mapping manually in a local instance:
0. Remove changes in
packages/kbn-check-mappings-update-cli/current_mappings.jsonincremental_id.textis not yet createdadminuser and assignsystem_indices_superuserandsuperuserroleGET /.kibana_alerting_cases_9.2.0_001/_mappingtextas multi-field forincremental_idPOST /.kibana_alerting_cases_9.2.0_001/_update_by_queryto update existing documentmapping update query
Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
release_note:breakinglabel should be applied in these situations.release_note:*label is applied per the guidelinesbackport:*labels.Identify risks
Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.
Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.