[Cases] - Use text field for phrase prefix queries by michaelolo24 · Pull Request #235290 · elastic/kibana

michaelolo24 · 2025-09-17T03:17:57Z

TLDR

We need to utilize text in place of keyword to not break phrase prefix queries, so introducing the field incremental_id.text

Background

A new issue appeared where the current search field title and description are of mapping type text. This typically shouldn't be a problem, but unfortunately when attempting to run a prefixed query with the new incremental_id.keyword field in this pr we received the following error:

To test, run this test on this PR (after changing all instances in the UI from incremental_id.text back to incremental_id.keyword:

search_phase_execution_exception Root causes: query_shard_exception: failed to create query: Can only use phrase prefix queries on text fields - not on [cases.incremental_id.keyword] which is of type [keyword]

After some digging as to exactly why this was happening, the only documentation regarding this we could find was in this comment: #129424 (comment) . We should most likely add this information to the documentation here as well: https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-prefix-query ?

For historical reference of mapping changes to the incremental_id see: #234054

The primary concern for this change is initializing a mapping change in the serverless environment, but this has been tested in this pr and tested against this PR that also enables the functionality that will populate this field. CI has also been run to make sure all tests pass with the new mapping to avoid any additional surprises here

For testing: see #230278 (comment)

kibanamachine · 2025-09-17T04:09:09Z

Cloud deployment initiated, see credentials at: https://buildkite.com/elastic/kibana-deploy-cloud-from-pr/builds/421

kibanamachine · 2025-09-17T04:25:47Z

Project deployed, see credentials at: https://buildkite.com/elastic/kibana-deploy-project-from-pr/builds/595

janmonschke

Nice catch, glad you caught it now!

elasticmachine · 2025-09-18T02:11:58Z

Pinging @elastic/kibana-cases (Feature:Cases)

afharo · 2025-09-18T14:21:34Z

x-pack/platform/plugins/shared/cases/server/saved_object_types/cases/schemas/v5.ts

+        schema.object({
+          keyword: schema.maybe(schema.string()),
+          text: schema.maybe(schema.string()),
+        }),


I don't think that we can store these. ES multifields shouldn't be part of the document.

keyword was added in the ./v4. Do you want me to remove this field?

Your document attributes will never have this shape:

{ ... incremental_id: { keyword: '123' } }

Only ever the shape you write them in:

{ ... incremental_id: 123 }

The multifield definition is an instruction to ES to index the same field in different ways, not to alter the shape of document attributes (a.k.a the source). So in both v4 and v5 your forward compatibility schema should be:

incremental_id: schema.maybe(schema.nullable(schema.number()))

I think this is a safe change to make retroactively to v4 as well since it's compatible with your document shape, although if your fwc schema is invoked on the next release it will allow for incremental_id: { keyword: string } | number which is not what your code expects I think. At least based on CasePersistedAttributes.

Looks like x-pack/platform/plugins/shared/cases/server/saved_object_types/cases/schemas/latest.ts still needs to be updated to point to v5.

Thanks @jloleysens , makes sense. I've made the udpates and will be testing locally and on serverless with the associated PR!

jloleysens

Left a comment about FWC schema expanding on what @afharo was saying that we should address.

Would you mind providing a simple example of the queries you are making? Just a set of requests to ES to repro what you are seeing and how this fixes it?

lenient

Core has prioritised the following work and we aim to deliver in the next month or so. It would give you far more control of the queries on your SOs so it might be worth waiting for. Although the fix to the FWC should still be done.

jloleysens · 2025-09-19T09:02:00Z

x-pack/platform/plugins/shared/cases/server/saved_object_types/cases/schemas/v5.ts

+        schema.object({
+          keyword: schema.maybe(schema.string()),
+          text: schema.maybe(schema.string()),
+        }),


Your document attributes will never have this shape:

{ ... incremental_id: { keyword: '123' } }

Only ever the shape you write them in:

{ ... incremental_id: 123 }

The multifield definition is an instruction to ES to index the same field in different ways, not to alter the shape of document attributes (a.k.a the source). So in both v4 and v5 your forward compatibility schema should be:

incremental_id: schema.maybe(schema.nullable(schema.number()))

I think this is a safe change to make retroactively to v4 as well since it's compatible with your document shape, although if your fwc schema is invoked on the next release it will allow for incremental_id: { keyword: string } | number which is not what your code expects I think. At least based on CasePersistedAttributes.

Looks like x-pack/platform/plugins/shared/cases/server/saved_object_types/cases/schemas/latest.ts still needs to be updated to point to v5.

…ing-2

jloleysens

I think this looks good. Thanks for addressing the feedback @michaelolo24 , I tried to suggest a couple of ways we might simplify some things!

x-pack/platform/plugins/shared/cases/server/saved_object_types/cases/schemas/v5.ts

x-pack/platform/plugins/shared/cases/server/saved_object_types/cases/schemas/v4.ts

x-pack/platform/plugins/shared/cases/server/saved_object_types/cases/versioning.test.ts

… src/core/server/integration_tests/ci_checks'

michaelolo24 · 2025-09-20T01:10:04Z

Would you mind providing a simple example of the queries you are making? Just a set of requests to ES to repro what you are seeing and how this fixes it?

@jloleysens This particular prefix+wildcard query failed in the previous implementation:

kibana/x-pack/platform/test/functional_with_es_ssl/apps/cases/group2/list_view.ts

Line 397 in 8ffa408

    
           it('only shows cases with a wildcard query "test*" matching the title', async () => {

It seems that this combination can't be done for fields mapped as keyword

…ing-2

elasticmachine · 2025-09-23T14:47:26Z

💔 Build Failed

Buildkite Build
Commit: 0e7fc82
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-235290-0e7fc82c0340
Security Deployment

Failed CI Steps

Metrics [docs]

✅ unchanged

History

💔 Build #342078 failed a35d368
💛 Build #341540 was flaky 5591214
💚 Build #340605 succeeded 14b006e
💔 Build #340134 failed a2cf356

### TLDR We need to utilize `text` in place of `keyword` to not break phrase prefix queries, so introducing the field `incremental_id.text` ### Background A new issue appeared where the current search field `title` and `description` are of mapping type text. This typically shouldn't be a problem, but unfortunately when attempting to run a prefixed query with the new `incremental_id.keyword` field in [this pr](elastic#230278) we received the following error: To test, run [this test](https://github.com/elastic/kibana/blob/8ffa408f560b59cde8f045ff440090a05bf7bdbf/x-pack/platform/test/functional_with_es_ssl/apps/cases/group2/list_view.ts#L397) on [this PR](elastic#230278) (after changing all instances in the UI from `incremental_id.text` back to `incremental_id.keyword`: ``` search_phase_execution_exception Root causes: query_shard_exception: failed to create query: Can only use phrase prefix queries on text fields - not on [cases.incremental_id.keyword] which is of type [keyword] ``` After some digging as to exactly why this was happening, the only documentation regarding this we could find was in this comment: elastic#129424 (comment) . We should most likely add this information to the documentation here as well: https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-prefix-query ? For historical reference of mapping changes to the `incremental_id` see: elastic#234054 The primary concern for this change is initializing a mapping change in the serverless environment, but this has been tested in this pr and tested against [this PR](elastic#230278) that also enables the functionality that will populate this field. CI has also been run to make sure all tests pass with the new mapping to avoid any additional surprises [here](elastic#230278 (comment)) For testing: see elastic#230278 (comment) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

### TLDR We need to utilize `text` in place of `keyword` to not break phrase prefix queries, so introducing the field `incremental_id.text` ### Background A new issue appeared where the current search field `title` and `description` are of mapping type text. This typically shouldn't be a problem, but unfortunately when attempting to run a prefixed query with the new `incremental_id.keyword` field in [this pr](#230278) we received the following error: To test, run [this test](https://github.com/elastic/kibana/blob/8ffa408f560b59cde8f045ff440090a05bf7bdbf/x-pack/platform/test/functional_with_es_ssl/apps/cases/group2/list_view.ts#L397) on [this PR](#230278) (after changing all instances in the UI from `incremental_id.text` back to `incremental_id.keyword`: ``` search_phase_execution_exception Root causes: query_shard_exception: failed to create query: Can only use phrase prefix queries on text fields - not on [cases.incremental_id.keyword] which is of type [keyword] ``` After some digging as to exactly why this was happening, the only documentation regarding this we could find was in this comment: #129424 (comment) . We should most likely add this information to the documentation here as well: https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-prefix-query ? For historical reference of mapping changes to the `incremental_id` see: #234054 The primary concern for this change is initializing a mapping change in the serverless environment, but this has been tested in this pr and tested against [this PR](#230278) that also enables the functionality that will populate this field. CI has also been run to make sure all tests pass with the new mapping to avoid any additional surprises [here](#230278 (comment)) For testing: see #230278 (comment) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

### TLDR We need to utilize `text` in place of `keyword` to not break phrase prefix queries, so introducing the field `incremental_id.text` ### Background A new issue appeared where the current search field `title` and `description` are of mapping type text. This typically shouldn't be a problem, but unfortunately when attempting to run a prefixed query with the new `incremental_id.keyword` field in [this pr](elastic#230278) we received the following error: To test, run [this test](https://github.com/elastic/kibana/blob/4da9288e4d3f13e24e991f91cc51c3ef9fcf5bf7/x-pack/platform/test/functional_with_es_ssl/apps/cases/group2/list_view.ts#L397) on [this PR](elastic#230278) (after changing all instances in the UI from `incremental_id.text` back to `incremental_id.keyword`: ``` search_phase_execution_exception Root causes: query_shard_exception: failed to create query: Can only use phrase prefix queries on text fields - not on [cases.incremental_id.keyword] which is of type [keyword] ``` After some digging as to exactly why this was happening, the only documentation regarding this we could find was in this comment: elastic#129424 (comment) . We should most likely add this information to the documentation here as well: https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-prefix-query ? For historical reference of mapping changes to the `incremental_id` see: elastic#234054 The primary concern for this change is initializing a mapping change in the serverless environment, but this has been tested in this pr and tested against [this PR](elastic#230278) that also enables the functionality that will populate this field. CI has also been run to make sure all tests pass with the new mapping to avoid any additional surprises [here](elastic#230278 (comment)) For testing: see elastic#230278 (comment) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

use text field for phrase prefix queries

a2cf356

janmonschke approved these changes Sep 17, 2025

View reviewed changes

fix type error

d68e178

michaelolo24 force-pushed the update-case-id-mapping-2 branch from 6adb189 to d68e178 Compare September 18, 2025 00:45

fix tests

14b006e

michaelolo24 marked this pull request as ready for review September 18, 2025 02:11

michaelolo24 requested review from a team as code owners September 18, 2025 02:11

janmonschke approved these changes Sep 18, 2025

View reviewed changes

afharo reviewed Sep 18, 2025

View reviewed changes

jloleysens reviewed Sep 19, 2025

View reviewed changes

michaelolo24 added 2 commits September 19, 2025 10:13

remove unnecessary config schema fields

2c5ab95

Merge remote-tracking branch 'upstream/main' into update-case-id-mapp…

e27251d

…ing-2

jloleysens approved these changes Sep 19, 2025

View reviewed changes

kibanamachine and others added 2 commits September 19, 2025 15:07

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

9fc6abb

… src/core/server/integration_tests/ci_checks'

additional pr feedback

5591214

Merge branch 'main' into update-case-id-mapping-2

a35d368

michaelolo24 added 2 commits September 23, 2025 09:44

Merge remote-tracking branch 'upstream/main' into update-case-id-mapp…

e6ccf10

…ing-2

update tests

0e7fc82

michaelolo24 added the ci:cloud-redeploy Always create a new Cloud deployment label Sep 23, 2025

michaelolo24 merged commit 73edb8c into elastic:main Sep 23, 2025
14 checks passed

michaelolo24 mentioned this pull request Sep 24, 2025

[Cases] Add extract observables setting to cases SO mapping #236028

Merged

10 tasks

jloleysens mentioned this pull request Sep 25, 2025

[Cases] Revert incremental id and add multifield #230278

Merged

10 tasks

Comments

Conversation

michaelolo24 commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

Background

Uh oh!

kibanamachine commented Sep 17, 2025

Uh oh!

kibanamachine commented Sep 17, 2025

Uh oh!

janmonschke left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Sep 18, 2025

Uh oh!

afharo Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

michaelolo24 Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

jloleysens Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelolo24 Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

jloleysens left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jloleysens Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jloleysens left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michaelolo24 commented Sep 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Metrics [docs]

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

michaelolo24 commented Sep 17, 2025 •

edited

Loading

jloleysens Sep 19, 2025 •

edited

Loading

jloleysens left a comment •

edited

Loading

jloleysens Sep 19, 2025 •

edited

Loading

michaelolo24 commented Sep 20, 2025 •

edited

Loading

elasticmachine commented Sep 23, 2025 •

edited

Loading