[ResponseOps] Cases analytics index creation and backfill by adcoelho · Pull Request #219211 · elastic/kibana

adcoelho · 2025-04-25T09:42:19Z

Closes #221234

Merging into a feature branch

Summary

This PR adds:

The analytics index creation when the cases plugin starts.
Backfill task registration.
Backfill task scheduling.
Mapping version 1 for:
- .internal.cases
- .internal.cases-comments
- .internal.cases-attachments
All corresponding painless scripts.

How to review

The main functionality is in analytics_index.ts. The class in this file implements the index creation algorithm of the RFC.
Three folders instantiate this class to create the indexes we need:

cases_analytics/cases_index/
cases_analytics/comments_index/
cases_analytics/attachments_index/

The backfill logic can be found in cases_analytics/tasks/backfill_task.
I moved the retry_service previously used by the cases connector to a common folder and started using it in the AnalyticsIndex class.
The call to create the indexes is done in x-pack/platform/plugins/shared/cases/server/plugin.ts.

How to test

A few things are needed.

Create a role that has index creation permission. Below is what works for me; it might be overkill.

Create a user with the role created in step 1. Let's call the user index_creator. The password should be changeme.

Start Kibana with this user as the Elasticsearch user - yarn start with the option --elasticsearch.username=index_creator.
Kibana startup logs should show relevant messages:

[INFO ][plugins.cases] [.internal.cases] Checking if index exists.
[INFO ][plugins.cases] [.internal.cases-attachments] Checking if index exists.
[INFO ][plugins.cases] [.internal.cases-comments] Checking if index exists.

You can check the content of these indexes in the Dev Tools.

GET /.internal.cases/_search
GET /.internal.cases-comments/_search
GET /.internal.cases-attachments/_search

You can check the mapping of each of these indexes in the Dev Tools.

GET /.internal.cases
GET /.internal.cases-comments
GET /.internal.cases-attachments

You can check the painless scripts in the Dev Tools.

GET /_scripts/cai_cases_script_1
GET /_scripts/cai_attachments_script_1
GET /_scripts/cai_comments_script_1

Backfill task implementation.

Add retry logic for index creation.

…urce-definitions/scripts/fix-location-collection.ts'

elasticmachine · 2025-05-28T07:53:16Z

Pinging @elastic/response-ops (Team:ResponseOps)

elasticmachine · 2025-05-28T07:53:41Z

Pinging @elastic/response-ops-cases (Feature:Cases)

...orm/plugins/shared/cases/server/cases_analytics/tasks/backfill_task/backfill_task_factory.ts

...form/plugins/shared/cases/server/cases_analytics/tasks/backfill_task/backfill_task_runner.ts

cnasikas

Great work! I left some comments 🙂!

x-pack/platform/plugins/shared/cases/server/plugin.ts

x-pack/platform/plugins/shared/cases/server/common/retry_service/retry_service.ts

x-pack/platform/plugins/shared/cases/server/cases_analytics/tasks/backfill_task/index.ts

...form/plugins/shared/cases/server/cases_analytics/tasks/backfill_task/backfill_task_runner.ts

…i-index-creation-and-backfill

cnasikas · 2025-06-10T15:22:27Z

...k/platform/plugins/shared/cases/server/cases_analytics/attachments_index/painless_scripts.ts

+        (
+            source["cases-comments"].type == "externalReference" &&
+            source["cases-comments"].externalReferenceAttachmentTypeId != ".files"
+        ) &&


This should be ||, correct?

elasticmachine · 2025-06-10T15:46:40Z

💔 Build Failed

Buildkite Build
Commit: 43fc951

Failed CI Steps

Linting (with types)

History

cc @adcoelho

This PR is for a feature branch that is being merged into main. The relevant PRs are: - #219211 - #222820 - #223241 - #224388 - #224682 ## Summary This PR adds 4 new indexes with case analytics data, which are created when the cases plugin starts. - `.internal.cases` - `.internal.cases-comments` - `.internal.cases-attachments` - `.internal.cases-activity` After the indexes are created, a backfill task for each of them is scheduled to run 1 minute after creation. This task populates the indexes with relevant data from `.kibana_alerting_cases`. A second type of task is registered, the index synchronization task. Four of these tasks, one for each index, are scheduled to run every 5 minutes. The synchronization tasks populated the indexes with data from `.kibana_alerting_cases` that was created or updated in the last five minutes. ## How to test You might want to start Kibana with `--verbose` to see relevant index messages in the console. Alternatively(what I normally do), is go to `analytics_index.ts`, `backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and change the `logDebug` function to call `this.logger.info` instead. This way, you will have less spam in the console. Every log message starts with the index name between square brackets, so you can look for `[.internal.cases-` and follow what is happening. 1. You should have some existing case data, so before anything else, please create some activity, attachments, etc. 2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml` and restart Kibana. 3. Check out [this branch](elastic/elasticsearch#129414) from the ES project. 4. Start Elastic Search with `yarn es source`. This will use the above version of Elasticsearch. 5. Wait a bit for the indexes to be created and populated(backfilled). 6. Using the dev tools: - Confirm the indexes exist. - Check the index mapping. Does it match the one in the code? Is the `_meta` field correct? - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts` - Check that the painless scripts match the code. - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts` - Confirm your existing case data is in the indexes. (See **Queries** section below.) 7. Play around with cases. Some examples: - Create a case - Change status/severity - Attach alerts - Add files - Change category/tags - Add comments - etc 8. Go to the dev tools again and confirm all this shows up in the relevant indexes. (See **Queries** section below.) ## Queries In addition to the ones, below I have a few more. Things like reindexing with specific scripts or fetching relevant data from `.kibana_alerting_cases`. Ping me if you want those queries. ### Checking index content ``` GET /.internal.cases/_search GET /.internal.cases-comments/_search GET /.internal.cases-attachments/_search GET /.internal.cases-activity/_search ``` ### Checking index mappings ``` GET /.internal.cases GET /.internal.cases-comments GET /.internal.cases-attachments GET /.internal.cases-activity ``` ### Fetching the painless scripts ``` GET /_scripts/cai_cases_script_1 GET /_scripts/cai_attachments_script_1 GET /_scripts/cai_comments_script_1 GET /_scripts/cai_activity_script_1 ``` ### Emptying the indexes It is sometimes useful for testing. ``` POST /.internal.cases/_delete_by_query POST /.internal.cases-comments/_delete_by_query POST /.internal.cases-attachments/_delete_by_query POST /.internal.cases-activity/_delete_by_query ``` ### Deleting the indexes It is sometimes useful for testing. ``` DELETE /.internal.cases DELETE /.internal.cases-comments DELETE /.internal.cases-attachments DELETE /.internal.cases-activity ``` ## Release notes Four dedicated case analytics indexes were created, allowing users to build dashboards and metrics over case data. These indexes are created on Kibana startup and updated periodically with cases, comments, attachments, and activity data. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>

This PR is for a feature branch that is being merged into main. The relevant PRs are: - elastic#219211 - elastic#222820 - elastic#223241 - elastic#224388 - elastic#224682 ## Summary This PR adds 4 new indexes with case analytics data, which are created when the cases plugin starts. - `.internal.cases` - `.internal.cases-comments` - `.internal.cases-attachments` - `.internal.cases-activity` After the indexes are created, a backfill task for each of them is scheduled to run 1 minute after creation. This task populates the indexes with relevant data from `.kibana_alerting_cases`. A second type of task is registered, the index synchronization task. Four of these tasks, one for each index, are scheduled to run every 5 minutes. The synchronization tasks populated the indexes with data from `.kibana_alerting_cases` that was created or updated in the last five minutes. ## How to test You might want to start Kibana with `--verbose` to see relevant index messages in the console. Alternatively(what I normally do), is go to `analytics_index.ts`, `backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and change the `logDebug` function to call `this.logger.info` instead. This way, you will have less spam in the console. Every log message starts with the index name between square brackets, so you can look for `[.internal.cases-` and follow what is happening. 1. You should have some existing case data, so before anything else, please create some activity, attachments, etc. 2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml` and restart Kibana. 3. Check out [this branch](elastic/elasticsearch#129414) from the ES project. 4. Start Elastic Search with `yarn es source`. This will use the above version of Elasticsearch. 5. Wait a bit for the indexes to be created and populated(backfilled). 6. Using the dev tools: - Confirm the indexes exist. - Check the index mapping. Does it match the one in the code? Is the `_meta` field correct? - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts` - Check that the painless scripts match the code. - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts` - Confirm your existing case data is in the indexes. (See **Queries** section below.) 7. Play around with cases. Some examples: - Create a case - Change status/severity - Attach alerts - Add files - Change category/tags - Add comments - etc 8. Go to the dev tools again and confirm all this shows up in the relevant indexes. (See **Queries** section below.) ## Queries In addition to the ones, below I have a few more. Things like reindexing with specific scripts or fetching relevant data from `.kibana_alerting_cases`. Ping me if you want those queries. ### Checking index content ``` GET /.internal.cases/_search GET /.internal.cases-comments/_search GET /.internal.cases-attachments/_search GET /.internal.cases-activity/_search ``` ### Checking index mappings ``` GET /.internal.cases GET /.internal.cases-comments GET /.internal.cases-attachments GET /.internal.cases-activity ``` ### Fetching the painless scripts ``` GET /_scripts/cai_cases_script_1 GET /_scripts/cai_attachments_script_1 GET /_scripts/cai_comments_script_1 GET /_scripts/cai_activity_script_1 ``` ### Emptying the indexes It is sometimes useful for testing. ``` POST /.internal.cases/_delete_by_query POST /.internal.cases-comments/_delete_by_query POST /.internal.cases-attachments/_delete_by_query POST /.internal.cases-activity/_delete_by_query ``` ### Deleting the indexes It is sometimes useful for testing. ``` DELETE /.internal.cases DELETE /.internal.cases-comments DELETE /.internal.cases-attachments DELETE /.internal.cases-activity ``` ## Release notes Four dedicated case analytics indexes were created, allowing users to build dashboards and metrics over case data. These indexes are created on Kibana startup and updated periodically with cases, comments, attachments, and activity data. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co> (cherry picked from commit e566fec) # Conflicts: # x-pack/platform/plugins/shared/task_manager/server/mocks.ts

This PR is for a feature branch that is being merged into main. The relevant PRs are: - elastic#219211 - elastic#222820 - elastic#223241 - elastic#224388 - elastic#224682 ## Summary This PR adds 4 new indexes with case analytics data, which are created when the cases plugin starts. - `.internal.cases` - `.internal.cases-comments` - `.internal.cases-attachments` - `.internal.cases-activity` After the indexes are created, a backfill task for each of them is scheduled to run 1 minute after creation. This task populates the indexes with relevant data from `.kibana_alerting_cases`. A second type of task is registered, the index synchronization task. Four of these tasks, one for each index, are scheduled to run every 5 minutes. The synchronization tasks populated the indexes with data from `.kibana_alerting_cases` that was created or updated in the last five minutes. ## How to test You might want to start Kibana with `--verbose` to see relevant index messages in the console. Alternatively(what I normally do), is go to `analytics_index.ts`, `backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and change the `logDebug` function to call `this.logger.info` instead. This way, you will have less spam in the console. Every log message starts with the index name between square brackets, so you can look for `[.internal.cases-` and follow what is happening. 1. You should have some existing case data, so before anything else, please create some activity, attachments, etc. 2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml` and restart Kibana. 3. Check out [this branch](elastic/elasticsearch#129414) from the ES project. 4. Start Elastic Search with `yarn es source`. This will use the above version of Elasticsearch. 5. Wait a bit for the indexes to be created and populated(backfilled). 6. Using the dev tools: - Confirm the indexes exist. - Check the index mapping. Does it match the one in the code? Is the `_meta` field correct? - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts` - Check that the painless scripts match the code. - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts` - Confirm your existing case data is in the indexes. (See **Queries** section below.) 7. Play around with cases. Some examples: - Create a case - Change status/severity - Attach alerts - Add files - Change category/tags - Add comments - etc 8. Go to the dev tools again and confirm all this shows up in the relevant indexes. (See **Queries** section below.) ## Queries In addition to the ones, below I have a few more. Things like reindexing with specific scripts or fetching relevant data from `.kibana_alerting_cases`. Ping me if you want those queries. ### Checking index content ``` GET /.internal.cases/_search GET /.internal.cases-comments/_search GET /.internal.cases-attachments/_search GET /.internal.cases-activity/_search ``` ### Checking index mappings ``` GET /.internal.cases GET /.internal.cases-comments GET /.internal.cases-attachments GET /.internal.cases-activity ``` ### Fetching the painless scripts ``` GET /_scripts/cai_cases_script_1 GET /_scripts/cai_attachments_script_1 GET /_scripts/cai_comments_script_1 GET /_scripts/cai_activity_script_1 ``` ### Emptying the indexes It is sometimes useful for testing. ``` POST /.internal.cases/_delete_by_query POST /.internal.cases-comments/_delete_by_query POST /.internal.cases-attachments/_delete_by_query POST /.internal.cases-activity/_delete_by_query ``` ### Deleting the indexes It is sometimes useful for testing. ``` DELETE /.internal.cases DELETE /.internal.cases-comments DELETE /.internal.cases-attachments DELETE /.internal.cases-activity ``` ## Release notes Four dedicated case analytics indexes were created, allowing users to build dashboards and metrics over case data. These indexes are created on Kibana startup and updated periodically with cases, comments, attachments, and activity data. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>

adcoelho added 2 commits April 23, 2025 09:49

Initial commit.

78ed174

Restructured code.

86fe720

Backfill task implementation.

adcoelho changed the title ~~[ResponseOps][WIP] Cases analytics index~~ [ResponseOps][WIP] Cases analytics index creation and backfill Apr 25, 2025

adcoelho added 2 commits May 21, 2025 09:42

Move RetryService to common folder.

f9d044f

Add retry logic for index creation.

Addressing PR comments.

802be31

adcoelho force-pushed the cai-index-creation-and-backfill branch 3 times, most recently from 1747935 to ca4f7ec Compare May 21, 2025 16:43

Adds analytics index tests.

7d515eb

adcoelho force-pushed the cai-index-creation-and-backfill branch from ca4f7ec to 7d515eb Compare May 22, 2025 14:00

adcoelho changed the title ~~[ResponseOps][WIP] Cases analytics index creation and backfill~~ [ResponseOps] Cases analytics index creation and backfill May 27, 2025

adcoelho marked this pull request as ready for review May 27, 2025 15:02

adcoelho requested a review from a team as a code owner May 27, 2025 15:02

[CI] Auto-commit changed files from 'ts-node .buildkite/pipeline-reso…

37dbdaf

…urce-definitions/scripts/fix-location-collection.ts'

adcoelho added the Team:ResponseOps Platform ResponseOps team (formerly the Cases and Alerting teams) t// label May 28, 2025

adcoelho self-assigned this May 28, 2025

adcoelho added Feature:Maintenance Window Feature:Cases Cases feature and removed Feature:Maintenance Window labels May 28, 2025

adcoelho commented May 28, 2025

View reviewed changes

...orm/plugins/shared/cases/server/cases_analytics/tasks/backfill_task/backfill_task_factory.ts Show resolved Hide resolved

adcoelho commented May 28, 2025

View reviewed changes

...orm/plugins/shared/cases/server/cases_analytics/tasks/backfill_task/backfill_task_factory.ts Show resolved Hide resolved

adcoelho commented May 28, 2025

View reviewed changes

...form/plugins/shared/cases/server/cases_analytics/tasks/backfill_task/backfill_task_runner.ts Show resolved Hide resolved

cnasikas reviewed May 29, 2025

View reviewed changes

adcoelho added 2 commits June 3, 2025 15:31

PR comments - part 1

7744009

PR comments - part 2

97b1a6e

adcoelho force-pushed the cai-index-creation-and-backfill branch from 4ed7e23 to 97b1a6e Compare June 4, 2025 13:02

PR changes - part 3

81063e5

adcoelho force-pushed the cai-index-creation-and-backfill branch from 57f4511 to 81063e5 Compare June 4, 2025 13:53

Merge remote-tracking branch 'upstream/cases-analytics-index' into ca…

43fc951

…i-index-creation-and-backfill

cnasikas approved these changes Jun 10, 2025

View reviewed changes

cnasikas reviewed Jun 10, 2025

View reviewed changes

adcoelho merged commit fa8ef65 into elastic:cases-analytics-index Jun 11, 2025
9 of 10 checks passed

adcoelho mentioned this pull request Jun 17, 2025

[ResponseOps] Cases analytics index #223405

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

[ResponseOps] Cases analytics index creation and backfill#219211

[ResponseOps] Cases analytics index creation and backfill#219211
adcoelho merged 10 commits intoelastic:cases-analytics-indexfrom
adcoelho:cai-index-creation-and-backfill

adcoelho commented Apr 25, 2025 •

edited

Loading

Uh oh!

elasticmachine commented May 28, 2025

Uh oh!

elasticmachine commented May 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cnasikas left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cnasikas Jun 10, 2025

Uh oh!

elasticmachine commented Jun 10, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Comments

Conversation

adcoelho commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to review

How to test

Uh oh!

elasticmachine commented May 28, 2025

Uh oh!

elasticmachine commented May 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cnasikas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cnasikas Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

adcoelho commented Apr 25, 2025 •

edited

Loading

elasticmachine commented Jun 10, 2025 •

edited

Loading