[ResponseOps] Cases analytics index configuration option#223241
Merged
adcoelho merged 4 commits intoelastic:cases-analytics-indexfrom Jun 16, 2025
Merged
[ResponseOps] Cases analytics index configuration option#223241adcoelho merged 4 commits intoelastic:cases-analytics-indexfrom
adcoelho merged 4 commits intoelastic:cases-analytics-indexfrom
Conversation
adcoelho
added
Team:ResponseOps
Contributor
|
Pinging @elastic/response-ops (Team:ResponseOps) |
Contributor
|
Pinging @elastic/response-ops-cases (Feature:Cases) |
adcoelho
force-pushed
the
cai-feature-flag
branch
from
bd6f2b8 to
d4832f5
Compare
adcoelho
force-pushed
the
cai-feature-flag
branch
from
d4832f5 to
b246cef
Compare
cnasikas
approved these changes
Jun 13, 2025
Contributor
💔 Build Failed
Failed CI StepsTest Failures
Metrics [docs]
History
cc @adcoelho |
adcoelho
merged commit 698a3fe
into
elastic:cases-analytics-index
9 of 10 checks passed
adcoelho
added a commit
that referenced
this pull request
Jun 24, 2025
This PR is for a feature branch that is being merged into main. The relevant PRs are: - #219211 - #222820 - #223241 - #224388 - #224682 ## Summary This PR adds 4 new indexes with case analytics data, which are created when the cases plugin starts. - `.internal.cases` - `.internal.cases-comments` - `.internal.cases-attachments` - `.internal.cases-activity` After the indexes are created, a backfill task for each of them is scheduled to run 1 minute after creation. This task populates the indexes with relevant data from `.kibana_alerting_cases`. A second type of task is registered, the index synchronization task. Four of these tasks, one for each index, are scheduled to run every 5 minutes. The synchronization tasks populated the indexes with data from `.kibana_alerting_cases` that was created or updated in the last five minutes. ## How to test You might want to start Kibana with `--verbose` to see relevant index messages in the console. Alternatively(what I normally do), is go to `analytics_index.ts`, `backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and change the `logDebug` function to call `this.logger.info` instead. This way, you will have less spam in the console. Every log message starts with the index name between square brackets, so you can look for `[.internal.cases-` and follow what is happening. 1. You should have some existing case data, so before anything else, please create some activity, attachments, etc. 2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml` and restart Kibana. 3. Check out [this branch](elastic/elasticsearch#129414) from the ES project. 4. Start Elastic Search with `yarn es source`. This will use the above version of Elasticsearch. 5. Wait a bit for the indexes to be created and populated(backfilled). 6. Using the dev tools: - Confirm the indexes exist. - Check the index mapping. Does it match the one in the code? Is the `_meta` field correct? - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts` - Check that the painless scripts match the code. - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts` - Confirm your existing case data is in the indexes. (See **Queries** section below.) 7. Play around with cases. Some examples: - Create a case - Change status/severity - Attach alerts - Add files - Change category/tags - Add comments - etc 8. Go to the dev tools again and confirm all this shows up in the relevant indexes. (See **Queries** section below.) ## Queries In addition to the ones, below I have a few more. Things like reindexing with specific scripts or fetching relevant data from `.kibana_alerting_cases`. Ping me if you want those queries. ### Checking index content ``` GET /.internal.cases/_search GET /.internal.cases-comments/_search GET /.internal.cases-attachments/_search GET /.internal.cases-activity/_search ``` ### Checking index mappings ``` GET /.internal.cases GET /.internal.cases-comments GET /.internal.cases-attachments GET /.internal.cases-activity ``` ### Fetching the painless scripts ``` GET /_scripts/cai_cases_script_1 GET /_scripts/cai_attachments_script_1 GET /_scripts/cai_comments_script_1 GET /_scripts/cai_activity_script_1 ``` ### Emptying the indexes It is sometimes useful for testing. ``` POST /.internal.cases/_delete_by_query POST /.internal.cases-comments/_delete_by_query POST /.internal.cases-attachments/_delete_by_query POST /.internal.cases-activity/_delete_by_query ``` ### Deleting the indexes It is sometimes useful for testing. ``` DELETE /.internal.cases DELETE /.internal.cases-comments DELETE /.internal.cases-attachments DELETE /.internal.cases-activity ``` ## Release notes Four dedicated case analytics indexes were created, allowing users to build dashboards and metrics over case data. These indexes are created on Kibana startup and updated periodically with cases, comments, attachments, and activity data. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>
adcoelho
added a commit
to adcoelho/kibana
that referenced
this pull request
Jun 24, 2025
This PR is for a feature branch that is being merged into main. The relevant PRs are: - elastic#219211 - elastic#222820 - elastic#223241 - elastic#224388 - elastic#224682 ## Summary This PR adds 4 new indexes with case analytics data, which are created when the cases plugin starts. - `.internal.cases` - `.internal.cases-comments` - `.internal.cases-attachments` - `.internal.cases-activity` After the indexes are created, a backfill task for each of them is scheduled to run 1 minute after creation. This task populates the indexes with relevant data from `.kibana_alerting_cases`. A second type of task is registered, the index synchronization task. Four of these tasks, one for each index, are scheduled to run every 5 minutes. The synchronization tasks populated the indexes with data from `.kibana_alerting_cases` that was created or updated in the last five minutes. ## How to test You might want to start Kibana with `--verbose` to see relevant index messages in the console. Alternatively(what I normally do), is go to `analytics_index.ts`, `backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and change the `logDebug` function to call `this.logger.info` instead. This way, you will have less spam in the console. Every log message starts with the index name between square brackets, so you can look for `[.internal.cases-` and follow what is happening. 1. You should have some existing case data, so before anything else, please create some activity, attachments, etc. 2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml` and restart Kibana. 3. Check out [this branch](elastic/elasticsearch#129414) from the ES project. 4. Start Elastic Search with `yarn es source`. This will use the above version of Elasticsearch. 5. Wait a bit for the indexes to be created and populated(backfilled). 6. Using the dev tools: - Confirm the indexes exist. - Check the index mapping. Does it match the one in the code? Is the `_meta` field correct? - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts` - Check that the painless scripts match the code. - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts` - Confirm your existing case data is in the indexes. (See **Queries** section below.) 7. Play around with cases. Some examples: - Create a case - Change status/severity - Attach alerts - Add files - Change category/tags - Add comments - etc 8. Go to the dev tools again and confirm all this shows up in the relevant indexes. (See **Queries** section below.) ## Queries In addition to the ones, below I have a few more. Things like reindexing with specific scripts or fetching relevant data from `.kibana_alerting_cases`. Ping me if you want those queries. ### Checking index content ``` GET /.internal.cases/_search GET /.internal.cases-comments/_search GET /.internal.cases-attachments/_search GET /.internal.cases-activity/_search ``` ### Checking index mappings ``` GET /.internal.cases GET /.internal.cases-comments GET /.internal.cases-attachments GET /.internal.cases-activity ``` ### Fetching the painless scripts ``` GET /_scripts/cai_cases_script_1 GET /_scripts/cai_attachments_script_1 GET /_scripts/cai_comments_script_1 GET /_scripts/cai_activity_script_1 ``` ### Emptying the indexes It is sometimes useful for testing. ``` POST /.internal.cases/_delete_by_query POST /.internal.cases-comments/_delete_by_query POST /.internal.cases-attachments/_delete_by_query POST /.internal.cases-activity/_delete_by_query ``` ### Deleting the indexes It is sometimes useful for testing. ``` DELETE /.internal.cases DELETE /.internal.cases-comments DELETE /.internal.cases-attachments DELETE /.internal.cases-activity ``` ## Release notes Four dedicated case analytics indexes were created, allowing users to build dashboards and metrics over case data. These indexes are created on Kibana startup and updated periodically with cases, comments, attachments, and activity data. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co> (cherry picked from commit e566fec) # Conflicts: # x-pack/platform/plugins/shared/task_manager/server/mocks.ts
akowalska622
pushed a commit
to akowalska622/kibana
that referenced
this pull request
Jun 25, 2025
This PR is for a feature branch that is being merged into main. The relevant PRs are: - elastic#219211 - elastic#222820 - elastic#223241 - elastic#224388 - elastic#224682 ## Summary This PR adds 4 new indexes with case analytics data, which are created when the cases plugin starts. - `.internal.cases` - `.internal.cases-comments` - `.internal.cases-attachments` - `.internal.cases-activity` After the indexes are created, a backfill task for each of them is scheduled to run 1 minute after creation. This task populates the indexes with relevant data from `.kibana_alerting_cases`. A second type of task is registered, the index synchronization task. Four of these tasks, one for each index, are scheduled to run every 5 minutes. The synchronization tasks populated the indexes with data from `.kibana_alerting_cases` that was created or updated in the last five minutes. ## How to test You might want to start Kibana with `--verbose` to see relevant index messages in the console. Alternatively(what I normally do), is go to `analytics_index.ts`, `backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and change the `logDebug` function to call `this.logger.info` instead. This way, you will have less spam in the console. Every log message starts with the index name between square brackets, so you can look for `[.internal.cases-` and follow what is happening. 1. You should have some existing case data, so before anything else, please create some activity, attachments, etc. 2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml` and restart Kibana. 3. Check out [this branch](elastic/elasticsearch#129414) from the ES project. 4. Start Elastic Search with `yarn es source`. This will use the above version of Elasticsearch. 5. Wait a bit for the indexes to be created and populated(backfilled). 6. Using the dev tools: - Confirm the indexes exist. - Check the index mapping. Does it match the one in the code? Is the `_meta` field correct? - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts` - Check that the painless scripts match the code. - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts` - Confirm your existing case data is in the indexes. (See **Queries** section below.) 7. Play around with cases. Some examples: - Create a case - Change status/severity - Attach alerts - Add files - Change category/tags - Add comments - etc 8. Go to the dev tools again and confirm all this shows up in the relevant indexes. (See **Queries** section below.) ## Queries In addition to the ones, below I have a few more. Things like reindexing with specific scripts or fetching relevant data from `.kibana_alerting_cases`. Ping me if you want those queries. ### Checking index content ``` GET /.internal.cases/_search GET /.internal.cases-comments/_search GET /.internal.cases-attachments/_search GET /.internal.cases-activity/_search ``` ### Checking index mappings ``` GET /.internal.cases GET /.internal.cases-comments GET /.internal.cases-attachments GET /.internal.cases-activity ``` ### Fetching the painless scripts ``` GET /_scripts/cai_cases_script_1 GET /_scripts/cai_attachments_script_1 GET /_scripts/cai_comments_script_1 GET /_scripts/cai_activity_script_1 ``` ### Emptying the indexes It is sometimes useful for testing. ``` POST /.internal.cases/_delete_by_query POST /.internal.cases-comments/_delete_by_query POST /.internal.cases-attachments/_delete_by_query POST /.internal.cases-activity/_delete_by_query ``` ### Deleting the indexes It is sometimes useful for testing. ``` DELETE /.internal.cases DELETE /.internal.cases-comments DELETE /.internal.cases-attachments DELETE /.internal.cases-activity ``` ## Release notes Four dedicated case analytics indexes were created, allowing users to build dashboards and metrics over case data. These indexes are created on Kibana startup and updated periodically with cases, comments, attachments, and activity data. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merging into a feature branch
Summary
This PR adds a configuration option to enable the cases analytics index logic.
xpack.cases.analyticsIndex.enabled: trueThe task types are registered, but by default, no task is scheduled, and no index is created.
How to review
The previous PRs are not merged yet, so only commits after bd6f2b8 should be reviewed.