[Telemetry][Security Solution] Index metadata collector#194004
Merged
szaffarano merged 62 commits intomainfrom Dec 13, 2024
Merged
[Telemetry][Security Solution] Index metadata collector#194004szaffarano merged 62 commits intomainfrom
szaffarano merged 62 commits intomainfrom
Conversation
szaffarano
force-pushed
the
szaffarano/datastreams-metadata-poc
branch
3 times, most recently
from
4d7a50f to
f0bf108
Compare
szaffarano
added
release_note:skip
szaffarano
force-pushed
the
szaffarano/datastreams-metadata-poc
branch
2 times, most recently
from
2412982 to
5c1a6da
Compare
Contributor
Author
|
@elasticmachine merge upstream |
Contributor
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
szaffarano
added
the
backport:all-open
pmuellr
reviewed
Oct 1, 2024
x-pack/plugins/security_solution/server/lib/telemetry/routes/index.ts
Outdated
Show resolved
Hide resolved
pmuellr
reviewed
Oct 1, 2024
pmuellr
approved these changes
Oct 1, 2024
Contributor
pmuellr
left a comment
pmuellr
left a comment
There was a problem hiding this comment.
ResponseOps changes LGTM, made some drive-by comments
szaffarano
added
ci:project-deploy-observability
Contributor
Author
|
@elasticmachine merge upstream |
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]Unknown metric groupsESLint disabled line counts
Total ESLint disabled count
History
To update your PR or re-run it, just comment with: cc @szaffarano |
donaherc
changed the title
Contributor
Author
|
@elasticmachine merge upstream |
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
5 similar comments
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
Contributor
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
szaffarano
added a commit
that referenced
this pull request
Jan 9, 2025
…194004) (#204311) (#204629) # Backport This will backport the following commits from `8.x` to `8.16`: - [[8.x] [Telemetry][Security Solution] Index metadata collector (#194004) (#204311)](#204311) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Sebastián Zaffarano","email":"sebastian.zaffarano@elastic.co"},"sourceCommit":{"committedDate":"2024-12-16T16:05:02Z","message":"[8.x] [Telemetry][Security Solution] Index metadata collector (#194004) (#204311)\n\n# Backport\n\nThis will backport the following commits from `main` to `8.x`:\n- [[Telemetry][Security Solution] Index metadata collector\n(#194004)](https://github.com/elastic/kibana/pull/194004)\n\n<!--- Backport version: 8.9.8 -->\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT [{\"author\":{\"name\":\"Sebastián\nZaffarano\",\"email\":\"sebastian.zaffarano@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2024-12-13T18:31:03Z\",\"message\":\"[Telemetry][Security\nSolution] Index metadata collector (#194004)\\n\\n##\nSummary\\r\\n\\r\\nImplements a security_solution task scheduled to run once\na day to\\r\\ncollect the following information:\\r\\n\\r\\n1. Datastreams\nstats\\r\\n2. Indices stats\\r\\n3. ILMs stats\\r\\n4. ILM configs\\r\\n\\r\\nThe\ntask allows a runtime configuration to limit the number of\nindices\\r\\nand data streams to analyze or event to disable the feature\nentirely.\\r\\n\\r\\nOnce the data is gathered, the task sends it as EBT\nevents.\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by: kibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\\r\\nCo-authored-by:\nElastic Machine\n<elasticmachine@users.noreply.github.com>\",\"sha\":\"36b344a4c58a3d78a892288e0eef71e9ff163b9d\",\"branchLabelMapping\":{\"^v9.0.0$\":\"main\",\"^v8.18.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:skip\",\"v9.0.0\",\"Team:\nSecuritySolution\",\"backport:all-open\",\"ci:cloud-deploy\"],\"number\":194004,\"url\":\"https://github.com/elastic/kibana/pull/194004\",\"mergeCommit\":{\"message\":\"[Telemetry][Security\nSolution] Index metadata collector (#194004)\\n\\n##\nSummary\\r\\n\\r\\nImplements a security_solution task scheduled to run once\na day to\\r\\ncollect the following information:\\r\\n\\r\\n1. Datastreams\nstats\\r\\n2. Indices stats\\r\\n3. ILMs stats\\r\\n4. ILM configs\\r\\n\\r\\nThe\ntask allows a runtime configuration to limit the number of\nindices\\r\\nand data streams to analyze or event to disable the feature\nentirely.\\r\\n\\r\\nOnce the data is gathered, the task sends it as EBT\nevents.\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by: kibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\\r\\nCo-authored-by:\nElastic Machine\n<elasticmachine@users.noreply.github.com>\",\"sha\":\"36b344a4c58a3d78a892288e0eef71e9ff163b9d\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[],\"targetPullRequestStates\":[{\"branch\":\"main\",\"label\":\"v9.0.0\",\"labelRegex\":\"^v9.0.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/194004\",\"number\":194004,\"mergeCommit\":{\"message\":\"[Telemetry][Security\nSolution] Index metadata collector (#194004)\\n\\n##\nSummary\\r\\n\\r\\nImplements a security_solution task scheduled to run once\na day to\\r\\ncollect the following information:\\r\\n\\r\\n1. Datastreams\nstats\\r\\n2. Indices stats\\r\\n3. ILMs stats\\r\\n4. ILM configs\\r\\n\\r\\nThe\ntask allows a runtime configuration to limit the number of\nindices\\r\\nand data streams to analyze or event to disable the feature\nentirely.\\r\\n\\r\\nOnce the data is gathered, the task sends it as EBT\nevents.\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by: kibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\\r\\nCo-authored-by:\nElastic Machine\n<elasticmachine@users.noreply.github.com>\",\"sha\":\"36b344a4c58a3d78a892288e0eef71e9ff163b9d\"}}]}]\nBACKPORT-->\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"7a3de72bbe94a87bd972602b29212764f2619951","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["backport"],"number":204311,"url":"https://github.com/elastic/kibana/pull/204311","mergeCommit":{"message":"[8.x] [Telemetry][Security Solution] Index metadata collector (#194004) (#204311)\n\n# Backport\n\nThis will backport the following commits from `main` to `8.x`:\n- [[Telemetry][Security Solution] Index metadata collector\n(#194004)](https://github.com/elastic/kibana/pull/194004)\n\n<!--- Backport version: 8.9.8 -->\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT [{\"author\":{\"name\":\"Sebastián\nZaffarano\",\"email\":\"sebastian.zaffarano@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2024-12-13T18:31:03Z\",\"message\":\"[Telemetry][Security\nSolution] Index metadata collector (#194004)\\n\\n##\nSummary\\r\\n\\r\\nImplements a security_solution task scheduled to run once\na day to\\r\\ncollect the following information:\\r\\n\\r\\n1. Datastreams\nstats\\r\\n2. Indices stats\\r\\n3. ILMs stats\\r\\n4. ILM configs\\r\\n\\r\\nThe\ntask allows a runtime configuration to limit the number of\nindices\\r\\nand data streams to analyze or event to disable the feature\nentirely.\\r\\n\\r\\nOnce the data is gathered, the task sends it as EBT\nevents.\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by: kibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\\r\\nCo-authored-by:\nElastic Machine\n<elasticmachine@users.noreply.github.com>\",\"sha\":\"36b344a4c58a3d78a892288e0eef71e9ff163b9d\",\"branchLabelMapping\":{\"^v9.0.0$\":\"main\",\"^v8.18.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:skip\",\"v9.0.0\",\"Team:\nSecuritySolution\",\"backport:all-open\",\"ci:cloud-deploy\"],\"number\":194004,\"url\":\"https://github.com/elastic/kibana/pull/194004\",\"mergeCommit\":{\"message\":\"[Telemetry][Security\nSolution] Index metadata collector (#194004)\\n\\n##\nSummary\\r\\n\\r\\nImplements a security_solution task scheduled to run once\na day to\\r\\ncollect the following information:\\r\\n\\r\\n1. Datastreams\nstats\\r\\n2. Indices stats\\r\\n3. ILMs stats\\r\\n4. ILM configs\\r\\n\\r\\nThe\ntask allows a runtime configuration to limit the number of\nindices\\r\\nand data streams to analyze or event to disable the feature\nentirely.\\r\\n\\r\\nOnce the data is gathered, the task sends it as EBT\nevents.\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by: kibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\\r\\nCo-authored-by:\nElastic Machine\n<elasticmachine@users.noreply.github.com>\",\"sha\":\"36b344a4c58a3d78a892288e0eef71e9ff163b9d\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[],\"targetPullRequestStates\":[{\"branch\":\"main\",\"label\":\"v9.0.0\",\"labelRegex\":\"^v9.0.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/194004\",\"number\":194004,\"mergeCommit\":{\"message\":\"[Telemetry][Security\nSolution] Index metadata collector (#194004)\\n\\n##\nSummary\\r\\n\\r\\nImplements a security_solution task scheduled to run once\na day to\\r\\ncollect the following information:\\r\\n\\r\\n1. Datastreams\nstats\\r\\n2. Indices stats\\r\\n3. ILMs stats\\r\\n4. ILM configs\\r\\n\\r\\nThe\ntask allows a runtime configuration to limit the number of\nindices\\r\\nand data streams to analyze or event to disable the feature\nentirely.\\r\\n\\r\\nOnce the data is gathered, the task sends it as EBT\nevents.\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by: kibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\\r\\nCo-authored-by:\nElastic Machine\n<elasticmachine@users.noreply.github.com>\",\"sha\":\"36b344a4c58a3d78a892288e0eef71e9ff163b9d\"}}]}]\nBACKPORT-->\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"7a3de72bbe94a87bd972602b29212764f2619951"}},"sourceBranch":"8.x","suggestedTargetBranches":[],"targetPullRequestStates":[{"url":"https://github.com/elastic/kibana/pull/204628","number":204628,"branch":"8.17","state":"OPEN"}]}] BACKPORT--> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
szaffarano
added a commit
that referenced
this pull request
Jan 9, 2025
…194004) (#204311) (#204628) # Backport This will backport the following commits from `8.x` to `8.17`: - [[8.x] [Telemetry][Security Solution] Index metadata collector (#194004) (#204311)](#204311) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Sebastián Zaffarano","email":"sebastian.zaffarano@elastic.co"},"sourceCommit":{"committedDate":"2024-12-16T16:05:02Z","message":"[8.x] [Telemetry][Security Solution] Index metadata collector (#194004) (#204311)\n\n# Backport\n\nThis will backport the following commits from `main` to `8.x`:\n- [[Telemetry][Security Solution] Index metadata collector\n(#194004)](https://github.com/elastic/kibana/pull/194004)\n\n<!--- Backport version: 8.9.8 -->\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT [{\"author\":{\"name\":\"Sebastián\nZaffarano\",\"email\":\"sebastian.zaffarano@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2024-12-13T18:31:03Z\",\"message\":\"[Telemetry][Security\nSolution] Index metadata collector (#194004)\\n\\n##\nSummary\\r\\n\\r\\nImplements a security_solution task scheduled to run once\na day to\\r\\ncollect the following information:\\r\\n\\r\\n1. Datastreams\nstats\\r\\n2. Indices stats\\r\\n3. ILMs stats\\r\\n4. ILM configs\\r\\n\\r\\nThe\ntask allows a runtime configuration to limit the number of\nindices\\r\\nand data streams to analyze or event to disable the feature\nentirely.\\r\\n\\r\\nOnce the data is gathered, the task sends it as EBT\nevents.\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by: kibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\\r\\nCo-authored-by:\nElastic Machine\n<elasticmachine@users.noreply.github.com>\",\"sha\":\"36b344a4c58a3d78a892288e0eef71e9ff163b9d\",\"branchLabelMapping\":{\"^v9.0.0$\":\"main\",\"^v8.18.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:skip\",\"v9.0.0\",\"Team:\nSecuritySolution\",\"backport:all-open\",\"ci:cloud-deploy\"],\"number\":194004,\"url\":\"https://github.com/elastic/kibana/pull/194004\",\"mergeCommit\":{\"message\":\"[Telemetry][Security\nSolution] Index metadata collector (#194004)\\n\\n##\nSummary\\r\\n\\r\\nImplements a security_solution task scheduled to run once\na day to\\r\\ncollect the following information:\\r\\n\\r\\n1. Datastreams\nstats\\r\\n2. Indices stats\\r\\n3. ILMs stats\\r\\n4. ILM configs\\r\\n\\r\\nThe\ntask allows a runtime configuration to limit the number of\nindices\\r\\nand data streams to analyze or event to disable the feature\nentirely.\\r\\n\\r\\nOnce the data is gathered, the task sends it as EBT\nevents.\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by: kibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\\r\\nCo-authored-by:\nElastic Machine\n<elasticmachine@users.noreply.github.com>\",\"sha\":\"36b344a4c58a3d78a892288e0eef71e9ff163b9d\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[],\"targetPullRequestStates\":[{\"branch\":\"main\",\"label\":\"v9.0.0\",\"labelRegex\":\"^v9.0.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/194004\",\"number\":194004,\"mergeCommit\":{\"message\":\"[Telemetry][Security\nSolution] Index metadata collector (#194004)\\n\\n##\nSummary\\r\\n\\r\\nImplements a security_solution task scheduled to run once\na day to\\r\\ncollect the following information:\\r\\n\\r\\n1. Datastreams\nstats\\r\\n2. Indices stats\\r\\n3. ILMs stats\\r\\n4. ILM configs\\r\\n\\r\\nThe\ntask allows a runtime configuration to limit the number of\nindices\\r\\nand data streams to analyze or event to disable the feature\nentirely.\\r\\n\\r\\nOnce the data is gathered, the task sends it as EBT\nevents.\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by: kibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\\r\\nCo-authored-by:\nElastic Machine\n<elasticmachine@users.noreply.github.com>\",\"sha\":\"36b344a4c58a3d78a892288e0eef71e9ff163b9d\"}}]}]\nBACKPORT-->\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"7a3de72bbe94a87bd972602b29212764f2619951","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["backport"],"number":204311,"url":"https://github.com/elastic/kibana/pull/204311","mergeCommit":{"message":"[8.x] [Telemetry][Security Solution] Index metadata collector (#194004) (#204311)\n\n# Backport\n\nThis will backport the following commits from `main` to `8.x`:\n- [[Telemetry][Security Solution] Index metadata collector\n(#194004)](https://github.com/elastic/kibana/pull/194004)\n\n<!--- Backport version: 8.9.8 -->\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT [{\"author\":{\"name\":\"Sebastián\nZaffarano\",\"email\":\"sebastian.zaffarano@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2024-12-13T18:31:03Z\",\"message\":\"[Telemetry][Security\nSolution] Index metadata collector (#194004)\\n\\n##\nSummary\\r\\n\\r\\nImplements a security_solution task scheduled to run once\na day to\\r\\ncollect the following information:\\r\\n\\r\\n1. Datastreams\nstats\\r\\n2. Indices stats\\r\\n3. ILMs stats\\r\\n4. ILM configs\\r\\n\\r\\nThe\ntask allows a runtime configuration to limit the number of\nindices\\r\\nand data streams to analyze or event to disable the feature\nentirely.\\r\\n\\r\\nOnce the data is gathered, the task sends it as EBT\nevents.\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by: kibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\\r\\nCo-authored-by:\nElastic Machine\n<elasticmachine@users.noreply.github.com>\",\"sha\":\"36b344a4c58a3d78a892288e0eef71e9ff163b9d\",\"branchLabelMapping\":{\"^v9.0.0$\":\"main\",\"^v8.18.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:skip\",\"v9.0.0\",\"Team:\nSecuritySolution\",\"backport:all-open\",\"ci:cloud-deploy\"],\"number\":194004,\"url\":\"https://github.com/elastic/kibana/pull/194004\",\"mergeCommit\":{\"message\":\"[Telemetry][Security\nSolution] Index metadata collector (#194004)\\n\\n##\nSummary\\r\\n\\r\\nImplements a security_solution task scheduled to run once\na day to\\r\\ncollect the following information:\\r\\n\\r\\n1. Datastreams\nstats\\r\\n2. Indices stats\\r\\n3. ILMs stats\\r\\n4. ILM configs\\r\\n\\r\\nThe\ntask allows a runtime configuration to limit the number of\nindices\\r\\nand data streams to analyze or event to disable the feature\nentirely.\\r\\n\\r\\nOnce the data is gathered, the task sends it as EBT\nevents.\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by: kibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\\r\\nCo-authored-by:\nElastic Machine\n<elasticmachine@users.noreply.github.com>\",\"sha\":\"36b344a4c58a3d78a892288e0eef71e9ff163b9d\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[],\"targetPullRequestStates\":[{\"branch\":\"main\",\"label\":\"v9.0.0\",\"labelRegex\":\"^v9.0.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/194004\",\"number\":194004,\"mergeCommit\":{\"message\":\"[Telemetry][Security\nSolution] Index metadata collector (#194004)\\n\\n##\nSummary\\r\\n\\r\\nImplements a security_solution task scheduled to run once\na day to\\r\\ncollect the following information:\\r\\n\\r\\n1. Datastreams\nstats\\r\\n2. Indices stats\\r\\n3. ILMs stats\\r\\n4. ILM configs\\r\\n\\r\\nThe\ntask allows a runtime configuration to limit the number of\nindices\\r\\nand data streams to analyze or event to disable the feature\nentirely.\\r\\n\\r\\nOnce the data is gathered, the task sends it as EBT\nevents.\\r\\n\\r\\n---------\\r\\n\\r\\nCo-authored-by: kibanamachine\n<42973632+kibanamachine@users.noreply.github.com>\\r\\nCo-authored-by:\nElastic Machine\n<elasticmachine@users.noreply.github.com>\",\"sha\":\"36b344a4c58a3d78a892288e0eef71e9ff163b9d\"}}]}]\nBACKPORT-->\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"7a3de72bbe94a87bd972602b29212764f2619951"}},"sourceBranch":"8.x","suggestedTargetBranches":[],"targetPullRequestStates":[]}] BACKPORT--> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
kibanamachine
removed
the
backport missing
2 tasks
szaffarano
added a commit
that referenced
this pull request
Apr 28, 2025
## Summary Adds `_source.mode` and `index.mode` fields to the [TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44) EBT event: ```typescript export interface IndexSettings { index_name: string; index_mode?: string; default_pipeline?: string; final_pipeline?: string; source_mode?: string; } ``` References: - #194004 Initial version for the updated EBT event. - #213822 This is a Similar PR updating the feature. The goal is to enhance the index settings EBT event with two new fields to enrich the telemetry information we collect about the indices used by the security integrations. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
szaffarano
added a commit
to szaffarano/kibana
that referenced
this pull request
Apr 29, 2025
## Summary Adds `_source.mode` and `index.mode` fields to the [TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44) EBT event: ```typescript export interface IndexSettings { index_name: string; index_mode?: string; default_pipeline?: string; final_pipeline?: string; source_mode?: string; } ``` References: - elastic#194004 Initial version for the updated EBT event. - elastic#213822 This is a Similar PR updating the feature. The goal is to enhance the index settings EBT event with two new fields to enrich the telemetry information we collect about the indices used by the security integrations. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit 8c596d3) # Conflicts: # x-pack/test/common/utils/security_solution/detections_response/tasks/indices_metadata.ts
szaffarano
added a commit
to szaffarano/kibana
that referenced
this pull request
Apr 29, 2025
## Summary Adds `_source.mode` and `index.mode` fields to the [TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44) EBT event: ```typescript export interface IndexSettings { index_name: string; index_mode?: string; default_pipeline?: string; final_pipeline?: string; source_mode?: string; } ``` References: - elastic#194004 Initial version for the updated EBT event. - elastic#213822 This is a Similar PR updating the feature. The goal is to enhance the index settings EBT event with two new fields to enrich the telemetry information we collect about the indices used by the security integrations. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit 8c596d3) # Conflicts: # x-pack/test/common/utils/security_solution/detections_response/tasks/indices_metadata.ts
szaffarano
added a commit
to szaffarano/kibana
that referenced
this pull request
Apr 29, 2025
## Summary Adds `_source.mode` and `index.mode` fields to the [TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44) EBT event: ```typescript export interface IndexSettings { index_name: string; index_mode?: string; default_pipeline?: string; final_pipeline?: string; source_mode?: string; } ``` References: - elastic#194004 Initial version for the updated EBT event. - elastic#213822 This is a Similar PR updating the feature. The goal is to enhance the index settings EBT event with two new fields to enrich the telemetry information we collect about the indices used by the security integrations. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit 8c596d3) # Conflicts: # x-pack/test/common/utils/security_solution/detections_response/tasks/indices_metadata.ts
szaffarano
added a commit
that referenced
this pull request
Apr 29, 2025
… (#219507) # Backport This will backport the following commits from `main` to `8.17`: - [[Security Solution] Add new fields to indices metadata (#219246)](#219246) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Sebastián Zaffarano","email":"sebastian.zaffarano@elastic.co"},"sourceCommit":{"committedDate":"2025-04-28T18:18:43Z","message":"[Security Solution] Add new fields to indices metadata (#219246)\n\n## Summary\n\nAdds `_source.mode` and `index.mode` fields to the\n[TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44)\nEBT event:\n\n```typescript\nexport interface IndexSettings {\n index_name: string;\n index_mode?: string;\n default_pipeline?: string;\n final_pipeline?: string;\n source_mode?: string;\n}\n```\nReferences:\n- #194004 Initial version for the\nupdated EBT event.\n- #213822 This is a Similar PR\nupdating the feature.\n\nThe goal is to enhance the index settings EBT event with two new fields\nto enrich the telemetry information we collect about the indices used by\nthe security integrations.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n\n---------\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"8c596d32010ad96eeac794f1938e686f9cbdb278","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","backport:all-open","ci:build-cloud-image","v9.1.0"],"title":"[Security Solution] Add new fields to indices metadata","number":219246,"url":"https://github.com/elastic/kibana/pull/219246","mergeCommit":{"message":"[Security Solution] Add new fields to indices metadata (#219246)\n\n## Summary\n\nAdds `_source.mode` and `index.mode` fields to the\n[TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44)\nEBT event:\n\n```typescript\nexport interface IndexSettings {\n index_name: string;\n index_mode?: string;\n default_pipeline?: string;\n final_pipeline?: string;\n source_mode?: string;\n}\n```\nReferences:\n- #194004 Initial version for the\nupdated EBT event.\n- #213822 This is a Similar PR\nupdating the feature.\n\nThe goal is to enhance the index settings EBT event with two new fields\nto enrich the telemetry information we collect about the indices used by\nthe security integrations.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n\n---------\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"8c596d32010ad96eeac794f1938e686f9cbdb278"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/219246","number":219246,"mergeCommit":{"message":"[Security Solution] Add new fields to indices metadata (#219246)\n\n## Summary\n\nAdds `_source.mode` and `index.mode` fields to the\n[TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44)\nEBT event:\n\n```typescript\nexport interface IndexSettings {\n index_name: string;\n index_mode?: string;\n default_pipeline?: string;\n final_pipeline?: string;\n source_mode?: string;\n}\n```\nReferences:\n- #194004 Initial version for the\nupdated EBT event.\n- #213822 This is a Similar PR\nupdating the feature.\n\nThe goal is to enhance the index settings EBT event with two new fields\nto enrich the telemetry information we collect about the indices used by\nthe security integrations.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n\n---------\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"8c596d32010ad96eeac794f1938e686f9cbdb278"}},{"url":"https://github.com/elastic/kibana/pull/219499","number":219499,"branch":"9.0","state":"OPEN"},{"url":"https://github.com/elastic/kibana/pull/219503","number":219503,"branch":"8.19","state":"OPEN"},{"url":"https://github.com/elastic/kibana/pull/219504","number":219504,"branch":"8.18","state":"OPEN"}]}] BACKPORT-->
szaffarano
added a commit
that referenced
this pull request
Apr 29, 2025
… (#219503) # Backport This will backport the following commits from `main` to `8.19`: - [[Security Solution] Add new fields to indices metadata (#219246)](#219246) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Sebastián Zaffarano","email":"sebastian.zaffarano@elastic.co"},"sourceCommit":{"committedDate":"2025-04-28T18:18:43Z","message":"[Security Solution] Add new fields to indices metadata (#219246)\n\n## Summary\n\nAdds `_source.mode` and `index.mode` fields to the\n[TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44)\nEBT event:\n\n```typescript\nexport interface IndexSettings {\n index_name: string;\n index_mode?: string;\n default_pipeline?: string;\n final_pipeline?: string;\n source_mode?: string;\n}\n```\nReferences:\n- #194004 Initial version for the\nupdated EBT event.\n- #213822 This is a Similar PR\nupdating the feature.\n\nThe goal is to enhance the index settings EBT event with two new fields\nto enrich the telemetry information we collect about the indices used by\nthe security integrations.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n\n---------\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"8c596d32010ad96eeac794f1938e686f9cbdb278","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","backport:all-open","ci:build-cloud-image","v9.1.0"],"title":"[Security Solution] Add new fields to indices metadata","number":219246,"url":"https://github.com/elastic/kibana/pull/219246","mergeCommit":{"message":"[Security Solution] Add new fields to indices metadata (#219246)\n\n## Summary\n\nAdds `_source.mode` and `index.mode` fields to the\n[TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44)\nEBT event:\n\n```typescript\nexport interface IndexSettings {\n index_name: string;\n index_mode?: string;\n default_pipeline?: string;\n final_pipeline?: string;\n source_mode?: string;\n}\n```\nReferences:\n- #194004 Initial version for the\nupdated EBT event.\n- #213822 This is a Similar PR\nupdating the feature.\n\nThe goal is to enhance the index settings EBT event with two new fields\nto enrich the telemetry information we collect about the indices used by\nthe security integrations.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n\n---------\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"8c596d32010ad96eeac794f1938e686f9cbdb278"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/219246","number":219246,"mergeCommit":{"message":"[Security Solution] Add new fields to indices metadata (#219246)\n\n## Summary\n\nAdds `_source.mode` and `index.mode` fields to the\n[TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44)\nEBT event:\n\n```typescript\nexport interface IndexSettings {\n index_name: string;\n index_mode?: string;\n default_pipeline?: string;\n final_pipeline?: string;\n source_mode?: string;\n}\n```\nReferences:\n- #194004 Initial version for the\nupdated EBT event.\n- #213822 This is a Similar PR\nupdating the feature.\n\nThe goal is to enhance the index settings EBT event with two new fields\nto enrich the telemetry information we collect about the indices used by\nthe security integrations.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n\n---------\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"8c596d32010ad96eeac794f1938e686f9cbdb278"}},{"url":"https://github.com/elastic/kibana/pull/219499","number":219499,"branch":"9.0","state":"OPEN"}]}] BACKPORT-->
szaffarano
added a commit
that referenced
this pull request
Apr 29, 2025
#219499) # Backport This will backport the following commits from `main` to `9.0`: - [[Security Solution] Add new fields to indices metadata (#219246)](#219246) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Sebastián Zaffarano","email":"sebastian.zaffarano@elastic.co"},"sourceCommit":{"committedDate":"2025-04-28T18:18:43Z","message":"[Security Solution] Add new fields to indices metadata (#219246)\n\n## Summary\n\nAdds `_source.mode` and `index.mode` fields to the\n[TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44)\nEBT event:\n\n```typescript\nexport interface IndexSettings {\n index_name: string;\n index_mode?: string;\n default_pipeline?: string;\n final_pipeline?: string;\n source_mode?: string;\n}\n```\nReferences:\n- #194004 Initial version for the\nupdated EBT event.\n- #213822 This is a Similar PR\nupdating the feature.\n\nThe goal is to enhance the index settings EBT event with two new fields\nto enrich the telemetry information we collect about the indices used by\nthe security integrations.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n\n---------\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"8c596d32010ad96eeac794f1938e686f9cbdb278","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","backport:all-open","ci:build-cloud-image","v9.1.0"],"title":"[Security Solution] Add new fields to indices metadata","number":219246,"url":"https://github.com/elastic/kibana/pull/219246","mergeCommit":{"message":"[Security Solution] Add new fields to indices metadata (#219246)\n\n## Summary\n\nAdds `_source.mode` and `index.mode` fields to the\n[TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44)\nEBT event:\n\n```typescript\nexport interface IndexSettings {\n index_name: string;\n index_mode?: string;\n default_pipeline?: string;\n final_pipeline?: string;\n source_mode?: string;\n}\n```\nReferences:\n- #194004 Initial version for the\nupdated EBT event.\n- #213822 This is a Similar PR\nupdating the feature.\n\nThe goal is to enhance the index settings EBT event with two new fields\nto enrich the telemetry information we collect about the indices used by\nthe security integrations.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n\n---------\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"8c596d32010ad96eeac794f1938e686f9cbdb278"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/219246","number":219246,"mergeCommit":{"message":"[Security Solution] Add new fields to indices metadata (#219246)\n\n## Summary\n\nAdds `_source.mode` and `index.mode` fields to the\n[TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44)\nEBT event:\n\n```typescript\nexport interface IndexSettings {\n index_name: string;\n index_mode?: string;\n default_pipeline?: string;\n final_pipeline?: string;\n source_mode?: string;\n}\n```\nReferences:\n- #194004 Initial version for the\nupdated EBT event.\n- #213822 This is a Similar PR\nupdating the feature.\n\nThe goal is to enhance the index settings EBT event with two new fields\nto enrich the telemetry information we collect about the indices used by\nthe security integrations.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n\n---------\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"8c596d32010ad96eeac794f1938e686f9cbdb278"}}]}] BACKPORT-->
szaffarano
added a commit
that referenced
this pull request
Apr 29, 2025
… (#219504) # Backport This will backport the following commits from `main` to `8.18`: - [[Security Solution] Add new fields to indices metadata (#219246)](#219246) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Sebastián Zaffarano","email":"sebastian.zaffarano@elastic.co"},"sourceCommit":{"committedDate":"2025-04-28T18:18:43Z","message":"[Security Solution] Add new fields to indices metadata (#219246)\n\n## Summary\n\nAdds `_source.mode` and `index.mode` fields to the\n[TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44)\nEBT event:\n\n```typescript\nexport interface IndexSettings {\n index_name: string;\n index_mode?: string;\n default_pipeline?: string;\n final_pipeline?: string;\n source_mode?: string;\n}\n```\nReferences:\n- #194004 Initial version for the\nupdated EBT event.\n- #213822 This is a Similar PR\nupdating the feature.\n\nThe goal is to enhance the index settings EBT event with two new fields\nto enrich the telemetry information we collect about the indices used by\nthe security integrations.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n\n---------\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"8c596d32010ad96eeac794f1938e686f9cbdb278","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","backport:all-open","ci:build-cloud-image","v9.1.0"],"title":"[Security Solution] Add new fields to indices metadata","number":219246,"url":"https://github.com/elastic/kibana/pull/219246","mergeCommit":{"message":"[Security Solution] Add new fields to indices metadata (#219246)\n\n## Summary\n\nAdds `_source.mode` and `index.mode` fields to the\n[TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44)\nEBT event:\n\n```typescript\nexport interface IndexSettings {\n index_name: string;\n index_mode?: string;\n default_pipeline?: string;\n final_pipeline?: string;\n source_mode?: string;\n}\n```\nReferences:\n- #194004 Initial version for the\nupdated EBT event.\n- #213822 This is a Similar PR\nupdating the feature.\n\nThe goal is to enhance the index settings EBT event with two new fields\nto enrich the telemetry information we collect about the indices used by\nthe security integrations.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n\n---------\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"8c596d32010ad96eeac794f1938e686f9cbdb278"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/219246","number":219246,"mergeCommit":{"message":"[Security Solution] Add new fields to indices metadata (#219246)\n\n## Summary\n\nAdds `_source.mode` and `index.mode` fields to the\n[TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44)\nEBT event:\n\n```typescript\nexport interface IndexSettings {\n index_name: string;\n index_mode?: string;\n default_pipeline?: string;\n final_pipeline?: string;\n source_mode?: string;\n}\n```\nReferences:\n- #194004 Initial version for the\nupdated EBT event.\n- #213822 This is a Similar PR\nupdating the feature.\n\nThe goal is to enhance the index settings EBT event with two new fields\nto enrich the telemetry information we collect about the indices used by\nthe security integrations.\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n\n---------\n\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"8c596d32010ad96eeac794f1938e686f9cbdb278"}},{"url":"https://github.com/elastic/kibana/pull/219499","number":219499,"branch":"9.0","state":"OPEN"},{"url":"https://github.com/elastic/kibana/pull/219503","number":219503,"branch":"8.19","state":"OPEN"}]}] BACKPORT--> --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
akowalska622
pushed a commit
to akowalska622/kibana
that referenced
this pull request
May 29, 2025
## Summary Adds `_source.mode` and `index.mode` fields to the [TELEMETRY_INDEX_SETTINGS_EVENT](https://github.com/elastic/kibana/blob/szaffarano/update-indices-metadata/x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts#L470C14-L470C44) EBT event: ```typescript export interface IndexSettings { index_name: string; index_mode?: string; default_pipeline?: string; final_pipeline?: string; source_mode?: string; } ``` References: - elastic#194004 Initial version for the updated EBT event. - elastic#213822 This is a Similar PR updating the feature. The goal is to enhance the index settings EBT event with two new fields to enrich the telemetry information we collect about the indices used by the security integrations. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
szaffarano
added a commit
that referenced
this pull request
Aug 15, 2025
## Summary This PR introduces a new plugin to relocate the [indices metadata telemetry feature](#194004) out of the `security_solution` plugin, making it available across all cluster setups, not just security clusters. ### Key Changes - **Plugin Location:** `x-pack/platform/plugins/private`. - **Scheduled Task:** Core logic runs as a periodic Kibana task (currently runs frequently for testing; intended to run daily) to collect indices and ILM information. - **EBT:** Collected data is published as EBT documents. Defined events include: * `DATA_STREAM_EVENT` * `INDEX_STATS_EVENTS` * `ILM_STATS_EVENTS` * `ILM_POLICY_EVENTS` See `server/lib/ebt/events.ts` for details. - **Internal Configuration Schema:** Allows tuning query behavior: ```ts export const IndicesMetadataConfigurationSchema = schema.object({ indices_threshold: schema.number(), datastreams_threshold: schema.number(), index_query_size: schema.number(), ilm_stats_query_size: schema.number(), ilm_policy_query_size: schema.number(), }); ``` Default values are zero, meaning no queries run unless explicitly configured. - **Remote Configuration via CDN:** Implements the same manifest mechanism used by `security_solution`: * A signed config manifest (zip with `manifest.json` + `manifest.sig`) is downloaded via CDN. * An `rxjs` timer polls for updates. * Once verified, new config values are applied dynamically. - **Plugin-Level Settings:** Supports configuring the CDN URL and public key to verify signatures: ```ts const pluginConfigSchema = schema.object({ cdnUrl: schema.maybe(schema.string()), publicKey: schema.maybe(schema.string()), }); ``` As a reference, below are stats for the last 90 days for the current implementation inside `security-solution` | event_type | max_doc_size_bytes | avg_doc_size_bytes | |--------------------------------|--------------------|--------------------------| | telemetry_index_settings_event | 2,235,887 | 25,286.12 | | telemetry_ilm_policy_event | 577,148 | 3,390.25 | | telemetry_ilm_stats_event | 2,731,533 | 19,607.14 | | telemetry_index_templates_event| 5,788,583 | 36,688.70 | | telemetry_index_stats_event | 3,829,341 | 55,705.83 | | telemetry_data_stream_event | 6,885,418 | 17,581.92 | --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
kibanamachine
added a commit
to kibanamachine/kibana
that referenced
this pull request
Aug 15, 2025
…12875) ## Summary This PR introduces a new plugin to relocate the [indices metadata telemetry feature](elastic#194004) out of the `security_solution` plugin, making it available across all cluster setups, not just security clusters. ### Key Changes - **Plugin Location:** `x-pack/platform/plugins/private`. - **Scheduled Task:** Core logic runs as a periodic Kibana task (currently runs frequently for testing; intended to run daily) to collect indices and ILM information. - **EBT:** Collected data is published as EBT documents. Defined events include: * `DATA_STREAM_EVENT` * `INDEX_STATS_EVENTS` * `ILM_STATS_EVENTS` * `ILM_POLICY_EVENTS` See `server/lib/ebt/events.ts` for details. - **Internal Configuration Schema:** Allows tuning query behavior: ```ts export const IndicesMetadataConfigurationSchema = schema.object({ indices_threshold: schema.number(), datastreams_threshold: schema.number(), index_query_size: schema.number(), ilm_stats_query_size: schema.number(), ilm_policy_query_size: schema.number(), }); ``` Default values are zero, meaning no queries run unless explicitly configured. - **Remote Configuration via CDN:** Implements the same manifest mechanism used by `security_solution`: * A signed config manifest (zip with `manifest.json` + `manifest.sig`) is downloaded via CDN. * An `rxjs` timer polls for updates. * Once verified, new config values are applied dynamically. - **Plugin-Level Settings:** Supports configuring the CDN URL and public key to verify signatures: ```ts const pluginConfigSchema = schema.object({ cdnUrl: schema.maybe(schema.string()), publicKey: schema.maybe(schema.string()), }); ``` As a reference, below are stats for the last 90 days for the current implementation inside `security-solution` | event_type | max_doc_size_bytes | avg_doc_size_bytes | |--------------------------------|--------------------|--------------------------| | telemetry_index_settings_event | 2,235,887 | 25,286.12 | | telemetry_ilm_policy_event | 577,148 | 3,390.25 | | telemetry_ilm_stats_event | 2,731,533 | 19,607.14 | | telemetry_index_templates_event| 5,788,583 | 36,688.70 | | telemetry_index_stats_event | 3,829,341 | 55,705.83 | | telemetry_data_stream_event | 6,885,418 | 17,581.92 | --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit 92acdae)
szaffarano
added a commit
to szaffarano/kibana
that referenced
this pull request
Aug 15, 2025
…12875) ## Summary This PR introduces a new plugin to relocate the [indices metadata telemetry feature](elastic#194004) out of the `security_solution` plugin, making it available across all cluster setups, not just security clusters. ### Key Changes - **Plugin Location:** `x-pack/platform/plugins/private`. - **Scheduled Task:** Core logic runs as a periodic Kibana task (currently runs frequently for testing; intended to run daily) to collect indices and ILM information. - **EBT:** Collected data is published as EBT documents. Defined events include: * `DATA_STREAM_EVENT` * `INDEX_STATS_EVENTS` * `ILM_STATS_EVENTS` * `ILM_POLICY_EVENTS` See `server/lib/ebt/events.ts` for details. - **Internal Configuration Schema:** Allows tuning query behavior: ```ts export const IndicesMetadataConfigurationSchema = schema.object({ indices_threshold: schema.number(), datastreams_threshold: schema.number(), index_query_size: schema.number(), ilm_stats_query_size: schema.number(), ilm_policy_query_size: schema.number(), }); ``` Default values are zero, meaning no queries run unless explicitly configured. - **Remote Configuration via CDN:** Implements the same manifest mechanism used by `security_solution`: * A signed config manifest (zip with `manifest.json` + `manifest.sig`) is downloaded via CDN. * An `rxjs` timer polls for updates. * Once verified, new config values are applied dynamically. - **Plugin-Level Settings:** Supports configuring the CDN URL and public key to verify signatures: ```ts const pluginConfigSchema = schema.object({ cdnUrl: schema.maybe(schema.string()), publicKey: schema.maybe(schema.string()), }); ``` As a reference, below are stats for the last 90 days for the current implementation inside `security-solution` | event_type | max_doc_size_bytes | avg_doc_size_bytes | |--------------------------------|--------------------|--------------------------| | telemetry_index_settings_event | 2,235,887 | 25,286.12 | | telemetry_ilm_policy_event | 577,148 | 3,390.25 | | telemetry_ilm_stats_event | 2,731,533 | 19,607.14 | | telemetry_index_templates_event| 5,788,583 | 36,688.70 | | telemetry_index_stats_event | 3,829,341 | 55,705.83 | | telemetry_data_stream_event | 6,885,418 | 17,581.92 | --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit 92acdae) # Conflicts: # .github/CODEOWNERS # package.json # tsconfig.base.json # yarn.lock
kibanamachine
added a commit
that referenced
this pull request
Aug 15, 2025
…2875) (#231939) # Backport This will backport the following commits from `main` to `9.1`: - [[Telemetry] Move indices metadata out of security solution (#212875)](#212875) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Sebastián Zaffarano","email":"sebastian.zaffarano@elastic.co"},"sourceCommit":{"committedDate":"2025-08-15T12:08:22Z","message":"[Telemetry] Move indices metadata out of security solution (#212875)\n\n## Summary\n\nThis PR introduces a new plugin to relocate the [indices metadata\ntelemetry feature](#194004) out of\nthe `security_solution` plugin, making it available across all cluster\nsetups, not just security clusters.\n\n### Key Changes\n\n- **Plugin Location:** `x-pack/platform/plugins/private`.\n- **Scheduled Task:** Core logic runs as a periodic Kibana task\n(currently runs frequently for testing; intended to run daily) to\ncollect indices and ILM information.\n- **EBT:** Collected data is published as EBT documents. Defined events\ninclude:\n * `DATA_STREAM_EVENT`\n * `INDEX_STATS_EVENTS`\n * `ILM_STATS_EVENTS`\n * `ILM_POLICY_EVENTS`\n See `server/lib/ebt/events.ts` for details.\n- **Internal Configuration Schema:** Allows tuning query behavior:\n ```ts\n export const IndicesMetadataConfigurationSchema = schema.object({\n indices_threshold: schema.number(),\n datastreams_threshold: schema.number(),\n index_query_size: schema.number(),\n ilm_stats_query_size: schema.number(),\n ilm_policy_query_size: schema.number(),\n });\n ```\nDefault values are zero, meaning no queries run unless explicitly\nconfigured.\n- **Remote Configuration via CDN:** Implements the same manifest\nmechanism used by `security_solution`:\n* A signed config manifest (zip with `manifest.json` + `manifest.sig`)\nis downloaded via CDN.\n * An `rxjs` timer polls for updates.\n * Once verified, new config values are applied dynamically.\n- **Plugin-Level Settings:** Supports configuring the CDN URL and public\nkey to verify signatures:\n ```ts\n const pluginConfigSchema = schema.object({\n cdnUrl: schema.maybe(schema.string()),\n publicKey: schema.maybe(schema.string()),\n });\n ```\n\nAs a reference, below are stats for the last 90 days for the current\nimplementation inside `security-solution`\n\n| event_type | max_doc_size_bytes | avg_doc_size_bytes |\n\n|--------------------------------|--------------------|--------------------------|\n| telemetry_index_settings_event | 2,235,887 | 25,286.12 |\n| telemetry_ilm_policy_event | 577,148 | 3,390.25 |\n| telemetry_ilm_stats_event | 2,731,533 | 19,607.14 |\n| telemetry_index_templates_event| 5,788,583 | 36,688.70 |\n| telemetry_index_stats_event | 3,829,341 | 55,705.83 |\n| telemetry_data_stream_event | 6,885,418 | 17,581.92 |\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"92acdae34279a8c96b856a742fe440f0f81dae2b","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","backport:current-major","v9.2.0"],"title":"[Telemetry] Move indices metadata out of security solution","number":212875,"url":"https://github.com/elastic/kibana/pull/212875","mergeCommit":{"message":"[Telemetry] Move indices metadata out of security solution (#212875)\n\n## Summary\n\nThis PR introduces a new plugin to relocate the [indices metadata\ntelemetry feature](#194004) out of\nthe `security_solution` plugin, making it available across all cluster\nsetups, not just security clusters.\n\n### Key Changes\n\n- **Plugin Location:** `x-pack/platform/plugins/private`.\n- **Scheduled Task:** Core logic runs as a periodic Kibana task\n(currently runs frequently for testing; intended to run daily) to\ncollect indices and ILM information.\n- **EBT:** Collected data is published as EBT documents. Defined events\ninclude:\n * `DATA_STREAM_EVENT`\n * `INDEX_STATS_EVENTS`\n * `ILM_STATS_EVENTS`\n * `ILM_POLICY_EVENTS`\n See `server/lib/ebt/events.ts` for details.\n- **Internal Configuration Schema:** Allows tuning query behavior:\n ```ts\n export const IndicesMetadataConfigurationSchema = schema.object({\n indices_threshold: schema.number(),\n datastreams_threshold: schema.number(),\n index_query_size: schema.number(),\n ilm_stats_query_size: schema.number(),\n ilm_policy_query_size: schema.number(),\n });\n ```\nDefault values are zero, meaning no queries run unless explicitly\nconfigured.\n- **Remote Configuration via CDN:** Implements the same manifest\nmechanism used by `security_solution`:\n* A signed config manifest (zip with `manifest.json` + `manifest.sig`)\nis downloaded via CDN.\n * An `rxjs` timer polls for updates.\n * Once verified, new config values are applied dynamically.\n- **Plugin-Level Settings:** Supports configuring the CDN URL and public\nkey to verify signatures:\n ```ts\n const pluginConfigSchema = schema.object({\n cdnUrl: schema.maybe(schema.string()),\n publicKey: schema.maybe(schema.string()),\n });\n ```\n\nAs a reference, below are stats for the last 90 days for the current\nimplementation inside `security-solution`\n\n| event_type | max_doc_size_bytes | avg_doc_size_bytes |\n\n|--------------------------------|--------------------|--------------------------|\n| telemetry_index_settings_event | 2,235,887 | 25,286.12 |\n| telemetry_ilm_policy_event | 577,148 | 3,390.25 |\n| telemetry_ilm_stats_event | 2,731,533 | 19,607.14 |\n| telemetry_index_templates_event| 5,788,583 | 36,688.70 |\n| telemetry_index_stats_event | 3,829,341 | 55,705.83 |\n| telemetry_data_stream_event | 6,885,418 | 17,581.92 |\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"92acdae34279a8c96b856a742fe440f0f81dae2b"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212875","number":212875,"mergeCommit":{"message":"[Telemetry] Move indices metadata out of security solution (#212875)\n\n## Summary\n\nThis PR introduces a new plugin to relocate the [indices metadata\ntelemetry feature](#194004) out of\nthe `security_solution` plugin, making it available across all cluster\nsetups, not just security clusters.\n\n### Key Changes\n\n- **Plugin Location:** `x-pack/platform/plugins/private`.\n- **Scheduled Task:** Core logic runs as a periodic Kibana task\n(currently runs frequently for testing; intended to run daily) to\ncollect indices and ILM information.\n- **EBT:** Collected data is published as EBT documents. Defined events\ninclude:\n * `DATA_STREAM_EVENT`\n * `INDEX_STATS_EVENTS`\n * `ILM_STATS_EVENTS`\n * `ILM_POLICY_EVENTS`\n See `server/lib/ebt/events.ts` for details.\n- **Internal Configuration Schema:** Allows tuning query behavior:\n ```ts\n export const IndicesMetadataConfigurationSchema = schema.object({\n indices_threshold: schema.number(),\n datastreams_threshold: schema.number(),\n index_query_size: schema.number(),\n ilm_stats_query_size: schema.number(),\n ilm_policy_query_size: schema.number(),\n });\n ```\nDefault values are zero, meaning no queries run unless explicitly\nconfigured.\n- **Remote Configuration via CDN:** Implements the same manifest\nmechanism used by `security_solution`:\n* A signed config manifest (zip with `manifest.json` + `manifest.sig`)\nis downloaded via CDN.\n * An `rxjs` timer polls for updates.\n * Once verified, new config values are applied dynamically.\n- **Plugin-Level Settings:** Supports configuring the CDN URL and public\nkey to verify signatures:\n ```ts\n const pluginConfigSchema = schema.object({\n cdnUrl: schema.maybe(schema.string()),\n publicKey: schema.maybe(schema.string()),\n });\n ```\n\nAs a reference, below are stats for the last 90 days for the current\nimplementation inside `security-solution`\n\n| event_type | max_doc_size_bytes | avg_doc_size_bytes |\n\n|--------------------------------|--------------------|--------------------------|\n| telemetry_index_settings_event | 2,235,887 | 25,286.12 |\n| telemetry_ilm_policy_event | 577,148 | 3,390.25 |\n| telemetry_ilm_stats_event | 2,731,533 | 19,607.14 |\n| telemetry_index_templates_event| 5,788,583 | 36,688.70 |\n| telemetry_index_stats_event | 3,829,341 | 55,705.83 |\n| telemetry_data_stream_event | 6,885,418 | 17,581.92 |\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"92acdae34279a8c96b856a742fe440f0f81dae2b"}}]}] BACKPORT--> Co-authored-by: Sebastián Zaffarano <sebastian.zaffarano@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
szaffarano
added a commit
that referenced
this pull request
Aug 15, 2025
…2875) (#231941) # Backport This will backport the following commits from `main` to `9.0`: - [[Telemetry] Move indices metadata out of security solution (#212875)](#212875) <!--- Backport version: 10.0.1 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Sebastián Zaffarano","email":"sebastian.zaffarano@elastic.co"},"sourceCommit":{"committedDate":"2025-08-15T12:08:22Z","message":"[Telemetry] Move indices metadata out of security solution (#212875)\n\n## Summary\n\nThis PR introduces a new plugin to relocate the [indices metadata\ntelemetry feature](#194004) out of\nthe `security_solution` plugin, making it available across all cluster\nsetups, not just security clusters.\n\n### Key Changes\n\n- **Plugin Location:** `x-pack/platform/plugins/private`.\n- **Scheduled Task:** Core logic runs as a periodic Kibana task\n(currently runs frequently for testing; intended to run daily) to\ncollect indices and ILM information.\n- **EBT:** Collected data is published as EBT documents. Defined events\ninclude:\n * `DATA_STREAM_EVENT`\n * `INDEX_STATS_EVENTS`\n * `ILM_STATS_EVENTS`\n * `ILM_POLICY_EVENTS`\n See `server/lib/ebt/events.ts` for details.\n- **Internal Configuration Schema:** Allows tuning query behavior:\n ```ts\n export const IndicesMetadataConfigurationSchema = schema.object({\n indices_threshold: schema.number(),\n datastreams_threshold: schema.number(),\n index_query_size: schema.number(),\n ilm_stats_query_size: schema.number(),\n ilm_policy_query_size: schema.number(),\n });\n ```\nDefault values are zero, meaning no queries run unless explicitly\nconfigured.\n- **Remote Configuration via CDN:** Implements the same manifest\nmechanism used by `security_solution`:\n* A signed config manifest (zip with `manifest.json` + `manifest.sig`)\nis downloaded via CDN.\n * An `rxjs` timer polls for updates.\n * Once verified, new config values are applied dynamically.\n- **Plugin-Level Settings:** Supports configuring the CDN URL and public\nkey to verify signatures:\n ```ts\n const pluginConfigSchema = schema.object({\n cdnUrl: schema.maybe(schema.string()),\n publicKey: schema.maybe(schema.string()),\n });\n ```\n\nAs a reference, below are stats for the last 90 days for the current\nimplementation inside `security-solution`\n\n| event_type | max_doc_size_bytes | avg_doc_size_bytes |\n\n|--------------------------------|--------------------|--------------------------|\n| telemetry_index_settings_event | 2,235,887 | 25,286.12 |\n| telemetry_ilm_policy_event | 577,148 | 3,390.25 |\n| telemetry_ilm_stats_event | 2,731,533 | 19,607.14 |\n| telemetry_index_templates_event| 5,788,583 | 36,688.70 |\n| telemetry_index_stats_event | 3,829,341 | 55,705.83 |\n| telemetry_data_stream_event | 6,885,418 | 17,581.92 |\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"92acdae34279a8c96b856a742fe440f0f81dae2b","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","backport:current-major","v9.2.0"],"title":"[Telemetry] Move indices metadata out of security solution","number":212875,"url":"https://github.com/elastic/kibana/pull/212875","mergeCommit":{"message":"[Telemetry] Move indices metadata out of security solution (#212875)\n\n## Summary\n\nThis PR introduces a new plugin to relocate the [indices metadata\ntelemetry feature](#194004) out of\nthe `security_solution` plugin, making it available across all cluster\nsetups, not just security clusters.\n\n### Key Changes\n\n- **Plugin Location:** `x-pack/platform/plugins/private`.\n- **Scheduled Task:** Core logic runs as a periodic Kibana task\n(currently runs frequently for testing; intended to run daily) to\ncollect indices and ILM information.\n- **EBT:** Collected data is published as EBT documents. Defined events\ninclude:\n * `DATA_STREAM_EVENT`\n * `INDEX_STATS_EVENTS`\n * `ILM_STATS_EVENTS`\n * `ILM_POLICY_EVENTS`\n See `server/lib/ebt/events.ts` for details.\n- **Internal Configuration Schema:** Allows tuning query behavior:\n ```ts\n export const IndicesMetadataConfigurationSchema = schema.object({\n indices_threshold: schema.number(),\n datastreams_threshold: schema.number(),\n index_query_size: schema.number(),\n ilm_stats_query_size: schema.number(),\n ilm_policy_query_size: schema.number(),\n });\n ```\nDefault values are zero, meaning no queries run unless explicitly\nconfigured.\n- **Remote Configuration via CDN:** Implements the same manifest\nmechanism used by `security_solution`:\n* A signed config manifest (zip with `manifest.json` + `manifest.sig`)\nis downloaded via CDN.\n * An `rxjs` timer polls for updates.\n * Once verified, new config values are applied dynamically.\n- **Plugin-Level Settings:** Supports configuring the CDN URL and public\nkey to verify signatures:\n ```ts\n const pluginConfigSchema = schema.object({\n cdnUrl: schema.maybe(schema.string()),\n publicKey: schema.maybe(schema.string()),\n });\n ```\n\nAs a reference, below are stats for the last 90 days for the current\nimplementation inside `security-solution`\n\n| event_type | max_doc_size_bytes | avg_doc_size_bytes |\n\n|--------------------------------|--------------------|--------------------------|\n| telemetry_index_settings_event | 2,235,887 | 25,286.12 |\n| telemetry_ilm_policy_event | 577,148 | 3,390.25 |\n| telemetry_ilm_stats_event | 2,731,533 | 19,607.14 |\n| telemetry_index_templates_event| 5,788,583 | 36,688.70 |\n| telemetry_index_stats_event | 3,829,341 | 55,705.83 |\n| telemetry_data_stream_event | 6,885,418 | 17,581.92 |\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"92acdae34279a8c96b856a742fe440f0f81dae2b"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212875","number":212875,"mergeCommit":{"message":"[Telemetry] Move indices metadata out of security solution (#212875)\n\n## Summary\n\nThis PR introduces a new plugin to relocate the [indices metadata\ntelemetry feature](#194004) out of\nthe `security_solution` plugin, making it available across all cluster\nsetups, not just security clusters.\n\n### Key Changes\n\n- **Plugin Location:** `x-pack/platform/plugins/private`.\n- **Scheduled Task:** Core logic runs as a periodic Kibana task\n(currently runs frequently for testing; intended to run daily) to\ncollect indices and ILM information.\n- **EBT:** Collected data is published as EBT documents. Defined events\ninclude:\n * `DATA_STREAM_EVENT`\n * `INDEX_STATS_EVENTS`\n * `ILM_STATS_EVENTS`\n * `ILM_POLICY_EVENTS`\n See `server/lib/ebt/events.ts` for details.\n- **Internal Configuration Schema:** Allows tuning query behavior:\n ```ts\n export const IndicesMetadataConfigurationSchema = schema.object({\n indices_threshold: schema.number(),\n datastreams_threshold: schema.number(),\n index_query_size: schema.number(),\n ilm_stats_query_size: schema.number(),\n ilm_policy_query_size: schema.number(),\n });\n ```\nDefault values are zero, meaning no queries run unless explicitly\nconfigured.\n- **Remote Configuration via CDN:** Implements the same manifest\nmechanism used by `security_solution`:\n* A signed config manifest (zip with `manifest.json` + `manifest.sig`)\nis downloaded via CDN.\n * An `rxjs` timer polls for updates.\n * Once verified, new config values are applied dynamically.\n- **Plugin-Level Settings:** Supports configuring the CDN URL and public\nkey to verify signatures:\n ```ts\n const pluginConfigSchema = schema.object({\n cdnUrl: schema.maybe(schema.string()),\n publicKey: schema.maybe(schema.string()),\n });\n ```\n\nAs a reference, below are stats for the last 90 days for the current\nimplementation inside `security-solution`\n\n| event_type | max_doc_size_bytes | avg_doc_size_bytes |\n\n|--------------------------------|--------------------|--------------------------|\n| telemetry_index_settings_event | 2,235,887 | 25,286.12 |\n| telemetry_ilm_policy_event | 577,148 | 3,390.25 |\n| telemetry_ilm_stats_event | 2,731,533 | 19,607.14 |\n| telemetry_index_templates_event| 5,788,583 | 36,688.70 |\n| telemetry_index_stats_event | 3,829,341 | 55,705.83 |\n| telemetry_data_stream_event | 6,885,418 | 17,581.92 |\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"92acdae34279a8c96b856a742fe440f0f81dae2b"}},{"url":"https://github.com/elastic/kibana/pull/231939","number":231939,"branch":"9.1","state":"OPEN"}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
NicholasPeretti
pushed a commit
to NicholasPeretti/kibana
that referenced
this pull request
Aug 18, 2025
…12875) ## Summary This PR introduces a new plugin to relocate the [indices metadata telemetry feature](elastic#194004) out of the `security_solution` plugin, making it available across all cluster setups, not just security clusters. ### Key Changes - **Plugin Location:** `x-pack/platform/plugins/private`. - **Scheduled Task:** Core logic runs as a periodic Kibana task (currently runs frequently for testing; intended to run daily) to collect indices and ILM information. - **EBT:** Collected data is published as EBT documents. Defined events include: * `DATA_STREAM_EVENT` * `INDEX_STATS_EVENTS` * `ILM_STATS_EVENTS` * `ILM_POLICY_EVENTS` See `server/lib/ebt/events.ts` for details. - **Internal Configuration Schema:** Allows tuning query behavior: ```ts export const IndicesMetadataConfigurationSchema = schema.object({ indices_threshold: schema.number(), datastreams_threshold: schema.number(), index_query_size: schema.number(), ilm_stats_query_size: schema.number(), ilm_policy_query_size: schema.number(), }); ``` Default values are zero, meaning no queries run unless explicitly configured. - **Remote Configuration via CDN:** Implements the same manifest mechanism used by `security_solution`: * A signed config manifest (zip with `manifest.json` + `manifest.sig`) is downloaded via CDN. * An `rxjs` timer polls for updates. * Once verified, new config values are applied dynamically. - **Plugin-Level Settings:** Supports configuring the CDN URL and public key to verify signatures: ```ts const pluginConfigSchema = schema.object({ cdnUrl: schema.maybe(schema.string()), publicKey: schema.maybe(schema.string()), }); ``` As a reference, below are stats for the last 90 days for the current implementation inside `security-solution` | event_type | max_doc_size_bytes | avg_doc_size_bytes | |--------------------------------|--------------------|--------------------------| | telemetry_index_settings_event | 2,235,887 | 25,286.12 | | telemetry_ilm_policy_event | 577,148 | 3,390.25 | | telemetry_ilm_stats_event | 2,731,533 | 19,607.14 | | telemetry_index_templates_event| 5,788,583 | 36,688.70 | | telemetry_index_stats_event | 3,829,341 | 55,705.83 | | telemetry_data_stream_event | 6,885,418 | 17,581.92 | --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
qn895
pushed a commit
to qn895/kibana
that referenced
this pull request
Aug 26, 2025
…12875) ## Summary This PR introduces a new plugin to relocate the [indices metadata telemetry feature](elastic#194004) out of the `security_solution` plugin, making it available across all cluster setups, not just security clusters. ### Key Changes - **Plugin Location:** `x-pack/platform/plugins/private`. - **Scheduled Task:** Core logic runs as a periodic Kibana task (currently runs frequently for testing; intended to run daily) to collect indices and ILM information. - **EBT:** Collected data is published as EBT documents. Defined events include: * `DATA_STREAM_EVENT` * `INDEX_STATS_EVENTS` * `ILM_STATS_EVENTS` * `ILM_POLICY_EVENTS` See `server/lib/ebt/events.ts` for details. - **Internal Configuration Schema:** Allows tuning query behavior: ```ts export const IndicesMetadataConfigurationSchema = schema.object({ indices_threshold: schema.number(), datastreams_threshold: schema.number(), index_query_size: schema.number(), ilm_stats_query_size: schema.number(), ilm_policy_query_size: schema.number(), }); ``` Default values are zero, meaning no queries run unless explicitly configured. - **Remote Configuration via CDN:** Implements the same manifest mechanism used by `security_solution`: * A signed config manifest (zip with `manifest.json` + `manifest.sig`) is downloaded via CDN. * An `rxjs` timer polls for updates. * Once verified, new config values are applied dynamically. - **Plugin-Level Settings:** Supports configuring the CDN URL and public key to verify signatures: ```ts const pluginConfigSchema = schema.object({ cdnUrl: schema.maybe(schema.string()), publicKey: schema.maybe(schema.string()), }); ``` As a reference, below are stats for the last 90 days for the current implementation inside `security-solution` | event_type | max_doc_size_bytes | avg_doc_size_bytes | |--------------------------------|--------------------|--------------------------| | telemetry_index_settings_event | 2,235,887 | 25,286.12 | | telemetry_ilm_policy_event | 577,148 | 3,390.25 | | telemetry_ilm_stats_event | 2,731,533 | 19,607.14 | | telemetry_index_templates_event| 5,788,583 | 36,688.70 | | telemetry_index_stats_event | 3,829,341 | 55,705.83 | | telemetry_data_stream_event | 6,885,418 | 17,581.92 | --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements a security_solution task scheduled to run once a day to collect the following information:
The task allows a runtime configuration to limit the number of indices and data streams to analyze or event to disable the feature entirely.
Once the data is gathered, the task sends it as EBT events.