[AI Assistant] Use semantic_text for internal knowledge base#186499
Merged
sorenlouv merged 34 commits intoelastic:mainfrom Nov 14, 2024
Merged
[AI Assistant] Use semantic_text for internal knowledge base#186499sorenlouv merged 34 commits intoelastic:mainfrom
semantic_text for internal knowledge base#186499sorenlouv merged 34 commits intoelastic:mainfrom
Conversation
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
…xt-internal-kb # Conflicts: # x-pack/plugins/observability_solution/observability_ai_assistant/server/service/client/index.ts # x-pack/plugins/observability_solution/observability_ai_assistant/server/service/index.ts # x-pack/plugins/observability_solution/observability_ai_assistant/server/service/knowledge_base_service/index.ts
…xt-internal-kb # Conflicts: # x-pack/plugins/observability_solution/observability_ai_assistant/server/service/knowledge_base_service/index.ts
sorenlouv
force-pushed
the
use-semantic-text-internal-kb
branch
from
9b9d65a to
6987fa5
Compare
botelastic
bot
added
ci:project-deploy-observability
💔 Build FailedFailed CI StepsHistoryTo update your PR or re-run it, just comment with: |
…xt-internal-kb # Conflicts: # x-pack/plugins/observability_solution/observability_ai_assistant/server/routes/knowledge_base/route.ts # x-pack/plugins/observability_solution/observability_ai_assistant/server/service/knowledge_base_service/index.ts
sorenlouv
commented
Aug 29, 2024
...bservability_solution/observability_ai_assistant/server/service/create_inference_endpoint.ts
Outdated
Show resolved
Hide resolved
…xt-internal-kb # Conflicts: # x-pack/packages/kbn-ai-assistant/src/chat/welcome_message_knowledge_base_setup_error_panel.tsx # x-pack/packages/kbn-ai-assistant/src/hooks/use_knowledge_base.tsx # x-pack/plugins/observability_solution/observability_ai_assistant/server/plugin.ts # x-pack/plugins/observability_solution/observability_ai_assistant/server/routes/knowledge_base/route.ts # x-pack/plugins/observability_solution/observability_ai_assistant/server/service/index.ts # x-pack/plugins/observability_solution/observability_ai_assistant/server/service/knowledge_base_service/index.ts
sorenlouv
force-pushed
the
use-semantic-text-internal-kb
branch
from
db31e60 to
035f911
Compare
sorenlouv
added a commit
to sorenlouv/kibana
that referenced
this pull request
Nov 14, 2024
…ic#186499) Closes elastic/obs-ai-team#162 Closes elastic#192757 This replaces the ML inference pipeline with `semantic_text` and adds a migration task that runs automatically when Kibana starts. Blocked by: - elastic/elasticsearch#110027 - elastic/elasticsearch#110033 - elastic/ml-team#1298 (cherry picked from commit 671ff30) # Conflicts: # x-pack/plugins/translations/translations/zh-CN.json
sorenlouv
added a commit
that referenced
this pull request
Nov 14, 2024
…186499) (#200243) # Backport This will backport the following commits from `main` to `8.x`: - [[AI Assistant] Use `semantic_text` for internal knowledge base (#186499)](#186499) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Søren Louv-Jansen","email":"soren.louv@elastic.co"},"sourceCommit":{"committedDate":"2024-11-14T12:30:13Z","message":"[AI Assistant] Use `semantic_text` for internal knowledge base (#186499)\n\nCloses elastic/obs-ai-team#162 \r\nCloses https://github.com/elastic/kibana/issues/192757\r\n\r\nThis replaces the ML inference pipeline with `semantic_text` and adds a\r\nmigration task that runs automatically when Kibana starts.\r\n\r\nBlocked by:\r\n - https://github.com/elastic/elasticsearch/pull/110027\r\n - https://github.com/elastic/elasticsearch/pull/110033\r\n - https://github.com/elastic/ml-team/issues/1298","sha":"671ff30516f2ca302962efffae7585dcd7ddfce9","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","v9.0.0","backport:prev-minor","Team:Obs AI Assistant","ci:project-deploy-observability","8.16 candidate"],"number":186499,"url":"https://github.com/elastic/kibana/pull/186499","mergeCommit":{"message":"[AI Assistant] Use `semantic_text` for internal knowledge base (#186499)\n\nCloses elastic/obs-ai-team#162 \r\nCloses https://github.com/elastic/kibana/issues/192757\r\n\r\nThis replaces the ML inference pipeline with `semantic_text` and adds a\r\nmigration task that runs automatically when Kibana starts.\r\n\r\nBlocked by:\r\n - https://github.com/elastic/elasticsearch/pull/110027\r\n - https://github.com/elastic/elasticsearch/pull/110033\r\n - https://github.com/elastic/ml-team/issues/1298","sha":"671ff30516f2ca302962efffae7585dcd7ddfce9"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/186499","number":186499,"mergeCommit":{"message":"[AI Assistant] Use `semantic_text` for internal knowledge base (#186499)\n\nCloses elastic/obs-ai-team#162 \r\nCloses https://github.com/elastic/kibana/issues/192757\r\n\r\nThis replaces the ML inference pipeline with `semantic_text` and adds a\r\nmigration task that runs automatically when Kibana starts.\r\n\r\nBlocked by:\r\n - https://github.com/elastic/elasticsearch/pull/110027\r\n - https://github.com/elastic/elasticsearch/pull/110033\r\n - https://github.com/elastic/ml-team/issues/1298","sha":"671ff30516f2ca302962efffae7585dcd7ddfce9"}}]}] BACKPORT-->
sorenlouv
added a commit
that referenced
this pull request
Nov 15, 2024
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Nov 15, 2024
Follow-up to elastic#186499 (cherry picked from commit 2296a79)
sorenlouv
added a commit
that referenced
this pull request
Nov 15, 2024
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Nov 15, 2024
Follow up to elastic#186499 This removes `modelId` from the config file. We don't need it anymore since it can be supplied as a url param. Currently it's only needed in the setup route (`POST /internal/observability_ai_assistant/kb/setup`) (cherry picked from commit 875313e)
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Nov 18, 2024
…ic#186499) Closes elastic/obs-ai-team#162 Closes elastic#192757 This replaces the ML inference pipeline with `semantic_text` and adds a migration task that runs automatically when Kibana starts. Blocked by: - elastic/elasticsearch#110027 - elastic/elasticsearch#110033 - elastic/ml-team#1298
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Nov 18, 2024
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Nov 18, 2024
Follow up to elastic#186499 This removes `modelId` from the config file. We don't need it anymore since it can be supplied as a url param. Currently it's only needed in the setup route (`POST /internal/observability_ai_assistant/kb/setup`)
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Nov 18, 2024
…ic#186499) Closes elastic/obs-ai-team#162 Closes elastic#192757 This replaces the ML inference pipeline with `semantic_text` and adds a migration task that runs automatically when Kibana starts. Blocked by: - elastic/elasticsearch#110027 - elastic/elasticsearch#110033 - elastic/ml-team#1298
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Nov 18, 2024
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Nov 18, 2024
Follow up to elastic#186499 This removes `modelId` from the config file. We don't need it anymore since it can be supplied as a url param. Currently it's only needed in the setup route (`POST /internal/observability_ai_assistant/kb/setup`)
This was referenced Nov 21, 2024
sorenlouv
added a commit
that referenced
this pull request
Dec 3, 2024
Currently the knowledge base creates index assets (index templates, index components) lazily when the user interacts with the assistant. This prevents running the semantic text migrations (added in #186499) when Kibana starts because the mappings have not yet been updated. Additionally, this PR also increases `min_number_of_allocations` to 1 to ensure at least one ML node is available at all times.
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Dec 3, 2024
Currently the knowledge base creates index assets (index templates, index components) lazily when the user interacts with the assistant. This prevents running the semantic text migrations (added in elastic#186499) when Kibana starts because the mappings have not yet been updated. Additionally, this PR also increases `min_number_of_allocations` to 1 to ensure at least one ML node is available at all times. (cherry picked from commit b217f1a)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Dec 3, 2024
Currently the knowledge base creates index assets (index templates, index components) lazily when the user interacts with the assistant. This prevents running the semantic text migrations (added in elastic#186499) when Kibana starts because the mappings have not yet been updated. Additionally, this PR also increases `min_number_of_allocations` to 1 to ensure at least one ML node is available at all times. (cherry picked from commit b217f1a)
kibanamachine
added a commit
that referenced
this pull request
Dec 3, 2024
…#202727) # Backport This will backport the following commits from `main` to `8.17`: - [[Obs AI Assistant] Perform index creation at startup (#201362)](#201362) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Søren Louv-Jansen","email":"soren.louv@elastic.co"},"sourceCommit":{"committedDate":"2024-12-03T13:48:19Z","message":"[Obs AI Assistant] Perform index creation at startup (#201362)\n\nCurrently the knowledge base creates index assets (index templates, index components) lazily when the user interacts with the assistant. This prevents running the semantic text migrations (added in #186499) when Kibana starts because the mappings have not yet been updated.\r\n\r\nAdditionally, this PR also increases `min_number_of_allocations` to 1 to\r\nensure at least one ML node is available at all times.","sha":"b217f1acbdce4d9c0288c87e9afa470038cf6557","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","Team:Obs AI Assistant","ci:project-deploy-observability","v8.17.0"],"title":"[Obs AI Assistant] Perform index creation at startup","number":201362,"url":"https://github.com/elastic/kibana/pull/201362","mergeCommit":{"message":"[Obs AI Assistant] Perform index creation at startup (#201362)\n\nCurrently the knowledge base creates index assets (index templates, index components) lazily when the user interacts with the assistant. This prevents running the semantic text migrations (added in #186499) when Kibana starts because the mappings have not yet been updated.\r\n\r\nAdditionally, this PR also increases `min_number_of_allocations` to 1 to\r\nensure at least one ML node is available at all times.","sha":"b217f1acbdce4d9c0288c87e9afa470038cf6557"}},"sourceBranch":"main","suggestedTargetBranches":["8.17"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/201362","number":201362,"mergeCommit":{"message":"[Obs AI Assistant] Perform index creation at startup (#201362)\n\nCurrently the knowledge base creates index assets (index templates, index components) lazily when the user interacts with the assistant. This prevents running the semantic text migrations (added in #186499) when Kibana starts because the mappings have not yet been updated.\r\n\r\nAdditionally, this PR also increases `min_number_of_allocations` to 1 to\r\nensure at least one ML node is available at all times.","sha":"b217f1acbdce4d9c0288c87e9afa470038cf6557"}},{"branch":"8.17","label":"v8.17.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Søren Louv-Jansen <soren.louv@elastic.co>
kibanamachine
added a commit
that referenced
this pull request
Dec 3, 2024
…202728) # Backport This will backport the following commits from `main` to `8.x`: - [[Obs AI Assistant] Perform index creation at startup (#201362)](#201362) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Søren Louv-Jansen","email":"soren.louv@elastic.co"},"sourceCommit":{"committedDate":"2024-12-03T13:48:19Z","message":"[Obs AI Assistant] Perform index creation at startup (#201362)\n\nCurrently the knowledge base creates index assets (index templates, index components) lazily when the user interacts with the assistant. This prevents running the semantic text migrations (added in #186499) when Kibana starts because the mappings have not yet been updated.\r\n\r\nAdditionally, this PR also increases `min_number_of_allocations` to 1 to\r\nensure at least one ML node is available at all times.","sha":"b217f1acbdce4d9c0288c87e9afa470038cf6557","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","Team:Obs AI Assistant","ci:project-deploy-observability","v8.17.0"],"title":"[Obs AI Assistant] Perform index creation at startup","number":201362,"url":"https://github.com/elastic/kibana/pull/201362","mergeCommit":{"message":"[Obs AI Assistant] Perform index creation at startup (#201362)\n\nCurrently the knowledge base creates index assets (index templates, index components) lazily when the user interacts with the assistant. This prevents running the semantic text migrations (added in #186499) when Kibana starts because the mappings have not yet been updated.\r\n\r\nAdditionally, this PR also increases `min_number_of_allocations` to 1 to\r\nensure at least one ML node is available at all times.","sha":"b217f1acbdce4d9c0288c87e9afa470038cf6557"}},"sourceBranch":"main","suggestedTargetBranches":["8.17"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/201362","number":201362,"mergeCommit":{"message":"[Obs AI Assistant] Perform index creation at startup (#201362)\n\nCurrently the knowledge base creates index assets (index templates, index components) lazily when the user interacts with the assistant. This prevents running the semantic text migrations (added in #186499) when Kibana starts because the mappings have not yet been updated.\r\n\r\nAdditionally, this PR also increases `min_number_of_allocations` to 1 to\r\nensure at least one ML node is available at all times.","sha":"b217f1acbdce4d9c0288c87e9afa470038cf6557"}},{"branch":"8.17","label":"v8.17.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Søren Louv-Jansen <soren.louv@elastic.co>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Dec 12, 2024
Currently the knowledge base creates index assets (index templates, index components) lazily when the user interacts with the assistant. This prevents running the semantic text migrations (added in elastic#186499) when Kibana starts because the mappings have not yet been updated. Additionally, this PR also increases `min_number_of_allocations` to 1 to ensure at least one ML node is available at all times.
This was referenced Jan 16, 2025
sorenlouv
added a commit
that referenced
this pull request
May 28, 2025
## Background The semantic text migration was added in 8.17 along with the move to semantic_text field (#186499). ## Suggestion We should remove the semantic_text migration starting in 9.1. Reasons: - The migration has recently caused severe disruption on a number of clusters (#220255). - The migration may no longer be needed. Users are required to upgrade Elasticsearch to 8.18 in order to update to 9.0, or 8.19 to upgrade to 9.1. This could mean that users are guaranteed to have run the migration previously, when upgrading Kibana to 9.1 (needs to be double checked with Kibana folks) --------- Co-authored-by: Viduni Wickramarachchi <viduni.ushanka@gmail.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
akowalska622
pushed a commit
to akowalska622/kibana
that referenced
this pull request
May 29, 2025
## Background The semantic text migration was added in 8.17 along with the move to semantic_text field (elastic#186499). ## Suggestion We should remove the semantic_text migration starting in 9.1. Reasons: - The migration has recently caused severe disruption on a number of clusters (elastic#220255). - The migration may no longer be needed. Users are required to upgrade Elasticsearch to 8.18 in order to update to 9.0, or 8.19 to upgrade to 9.1. This could mean that users are guaranteed to have run the migration previously, when upgrading Kibana to 9.1 (needs to be double checked with Kibana folks) --------- Co-authored-by: Viduni Wickramarachchi <viduni.ushanka@gmail.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
zacharyparikh
pushed a commit
to zacharyparikh/kibana
that referenced
this pull request
Jun 4, 2025
## Background The semantic text migration was added in 8.17 along with the move to semantic_text field (elastic#186499). ## Suggestion We should remove the semantic_text migration starting in 9.1. Reasons: - The migration has recently caused severe disruption on a number of clusters (elastic#220255). - The migration may no longer be needed. Users are required to upgrade Elasticsearch to 8.18 in order to update to 9.0, or 8.19 to upgrade to 9.1. This could mean that users are guaranteed to have run the migration previously, when upgrading Kibana to 9.1 (needs to be double checked with Kibana folks) --------- Co-authored-by: Viduni Wickramarachchi <viduni.ushanka@gmail.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes https://github.com/elastic/obs-ai-assistant-team/issues/162
Closes #192757
This replaces the ML inference pipeline with
semantic_textand adds a migration task that runs automatically when Kibana starts.Dependencies: