remove old maps stuff by spalger · Pull Request #13657 · elastic/kibana

spalger · 2017-08-22T23:38:22Z

This is no longer relevant since we have updated the tile service to use Kartotherian.

thomasneirynck · 2017-08-23T04:01:09Z

i had no idea this folder was there still. 👍 I'd set it to exclude in my IDE, so it hadn't showed up in forever..

jenkins, test this

epixa

LGTM

… and interface (elastic#13657)

…ce (elastic#13657)

@timestamp

… and interface (#13657) (#232288) ## Summary Epic: elastic/security-team#12768 Meta: elastic/security-team#13657 RFC: [internal link](https://docs.google.com/document/d/13jAJ5Q3_At_zAuwKjvpYehnM5uzKJSwZIZ1aJJNAf5k) With these changes we update the conversation summary schema to accommodate new fields to allow conversation summarization and past conversation search. Also, as part of these changes, the OLD (unused) summary fields are removed from the APIs. ### Mapping changes The conversations index mapping already has a `summary` field which is an object that looks like: <details> <summary>OLD summary schema</summary> ```json "summary": { "properties": { "@timestamp": { "type": "date" }, "confidence": { "type": "keyword" }, "content": { "type": "text" }, "public": { "type": "boolean" } } } ``` </details> To be able to summarize conversations and semantically search through existing summaries, the new fields (`semantic_content` and `summarized_message_ids`) are added into the mapping: <details> <summary>Updated summary schema</summary> ```json "summary": { "properties": { "@timestamp": { "type": "date" }, "confidence": { "type": "keyword" }, "content": { "type": "text" }, "public": { "type": "boolean" }, "semantic_content": { "type": "semantic_text", "inference_id": ".elser-2-elasticsearch" }, "summarized_message_ids": { "type": "keyword", "array": true } } } ``` </details> ### New fields description `semantic_content` field will be used to store conversation summary and allows semantical search through the ELSER v2 or E5 models. `summarized_message_ids` field will contain a list of all messages that are summarized and part of the summary stored within the `semantic_content` field. ### Legacy fields and API interface changes There are bunch of fields that were never used and won't be supported or used in future - `summary.confidence`, `summary.content` and `summary.public`. After discussion with @YulNaumenko and @elastic/security-generative-ai, this fields will be marked as legacy on the mappings level for compatibility with the installed indices and will be removed on the API level. Previously, we allowed to update `summary.confidence`, `summary.content` and `summary.public` fields via API calls and never used in kibana UI. **NOTE**: Thanks @spong to pointing to [this cluster](https://overview.elastic-cloud.com/app/dashboards#/view/serverless-api-services-http-requests-overview?_g=h@6558260) to see the API usage in production. It shows that within last 90 days, the update conversation API (the only way for users to update conversations and potentially add a summary to it) was used only 41 times which looks low and I believe negligible. <img width="1144" height="423" alt="Screenshot 2025-08-20 at 10 50 35" src="https://github.com/user-attachments/assets/6cb8e1a2-4d9d-44d2-8e66-2de6d8ac74e2" /> From now on, the conversation will have next summary fields on the **API level**: ```typescript interface ConversationSummary { /** * The timestamp summary was updated. */ timestamp: string; /** * Summary text of the conversation over time. */ semanticContent?: string; /** * The list of summarized messages. */ summarizedMessageIds?: string[]; } ``` ### Testing To test, you can use next API calls: <details> <summary>Fetch all existing conversations</summary> This call will fetch all existing conversation. Good for overview of existing conversations and verifying expected summary values. ```curl curl --location 'http://localhost:5601/sbb/api/security_ai_assistant/current_user/conversations/_find' \ --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \ --header 'kbn-xsrf: true' \ --header 'elastic-api-version: 2023-10-31' ``` </details> <details> <summary>Update a conversation</summary> This call will update a conversation and add/update a summary. ```curl curl --location --request PUT 'http://localhost:5601/sbb/api/security_ai_assistant/current_user/conversations/{{CONVERSATION_ID}}' \ --header 'kbn-xsrf: true' \ --header 'elastic-api-version: 2023-10-31' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \ --data '{ "id": "a565baa8-5566-47b2-ab69-807248b2fc46", "summary": { "semanticContent": "Very nice demo semantic content." } }' ``` </details> <details> <summary>Bulk Update existing conversation(s)</summary> This call will update a conversation and add/update a summary. ```curl curl --location 'http://localhost:5601/sbb/internal/elastic_assistant/current_user/conversations/_bulk_action' \ --header 'kbn-xsrf: true' \ --header 'elastic-api-version: 1' \ --header 'x-elastic-internal-origin: Kibana' \ --header 'kbn-version: 9.2.0' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \ --data '{ "update": [ { "id": "{{CONVERSATION_ID}}", "summary": { "semanticContent": "Very nice demo semantic content." } } ] }' ``` </details> Some test cases: 1. Check that if not updated, a new conversation does not have a summary 2. Check that `summary` contains expected value after it has been updated via one of the above APIs 3. Check that we do not return legacy fields (`summary.confidence`, `summary.content` and `summary.public`) even if you add a document with those fields set. You can set legacy fields, either via DevTools or via update APIs from above in previous kibana version. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…ce (#13657) (#232394) ## Summary Epic: elastic/security-team#12768 Meta: elastic/security-team#13657 RFC: [internal link](https://docs.google.com/document/d/13jAJ5Q3_At_zAuwKjvpYehnM5uzKJSwZIZ1aJJNAf5k) With these changes we update the conversation `Message` schema by adding a new field `id`. This will allow us to easily refer messages that are summarized. ### New messages All new messages will be added with the `id` field set. ### Existing messages The `id` field for all existing messages will be `undefined` and will be updated whenever a user updates the conversation: - appends new messages - regenerates a last assistant respond In those case above, all messages within the ongoing conversation will be updated and will have a `id` field set. ### Testing scenarios 1. **Existing messages without `id`** - Create conversation with messages in "pre this PR" code (e.g. main branch) - Checkout code from this PR - Check via Dev Tools that messages within the conversations index do not have `id` field set - Add a new message to one of the conversations - Check that all messages within that conversation are updated and `id` field set for all of them 2. **New messages** - Checkout code from this PR - Create a new chat and add new messages - Check via Dev Tools that all messages within the new conversation have `id` field set --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

@timestamp

… and interface (elastic#13657) (elastic#232288) ## Summary Epic: elastic/security-team#12768 Meta: elastic/security-team#13657 RFC: [internal link](https://docs.google.com/document/d/13jAJ5Q3_At_zAuwKjvpYehnM5uzKJSwZIZ1aJJNAf5k) With these changes we update the conversation summary schema to accommodate new fields to allow conversation summarization and past conversation search. Also, as part of these changes, the OLD (unused) summary fields are removed from the APIs. ### Mapping changes The conversations index mapping already has a `summary` field which is an object that looks like: <details> <summary>OLD summary schema</summary> ```json "summary": { "properties": { "@timestamp": { "type": "date" }, "confidence": { "type": "keyword" }, "content": { "type": "text" }, "public": { "type": "boolean" } } } ``` </details> To be able to summarize conversations and semantically search through existing summaries, the new fields (`semantic_content` and `summarized_message_ids`) are added into the mapping: <details> <summary>Updated summary schema</summary> ```json "summary": { "properties": { "@timestamp": { "type": "date" }, "confidence": { "type": "keyword" }, "content": { "type": "text" }, "public": { "type": "boolean" }, "semantic_content": { "type": "semantic_text", "inference_id": ".elser-2-elasticsearch" }, "summarized_message_ids": { "type": "keyword", "array": true } } } ``` </details> ### New fields description `semantic_content` field will be used to store conversation summary and allows semantical search through the ELSER v2 or E5 models. `summarized_message_ids` field will contain a list of all messages that are summarized and part of the summary stored within the `semantic_content` field. ### Legacy fields and API interface changes There are bunch of fields that were never used and won't be supported or used in future - `summary.confidence`, `summary.content` and `summary.public`. After discussion with @YulNaumenko and @elastic/security-generative-ai, this fields will be marked as legacy on the mappings level for compatibility with the installed indices and will be removed on the API level. Previously, we allowed to update `summary.confidence`, `summary.content` and `summary.public` fields via API calls and never used in kibana UI. **NOTE**: Thanks @spong to pointing to [this cluster](https://overview.elastic-cloud.com/app/dashboards#/view/serverless-api-services-http-requests-overview?_g=h@6558260) to see the API usage in production. It shows that within last 90 days, the update conversation API (the only way for users to update conversations and potentially add a summary to it) was used only 41 times which looks low and I believe negligible. <img width="1144" height="423" alt="Screenshot 2025-08-20 at 10 50 35" src="https://github.com/user-attachments/assets/6cb8e1a2-4d9d-44d2-8e66-2de6d8ac74e2" /> From now on, the conversation will have next summary fields on the **API level**: ```typescript interface ConversationSummary { /** * The timestamp summary was updated. */ timestamp: string; /** * Summary text of the conversation over time. */ semanticContent?: string; /** * The list of summarized messages. */ summarizedMessageIds?: string[]; } ``` ### Testing To test, you can use next API calls: <details> <summary>Fetch all existing conversations</summary> This call will fetch all existing conversation. Good for overview of existing conversations and verifying expected summary values. ```curl curl --location 'http://localhost:5601/sbb/api/security_ai_assistant/current_user/conversations/_find' \ --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \ --header 'kbn-xsrf: true' \ --header 'elastic-api-version: 2023-10-31' ``` </details> <details> <summary>Update a conversation</summary> This call will update a conversation and add/update a summary. ```curl curl --location --request PUT 'http://localhost:5601/sbb/api/security_ai_assistant/current_user/conversations/{{CONVERSATION_ID}}' \ --header 'kbn-xsrf: true' \ --header 'elastic-api-version: 2023-10-31' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \ --data '{ "id": "a565baa8-5566-47b2-ab69-807248b2fc46", "summary": { "semanticContent": "Very nice demo semantic content." } }' ``` </details> <details> <summary>Bulk Update existing conversation(s)</summary> This call will update a conversation and add/update a summary. ```curl curl --location 'http://localhost:5601/sbb/internal/elastic_assistant/current_user/conversations/_bulk_action' \ --header 'kbn-xsrf: true' \ --header 'elastic-api-version: 1' \ --header 'x-elastic-internal-origin: Kibana' \ --header 'kbn-version: 9.2.0' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \ --data '{ "update": [ { "id": "{{CONVERSATION_ID}}", "summary": { "semanticContent": "Very nice demo semantic content." } } ] }' ``` </details> Some test cases: 1. Check that if not updated, a new conversation does not have a summary 2. Check that `summary` contains expected value after it has been updated via one of the above APIs 3. Check that we do not return legacy fields (`summary.confidence`, `summary.content` and `summary.public`) even if you add a document with those fields set. You can set legacy fields, either via DevTools or via update APIs from above in previous kibana version. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

@timestamp

… and interface (elastic#13657) (elastic#232288) ## Summary Epic: elastic/security-team#12768 Meta: elastic/security-team#13657 RFC: [internal link](https://docs.google.com/document/d/13jAJ5Q3_At_zAuwKjvpYehnM5uzKJSwZIZ1aJJNAf5k) With these changes we update the conversation summary schema to accommodate new fields to allow conversation summarization and past conversation search. Also, as part of these changes, the OLD (unused) summary fields are removed from the APIs. ### Mapping changes The conversations index mapping already has a `summary` field which is an object that looks like: <details> <summary>OLD summary schema</summary> ```json "summary": { "properties": { "@timestamp": { "type": "date" }, "confidence": { "type": "keyword" }, "content": { "type": "text" }, "public": { "type": "boolean" } } } ``` </details> To be able to summarize conversations and semantically search through existing summaries, the new fields (`semantic_content` and `summarized_message_ids`) are added into the mapping: <details> <summary>Updated summary schema</summary> ```json "summary": { "properties": { "@timestamp": { "type": "date" }, "confidence": { "type": "keyword" }, "content": { "type": "text" }, "public": { "type": "boolean" }, "semantic_content": { "type": "semantic_text", "inference_id": ".elser-2-elasticsearch" }, "summarized_message_ids": { "type": "keyword", "array": true } } } ``` </details> ### New fields description `semantic_content` field will be used to store conversation summary and allows semantical search through the ELSER v2 or E5 models. `summarized_message_ids` field will contain a list of all messages that are summarized and part of the summary stored within the `semantic_content` field. ### Legacy fields and API interface changes There are bunch of fields that were never used and won't be supported or used in future - `summary.confidence`, `summary.content` and `summary.public`. After discussion with @YulNaumenko and @elastic/security-generative-ai, this fields will be marked as legacy on the mappings level for compatibility with the installed indices and will be removed on the API level. Previously, we allowed to update `summary.confidence`, `summary.content` and `summary.public` fields via API calls and never used in kibana UI. **NOTE**: Thanks @spong to pointing to [this cluster](https://overview.elastic-cloud.com/app/dashboards#/view/serverless-api-services-http-requests-overview?_g=h@6558260) to see the API usage in production. It shows that within last 90 days, the update conversation API (the only way for users to update conversations and potentially add a summary to it) was used only 41 times which looks low and I believe negligible. <img width="1144" height="423" alt="Screenshot 2025-08-20 at 10 50 35" src="https://github.com/user-attachments/assets/6cb8e1a2-4d9d-44d2-8e66-2de6d8ac74e2" /> From now on, the conversation will have next summary fields on the **API level**: ```typescript interface ConversationSummary { /** * The timestamp summary was updated. */ timestamp: string; /** * Summary text of the conversation over time. */ semanticContent?: string; /** * The list of summarized messages. */ summarizedMessageIds?: string[]; } ``` ### Testing To test, you can use next API calls: <details> <summary>Fetch all existing conversations</summary> This call will fetch all existing conversation. Good for overview of existing conversations and verifying expected summary values. ```curl curl --location 'http://localhost:5601/sbb/api/security_ai_assistant/current_user/conversations/_find' \ --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \ --header 'kbn-xsrf: true' \ --header 'elastic-api-version: 2023-10-31' ``` </details> <details> <summary>Update a conversation</summary> This call will update a conversation and add/update a summary. ```curl curl --location --request PUT 'http://localhost:5601/sbb/api/security_ai_assistant/current_user/conversations/{{CONVERSATION_ID}}' \ --header 'kbn-xsrf: true' \ --header 'elastic-api-version: 2023-10-31' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \ --data '{ "id": "a565baa8-5566-47b2-ab69-807248b2fc46", "summary": { "semanticContent": "Very nice demo semantic content." } }' ``` </details> <details> <summary>Bulk Update existing conversation(s)</summary> This call will update a conversation and add/update a summary. ```curl curl --location 'http://localhost:5601/sbb/internal/elastic_assistant/current_user/conversations/_bulk_action' \ --header 'kbn-xsrf: true' \ --header 'elastic-api-version: 1' \ --header 'x-elastic-internal-origin: Kibana' \ --header 'kbn-version: 9.2.0' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \ --data '{ "update": [ { "id": "{{CONVERSATION_ID}}", "summary": { "semanticContent": "Very nice demo semantic content." } } ] }' ``` </details> Some test cases: 1. Check that if not updated, a new conversation does not have a summary 2. Check that `summary` contains expected value after it has been updated via one of the above APIs 3. Check that we do not return legacy fields (`summary.confidence`, `summary.content` and `summary.public`) even if you add a document with those fields set. You can set legacy fields, either via DevTools or via update APIs from above in previous kibana version. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…ce (elastic#13657) (elastic#232394) ## Summary Epic: elastic/security-team#12768 Meta: elastic/security-team#13657 RFC: [internal link](https://docs.google.com/document/d/13jAJ5Q3_At_zAuwKjvpYehnM5uzKJSwZIZ1aJJNAf5k) With these changes we update the conversation `Message` schema by adding a new field `id`. This will allow us to easily refer messages that are summarized. ### New messages All new messages will be added with the `id` field set. ### Existing messages The `id` field for all existing messages will be `undefined` and will be updated whenever a user updates the conversation: - appends new messages - regenerates a last assistant respond In those case above, all messages within the ongoing conversation will be updated and will have a `id` field set. ### Testing scenarios 1. **Existing messages without `id`** - Create conversation with messages in "pre this PR" code (e.g. main branch) - Checkout code from this PR - Check via Dev Tools that messages within the conversations index do not have `id` field set - Add a new message to one of the conversations - Check that all messages within that conversation are updated and `id` field set for all of them 2. **New messages** - Checkout code from this PR - Create a new chat and add new messages - Check via Dev Tools that all messages within the new conversation have `id` field set --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

remove old maps stuff

11c2080

spalger added review v7.0.0 labels Aug 22, 2017

spalger requested a review from epixa August 22, 2017 23:38

thomasneirynck added the Feature:Visualizations Generic visualization features (in case no more specific feature label is available) label Aug 23, 2017

epixa approved these changes Aug 23, 2017

View reviewed changes

spalger merged commit 8ba33e7 into elastic:master Aug 25, 2017

spalger deleted the remove/tile-service-stuff branch October 18, 2019 17:44

spalger restored the remove/tile-service-stuff branch October 18, 2019 17:44

spalger deleted the remove/tile-service-stuff branch October 18, 2019 17:45

spalger restored the remove/tile-service-stuff branch October 18, 2019 17:45

spalger deleted the remove/tile-service-stuff branch October 18, 2019 17:46

e40pud added a commit to e40pud/kibana that referenced this pull request Aug 19, 2025

[Security Solution][AI Assistant] Update ConversationSummary schema…

3cac4fa

… and interface (elastic#13657)

e40pud added a commit to e40pud/kibana that referenced this pull request Aug 20, 2025

[Security Solution][AI Assistant] Update Message schema and interfa…

c215d10

…ce (elastic#13657)

e40pud added a commit to e40pud/kibana that referenced this pull request Aug 20, 2025

[Security Solution][AI Assistant] Update Message schema and interfa…

c7b8253

…ce (elastic#13657)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

remove old maps stuff#13657

remove old maps stuff#13657
spalger merged 1 commit intoelastic:masterfrom
spalger:remove/tile-service-stuff

spalger commented Aug 22, 2017

Uh oh!

thomasneirynck commented Aug 23, 2017

Uh oh!

epixa left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

spalger commented Aug 22, 2017

Uh oh!

thomasneirynck commented Aug 23, 2017

Uh oh!

epixa left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants