[Security AI Assistant] Chat complete API

[YulNaumenko]

Summary

Current PR introduces the new public chat/complete API for Security Assistant. Implementation support both LangGraph and non-LangGraph implementations (managed by experimental feature assistantKnowledgeBaseByDefault).
Use cases:
POST http://localhost:5601/api/security_ai_assistant/chat/complete

1. Simplest Q&A:

User:
{
    "connectorId": "my-gen-ai",
    "persist": false,
    "messages": [
        {
            "role": "user",
            "content": "As an expert user of Elastic Security, please generate an accurate and valid ESQL query to detect the use case below. Your response should be formatted to be able to use immediately in an Elastic Security timeline or detection rule. Take your time with the answer, check your knowledge really well on all the functions I am asking for. For ES|QL answers specifically, you should only ever answer with what's available in your private knowledge. I cannot afford for queries to be inaccurate. Assume I am using the Elastic Common Schema and Elastic Agent.Ensure the answers are formatted in a way which is easily copyable as a separate code block in markdown."
        }
    ]
}

Assistant:
{
    "connector_id": "my-gen-ai",
    "data": "Of course, I'd be happy to help with that. Could you please provide me with the specific use case or details of the activity you're looking to detect using the ESQL query?",
    "trace_data": {
        "transaction_id": "0aa30f21b107cd03",
        "trace_id": "e5c63c37804e974b8022a86b3a3d88bc"
    },
    "replacements": {},
    "status": "ok"
}

2. Q&A with the data and anonymization fields list + centralized anonymization applied:

User:
{
    "connectorId": "my-gen-ai",
    "persist": false,
    "messages": [
        {
            "role": "user",
            "content": "Evaluate the event from the context and format your output neatly in markdown syntax for my Elastic Security case.\nAdd your description, recommended actions and bulleted triage steps. Use the MITRE ATT&CK data provided to add more context and recommendations from MITRE, and hyperlink to the relevant pages on MITRE's website. Be sure to include the user and host risk score data from the context. Your response should include steps that point to Elastic Security specific features, including endpoint response actions, the Elastic Agent OSQuery manager integration (with example osquery queries), timelines and entity analytics and link to all the relevant Elastic Security documentation.",
            "data": {
                "event.category": "process",
                "process.pid": 69516,
                "host.os.version": 14.5,
                "host.os.name": "macOS",
                "host.name": "Yuliias-MBP",
                "process.name": "biomesyncd",
                "user.name": "yuliianaumenko",
                "process.working_directory":"/",
                "event.module":"system",
                "process.executable": "/usr/libexec/biomesyncd",
                "process.args": "/usr/libexec/biomesyncd",
                "message": "Process biomesyncd (PID: 69516) by user yuliianaumenko STOPPED"
            },
            "fields_to_anonymize": ["host.os.name", "event.module"]
        }
    ]
}
Assistant:

{
    "connector_id": "my-gen-ai",
    "data": "## Event Evaluation\n\n### Description\n\nThe event details a process named `biomesyncd` with PID 69516, initiated by the user `yuliianaumenko`, being stopped on a host with the name `138fea45-b182-450d-bfc7-169baf807073`. The process was executed from `/usr/libexec/biomesyncd` with the same name as the executable, and it was operating from the root directory `/`. The host is running an operating system named `509de90a-a77f-4a48-97b0-453fce3a9846` with version `14.5`.\n\n### Recommended Actions\n\n1. **Initial Assessment**: Determine if `biomesyncd` is a legitimate process for the host's operating system and the user's typical behavior.\n2. **Investigate Process History**: Use Elastic Security's timeline feature to trace the history of `biomesyncd` executions and any related events.\n3. **User Behavior Analysis**: Analyze the user `yuliianaumenko`'s behavior for any anomalies using Elastic Security's entity analytics.\n4. **Host Risk Score Evaluation**: Review the risk score assigned to the host `138fea45-b182-450d-bfc7-169baf807073` to prioritize investigation.\n5. **Endpoint Response**: Consider using Elastic Security's endpoint response actions to isolate the host if malicious activity is suspected.\n6. **OSQuery Manager Integration**: Utilize the Elastic Agent OSQuery manager to run queries for further investigation.\n\n### Triage Steps\n\n1. **Verify Process Authenticity**:\n   - Check if `biomesyncd` is known to be a part of the host's operating system or if it's potentially malicious.\n2. **Timeline Analysis**:\n   - Navigate to Elastic Security's [Timelines](https://www.elastic.co/guide/en/security/current/timelines-ui.html) to investigate related events before and after the process stoppage.\n3. **User and Host Risk Score**:\n   - Assess the risk scores provided by Elastic Security for both the user and the host to prioritize your investigation.\n4. **Endpoint Isolation**:\n   - If suspicious activity is confirmed, use [Endpoint Response Actions](https://www.elastic.co/guide/en/security/current/responses-ui.html) to isolate the host.\n5. **OSQuery for Further Investigation**:\n   - Use the [Elastic Agent OSQuery Manager](https://www.elastic.co/guide/en/fleet/current/osquery.html) integration to execute queries like `SELECT * FROM processes WHERE name = 'biomesyncd';` to gather more information about the process and its activities.\n\n### MITRE ATT&CK Context\n\nGiven the information, it's unclear which specific MITRE ATT&CK technique might be relevant without further investigation. However, monitoring and investigating unexpected or unauthorized process terminations can be related to various techniques such as [T1569.002: System Services: Service Stop](https://attack.mitre.org/techniques/T1569/002/). It's crucial to determine whether this event is part of regular maintenance or indicative of malicious activity.\n\n### Additional Resources\n\n- Elastic Security Documentation: [Elastic Security](https://www.elastic.co/guide/en/security/current/index.html)\n- MITRE ATT&CK: [MITRE ATT&CK Techniques](https://attack.mitre.org/techniques/enterprise/)\n- Elastic Agent OSQuery Manager: [OSQuery Manager](https://www.elastic.co/guide/en/fleet/current/osquery.html)\n\nPlease proceed with the recommended actions and triage steps to further investigate this event and determine the appropriate response.",
    "trace_data": {
        "transactionId": "bd446038ccb07011",
        "traceId": "58727f2954b316f45182dd3e55c5564d"
    },
    "replacements": {
        "509de90a-a77f-4a48-97b0-453fce3a9846": "macOS",
        "138fea45-b182-450d-bfc7-169baf807073": "Yuliias-MBP",
        "ed31954a-42db-4cef-be6e-cd53d3b71a21": "yuliianaumenko",
        "9c8740a5-6f23-4469-829f-ddcb35809723": "system"
    },
    "status": "ok"
}

3. Q&A as a part of the existing conversation by id:

User:
{
    "conversationId": "1d59bf07-144e-407b-9da0-3a55c69e5c10",
    "connectorId": "my-gen-ai",
    "persist": true,
    "messages": [
        {
            "role": "user",
            "content": "As an expert user of Elastic Security, please generate an accurate and valid ESQL query to detect the use case below. Your response should be formatted to be able to use immediately in an Elastic Security timeline or detection rule. Take your time with the answer, check your knowledge really well on all the functions I am asking for. For ES|QL answers specifically, you should only ever answer with what's available in your private knowledge. I cannot afford for queries to be inaccurate. Assume I am using the Elastic Common Schema and Elastic Agent.Ensure the answers are formatted in a way which is easily copyable as a separate code block in markdown."
        }
    ]
}

Assistant:
{
    "connector_id": "my-gen-ai",
    "data": "It seems like there might have been a misunderstanding or a technical issue, as the specific use case details you're looking to detect with the ESQL query were not provided in your message. To generate an accurate and valid ESQL query, I need to know the specific behaviors, events, or conditions you wish to identify. This could include, but is not limited to, suspicious process executions, unusual network connections, file modifications, or any other security-relevant events.\n\nPlease provide the details of the use case you're interested in, and I'll be more than happy to assist you with crafting the appropriate ESQL query.",
    "trace_data": {
        "transactionId": "e4fdba10ed34e0b2",
        "traceId": "c06fc9704498ee026f832443154c8033"
    },
    "replacements": {},
    "status": "ok"
}

4. Q&A with creation of the new conversation:

User:
{
    "connectorId": "my-gen-ai",
    "persist": true,
    "messages": [
        {
            "role": "user",
            "content": "As an expert user of Elastic Security, please generate an accurate and valid ESQL query to detect the use case below. Your response should be formatted to be able to use immediately in an Elastic Security timeline or detection rule. Take your time with the answer, check your knowledge really well on all the functions I am asking for. For ES|QL answers specifically, you should only ever answer with what's available in your private knowledge. I cannot afford for queries to be inaccurate. Assume I am using the Elastic Common Schema and Elastic Agent.Ensure the answers are formatted in a way which is easily copyable as a separate code block in markdown."
        }
    ]
}
Assistant:
{
    "connector_id": "my-gen-ai",
    "data": "To provide you with an accurate and valid ESQL query, I need the specific use case you're looking to detect with Elastic Security. Could you please provide me with the details of the scenario or behavior you're aiming to monitor or detect?",
    "trace_data": {
        "transactionId": "09f6bc2cf03931b6",
        "traceId": "8dd3743ac6fc92d33194ea84222ba58c"
    },
    "replacements": {},
    "status": "ok"
}

5. The most complex use case with the data, anonymization and adding the Q&A to the existing conversation:

User:
{
    "connectorId": "my-gen-ai",
    "conversationId": "d574ec43-03d2-4c24-83ce-08002f755f73",
    "persist": true,
    "messages": [
        {
            "role": "user",
            "content": "Evaluate the event from the context and format your output neatly in markdown syntax for my Elastic Security case.\nAdd your description, recommended actions and bulleted triage steps. Use the MITRE ATT&CK data provided to add more context and recommendations from MITRE, and hyperlink to the relevant pages on MITRE's website. Be sure to include the user and host risk score data from the context. Your response should include steps that point to Elastic Security specific features, including endpoint response actions, the Elastic Agent OSQuery manager integration (with example osquery queries), timelines and entity analytics and link to all the relevant Elastic Security documentation.",
            "data": {
                "event.category": "process",
                "process.pid": 69516,
                "host.os.version": 14.5,
                "host.os.name": "macOS",
                "host.name": "Yuliias-MBP",
                "process.name": "biomesyncd",
                "user.name": "yuliianaumenko",
                "process.working_directory":"/",
                "event.module":"system",
                "process.executable": "/usr/libexec/biomesyncd",
                "process.args": "/usr/libexec/biomesyncd",
                "message": "Process biomesyncd (PID: 69516) by user yuliianaumenko STOPPED"
            },
            "fields_to_anonymize": ["host.os.name", "event.module"]
        }
    ]
}
Assistant:

6. Streaming support:

User:
{
    "connectorId": "my-gen-ai",
    "isStream": true,
    "persist": false,
    "messages": [
        {
            "role": "user",
            "content": "Evaluate the event from the context and format your output neatly in markdown syntax for my Elastic Security case.\nAdd your description, recommended actions and bulleted triage steps. Use the MITRE ATT&CK data provided to add more context and recommendations from MITRE, and hyperlink to the relevant pages on MITRE's website. Be sure to include the user and host risk score data from the context. Your response should include steps that point to Elastic Security specific features, including endpoint response actions, the Elastic Agent OSQuery manager integration (with example osquery queries), timelines and entity analytics and link to all the relevant Elastic Security documentation.",
            "data": {
                "event.category": "process",
                "process.pid": 69516,
                "host.os.version": 14.5,
                "host.os.name": "macOS",
                "host.name": "Yuliias-MBP",
                "process.name": "biomesyncd",
                "user.name": "yuliianaumenko",
                "process.working_directory":"/",
                "event.module":"system",
                "process.executable": "/usr/libexec/biomesyncd",
                "process.args": "/usr/libexec/biomesyncd",
                "message": "Process biomesyncd (PID: 69516) by user yuliianaumenko STOPPED"
            },
            "fields_to_anonymize": ["host.os.name", "event.module"]
        }
    ]
}

[joemcelroy]

Works as expected from playground side.

[spong]

Do we want to add the same authUser check here for updateConversation() as you did above for createConversation()?

if (!this.options.currentUser) {
  throw new Error('AIAssistantConversationsDataClient currentUser is not defined.');
}

[spong]

Thoughts on capturing the chat/complete (non-stream) response interface here as well?

I was going to comment and see if we wanted to return conversationId when persist:true, but then noticed we are just using the ExecuteConnectorResponse interface as part of StaticReturnType, which is what langChainExecute implicitly returns.

I guess more generally, what are your thoughts on chat/complete replacing /actions/connector/{connectorId}/_execute route? Would be nice to get away from that initial implementation that was mirroring the actions framework.

[spong]

nit: This had me do a double-take as I was like "Why is the conversation response being stored in state, wouldn't that be computed from messages?", then I realized it's just the current persisted state of the conversation.

Maybe we should alias ConversationResponse to Conversation for more general use? I do like that we're aligning on the API response interface, that's nice.

[spong]

FYI, @patrykkopycinski and I noticed LangSmith tracing wasn't working with streaming. Turns out I forgot to pass in traceOptions here which includes the necessary tracer. It's plumbed through the rest of the way, so only need to add it to this call.

Patryk is going to fix this in his esql PR, so need to add it now if you don't want, but wanted to let you know in case you were looking for traces it in LangSmith.

[spong]

Any reason you're using empty string vs undefined for denoting when we'll use the title from the persisted conversation?

[YulNaumenko]

no reason

[spong]

🙏

Will be so nice to then separate out the system prompt as well once prompt persistence lands. Thank you @YulNaumenko!!

[stephmilovic]

Without stringify, this will not get parsed when called:

[spong]

FYI, this might be resolved with Patryk's work on BedrockChat: #186809. Will still be needed for SimpleChatModel though

[spong]

When creating conversations via persist:true using the chat/complete API, I noticed that the conversations were not showing up in the UI. This appears to be because the user object doesn't have profile_uid when using the API, but this is the unique identifier we use when fetching.

I used this API request to create the conversation:

### 4. Q&A with creation of the new conversation:
POST http://localhost:5601/kbn/api/security_ai_assistant/chat/complete
kbn-xsrf
Elastic-Api-Version: 2023-10-31
Authorization: Basic elastic changeme
X-Kbn-Context: %7B%22type%22%3A%22application%22%2C%22name%22%3A%22securitySolutionUI%22%2C%22url%22%3A%22%2Fkbn%2Fapp%2Fsecurity%22%7D
Content-Type: application/json

{
  "connectorId": "azure-open-ai",
  "persist": true,
  "responseLanguage": "dutch",
  "messages": [
    {
      "role": "user",
      "content": "Tell me funny joke"
    }
  ]
}

And the conversation created was:

      {
        "_index": ".ds-.kibana-elastic-ai-assistant-conversations-default-2024.06.28-000001",
        "_id": "f0d5694a-51e1-41a9-84ed-cca483321675",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-07-01T22:19:54.563Z",
          "updated_at": "2024-07-01T22:19:56.210Z",
          "api_config": {
            "action_type_id": ".gen-ai",
            "connector_id": "azure-open-ai"
          },
          "namespace": "default",
          "created_at": "2024-07-01T22:19:54.563Z",
          "messages": [
            {
              "@timestamp": "2024-07-01T22:19:54.563Z",
              "role": "user",
              "reader": null,
              "is_error": null,
              "content": "Tell me funny joke"
            },
            {
              "@timestamp": "2024-07-01T22:19:56.210Z",
              "role": "assistant",
              "reader": null,
              "is_error": false,
              "trace_data": {},
              "content": """Why don't scientists trust atoms?

Because they make up everything!"""
            }
          ],
          "replacements": [],
          "title": "Cybersecurity Humor",
          "category": "assistant",
          "users": [
            {
              "name": "elastic"
            }
          ]
        }
      },

And this was the `user` object logged out right when we create the resource

{
  "username": "elastic",
  "roles": [
    "superuser"
  ],
  "full_name": null,
  "email": null,
  "metadata": {
    "_reserved": true
  },
  "enabled": true,
  "authentication_realm": {
    "name": "reserved",
    "type": "reserved"
  },
  "lookup_realm": {
    "name": "reserved",
    "type": "reserved"
  },
  "authentication_type": "realm",
  "authentication_provider": {
    "type": "http",
    "name": "__http__"
  },
  "elastic_cloud_user": false
}

[spong]

Checked out, code reviewed, and tested locally -- LGTM! Thanks so much @YulNaumenko and @stephmilovic for this streamlined API and enhancements in simplifying our flows. Looks great! 🚀

Note: I did find one bug where conversations created via the chat/complete API were not included in the Conversations UI, but will leave that up to you to fix now or in a follow-up.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: cf52d5f

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
integrationAssistant	538	539	+1
securitySolution	5580	5581	+1
total			+2

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/elastic-assistant-common	309	323	+14

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	15.6MB	15.6MB	+4.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/elastic-assistant-common	333	349	+16

ESLint disabled line counts

id	before	after	diff
elasticAssistant	49	51	+2

Total ESLint disabled count

id	before	after	diff
elasticAssistant	49	51	+2

History

• 💔 Build #218939 failed 263ddc5
• 💔 Build #218913 failed 5aa9c72
• 💔 Build #218878 failed 15fc400
• 💔 Build #218867 failed b289452
• 💔 Build #218678 failed ff4f1b8

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @YulNaumenko