[Request]: Obs AI Assistant supports other models than ELSER when field type semantic_text is used

[sorenlouv]

Description

The Use search connectors currently states that Elser must be used in order for the AI Assistant to support it.

Since version 8.17, we now support querying search connectors using the semantic_text field. This update allows users to:

1. Choose their preferred embedding model: Users can opt for models like e5 (multilingual) instead of being limited to ELSER
2. Incorporate custom indices: Users can create their own indices and include them in the knowledge base. This can be configured using the UI setting observability:aiAssistantSearchConnectorIndexPattern. In this case they can also use any model they want, as long as they use semantic_text

Resources

• The ability to query semantic_text fields was added in [Obs AI Assistant] Support querying semantic_text fields in search connectors kibana#200184
• The ability to include custom indices in the knowledge base was added in [Obs AI Assistant] Query all search connectors by default but allow user to override kibana#183712

Which documentation set does this change impact?

Stateful and Serverless

Feature differences

This feature is identical in serverless and stateful

What release is this request related to?

8.17

Collaboration model

The documentation team

Point of contact.

Main contact: @mdbirnstiehl @dedemorton
Stakeholders: @sorenlouv @teknogeek0

[mdbirnstiehl]

Hi @sorenlouv, just looking through this issue and the linked PRs and want to verify:

Is this correct? By default, all indexes associated with a connector are queried, but users would use the observability:aiAssistantSearchConnectorIndexPattern setting to limit the index patterns are queried?

[sorenlouv]

Is this correct? By default, all indexes associated with a connector are queried, but users would use the observability:aiAssistantSearchConnectorIndexPattern setting to limit the index patterns are queried?

Users don't need to change observability:aiAssistantSearchConnectorIndexPattern. If they setup a search connector that has semantic_text fields we will query it, and use the model the user decided on.