[Obs AI] Supports for Logs AI Insight in ES|QL mode in Discover

[neptunian]

Summary

Log AI Insight in Discover was unavailable in ES|QL mode because ES|QL doesn't return _id and _index metadata by default, which the component relied on to re-fetch the document from Elasticsearch. The ideal solution would be an ES-level setting to include metadata fields by default (under discussion with the ES team but not yet prioritized). As a workaround, this adds a fallback path that passes the log entry fields directly from the client when _id/_index aren't available.
This is feasible because the existing server-side fetch (getLogDocumentById) only retrieves 8 specific fields (@timestamp, message, log.level, service.name, trace.id, span.id, http.response.status_code, error.exception.message) — all of which are already present in the default ES|QL query results (FROM logs-*). So for the majority of queries we send the same fields to the LLM in either mode and no context is lost.

Limitations: If a user writes an ES|QL query that drops fields (e.g. FROM logs-* | KEEP message), the LLM will have less context to work with compared to the KQL path, which always re-fetches the full set of fields by document ID. The insight will still render but may produce a less detailed analysis. We can also no longer pass the log document attachment in esql mode without the id so that will not be available but its usually not needed since the fields available (logEntry) are already embedded in the context.

The existing KQL/Lucene path (fetch by _id/_index) is unchanged.

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: a1fbb30

Failed CI Steps

• Jest Integration Tests #9
• FTR Configs #138
• Scout: [ security / entity_store ] plugin

Test Failures

• [job] [logs] Jest Integration Tests #9 / Knowledge Base End-to-End Integration Test should save and retrieve knowledge base content through the complete flow
• [job] [logs] Scout: [ security / entity_store ] plugin / local-stateful-classic - Automated email resolution integration tests - Basic email matching — two entities with same email

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
observabilityAgentBuilder	17.4KB	17.8KB	+385.0B

History

• 💔 Build #413173 failed fb20ab7

[viduni94]

LGTM (CR only)
Left one comment

[viduni94]

Should we add a size/limit on fields or accept only a list of fields (LOG_DOCUMENT_FIELDS) to avoid sending a massive list of fields to the LLM?

[neptunian]

I am leaning towards no, because I don't think it's going to hurt but only benefit having more fields. I'm not sure why we bother limiting the log fetch to begin with. Maybe we should remove that filter. CC @arturoliduena . I don't think size is going to be an issue, a single doc is small enough to be trivial for an http request.

[arturoliduena]

think the main concern here is controlling token usage rather than request size.
I agree that more fields can help the LLM context, it usually doesn’t hurt.
Maybe we don’t need a strict limit here, but at least some maxFields or something.

[neptunian]

It's one filtered document so I don't think we should be too concerned about it. I think we should be more concerned about the level of trace data we are fetching and injecting into the prompt:

    const { traces } = await getTraces({
      core,
      plugins,
      logger,
      esClient,
      index,
      start: windowStart,
      end: windowEnd,
      kqlFilter: `_id: ${id}`,
      maxTraces: 10,
      maxDocsPerTrace: 100,
    });

[arturoliduena]

Could we filter out null on the client side?

[neptunian]

It is. Maybe I don't need this server side one, but I put it in both places. https://github.com/elastic/kibana/pull/258595/changes#diff-55daf8fc97ac45e642ebaa371c2f8b29f77836e3aa41a74b681e3ff7eea9e881R47

[arturoliduena]

when hasDocIdentity is false and hasFields is true, should we add a attachment:

{
  type: OBSERVABILITY_LOG_ATTACHMENT_TYPE_ID,
  data: { fields },
}

[neptunian]

I guess I don't see the point since the whole document is already in the conversation. What would it give us?

[arturoliduena]

Thanks @neptunian, for fixing this.

[kibanamachine]

Starting backport for target branches: 9.3

https://github.com/elastic/kibana/actions/runs/23506730912

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	9.3	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 258595

Questions ?

Please refer to the Backport tool documentation

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 258595 locally
cc: @neptunian

[neptunian]

💚 All backports created successfully

Status	Branch	Result
✅	9.3

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[kibanamachine]

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @neptunian