[Observability AI Assistant] Screen Context by CoenWarmer · Pull Request #175885 · elastic/kibana

CoenWarmer · 2024-01-30T13:36:41Z

Summary

This adds functionality to allow consumers of the AI Assistant for Observability to add context to the LLM conversation, at the start of a conversation and contextual after every prompt.

Screen.Recording.2024-01-30.at.14.37.41.mov

How to use

The service now exposes a setApplicationContext function, that returns a hook to unregister the context. Here's an example:

Consumers can use this to add context relevant to the route or settings of Kibana that might be relevant for the LLM within that conversation.

Example:

useEffect(() => {
  return setApplicationContext({
    data: [
      {
        name: 'top_transactions',
        description: 'The visible transaction groups',
        value: mainStatistics.transactionGroups.map((group) => {
          return {
            name: group.name,
            alertsCount: group.alertsCount,
          };
        }),
      },
    ],
  });
}, [setApplicationContext, mainStatistics]);

By default the URL that the user is currently on is always included in the context so the Assistant will always take that into account.

Details for reviewers

recall function has been renamed to context
context function now returns both Knowledge base entries as well as chat context that is set by setApplicationContext.[Security Solution] GenAI API Integration Tests #176357
part of the function logic was moved from the ObservabilityAIAssistantService into the ChatFunctionClient, for easier testing

ghost · 2024-01-30T13:36:56Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
/oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

dgieselaar · 2024-01-31T10:24:08Z

x-pack/plugins/observability_ai_assistant/server/routes/chat/route.ts


    const client = await service.getClient({ request });

+    client.setChatContext(params.body.chatContext);


I don't think we should do this on the server (having side-effects). I also don't think we need it for chat, just /chat/complete. Let's make it a property of client.complete({ chatContext, ... })

Could you elaborate why this doesn't need to happen on the server when other functions do, from a conceptual standpoint? What is it about this that makes it doesn't need to happen on the server?

How would that work when the context function now retrieves both knowledge base entries and chat context? The kb entry getting happens on the server, so how should it then get stuff from the client here?

Re: also adding it to /chat: wouldn't having that also unlock augmenting the contextual insights with context?

Could you elaborate why this doesn't need to happen on the server when other functions do, from a conceptual standpoint? What is it about this that makes it doesn't need to happen on the server?

How would that work when the context function now retrieves both knowledge base entries and chat context? The kb entry getting happens on the server, so how should it then get stuff from the client here?

Sorry, I realize my wording is ambiguous. What I mean is that I don't think we should have side-effects on the server. The client is unique per request, but I still think it is unexpected for it to have side-effects like this. Maybe side-effects is the wrong term anyway, but what I mean is that it should be a property of /chat/complete. E.g. we don't use setMessages either.

Re: also adding it to /chat: wouldn't having that also unlock augmenting the contextual insights with context?

The contextual insights use /chat/complete.

dgieselaar

I've not finished going through all of this, so consider this early feedback - that I expect will amount to some significant changes.

dgieselaar · 2024-01-31T13:01:05Z

x-pack/plugins/observability_ai_assistant/public/components/insight/insight.tsx

  initialMessages: Message[];
  connectorId: string;
 }) {
+  const service = useObservabilityAIAssistant();


why the plugin service, and not the chat service?

Because the chat service is not available until after the chat has started.

...k/plugins/observability_ai_assistant/public/components/action_menu_item/action_menu_item.tsx

dgieselaar · 2024-01-31T15:11:02Z

...ugins/observability_ai_assistant/public/hooks/use_observability_ai_assistant_chat_context.ts

+
+import { ObservabilityAIAssistantService } from '../types';
+
+export function useObservabilityAIAssistantChatContext(service: ObservabilityAIAssistantService) {


I think we need to run a function here on unmount to remove the context set by the consuming component.

dgieselaar · 2024-01-31T15:11:58Z

x-pack/plugins/observability_ai_assistant/public/service/create_chat_service.ts

        shareReplay()
      );
    },
+    complete({ chatContext, connectorId, conversationId, messages, persist, signal }) {


I assume this is just re-sorting, but it makes it harder to see the diff

dgieselaar · 2024-01-31T15:12:48Z

x-pack/plugins/observability_ai_assistant/public/service/create_service.ts

    getLicense: () => licenseStart.license$,
    getLicenseManagementLocator: () => shareStart,
+    setChatContext: (newChatContext: ChatContext) => {
+      chatContext = {


as discussed elsewhere, this just keeps adding context indefinitely

dgieselaar · 2024-01-31T16:08:16Z

x-pack/plugins/observability_ai_assistant/server/routes/chat/route.ts


    const client = await service.getClient({ request });

+    client.setChatContext(params.body.chatContext);


Could you elaborate why this doesn't need to happen on the server when other functions do, from a conceptual standpoint? What is it about this that makes it doesn't need to happen on the server?

How would that work when the context function now retrieves both knowledge base entries and chat context? The kb entry getting happens on the server, so how should it then get stuff from the client here?

Sorry, I realize my wording is ambiguous. What I mean is that I don't think we should have side-effects on the server. The client is unique per request, but I still think it is unexpected for it to have side-effects like this. Maybe side-effects is the wrong term anyway, but what I mean is that it should be a property of /chat/complete. E.g. we don't use setMessages either.

Re: also adding it to /chat: wouldn't having that also unlock augmenting the contextual insights with context?

The contextual insights use /chat/complete.

CoenWarmer · 2024-01-31T17:19:06Z

I've not finished going through all of this, so consider this early feedback - that I expect will amount to some significant changes.

I'm surprised 20 files is too much to go through in one go. Wouldn't it make sense to finish it and get a complete picture?

In any case, I replied to one of your remarks that I think is valuable here. FWIW, I also wanted to have it on the client and not the service but the client is only available after the chat is started. That would mean consumers would have to pass in the client to the hook which isn't great either. Let me know your thoughts.

dgieselaar · 2024-01-31T21:48:28Z

@CoenWarmer I have other things to do :) I'm doing this to give you quicker feedback so you'll be able to work on it.

elasticmachine · 2024-02-07T14:23:01Z

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

elasticmachine · 2024-02-07T14:23:02Z

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

x-pack/plugins/observability_ai_assistant/server/functions/context.ts

This reverts commit 3018007.

dgieselaar · 2024-02-09T08:26:25Z

Thanks for your work so far @CoenWarmer! Discussed with @sorenlouv over Zoom, I plan on making a couple of changes:

reverting the suggestions so we can split that out into a separate PR. Looks like the work is not really related and I would rather have a separate review process for it.
We'll flatten the context object to strings that we'll concatenate. This should feel more natural (e.g., consider "The user is looking at the opbeans-go service. There are no anomalies for this service" vs JSON that describes the same).
I'll add support for data, which are pieces of data that we send over to the /chat/complete endpoint, keep in memory, and tell the LLM that it can request it. This saves on tokens initially, but still allows for the LLM to "see" what the user is looking at when it wants to.

sorenlouv · 2024-02-09T22:08:37Z

x-pack/plugins/apm/public/components/app/service_inventory/index.tsx

+  const setApplicationContext =
+    useApmPluginContext().observabilityAIAssistant.service
+      .setApplicationContext;


Nit: with destructuring we might be able to get a one-liner.

Suggested change

const setApplicationContext =

useApmPluginContext().observabilityAIAssistant.service

.setApplicationContext;

const { setApplicationContext } = useApmPluginContext().observabilityAIAssistant.service;

x-pack/plugins/apm/public/components/app/service_inventory/index.tsx

sorenlouv · 2024-02-13T09:08:46Z

x-pack/plugins/apm/public/components/app/service_inventory/index.tsx

+      });
+    }
+
+    return setApplicationContext({


Can we call this chatContext? It will be helpful for consumers to highlight that this is context for the AI Assistant - and not application context in general.

Suggested change

return setApplicationContext({

return setChatContext({

Also: I think we to consider the different types of "context" and how we refer to them. Afaict we have:

chat context supplied by consumers

knowledge base articles

data from connectors (third parties)

Is all of it context? Is all of it knowledge base?

It was chatContext originally, but I'm thinking there are use cases without chat, e.g. on a one-off interaction with the LLM, like contextual insights, or generating an ES|QL query. I agree that applicationContext is not very clear. Maybe setAssistantContext?

setAssistantContext is definitely better.

x-pack/plugins/observability/public/pages/slo_details/slo_details.tsx

x-pack/plugins/observability/public/pages/slos/components/slo_list.tsx

x-pack/plugins/observability_ai_assistant/README.md

...k/plugins/observability_ai_assistant/public/components/action_menu_item/action_menu_item.tsx

sorenlouv · 2024-02-13T09:25:30Z

x-pack/plugins/observability_ai_assistant/public/components/chat/chat_body.test.tsx

-            name: 'recall',
-            arguments: '{"queries":[],"categories":[]}',
+            name: 'context',
+            arguments: '{"learnings": { "queries":[],"categories":[]} }',


Why the nesting under learnings?

oh, this is a mistake. will correct

x-pack/plugins/observability_ai_assistant/public/mock.tsx

x-pack/plugins/observability_ai_assistant/server/functions/context.ts

sorenlouv · 2024-02-13T10:06:33Z

x-pack/plugins/observability_ai_assistant/server/functions/context.ts

+        const screenDescription = appContexts.map((context) => context.description).join('\n\n');

-      const nonEmptyQueries = compact(queries);
+        const content = { screen_description: screenDescription, learnings: [] };


What about moving the dataWithinTokenLimit to the getContext function?

Suggested change

const content = { screen_description: screenDescription, learnings: [] };

// any data that falls within the token limit, send it automatically

const dataWithinTokenLimit = compact(appContexts.flatMap((context) => context.data)).filter(

(data) => encode(JSON.stringify(data.value)).length <= MAX_TOKEN_COUNT_FOR_DATA_ON_SCREEN

);

const content = {

screen_description: screenDescription,

data_on_screen: dataWithinTokenLimit,

learnings: [],

};

sorenlouv · 2024-02-13T10:06:51Z

x-pack/plugins/observability_ai_assistant/server/functions/context.ts

+            // any data that falls within the token limit, send it automatically
+
+            const dataWithinTokenLimit = compact(
+              appContexts.flatMap((context) => context.data)
+            ).filter(
+              (data) =>
+                encode(JSON.stringify(data.value)).length <= MAX_TOKEN_COUNT_FOR_DATA_ON_SCREEN
+            );


Why is this not part of the getContext function?

It wasn't part of it because I originally implemented it as separate messages (get_data_on_screen function calls). But I was worried that that would cause the LLM to no longer call get_data_on_screen, so I removed it again. I've moved it to the getContext function.

x-pack/plugins/observability_ai_assistant/server/functions/context.ts

x-pack/plugins/observability_ai_assistant/server/functions/query/index.ts

sorenlouv · 2024-02-13T10:09:14Z

x-pack/plugins/observability_ai_assistant/server/routes/chat/route.ts

    });

-    const response$ = await client.complete({
+    const response$ = client.complete({


Thanks! This dangling await has been triggering severe OCD!

😄 @miltonhultgren was complaining about it as well

sorenlouv · 2024-02-13T10:13:42Z

x-pack/plugins/observability_ai_assistant/server/service/chat_function_client/index.ts

+    if (allData.length) {
+      this.registerFunction(
+        {
+          name: 'get_data_on_screen',


I want to make sure I understand this correctly: The LLM can now get the chat context in two ways:

calling the context function will return the chat context (possibly truncated due to token limit) or

calling the get_data_on_screen function which will return the untruncated chat context

Btw. it looks like get_data_on_screen doesn't include the appContexts.description (aka "screenDescription"). Is that intentional? Afaict the context for the SLOList only has screen description and no data attribute so that would result in no context being provided here

Yes, the descriptions are automatically added in the context function call, so there is no need for the LLM to request it.

sorenlouv · 2024-02-13T10:38:32Z

x-pack/plugins/observability/public/pages/alert_details/alert_details.tsx

+    `);
+
+    return setApplicationContext({
+      description,


nit: should we call the top level description for screenDescription to indicate that it should describe what the user is looking at, whereas the description under data is to describe what the data entities are.

Suggested change

description,

screenDescription,

sorenlouv

Awesome improvement! Excited to see the impact of this!

kibana-ci · 2024-02-13T14:32:22Z

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`observability`	598	599	+1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`observabilityAIAssistant`	73	76	+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`apm`	3.2MB	3.2MB	+1.2KB
`observability`	638.5KB	640.8KB	+2.2KB
`observabilityAIAssistant`	168.5KB	168.8KB	+263.0B
total			+3.7KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`observabilityAIAssistant`	13	14	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`observability`	102.8KB	102.8KB	-48.0B
`observabilityAIAssistant`	14.5KB	14.6KB	+101.0B
total			+53.0B

Unknown metric groups

API count

id	before	after	diff
`observabilityAIAssistant`	75	78	+3

History

💚 Build #193239 succeeded 2177d37
💔 Build #193065 failed 7fade90
💛 Build #192665 was flaky 91d4c25
💔 Build #192639 failed f0fad69
💔 Build #192620 failed 28269fe
💔 Build #192535 failed 5631579

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

shahzad31 · 2024-02-13T15:00:20Z

x-pack/plugins/observability/public/pages/alert_details/alert_details.tsx

+
+    ${
+      alertDetail.formatted.reason
+        ? `The reason given for the alert is ${alertDetail.formatted.reason}.`


Not needed, everything we sent to the LLM needs to be in English.

shahzad31 · 2024-02-13T15:00:29Z

x-pack/plugins/observability/public/pages/alert_details/alert_details.tsx

+      return;
+    }
+
+    const screenDescription = dedent(`The user is looking at an ${


@shahzad31 We always use English when communicating with the LLM (think of this as an API call where you wouldn't translate the params either)

shahzad31 · 2024-02-13T15:03:51Z

x-pack/plugins/observability/public/pages/slo_details/slo_details.tsx

+        The user is looking at the detail page for the following SLO
+
+        Name: ${slo.name}.
+        Id: ${slo.id}


i guess we also need to slo.instanceId if it's not equal to *

it's available in the metadata below

What metadata are you referring to?

I wonder if using the actual doc fields slo.name, slo.id etc, would help the LLM be able to match the data to parameters for functions when those parameters use the field name.

the data prop - that's also sent over to the LLM

re: your second question: yes, that's useful to the LLM, but also available in the data prop

@dominiqueclarke sorry I spoke to soon, I have not actually verified whether field names are available in the metadata. Where are the field names used (outside of SOs)?

dominiqueclarke

LGTM just wondering about the instanceId value as Shahzad has mentioned but SLO team can always refine the data provided on the pages we own.

## Summary This adds functionality to allow consumers of the AI Assistant for Observability to add context to the LLM conversation, at the start of a conversation and contextual after every prompt. https://github.com/elastic/kibana/assets/535564/b4d62897-d701-4c23-b90b-464cad21e9d0 ![image](https://github.com/elastic/kibana/assets/352732/85a7e27a-e715-4273-a3a3-0dd68a7c9c5c) ## How to use The service now exposes a `setApplicationContext` function, that returns a hook to unregister the context. Here's an example: Consumers can use this to add context relevant to the route or settings of Kibana that might be relevant for the LLM within that conversation. Example: ```ts useEffect(() => { return setApplicationContext({ data: [ { name: 'top_transactions', description: 'The visible transaction groups', value: mainStatistics.transactionGroups.map((group) => { return { name: group.name, alertsCount: group.alertsCount, }; }), }, ], }); }, [setApplicationContext, mainStatistics]); ``` By default the URL that the user is currently on is always included in the context so the Assistant will always take that into account. ## Details for reviewers - `recall` function has been renamed to `context` - `context` function now returns both Knowledge base entries as well as chat context that is set by `setApplicationContext`.elastic#176357 - part of the function logic was moved from the ObservabilityAIAssistantService into the ChatFunctionClient, for easier testing --------- Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

Add chatContext functionality

ca57268

CoenWarmer requested a review from a team January 30, 2024 13:36

CoenWarmer added the release_note:skip Skip the PR/issue when compiling release notes label Jan 30, 2024

CoenWarmer changed the title ~~Add chatContext functionality~~ [Observablity AI Assistant] Add chat context functionality Jan 30, 2024

Correctly set URL

d05bfce

CoenWarmer force-pushed the feat/chat-context branch from b3e75ca to d05bfce Compare January 30, 2024 15:10

dgieselaar reviewed Jan 31, 2024

View reviewed changes

CoenWarmer changed the title ~~[Observablity AI Assistant] Add chat context functionality~~ [Observability AI Assistant] Add chat context functionality Jan 31, 2024

CoenWarmer requested a review from dgieselaar January 31, 2024 12:48

dgieselaar reviewed Jan 31, 2024

View reviewed changes

CoenWarmer requested a review from dgieselaar January 31, 2024 17:01

CoenWarmer added 2 commits February 7, 2024 00:03

Merge branch 'main' of github.com:elastic/kibana into feat/chat-context

9862d17

Add starter and contextual suggestions

3018007

CoenWarmer requested a review from a team as a code owner February 7, 2024 14:22

CoenWarmer requested a review from a team February 7, 2024 14:22

botelastic bot added Team:obs-ux-infra_services - DEPRECATED DEPRECATED - Use Team:obs-presentation. Team:actionable-obs Formerly "obs-ux-management", responsible for SLO, o11y alerting, significant events, & synthetics. labels Feb 7, 2024

CoenWarmer marked this pull request as draft February 7, 2024 14:25

CoenWarmer changed the title ~~[Observability AI Assistant] Add chat context functionality~~ [Observability AI Assistant] Chat Context + Suggestions functionality Feb 7, 2024

sorenlouv reviewed Feb 8, 2024

View reviewed changes

x-pack/plugins/observability_ai_assistant/server/functions/context.ts Outdated Show resolved Hide resolved

dgieselaar added 2 commits February 9, 2024 09:07

Merge branch 'main' of github.com:elastic/kibana into feat/chat-context

d14a1ab

Revert "Add starter and contextual suggestions"

e4d60dd

This reverts commit 3018007.

dgieselaar added 2 commits February 9, 2024 12:29

Support for data in appContext

f4a1fb9

Merge branch 'main' of github.com:elastic/kibana into feat/chat-context

37defe5

dgieselaar added 3 commits February 13, 2024 10:41

Merge branch 'main' of github.com:elastic/kibana into feat/chat-context

cd540f6

Add screen description as a query

3bbe608

Merge branch 'main' of github.com:elastic/kibana into feat/chat-context

23457ab

sorenlouv reviewed Feb 13, 2024

View reviewed changes

x-pack/plugins/observability_ai_assistant/server/functions/context.ts Show resolved Hide resolved

sorenlouv reviewed Feb 13, 2024

View reviewed changes

x-pack/plugins/observability_ai_assistant/server/functions/query/index.ts Show resolved Hide resolved

sorenlouv reviewed Feb 13, 2024

View reviewed changes

Review feedback

2177d37

sorenlouv reviewed Feb 13, 2024

View reviewed changes

dgieselaar added 2 commits February 13, 2024 13:42

Rename setApplicationContext to setScreenContext

67e1f75

Merge branch 'main' of github.com:elastic/kibana into feat/chat-context

1ee69c5

dgieselaar changed the title ~~[Observability AI Assistant] App Context~~ [Observability AI Assistant] Screen Context Feb 13, 2024

Remove console.log statement

d87b4f9

sorenlouv approved these changes Feb 13, 2024

View reviewed changes

dgieselaar enabled auto-merge (squash) February 13, 2024 12:58

dgieselaar approved these changes Feb 13, 2024

View reviewed changes

dgieselaar requested a review from a team February 13, 2024 14:53

shahzad31 reviewed Feb 13, 2024

View reviewed changes

dominiqueclarke approved these changes Feb 13, 2024

View reviewed changes

dgieselaar merged commit 853b5f6 into elastic:main Feb 13, 2024

kibanamachine added the backport:skip This PR does not require backporting label Feb 13, 2024


		const client = await service.getClient({ request });

		client.setChatContext(params.body.chatContext);


		import { ObservabilityAIAssistantService } from '../types';

		export function useObservabilityAIAssistantChatContext(service: ObservabilityAIAssistantService) {

Conversation

CoenWarmer commented Jan 30, 2024 • edited by dgieselaar Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to use

Details for reviewers

Uh oh!

ghost commented Jan 30, 2024

🤖 GitHub comments

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CoenWarmer Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dgieselaar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CoenWarmer commented Jan 31, 2024

Uh oh!

dgieselaar commented Jan 31, 2024

Uh oh!

elasticmachine commented Feb 7, 2024

Uh oh!

elasticmachine commented Feb 7, 2024

Uh oh!

Uh oh!

dgieselaar commented Feb 9, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CoenWarmer commented Jan 30, 2024 •

edited by dgieselaar

Loading

CoenWarmer Jan 31, 2024 •

edited

Loading