Skip to content

[Security solution] Implement dashboard to track Gen AI Token Usage#159075

Merged
stephmilovic merged 79 commits intoelastic:mainfrom
stephmilovic:token_lens
Jun 15, 2023
Merged

[Security solution] Implement dashboard to track Gen AI Token Usage#159075
stephmilovic merged 79 commits intoelastic:mainfrom
stephmilovic:token_lens

Conversation

@stephmilovic
Copy link
Copy Markdown
Contributor

@stephmilovic stephmilovic commented Jun 5, 2023 •
edited
Loading

Summary

Resolves https://github.com/elastic/security-team/issues/6712

  1. Implements new gen ai subaction called getDashboard. The subaction checks if a dashbord with a given id exists, and if not creates the dashboard. Returns { exists: true } upon successful retreival/creation
  2. The subaction also checks the users permissions on the index pattern used by the dashboard, and will return { exists: false } if the user does not have permissions.
  3. When the getDashboard subaction returns { exists: true } and isEdit is true, the connector form displays a link to the dashboard filtered by the connector id
Screenshot 2023-06-05 at 5 17 46 PM Screenshot 2023-06-05 at 5 21 09 PM

Type confusion

It seems the stack_connectors uses the triggers_actions_ui CONNECTORS_PLUGIN_ID which has its own TriggersAndActionsUiServices type. However, when I import useKibana it is tied to the type for PLUGIN_ID. That is why I extended both versions of the app to include dashboard. Please let me know if you think I should instead maybe export a useKibana that is specifically typed to the connector?

To test

Credentials for OpenAI and Azure OpenAI: https://p.elstc.co/paste/+6nHnsHt#WJOOGIkVoJjTq7zPLP77BzSG1AUxNFUDvcnSfLPtO7m

  1. Create at least two GenAI connectors with OpenAI or Azure OpenAI credentials
  2. Create multiple users
  3. Log in with each user and send a variety of prompts using the "Test" panel for each connector
  4. Open one of the connectors and click the new link to dashboard
  5. Confirm that the different users token usage is reflected
  6. Confirm that the view is filtered to the connector id:
Screenshot 2023-06-05 at 5 29 50 PM 8. Remove the filter, confirm that the Lens charts reflect that change and shows usage for both of your connectors

Testing permissions

  1. Create a user with the "Viewer" role.
  2. Login with that user and open a Gen AI connector edit view in stack management
  3. Notice that the link to the dashboard is not there

Update 6/12/23: Updated to selectively show the link to the dashboard based on user permissions
Update 6/13/23: Added a markdown text block to explain dashboard permissions, content to be updated with a doc link
Screenshot 2023-06-13 at 11 09 58 AM

stephmilovic and others added 27 commits May 25, 2023 13:47
@stephmilovic
wip
1a99899
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…
aa9dfb7 
…fix'
@stephmilovic
wip user name
06843a0
@stephmilovic
Merge branch 'gen_ai_tokens' of github.com:stephmilovic/kibana into g…
a0625be 
…en_ai_tokens
@stephmilovic
merge main
8161281
@stephmilovic
add user name to event log on gen ai events
c99edf9
@stephmilovic
add tests
388e8fa
@stephmilovic
add comment
fde9446
@stephmilovic
Merge branch 'main' into gen_ai_tokens
6f898c1
@stephmilovic
rm silly line
43181f1
@stephmilovic
remove imports from stack-connectors
c3a323b
@stephmilovic
Update action_executor.ts
c18afc0
@stephmilovic
update user
31d69cc
@stephmilovic
add user to every event log event
ce09752
@kibanamachine
Merge branch 'main' into gen_ai_tokens
45a32da
@stephmilovic
Merge remote-tracking branch 'upstream/main' into gen_ai_tokens
293c846
@stephmilovic
fix bad name
38ff3c3
@stephmilovic
ys
57a0e07
@stephmilovic
Changes for Patrick
e2c5818
@stephmilovic
Merge remote-tracking branch 'upstream/main' into gen_ai_tokens
332e404
@stephmilovic
Merge branch 'gen_ai_tokens' into token_lens
797b147
@stephmilovic
wip still
3da6f77
@stephmilovic
wip still
98bbc5a
@stephmilovic
one more
2872be8
@stephmilovic
Merge branch 'gen_ai_tokens' into token_lens
1926ef3
@stephmilovic
add usage dashboard to gen ai connector
b38ef81
@stephmilovic
Merge branch 'main' into token_lens
2c87991
@stephmilovic stephmilovic added release_note:enhancement Team:Threat Hunting Security Solution Threat Hunting Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Jun 5, 2023
@stephmilovic
Copy link
Copy Markdown
Contributor Author

stephmilovic commented Jun 14, 2023

@elasticmachine merge upstream

stephmilovic
stephmilovic commented Jun 14, 2023
View reviewed changes
x-pack/test/alerting_api_integration/common/lib/object_remover.ts
)
.set('kbn-xsrf', 'foo')
.expect(204);
.expect(plugin === 'saved_objects' ? 200 : 204);
Copy link
Copy Markdown
Contributor Author

@stephmilovic stephmilovic Jun 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm deleting a saved object, and the successful response for this api is 200

pmuellr
pmuellr approved these changes Jun 15, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@pmuellr pmuellr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kibana-ci
Copy link
Copy Markdown

kibana-ci commented Jun 15, 2023

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
stackConnectors 202 204 +2

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
actions 259 261 +2
triggersActionsUi 518 519 +1
total +3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
stackConnectors 449.8KB 457.4KB +7.6KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
stackConnectors 34.7KB 34.7KB +27.0B
triggersActionsUi 85.6KB 85.6KB +44.0B
total +71.0B
Unknown metric groups

API count

id before after diff
actions 264 266 +2
triggersActionsUi 544 545 +1
total +3

ESLint disabled line counts

id before after diff
enterpriseSearch 13 15 +2
securitySolution 410 414 +4
total +6

Total ESLint disabled count

id before after diff
enterpriseSearch 14 16 +2
securitySolution 493 497 +4
total +6

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@stephmilovic stephmilovic enabled auto-merge (squash) June 15, 2023 18:23
angorayc
angorayc approved these changes Jun 15, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@angorayc angorayc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nicely done! Thank you Steph! LGTM!

@stephmilovic stephmilovic merged commit 6c80b49 into elastic:main Jun 15, 2023
@kibanamachine kibanamachine added the backport:skip This PR does not require backporting label Jun 15, 2023
@joepeeples joepeeples mentioned this pull request Jun 22, 2023
Closed
11 tasks
@lcawl lcawl mentioned this pull request Jul 24, 2023
Merged
@stephmilovic stephmilovic mentioned this pull request Oct 17, 2023
Merged
@stephmilovic stephmilovic mentioned this pull request Nov 14, 2023
Merged
@pgayvallet pgayvallet mentioned this pull request Jan 24, 2025
Merged
pgayvallet added a commit that referenced this pull request Jan 28, 2025
@pgayvallet
Remove TS reference from stack_connectors to dashboard (#208165)
5b62617 
## Summary

In #159075 was introduced a reference from the `stack_connectors` plugin
to the `dashboard` plugin. This dependency was made so that the logic
creating the GenAI dashboard has proper typing for the dashboard SO
attributes type.

However, that is a bad isolation of concerns, introducing a high risk of
cyclic dependencies: the `dashboard` plugin, or any of its dependencies,
can't depend on `stack_connectors`, or any plugin depending on
stack_connectors.

This is especially problematic because of another bad design decision of
having `dashboard` depending on `observability_ai_assistant`, as it
increases the risk of circular dependencies.

This PR addresses it by removing the few usages we have of the dashboard
plugin's type in stack_connectors. Of course the ideal fix would be to
extract the type from the dashboard plugin instead, but those types
we're using here are inferred from config schema concrete objects,
making it complicated, so I went the pragmatic road even if it means
loosing a bit of type safety.
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jan 28, 2025
@pgayvallet
Remove TS reference from stack_connectors to dashboard (elastic#2…
fe694db 
…08165)

## Summary

In elastic#159075 was introduced a reference from the `stack_connectors` plugin
to the `dashboard` plugin. This dependency was made so that the logic
creating the GenAI dashboard has proper typing for the dashboard SO
attributes type.

However, that is a bad isolation of concerns, introducing a high risk of
cyclic dependencies: the `dashboard` plugin, or any of its dependencies,
can't depend on `stack_connectors`, or any plugin depending on
stack_connectors.

This is especially problematic because of another bad design decision of
having `dashboard` depending on `observability_ai_assistant`, as it
increases the risk of circular dependencies.

This PR addresses it by removing the few usages we have of the dashboard
plugin's type in stack_connectors. Of course the ideal fix would be to
extract the type from the dashboard plugin instead, but those types
we're using here are inferred from config schema concrete objects,
making it complicated, so I went the pragmatic road even if it means
loosing a bit of type safety.

(cherry picked from commit 5b62617)
kibanamachine added a commit that referenced this pull request Jan 28, 2025
@kibanamachine @pgayvallet
[8.x] Remove TS reference from `stack_connectors` to `…
b7f67ea 
…dashboard` (#208165) (#208604)

# Backport

This will backport the following commits from `main` to `8.x`:
- [Remove TS reference from `stack_connectors` to
`dashboard`
(#208165)](#208165)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Pierre
Gayvallet","email":"pierre.gayvallet@elastic.co"},"sourceCommit":{"committedDate":"2025-01-28T18:58:57Z","message":"Remove
TS reference from `stack_connectors` to `dashboard` (#208165)\n\n##
Summary\r\n\r\nIn #159075 was introduced a reference from the
`stack_connectors` plugin\r\nto the `dashboard` plugin. This dependency
was made so that the logic\r\ncreating the GenAI dashboard has proper
typing for the dashboard SO\r\nattributes type.\r\n\r\nHowever, that is
a bad isolation of concerns, introducing a high risk of\r\ncyclic
dependencies: the `dashboard` plugin, or any of its
dependencies,\r\ncan't depend on `stack_connectors`, or any plugin
depending on\r\nstack_connectors.\r\n\r\nThis is especially problematic
because of another bad design decision of\r\nhaving `dashboard`
depending on `observability_ai_assistant`, as it\r\nincreases the risk
of circular dependencies.\r\n\r\nThis PR addresses it by removing the
few usages we have of the dashboard\r\nplugin's type in
stack_connectors. Of course the ideal fix would be to\r\nextract the
type from the dashboard plugin instead, but those types\r\nwe're using
here are inferred from config schema concrete objects,\r\nmaking it
complicated, so I went the pragmatic road even if it means\r\nloosing a
bit of type
safety.","sha":"5b6261782e6c93ccd11c3b145cc01bf671780229","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:version","v8.18.0"],"title":"Remove
TS reference from `stack_connectors` to
`dashboard`","number":208165,"url":"https://github.com/elastic/kibana/pull/208165","mergeCommit":{"message":"Remove
TS reference from `stack_connectors` to `dashboard` (#208165)\n\n##
Summary\r\n\r\nIn #159075 was introduced a reference from the
`stack_connectors` plugin\r\nto the `dashboard` plugin. This dependency
was made so that the logic\r\ncreating the GenAI dashboard has proper
typing for the dashboard SO\r\nattributes type.\r\n\r\nHowever, that is
a bad isolation of concerns, introducing a high risk of\r\ncyclic
dependencies: the `dashboard` plugin, or any of its
dependencies,\r\ncan't depend on `stack_connectors`, or any plugin
depending on\r\nstack_connectors.\r\n\r\nThis is especially problematic
because of another bad design decision of\r\nhaving `dashboard`
depending on `observability_ai_assistant`, as it\r\nincreases the risk
of circular dependencies.\r\n\r\nThis PR addresses it by removing the
few usages we have of the dashboard\r\nplugin's type in
stack_connectors. Of course the ideal fix would be to\r\nextract the
type from the dashboard plugin instead, but those types\r\nwe're using
here are inferred from config schema concrete objects,\r\nmaking it
complicated, so I went the pragmatic road even if it means\r\nloosing a
bit of type
safety.","sha":"5b6261782e6c93ccd11c3b145cc01bf671780229"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/208165","number":208165,"mergeCommit":{"message":"Remove
TS reference from `stack_connectors` to `dashboard` (#208165)\n\n##
Summary\r\n\r\nIn #159075 was introduced a reference from the
`stack_connectors` plugin\r\nto the `dashboard` plugin. This dependency
was made so that the logic\r\ncreating the GenAI dashboard has proper
typing for the dashboard SO\r\nattributes type.\r\n\r\nHowever, that is
a bad isolation of concerns, introducing a high risk of\r\ncyclic
dependencies: the `dashboard` plugin, or any of its
dependencies,\r\ncan't depend on `stack_connectors`, or any plugin
depending on\r\nstack_connectors.\r\n\r\nThis is especially problematic
because of another bad design decision of\r\nhaving `dashboard`
depending on `observability_ai_assistant`, as it\r\nincreases the risk
of circular dependencies.\r\n\r\nThis PR addresses it by removing the
few usages we have of the dashboard\r\nplugin's type in
stack_connectors. Of course the ideal fix would be to\r\nextract the
type from the dashboard plugin instead, but those types\r\nwe're using
here are inferred from config schema concrete objects,\r\nmaking it
complicated, so I went the pragmatic road even if it means\r\nloosing a
bit of type
safety.","sha":"5b6261782e6c93ccd11c3b145cc01bf671780229"}},{"branch":"8.x","label":"v8.18.0","branchLabelMappingKey":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Pierre Gayvallet <pierre.gayvallet@elastic.co>
@mgiota mgiota mentioned this pull request Jun 19, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pmuellr pmuellr pmuellr approved these changes

@kobelb kobelb kobelb approved these changes

@angorayc angorayc angorayc approved these changes

Assignees

No one assigned

Labels

backport:skip This PR does not require backporting release_note:enhancement Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Threat Hunting Security Solution Threat Hunting Team v8.9.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@stephmilovic @elasticmachine @jamesspi @kobelb @pmuellr @kibana-ci @angorayc @kibanamachine