[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments#181159
Merged
andrew-goldstein merged 1 commit intoelastic:mainfrom Apr 18, 2024
Merged
Conversation
andrew-goldstein
added
release_note:skip
Contributor
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
…d deployments ### Summary This PR enables LangSmith tracing for the [AI Insights](elastic#180611) feature in cloud deployments. LangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <elastic#180227> ### Details To enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags: ``` xpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights'] ``` - The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here - The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing After enabling the feature flags above: 1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant 2) Click the settings gear in the assistant 3) Click the `Evaluation` settings category 4) Click `Show Trace Options (for internal use only)` 5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below: 
andrew-goldstein
force-pushed
the
insights-langsmith
branch
from
6b61234 to
f08f960
Compare
spong
reviewed
Apr 18, 2024
| projectName?: string; | ||
| runName?: string; | ||
| tags?: string[]; | ||
| tracers?: LangChainTracer[]; |
Member
There was a problem hiding this comment.
As noted when we paired, we should update this type to support our internal APMTracer type as well so we can just add it to the array and pass along TraceOptions wherever it needs to go. No change needed here, but something I made a note of for when I'm back working some eval changes. 👍
spong
reviewed
Apr 18, 2024
Comment on lines
30
to
+31
| traceId?: string; | ||
| traceOptions?: TraceOptions; |
Member
There was a problem hiding this comment.
Again no change necessary here, but we should look to consolidate traceId into traceOptions as this is getting a little confusing. The existing traceId is intended to allow users of the ActionsClientLLM to link any calls together so there's a single traceId for all calls initiated from a single request from the kibana client. Again, I've made a note of this for when I work some eval changes here soon. We'll also want to standardize with the new usage of the ActionsClientChatOpenAI over ActionsClientLLM.
spong
approved these changes
Apr 18, 2024
Member
spong
left a comment
spong
left a comment
There was a problem hiding this comment.
Checked out, code reviewed, and tested locally -- LGTM! 👍 🚀
Added a few notes around forward looking statements that we can look to address in the future, but no changes necessary here. Nice to see how little code it takes to instrument calls -- thanks for getting this improvement in @andrew-goldstein! 🙌
spong
reviewed
Apr 18, 2024
| }: ActionsClientLlmParams) { | ||
| super({}); | ||
| super({ | ||
| callbacks: [...(traceOptions?.tracers ?? [])], |
Member
There was a problem hiding this comment.
Just noting from our pairing discussions yesterday that we wanted to include the APMTracer() here as well, but since it lives in the plugin (and ActionsClientLLM has been moved to the common package), we need to do a little refactoring first before adding something like:
💚 Build Succeeded
Metrics [docs]Async chunks
History
To update your PR or re-run it, just comment with: |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Apr 18, 2024
…eployments (elastic#181159) ## [Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments ### Summary This PR enables LangSmith tracing for the [AI Insights](elastic#180611) feature in cloud deployments. LangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <elastic#180227> ### Details To enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags: ``` xpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights'] ``` - The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here - The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing After enabling the feature flags above: 1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant 2) Click the settings gear in the assistant 3) Click the `Evaluation` settings category 4) Click `Show Trace Options (for internal use only)` 5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below:  (cherry picked from commit 89609fe)
Contributor
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Apr 18, 2024
…cloud deployments (#181159) (#181203) # Backport This will backport the following commits from `main` to `8.14`: - [[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments (#181159)](#181159) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Andrew Macri","email":"andrew.macri@elastic.co"},"sourceCommit":{"committedDate":"2024-04-18T19:01:27Z","message":"[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments (#181159)\n\n## [Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments\r\n\r\n### Summary\r\n\r\nThis PR enables LangSmith tracing for the [AI Insights](#180611) feature in cloud deployments.\r\n\r\nLangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <https://github.com/elastic/kibana/pull/180227>\r\n\r\n### Details\r\n\r\nTo enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights']\r\n```\r\n\r\n- The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here\r\n- The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing\r\n\r\nAfter enabling the feature flags above:\r\n\r\n1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant\r\n\r\n2) Click the settings gear in the assistant\r\n\r\n3) Click the `Evaluation` settings category\r\n\r\n4) Click `Show Trace Options (for internal use only)`\r\n\r\n5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below:\r\n\r\n","sha":"89609fe596d79b7d2eb4f209c5388824f9b279c1","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","Team:Security Generative AI","v8.14.0","v8.15.0","Feature:AI Insights"],"title":"[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments","number":181159,"url":"https://github.com/elastic/kibana/pull/181159","mergeCommit":{"message":"[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments (#181159)\n\n## [Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments\r\n\r\n### Summary\r\n\r\nThis PR enables LangSmith tracing for the [AI Insights](#180611) feature in cloud deployments.\r\n\r\nLangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <https://github.com/elastic/kibana/pull/180227>\r\n\r\n### Details\r\n\r\nTo enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights']\r\n```\r\n\r\n- The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here\r\n- The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing\r\n\r\nAfter enabling the feature flags above:\r\n\r\n1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant\r\n\r\n2) Click the settings gear in the assistant\r\n\r\n3) Click the `Evaluation` settings category\r\n\r\n4) Click `Show Trace Options (for internal use only)`\r\n\r\n5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below:\r\n\r\n","sha":"89609fe596d79b7d2eb4f209c5388824f9b279c1"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/181159","number":181159,"mergeCommit":{"message":"[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments (#181159)\n\n## [Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments\r\n\r\n### Summary\r\n\r\nThis PR enables LangSmith tracing for the [AI Insights](#180611) feature in cloud deployments.\r\n\r\nLangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <https://github.com/elastic/kibana/pull/180227>\r\n\r\n### Details\r\n\r\nTo enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights']\r\n```\r\n\r\n- The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here\r\n- The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing\r\n\r\nAfter enabling the feature flags above:\r\n\r\n1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant\r\n\r\n2) Click the settings gear in the assistant\r\n\r\n3) Click the `Evaluation` settings category\r\n\r\n4) Click `Show Trace Options (for internal use only)`\r\n\r\n5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below:\r\n\r\n","sha":"89609fe596d79b7d2eb4f209c5388824f9b279c1"}}]}] BACKPORT--> Co-authored-by: Andrew Macri <andrew.macri@elastic.co>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments
Summary
This PR enables LangSmith tracing for the AI Insights feature in cloud deployments.
LangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in #180227
Details
To enable LangSmith tracing in cloud deployments, configure the following
xpack.securitySolution.enableExperimentalfeature flags:assistantModelEvaluationfeature flag enables theEvaluationsettings category in the assistant. The LangSmith project name and API key are configured hereassistantAlertsInsightsfeature flag enables the AI Insights feature, which is off by default at the time of this writingAfter enabling the feature flags above:
Click the
AI Assistantbutton anywhere in the Security Solution to launch the assistantClick the settings gear in the assistant
Click the
Evaluationsettings categoryClick
Show Trace Options (for internal use only)Specify a
LangSmith ProjectandLangSmith API Keyper the screenshot below: