Skip to content

Comments

[Security Assistant] Enable InferenceChatModel by default#227856

Merged
stephmilovic merged 2 commits intoelastic:mainfrom
stephmilovic:inference_chat_model_disabled
Jul 15, 2025
Merged

[Security Assistant] Enable InferenceChatModel by default#227856
stephmilovic merged 2 commits intoelastic:mainfrom
stephmilovic:inference_chat_model_disabled

Conversation

@stephmilovic
Copy link
Contributor

@stephmilovic stephmilovic commented Jul 14, 2025 •
edited by kibanamachine
Loading

Summary

Updates the feature flag to disable rather than enable InferenceChatModel. This means InferenceChatModel is the default chat model, and the feature flag will disable and revert us back to the former chat models (ActionsClientChatBedrockConverse, ActionsClientChatOpenAI, ActionsClientChatVertexAI). The feature flag look like this:

# Disable InferenceChatModel
feature_flags.overrides.securitySolution.inferenceChatModelDisabled: true

Testing

  1. Send messages in the Security Assistant with Gemini, Bedrock, and OpenAI
  2. Open LangSmith to ensure that these messages were sent with InferenceChatModel
  3. Add the feature flag and send messages with the three providers
  4. Open LangSmith to ensure that these messages were sent with the former chat models

@stephmilovic stephmilovic requested a review from a team as a code owner July 14, 2025 15:19
@stephmilovic stephmilovic added release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. ci:cloud-deploy Create or update a Cloud deployment Team:Security Generative AI Security Generative AI backport:version Backport to applied version labels v9.1.0 v8.19.0 ci:security-genai-run-evals labels Jul 14, 2025
@elasticmachine
Copy link
Contributor

elasticmachine commented Jul 14, 2025

Pinging @elastic/security-solution (Team: SecuritySolution)

@stephmilovic
Copy link
Contributor Author

stephmilovic commented Jul 14, 2025

run docs-build

@elasticmachine
Copy link
Contributor

elasticmachine commented Jul 14, 2025

⏳ Build in-progress

History

spong
spong approved these changes Jul 15, 2025
View reviewed changes
Copy link
Member

@spong spong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked out, tested all relevant models locally, verifying traces, and also tested the old code path with FF disabled. Pair reviewed eval traces against the last weekly run with @stephmilovic and everything looks good! Thanks for enabling and getting everything ready to move on over to the inference plugin here Steph! 🎉 LGTM! 👍

Note: We did see some regressions with the KnowledgeBaseRetrievalTool, however upon further investigation this was already present with the existing ActionsClientChatModels, so we tested further and were able to resolve with some prompt updates, so those updates will following shortly.

@stephmilovic stephmilovic merged commit 63bde26 into elastic:main Jul 15, 2025
11 of 12 checks passed
@kibanamachine
Copy link
Contributor

kibanamachine commented Jul 15, 2025

Starting backport for target branches: 8.19, 9.1

https://github.com/elastic/kibana/actions/runs/16306304502

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jul 15, 2025
@stephmilovic
[Security Assistant] Enable InferenceChatModel by default (elastic#…
bd16226 
…227856)

(cherry picked from commit 63bde26)
@kibanamachine kibanamachine mentioned this pull request Jul 15, 2025
Merged
@kibanamachine
Copy link
Contributor

kibanamachine commented Jul 15, 2025

💔 Some backports could not be created

Status Branch Result
❌ 8.19 Backport failed because of merge conflicts
✅ 9.1

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 227856

Questions ?

Please refer to the Backport tool documentation

stephmilovic added a commit to stephmilovic/kibana that referenced this pull request Jul 15, 2025
@stephmilovic
[Security Assistant] Enable InferenceChatModel by default (elastic#…
b420c16 
…227856)

(cherry picked from commit 63bde26)

# Conflicts:
#	api_docs/kbn_elastic_assistant_common.devdocs.json
#	x-pack/platform/packages/shared/kbn-elastic-assistant-common/constants.ts
#	x-pack/solutions/security/plugins/elastic_assistant/server/routes/evaluate/post_evaluate.ts
#	x-pack/solutions/security/plugins/elastic_assistant/server/routes/post_actions_connector_execute.ts
@stephmilovic stephmilovic mentioned this pull request Jul 15, 2025
Merged
@stephmilovic
Copy link
Contributor Author

stephmilovic commented Jul 15, 2025

💚 All backports created successfully

Status Branch Result
✅ 8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

@kibanamachine kibanamachine mentioned this pull request Jul 16, 2025
Merged
1 task
kibanamachine added a commit that referenced this pull request Jul 16, 2025
@kibanamachine @stephmilovic @elasticmachine
[9.1] [Security Assistant] Enable InferenceChatModel by default (#2…
fe6000b 
…27856) (#228113)

# Backport

This will backport the following commits from `main` to `9.1`:
- [[Security Assistant] Enable `InferenceChatModel` by default
(#227856)](#227856)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Steph
Milovic","email":"stephanie.milovic@elastic.co"},"sourceCommit":{"committedDate":"2025-07-15T23:11:15Z","message":"[Security
Assistant] Enable `InferenceChatModel` by default
(#227856)","sha":"63bde264707b4dcdbe06e4c06cb8136ee2cfea37","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:
SecuritySolution","ci:cloud-deploy","Team:Security Generative
AI","backport:version","v9.1.0","v8.19.0","ci:security-genai-run-evals","v9.2.0"],"title":"[Security
Assistant] Enable `InferenceChatModel` by
default","number":227856,"url":"https://github.com/elastic/kibana/pull/227856","mergeCommit":{"message":"[Security
Assistant] Enable `InferenceChatModel` by default
(#227856)","sha":"63bde264707b4dcdbe06e4c06cb8136ee2cfea37"}},"sourceBranch":"main","suggestedTargetBranches":["9.1","8.19"],"targetPullRequestStates":[{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/227856","number":227856,"mergeCommit":{"message":"[Security
Assistant] Enable `InferenceChatModel` by default
(#227856)","sha":"63bde264707b4dcdbe06e4c06cb8136ee2cfea37"}}]}]
BACKPORT-->

Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
stephmilovic added a commit that referenced this pull request Jul 16, 2025
@stephmilovic @kibanamachine @elasticmachine
[8.19] [Security Assistant] Enable InferenceChatModel by default (#…
5faa3ef 
…227856) (#228114)

# Backport

This will backport the following commits from `main` to `8.19`:
- [[Security Assistant] Enable `InferenceChatModel` by default
(#227856)](#227856)

<!--- Backport version: 10.0.1 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Steph
Milovic","email":"stephanie.milovic@elastic.co"},"sourceCommit":{"committedDate":"2025-07-15T23:11:15Z","message":"[Security
Assistant] Enable `InferenceChatModel` by default
(#227856)","sha":"63bde264707b4dcdbe06e4c06cb8136ee2cfea37","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:
SecuritySolution","ci:cloud-deploy","Team:Security Generative
AI","backport:version","v9.1.0","v8.19.0","ci:security-genai-run-evals","v9.2.0"],"title":"[Security
Assistant] Enable `InferenceChatModel` by
default","number":227856,"url":"https://github.com/elastic/kibana/pull/227856","mergeCommit":{"message":"[Security
Assistant] Enable `InferenceChatModel` by default
(#227856)","sha":"63bde264707b4dcdbe06e4c06cb8136ee2cfea37"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/228113","number":228113,"state":"OPEN"},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/227856","number":227856,"mergeCommit":{"message":"[Security
Assistant] Enable `InferenceChatModel` by default
(#227856)","sha":"63bde264707b4dcdbe06e4c06cb8136ee2cfea37"}}]}]
BACKPORT-->

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Bluefinger pushed a commit to Bluefinger/kibana that referenced this pull request Jul 22, 2025
@stephmilovic @Bluefinger
[Security Assistant] Enable InferenceChatModel by default (elastic#…
11ddc3e 
…227856)
kertal pushed a commit to kertal/kibana that referenced this pull request Jul 25, 2025
@stephmilovic @kertal
[Security Assistant] Enable InferenceChatModel by default (elastic#…
71301d1 
…227856)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@spong spong spong approved these changes

Assignees

No one assigned

Labels

backport:version Backport to applied version labels ci:cloud-deploy Create or update a Cloud deployment ci:security-genai-run-evals release_note:skip Skip the PR/issue when compiling release notes Team:Security Generative AI Security Generative AI Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.19.0 v9.1.0 v9.2.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@stephmilovic @elasticmachine @kibanamachine @spong