Repository navigation
Upgrade query-generator SQL step to Claude Sonnet 5 - #49187
Conversation
The osquery-SQL-generation call in the /query-generator page was still on claude-sonnet-4-6. Move it to claude-sonnet-5, which follows the platform/ schema constraints in the prompt more literally and should produce more reliable SQL, since Claude Sonnet 5 now defaults to adaptive thinking when `thinking` is omitted (which this helper does), add `output_config.effort` support to the shared ai.prompt helper and set it to "low" here -- the Haiku pre-filtering step already narrows the schema, so the SQL-generation step doesn't need much additional deliberation. Also bump max_tokens from 4096 to 8192 so thinking tokens don't eat into the JSON response budget and cause truncation. The schema-filtration step stays on claude-haiku-4-5, which is already the latest Haiku release.
The sqlReport call passed the system prompt as a bare object-shorthand key named systemPromptForQueryGeneration, but the ai.prompt helper's declared input is systemPrompt -- Sails silently drops unrecognized keys passed to .with(), so the "Return ONLY a raw JSON object..." system prompt was never actually reaching the model for this call. This has been broken since the query generator was switched to Anthropic (f7c20c4); the sibling filteredTables call above was unaffected since it passes systemPrompt positionally.
WalkthroughThe SQL generation controller now uses Claude Sonnet 5 with low effort. The AI prompt helper accepts an optional effort input, forwards it through Anthropic requests as 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@website/api/helpers/ai/prompt.js`:
- Around line 87-99: Update the Anthropic response handling that derives
rawPromptResponse from anthropicResponse.content to scan for the first block
whose type is "text" rather than assuming content[0] is text. Throw an explicit
error when no text block exists, preserving the existing downstream parsing flow
for valid text responses.
- Around line 97-99: Update the effort handling in the prompt helper so
output_config.effort is attached only when baseModel identifies a supported
Anthropic model; ensure claude-haiku-* requests omit it or reject the
unsupported baseModel/effort combination before sending the request.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 65ea3077-960f-44c3-9ca4-084cbb45ef4d
📒 Files selected for processing (2)
website/api/controllers/query-generator/get-llm-generated-sql.jswebsite/api/helpers/ai/prompt.js
There was a problem hiding this comment.
Pull request overview
This PR updates the website query generator’s LLM SQL-generation step to use a newer Anthropic model and adds support for configuring “effort” (adaptive thinking depth) in the shared ai.prompt helper.
Changes:
- Add an
effortinput tosails.helpers.ai.promptand forward it to Anthropic requests asoutput_config.effort. - Increase Anthropic
max_tokensfrom 4096 → 8192 in the shared helper. - Update
/query-generatorSQL generation to useclaude-sonnet-5, pass the correctsystemPromptkey, and seteffort: "low".
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| website/api/helpers/ai/prompt.js | Adds optional effort support for Anthropic calls and increases max_tokens headroom. |
| website/api/controllers/query-generator/get-llm-generated-sql.js | Switches the SQL-generation call to Sonnet 5, fixes system prompt forwarding, and sets low effort. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (2)
website/api/helpers/ai/prompt.js:42
effortis passed through to Anthropic asoutput_config.effort, but the helper input accepts any string. A typo/unsupported value would result in a 4xx from Anthropic and fail the helper. Since this helper already validates other enum-like inputs elsewhere in the codebase (viaisIn), it would be safer to constraineffortto the supported set.
effort: {
type: 'string',
description: 'Optional effort level for adaptive thinking (controls thinking depth vs. token/latency cost). Only supported on Anthropic models with output_config.effort support (e.g. Claude Sonnet 5, Claude Opus 4.6+). Ignored for models that don\'t support it (e.g. Claude Haiku 4.5, OpenAI models).',
example: 'low'
},
website/api/helpers/ai/prompt.js:127
- Anthropic responses can include multiple
textcontent blocks. Taking only the firsttextblock can truncate the answer and cause downstream JSON parsing failures. Joining alltextblocks is more robust while still skippingthinking/redacted_thinkingblocks.
let textBlock = _.find(anthropicResponse.content, { type: 'text' });
if (!textBlock) {
throw new Error('The LLM responded, but its response did not contain a text block. Full response content: '+require('util').inspect(anthropicResponse.content, {depth: 3}));
}
rawPromptResponse = textBlock.text;
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@website/api/helpers/ai/prompt.js`:
- Around line 124-125: Update the `!textBlock` error path in `prompt.js` to omit
the full `anthropicResponse.content` and include only bounded metadata, such as
content-block types and `stop_reason`; preserve the existing error behavior
while ensuring generated thinking or other response content cannot reach logs
through `get-llm-generated-sql.js`. Add a test that verifies sensitive response
blocks are absent from the emitted error.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 87912b90-6a6c-4449-b0de-310168fe492b
📒 Files selected for processing (1)
website/api/helpers/ai/prompt.js
| if (!textBlock) { | ||
| throw new Error('The LLM responded, but its response did not contain a text block. Full response content: '+require('util').inspect(anthropicResponse.content, {depth: 3})); |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 6 \
'Full response content|did not contain a text block|helpers\.ai\.prompt|sails\.log\.(warn|error).*err' \
websiteRepository: fleetdm/fleet
Length of output: 50372
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## prompt.js relevant section"
sed -n '90,145p' website/api/helpers/ai/prompt.js | cat -n
echo
echo "## get-llm-generated-sql.js relevant section"
sed -n '55,95p' website/api/controllers/query-generator/get-llm-generated-sql.js | cat -n
echo
echo "## references to anthropic usage in website/api helpers/controllers"
rg -n -C 4 "anthropic|baseModel|think|thinking|contentBlock|Claude|claude|messages" website/api website/scripts --glob '*.js' || trueRepository: fleetdm/fleet
Length of output: 50371
🌐 Web query:
Anthropic messages API content blocks thinking type documentation
💡 Result:
In the Anthropic Messages API, thinking is a feature that allows models to output their step-by-step reasoning process before providing a final answer [1][2]. This reasoning is encapsulated in dedicated thinking content blocks [1][3]. Thinking Configuration You control thinking through the thinking parameter in the request body [4][2]. There are two primary modes: - Adaptive Thinking: Enabled by setting thinking: {"type": "adaptive"}. This is the recommended mode for models that support it; it allows you to control reasoning depth using an effort parameter (e.g., output_config: {"effort": "medium"}) rather than a fixed token budget [4][5][2]. - Manual Extended Thinking: Enabled by setting thinking: {"type": "enabled", "budget_tokens": N}. This requires a specific token budget [4][5]. This mode is deprecated on newer models (Claude 4.7+) but remains the only option for certain older models (e.g., Claude 3.5, 4.5) [5]. Content Block Structure When thinking is enabled, the API returns content blocks of type thinking [1][6]. - Block Contents: These blocks contain the raw reasoning text and a cryptographic signature [6][7]. - Preserving History: If you are engaging in multi-turn conversations or tool use, you must preserve these thinking blocks in your conversation history and send them back to the API exactly as received [8][9]. Modifying them will cause the API to reject the request [8]. - Redacted Thinking: In some cases, safety systems may redact parts of the reasoning; these are returned as redacted_thinking blocks, which must also be preserved and returned in subsequent turns to maintain conversation context [3][8]. Display Settings You can control whether the thinking output is returned to you by using the display field within the thinking configuration (e.g., thinking: {"type": "adaptive", "display": "visible"}) [2]. Streaming When using streaming, thinking output is delivered via thinking_delta events within content_block_delta events, followed by a signature_delta event before the content_block_stop event [2]. Text blocks are streamed afterward [2].
Citations:
- 1: https://github.com/anthropics/anthropic-sdk-python/blob/main/src/anthropic/resources/messages/messages.py
- 2: https://platform.claude.com/docs/en/build-with-claude/thinking
- 3: https://platform.claude.com/cookbook/extended-thinking-extended-thinking
- 4: https://platform.claude.com/docs/en/build-with-claude/extended-thinking
- 5: https://platform.claude.com/docs/en/build-with-claude/extended-thinking?_rsc=1ebj0
- 6: https://docs.rs/anthropic-rs-sdk/latest/anthropic/types/messages/enum.ContentBlock.html
- 7: https://repos.supermodeltools.com/anthropic-sdk-python/file-src-anthropic-types-thinking-block-py.html
- 8: https://platform.claude.com/cookbook/extended-thinking-extended-thinking-with-tool-use
- 9: https://platform.claude.com/docs/en/build-with-claude/thinking-tool-workflows
Do not include the full Anthropic response in the exception.
website/api/helpers/ai/prompt.js:124-125 formats every Anthropic response block into the error. website/api/controllers/query-generator/get-llm-generated-sql.js:74-75 and 179-183 then inspect/log that error, which can expose generated thinking or other generated content to application logs. Log only bounded metadata such as content-block types and stop_reason, and add a test asserting sensitive response blocks are absent from the emitted error.
Proposed fix
let textBlock = _.find(anthropicResponse.content, { type: 'text' });
if (!textBlock) {
- throw new Error('The LLM responded, but its response did not contain a text block. Full response content: '+require('util').inspect(anthropicResponse.content, {depth: 3}));
+ let contentBlockTypes = Array.isArray(anthropicResponse.content)
+ ? anthropicResponse.content.map((block) => block && block.type).join(', ')
+ : 'unknown';
+ throw new Error('The LLM responded, but its response did not contain a text block. Content block types: '+contentBlockTypes);
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if (!textBlock) { | |
| throw new Error('The LLM responded, but its response did not contain a text block. Full response content: '+require('util').inspect(anthropicResponse.content, {depth: 3})); | |
| if (!textBlock) { | |
| let contentBlockTypes = Array.isArray(anthropicResponse.content) | |
| ? anthropicResponse.content.map((block) => block && block.type).join(', ') | |
| : 'unknown'; | |
| throw new Error('The LLM responded, but its response did not contain a text block. Content block types: '+contentBlockTypes); | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@website/api/helpers/ai/prompt.js` around lines 124 - 125, Update the
`!textBlock` error path in `prompt.js` to omit the full
`anthropicResponse.content` and include only bounded metadata, such as
content-block types and `stop_reason`; preserve the existing error behavior
while ensuring generated thinking or other response content cannot reach logs
through `get-llm-generated-sql.js`. Add a test that verifies sensitive response
blocks are absent from the emitted error.
Source: MCP tools
Related issue: N/A
What this does
The
/query-generatorpage's osquery-SQL-generation step (get-llm-generated-sql.js) was onclaude-sonnet-4-6, which is now one generation behind. This PR:claude-sonnet-5. The schema-filtration step stays onclaude-haiku-4-5, which is already the latest Haiku release, so no change needed there.effortsupport to the sharedai.prompthelper, forwarded asoutput_config.efforton Anthropic requests, and sets it to"low"for the SQL-generation call. Effort controls how much the model deliberates (and how many tokens/how much latency that costs)."low"was chosen because the Haiku pre-filtering step already narrows the osquery schema down to relevant tables, so the Sonnet step isn't starting from scratch and doesn't need to spend much effort re-deriving that context.max_tokensin the Anthropic branch of the helper from 4096 to 8192. Claude Sonnet 5 turns on adaptive thinking by default when thethinkingparam is omitted (which this helper does), andmax_tokensis a hard cap on total output including thinking tokens — at 4096 there was a real risk of thinking tokens eating into the budget and truncating the JSON response the SQL step needs to return.sqlReportcall passed the system prompt as a bare object-shorthand key namedsystemPromptForQueryGeneration, but theai.prompthelper's declared input issystemPrompt. Sails silently drops unrecognized keys passed to.with(...), so the "Return ONLY a raw JSON object..." system prompt was never actually reaching the model for this call. This has been broken since the query generator was switched to Anthropic (f7c20c4731); the siblingfilteredTablescall above it was unaffected since it passessystemPromptpositionally. Now fixed tosystemPrompt: systemPromptForQueryGeneration.Why
Claude Sonnet 5 follows structured/constrained instructions (don't alias tables, use
LIKEwith wildcards, only reference documented columns, etc.) more literally than 4.6, which should make the generated SQL more reliable. It's priced the same or cheaper than 4.6 during the current introductory period.Trade-offs called out for review
sails.helpers.http.post, single blocking call over a socket), so any added thinking time is invisible wait time for the user rather than a visible "thinking" indicator.effort: "low"should keep this modest, but worth confirming with a manual QA pass on a few representative questions before merging.output_config.effort(addedeffortis a no-op if passed to it), so it was left as-is.Checklist for submitter
If some of the following don't apply, delete the relevant line.
Changes file added for user-visible changes in
changes/,orbit/changes/oree/fleetd-chrome/changes.See Changes files for more information.
Input data is properly validated,
SELECT *is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters.Timeouts are implemented and retries are limited to avoid infinite loops
If paths of existing endpoints are modified without backwards compatibility, checked the frontend/CLI for any necessary changes
Testing
Added/updated automated tests
Where appropriate, automated tests simulate multiple hosts and test for host isolation (updates to one hosts's records do not affect another)
QA'd all new/changed functionality manually
Summary by CodeRabbit