Skip to content

utility: route classifier and analyzer LLM calls through call-site IDs #26111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
siddseethepalli merged 1 commit into from
Apr 16, 2026
Merged

utility: route classifier and analyzer LLM calls through call-site IDs #26111

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion assistant/src/daemon/classifier.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ export async function classifyInteraction(
"You are a classifier. Determine whether the user's request requires computer use (controlling the GUI — clicking, scrolling, typing into app windows, navigating between apps) or can be handled with local tools (answering questions, running terminal commands, creating/editing/reading files, web searches, writing code). GUI tasks → computer_use. Everything else → text_qa.",
{
config: {
modelIntent: "latency-optimized",
callSite: "interactionClassifier",

@chatgpt-codex-connector chatgpt-codex-connector Bot Apr 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Route call-site requests through matching provider selection

This call now opts into config.callSite, but provider resolution in these paths still uses getConfiguredProvider()/resolveConfiguredProvider() without a call-site argument. That means transport selection still follows services.inference.provider, while RetryProvider.normalizeViaCallSite() rewrites the model from resolveCallSiteConfig(callSite, getConfig().llm). If those two configs diverge (which can happen after setModel, which only updates services.inference.*), the request can be sent to the wrong provider with an incompatible model ID (e.g. OpenAI transport + Claude model), causing hard failures and fallback behavior.

Useful? React with 👍 / 👎.

@devin-ai-integration devin-ai-integration Bot Apr 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 callSite not passed to getConfiguredProvider() in classifier — per-call-site provider overrides ignored

The PR adds callSite: "interactionClassifier" to the config at line 68, but getConfiguredProvider() at assistant/src/daemon/classifier.ts:31 is called without the callSite argument. This means the provider is selected via the legacy services.inference.provider path, while the RetryProvider resolves model/params from the call-site config. If a user configures llm.callSites.interactionClassifier.provider = "openai", the classifier would still use the default provider (e.g. anthropic), and the resolved model (e.g. gpt-4o) would be sent to the wrong provider, causing an API error. The test at assistant/src/providers/__tests__/retry-callsite.test.ts:302-314 demonstrates the expected pattern: getConfiguredProvider("heartbeatAgent") with the call site passed as an argument.

Prompt for agents 
In assistant/src/daemon/classifier.ts, the getConfiguredProvider() call at line 31 needs to receive the callSite argument so that per-call-site provider overrides are respected. Change line 31 from `const provider = await getConfiguredProvider()` to `const provider = await getConfiguredProvider("interactionClassifier")`. The same fix pattern applies to the other three files changed in this PR: style-analyzer.ts:130, skills.ts:1520, and invite-instruction-generator.ts:73 (which uses resolveConfiguredProvider instead).
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

max_tokens: 128,
tool_choice: {
type: "tool" as const,
Expand Down
2 changes: 1 addition & 1 deletion assistant/src/daemon/handlers/skills.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1539,7 +1539,7 @@ export async function draftSkill(
[],
undefined,
{
config: { modelIntent: "latency-optimized", max_tokens: 256 },
config: { callSite: "skillCategoryInference", max_tokens: 256 },

@devin-ai-integration devin-ai-integration Bot Apr 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 callSite not passed to getConfiguredProvider() in skills handler — per-call-site provider overrides ignored

The PR adds callSite: "skillCategoryInference" at line 1542, but getConfiguredProvider() at assistant/src/daemon/handlers/skills.ts:1520 is called without the callSite argument. Same root cause as the other sites: per-call-site provider overrides in llm.callSites.skillCategoryInference are silently ignored.

Prompt for agents 
In assistant/src/daemon/handlers/skills.ts, the getConfiguredProvider() call at line 1520 needs to receive the callSite argument: change `const provider = await getConfiguredProvider()` to `const provider = await getConfiguredProvider("skillCategoryInference")` so that per-call-site provider overrides are respected.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

signal,
},
);
Expand Down
5 changes: 4 additions & 1 deletion assistant/src/messaging/style-analyzer.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,10 @@ export async function extractStylePatterns(
promptMessages,
[storeStyleAnalysisTool],
STYLE_EXTRACTION_SYSTEM_PROMPT,
{ signal: AbortSignal.timeout(30_000) },
{
signal: AbortSignal.timeout(30_000),
config: { callSite: "styleAnalyzer" },

@devin-ai-integration devin-ai-integration Bot Apr 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 callSite not passed to getConfiguredProvider() in style-analyzer — per-call-site provider overrides ignored

The PR adds callSite: "styleAnalyzer" at line 152, but getConfiguredProvider() at assistant/src/messaging/style-analyzer.ts:130 is called without the callSite argument. Same root cause as the classifier: the provider is selected via the legacy path while model/params are resolved via the call-site config, so llm.callSites.styleAnalyzer.provider overrides are silently ignored. If a user configures a different provider for this call site, the model resolved from the call-site config may be incompatible with the selected provider.

Prompt for agents 
In assistant/src/messaging/style-analyzer.ts, the getConfiguredProvider() call at line 130 needs to receive the callSite argument: change `const provider = await getConfiguredProvider()` to `const provider = await getConfiguredProvider("styleAnalyzer")` so that per-call-site provider overrides are respected.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

},
);

const toolBlock = response.content.find((b) => b.type === "tool_use");
Expand Down
2 changes: 1 addition & 1 deletion assistant/src/runtime/invite-instruction-generator.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ export async function generateInviteInstruction(params: {
[userMessage(prompt)],
undefined,
undefined,
{ signal, config: { modelIntent: "latency-optimized" } },
{ signal, config: { callSite: "inviteInstructionGenerator" } },

@chatgpt-codex-connector chatgpt-codex-connector Bot Apr 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep invite generation on latency-optimized defaults

Replacing the explicit modelIntent: "latency-optimized" with callSite changes behavior when llm.callSites.inviteInstructionGenerator is not configured. LLMSchema defaults callSites to {}, and migration 038 does not seed this call site, so resolution falls back to llm.default (high-latency model by default). Because this path has a 5s timeout, the slower model will increase fallback-to-template frequency and regress instruction quality/latency for default installations.

Useful? React with 👍 / 👎.

@devin-ai-integration devin-ai-integration Bot Apr 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 callSite not passed to resolveConfiguredProvider() in invite-instruction-generator — per-call-site provider overrides ignored

The PR adds callSite: "inviteInstructionGenerator" at line 126, but resolveConfiguredProvider() at assistant/src/runtime/invite-instruction-generator.ts:73 is called without the callSite argument. Same root cause as the other sites: per-call-site provider overrides in llm.callSites.inviteInstructionGenerator are silently ignored.

Prompt for agents 
In assistant/src/runtime/invite-instruction-generator.ts, the resolveConfiguredProvider() call at line 73 needs to receive the callSite argument: change `const resolved = await resolveConfiguredProvider()` to `const resolved = await resolveConfiguredProvider("inviteInstructionGenerator")` so that per-call-site provider overrides are respected.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

);

const text = extractText(response).trim();
Expand Down
Loading