Repository navigation
fix(sdk): Add structured output instructions to prevent conversational filler in JSON responses - #576
Conversation
|
Important Review skippedBot user detected. To trigger a single review, invoke the You can disable this status message by setting the Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughAdds Claude 4.5 and Gemini 3/2.0 model entries across providers, extends vision/model enums and token limits, introduces structured-output plumbing (schema + disableTools), injects structured-output instructions into prompts, improves generation parsing fallbacks, and adds docs, examples, and tests for schema-driven outputs and Google Gemini limitations. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Client
participant SDK as NeuroLink SDK
participant GenHandler as GenerationHandler
participant Provider
participant Tools
participant Validator as Zod Validator
Client->>SDK: call generate({ prompt, schema?, disableTools? })
SDK->>GenHandler: build messages (include STRUCTURED_OUTPUT_INSTRUCTIONS if schema)
GenHandler->>Tools: (if !disableTools & tools present) prepare tool calls
GenHandler->>Provider: send generate request (messages, tools?, toolChoice?)
Provider-->>GenHandler: streaming/response (experimental_output?, text)
GenHandler->>GenHandler: extract structured JSON (prefer experimental_output, strip fences fallback)
GenHandler->>Validator: validate parsed JSON against schema
alt validation success
GenHandler-->>SDK: return GenerateResult{ content: parsedJSON, raw: ... }
SDK-->>Client: resolved result
else validation failure
GenHandler-->>SDK: return error / NoObjectGeneratedError
SDK-->>Client: error
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Areas requiring extra attention:
Possibly related PRs
Suggested labels
Poem
Pre-merge checks and finishing touches❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
Comment |
There was a problem hiding this comment.
Pull request overview
This PR fixes a JSON parsing issue where AI providers (particularly Vertex) return conversational filler text like "Excellent!" before the JSON output when using structured output with Zod schemas, causing NoObjectGeneratedError failures. The fix adds explicit JSON-only instructions to the system prompt when structured output is requested.
Key Changes:
- Added
STRUCTURED_OUTPUT_INSTRUCTIONSconstant with explicit JSON-only output requirements - Enhanced
buildMessagesArrayandbuildMultimodalMessagesArrayto conditionally append structured output instructions when schema + json/structured format are detected - Added 10 unit tests validating the instruction injection behavior
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
src/lib/config/conversationMemory.ts |
Adds new constant STRUCTURED_OUTPUT_INSTRUCTIONS with explicit JSON formatting requirements |
src/lib/utils/messageBuilder.ts |
Implements shouldUseStructuredOutput() helper and integrates instruction injection into both message builder functions |
test/unit/structured-output.test.ts |
Adds comprehensive test suite validating instruction injection behavior for various scenarios |
src/lib/neurolink.ts |
Code formatting: multi-line import statement reformatting |
src/lib/factories/providerRegistry.ts |
Code formatting: multi-line import statement reformatting |
src/lib/sdk/toolRegistration.ts |
Code formatting: multi-line type definition reformatting |
src/cli/index.ts |
Code formatting: multi-line import statement reformatting |
src/cli/factories/commandFactory.ts |
Code formatting: multi-line import statement reformatting |
src/cli/commands/setup-gcp.ts |
Code formatting: console log statement reformatting |
neurolink-demo/public/mcp-automatic-demo.html |
Code formatting: HTML onclick attribute reformatting |
neurolink-demo/mcp-automatic-demo.html |
Code formatting: HTML onclick attribute reformatting |
docs/getting-started/providers/litellm.md |
Code formatting: blank line addition |
CLAUDE.md |
Code formatting: import statement reformatting |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.
| const systemContent = systemMessage?.content as string; | ||
| expect(systemContent).toContain("JSON"); | ||
| }); | ||
| }); |
There was a problem hiding this comment.
Missing test case: There's no test case for when a schema is provided but output.format is undefined/not specified. According to the shouldUseStructuredOutput logic, this scenario would NOT add structured output instructions. This edge case should be tested to verify that the instructions are only added when BOTH schema AND explicit format ("json" or "structured") are present.
| }); | ||
|
|
||
| describe("Structured Output Instructions Constant", () => { | ||
| it("should have defined structured output instructions", async () => { | ||
| // This test will verify that the STRUCTURED_OUTPUT_INSTRUCTIONS constant exists | ||
| // after we implement it | ||
| const { STRUCTURED_OUTPUT_INSTRUCTIONS } = | ||
| await import("../../src/lib/config/conversationMemory.js"); | ||
|
|
||
| // If the constant exists, verify its content | ||
| if (STRUCTURED_OUTPUT_INSTRUCTIONS) { | ||
| expect(STRUCTURED_OUTPUT_INSTRUCTIONS).toContain("JSON"); | ||
| // Should instruct to avoid preamble | ||
| expect(STRUCTURED_OUTPUT_INSTRUCTIONS.toLowerCase()).toMatch( | ||
| /only|pure|valid|no.*text|no.*preamble|directly/, | ||
| ); | ||
| } | ||
| }); | ||
| }); |
There was a problem hiding this comment.
Missing test coverage: The buildMultimodalMessagesArray function has the same structured output instruction logic added (lines 652-655 in messageBuilder.ts) but there are no tests covering this behavior in the multimodal context. Tests should validate that structured output instructions are properly added for multimodal messages when schema and json/structured format are specified.
| it("should NOT include JSON-only instructions when output format is text", async () => { | ||
| const options: TextGenerationOptions = { | ||
| prompt: "Tell me a story", | ||
| systemPrompt: "You are a helpful assistant.", | ||
| output: { format: "text" }, | ||
| }; | ||
|
|
||
| const messages = await buildMessagesArray(options); | ||
|
|
||
| // Find the system message | ||
| const systemMessage = messages.find((m) => m.role === "system"); | ||
| expect(systemMessage).toBeDefined(); | ||
| expect(typeof systemMessage?.content).toBe("string"); | ||
|
|
||
| const systemContent = systemMessage?.content as string; | ||
|
|
||
| // Should NOT contain structured output instructions | ||
| expect(systemContent).not.toContain("STRUCTURED OUTPUT REQUIREMENT"); | ||
| expect(systemContent).not.toContain("conversational filler"); | ||
| }); |
There was a problem hiding this comment.
Missing test case: The tests validate that instructions are NOT added when output.format is "text" (line 105-124), but there's no test for the edge case where a schema is provided WITH output.format: "text". This combination could occur in practice and the current implementation would not add the structured output instructions (by design), but this behavior should be explicitly tested to document the expected behavior.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // Add structured output instructions when schema is provided with json/structured format | ||
| if (shouldUseStructuredOutput(options)) { | ||
| systemPrompt = `${systemPrompt}${STRUCTURED_OUTPUT_INSTRUCTIONS}`; | ||
| } |
There was a problem hiding this comment.
The new structured-output block in buildMultimodalMessagesArray is guarded by shouldUseStructuredOutput(options), but the caller (MessageBuilder) builds multimodalOptions without forwarding schema or output.format, so any request that includes images/files (and thus takes the multimodal path) will arrive here with those fields undefined and the guard remains false. As a result structured-output requests with attachments still omit the JSON-only instructions and can continue returning conversational preambles, defeating the intended fix. Pass through the schema/output to the multimodal builder so these instructions are actually injected.
Useful? React with 👍 / 👎.
🤖 AI Review & Build Compliance ✅Status: AI analysis complete • Build rules validated • Ready for review 📊 View detailed analysis results🛡️ Analysis Complete
📋 Ready for Merge When
🤖 AI analysis complete - check individual code comments for specific feedback |
c9e2576 to
c645b66
Compare
✅ Single Commit Policy - COMPLIANTStatus: Policy requirements met • 1 commit • Valid format • Ready for merge 📊 View validation details📝 Commit Details
✅ Validation Results
🤖 Automated validation by NeuroLink Single Commit Enforcement |
🤖 AI Review & Build Compliance ✅Status: AI analysis complete • Build rules validated • Ready for review 📊 View detailed analysis results🛡️ Analysis Complete
📋 Ready for Merge When
🤖 AI analysis complete - check individual code comments for specific feedback |
|
@coderabbitai full review |
✅ Actions performedFull review triggered. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 26 out of 26 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Claude 4 Series (Latest - May 2025) | ||
| // Claude 4.5 Series (Latest - December 2025) | ||
| CLAUDE_4_5_SONNET = "claude-sonnet-4-5@20250929", | ||
| CLAUDE_4_5_OPUS = "claude-opus-4-5@20251124", |
There was a problem hiding this comment.
Inconsistent date format in model version. This model ID uses @20251124 (November 24) which differs from:
- BedrockModels.CLAUDE_4_5_OPUS:
"anthropic.claude-opus-4-5-20251101-v1:0"(Nov 1) - AnthropicModels.CLAUDE_OPUS_4_5:
"claude-opus-4-5-20251101"(Nov 1) - modelRegistry.ts releaseDate:
"2025-11-01"(Nov 1) - models.json releaseDate:
"2025-11-01"(Nov 1)
Should be "claude-opus-4-5@20251101" to match other definitions.
| CLAUDE_4_5_OPUS = "claude-opus-4-5@20251124", | |
| CLAUDE_4_5_OPUS = "claude-opus-4-5@20251101", |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
♻️ Duplicate comments (4)
src/lib/utils/messageBuilder.ts (3)
325-328: Address inconsistent string concatenation pattern.When appending
CONVERSATION_INSTRUCTIONSat line 322,systemPromptis trimmed first:${systemPrompt.trim()}${CONVERSATION_INSTRUCTIONS}. However, at line 327 when appendingSTRUCTURED_OUTPUT_INSTRUCTIONS, there's no trim:${systemPrompt}${STRUCTURED_OUTPUT_INSTRUCTIONS}.For consistency and to avoid potential whitespace handling issues, both should use the same pattern.
Apply this diff to match the trimming pattern used at line 322:
- systemPrompt = `${systemPrompt}${STRUCTURED_OUTPUT_INSTRUCTIONS}`; + systemPrompt = `${systemPrompt.trim()}${STRUCTURED_OUTPUT_INSTRUCTIONS}`;
652-655: Address inconsistent string concatenation pattern.Similar to the issue in
buildMessagesArray, line 649 trimssystemPromptbefore appendingCONVERSATION_INSTRUCTIONS, but line 654 doesn't trim before appendingSTRUCTURED_OUTPUT_INSTRUCTIONS.Apply this diff for consistency with line 649:
- systemPrompt = `${systemPrompt}${STRUCTURED_OUTPUT_INSTRUCTIONS}`; + systemPrompt = `${systemPrompt.trim()}${STRUCTURED_OUTPUT_INSTRUCTIONS}`;
652-655: Structured output instructions are not reaching multimodal requests — schema and output.format properties must be forwardedAll callers of
buildMultimodalMessagesArrayconstructmultimodalOptionswithout includingschemaoroutput.formatfrom the original request. This causesshouldUseStructuredOutput()to always return false for multimodal requests (line 653), preventing structured output instructions from being added.Fix locations:
- MessageBuilder.ts (lines 62-81, 155-176): Add
schema: options.schemaandoutput: options.outputto multimodalOptions- buildMultimodalOptions() in multimodalOptionsBuilder.ts (lines 45-69): Add same properties to returned object
test/unit/structured-output.test.ts (1)
183-183: Add test for schema with undefined output format.As noted in a previous review, there's no test for when
schemais provided butoutput.formatis undefined. This edge case should be explicitly tested to document expected behavior (instructions should NOT be added per theshouldUseStructuredOutputlogic).
🧹 Nitpick comments (12)
src/lib/constants/enums.ts (1)
131-134: Minor naming inconsistency in AnthropicModels enum.The naming convention differs from other enums:
AnthropicModels:CLAUDE_SONNET_4_5,CLAUDE_OPUS_4_5(model name, then version)BedrockModels/VertexModels:CLAUDE_4_5_SONNET,CLAUDE_4_5_OPUS(version, then model name)This inconsistency may cause confusion when users switch between providers. Consider aligning the naming pattern.
src/lib/config/conversationMemory.ts (1)
36-49: Well-crafted instructions to enforce JSON-only output.This directly addresses the PR objective of preventing AI providers from prepending conversational filler (e.g., "Excellent! Now I have all the information...") before JSON output. The instructions cover common failure modes effectively.
One minor consideration: Line 48 states "starting with { and ending with }" which assumes object output. If array schemas are supported (starting with
[), you may want to generalize this:-- Output ONLY the raw JSON object, starting with { and ending with } +- Output ONLY the raw JSON value, starting with { or [ and ending with } or ]docs/features/structured-output.md (1)
87-125: Verify: Potential content duplication in this file.According to the AI summary, this Google Gemini limitation section appears duplicated in two locations within this file. While the guidance itself is clear and helpful, having identical content in multiple places within the same document can confuse readers and creates maintenance overhead.
Please verify whether this content block appears elsewhere in the file and consolidate if needed.
docs/getting-started/providers/google-vertex.md (1)
730-795: Verify: Content duplication detected in this file.The AI summary indicates this "Known Limitations" section appears duplicated within this file (two inserted copies with identical content). While the guidance is valuable, duplicating entire sections can confuse readers and make maintenance more difficult.
Please check if this section appears multiple times in the document and consolidate to a single location.
docs/getting-started/providers/google-ai.md (1)
875-929: Verify: Content duplication within this file.Consistent with other documentation files in this PR, the AI summary indicates this "Known Limitations" section may be duplicated within the document. Please verify and consolidate to avoid reader confusion and maintenance overhead.
docs/TROUBLESHOOTING.md (1)
930-1034: Helpful troubleshooting guidance with potential duplication.The troubleshooting sections provide clear symptom-solution patterns that will help users quickly resolve these common issues. The industry context is valuable for setting proper expectations.
However, the AI summary indicates duplicate content appears within this section. Please verify and consolidate if needed.
examples/structured-output-google-providers.ts (2)
37-38: Add error handling for JSON.parse.
JSON.parsecan throw if the response contains malformed JSON. In an example meant to demonstrate structured output, showing proper error handling would be valuable.- const analysis = JSON.parse(result.content); - console.log("Analysis:", JSON.stringify(analysis, null, 2)); + try { + const analysis = JSON.parse(result.content); + console.log("Analysis:", JSON.stringify(analysis, null, 2)); + } catch (parseError) { + console.error("Failed to parse JSON response:", result.content); + throw parseError; + }
47-59: Unused variable in error demonstration.The
resultvariable is declared but never used since the call is expected to fail. Consider using an underscore prefix to indicate intentional non-use, or remove the variable declaration entirely.try { - const result = await neurolink.generate({ + await neurolink.generate({ input: { text: "Analyze TechCorp as an investment opportunity", }, schema: CompanyAnalysisSchema, output: { format: "json" }, provider: "vertex", // Missing: disableTools: true }); + console.log("Unexpected success - this should have failed"); } catch (error) { console.error("Expected error:", error); }test/unit/structured-output.test.ts (1)
205-223: Test assertions could be more specific to optional field handling.The test is named "should handle schema with optional fields correctly" but only validates basic message structure. Consider adding assertions that specifically verify the optional field behavior if that's the intent.
src/lib/models/modelRegistry.ts (1)
558-609: Consider updating USE_CASE_RECOMMENDATIONS with new Claude 4.5 models.The new Claude 4.5 Opus and Sonnet models have higher use case scores than Claude 3.5 Sonnet (10 vs 9-10), but
USE_CASE_RECOMMENDATIONSstill references only Claude 3.5 models. Consider adding the 4.5 variants for use cases where they excel.docs/reference/provider-feature-compatibility.md (1)
129-129: Consider rewording speculative future support claim.The statement about Gemini 3 Pro Preview supporting both tools and schemas in Nov 2025 may become outdated or inaccurate. Consider linking to official Google documentation or rewording to be less definitive.
-- **Future:** Gemini 3 Pro Preview (Nov 2025) will support both +- **Future:** Gemini 3 Pro Preview may support both (refer to Google's official documentation for updates)test/zod-schema-test-function.ts (1)
355-358: Consider extracting expected values from fixtures dynamically.The hardcoded
expectedTopPerformers,expectedAccountId, andexpectedCampaignIdsvalues create a maintenance burden if fixtures change. Consider loading these values from the fixture files or defining them in a shared constants file.Also applies to: 628-630
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
test/fixtures/meta-ads-campaign-performance.csvis excluded by!**/*.csv
📒 Files selected for processing (25)
config/models.json(4 hunks)docs/TROUBLESHOOTING.md(1 hunks)docs/features/structured-output.md(1 hunks)docs/getting-started/providers/google-ai.md(1 hunks)docs/getting-started/providers/google-vertex.md(1 hunks)docs/reference/provider-comparison.md(2 hunks)docs/reference/provider-feature-compatibility.md(2 hunks)docs/sdk/api-reference.md(1 hunks)examples/structured-output-google-providers.ts(1 hunks)examples/structured-output-test.js(4 hunks)src/lib/adapters/providerImageAdapter.ts(5 hunks)src/lib/config/conversationMemory.ts(1 hunks)src/lib/constants/enums.ts(6 hunks)src/lib/constants/tokens.ts(3 hunks)src/lib/core/modules/GenerationHandler.ts(2 hunks)src/lib/models/modelRegistry.ts(1 hunks)src/lib/providers/googleAiStudio.ts(1 hunks)src/lib/providers/googleVertex.ts(2 hunks)src/lib/types/generateTypes.ts(1 hunks)src/lib/utils/messageBuilder.ts(4 hunks)test/continuous-test-suite.ts(2 hunks)test/fixtures/meta-ads-account-metrics.json(1 hunks)test/fixtures/zod-sample.ts(1 hunks)test/unit/structured-output.test.ts(1 hunks)test/zod-schema-test-function.ts(1 hunks)
🧰 Additional context used
🧠 Learnings (11)
📚 Learning: 2025-09-17T17:55:15.261Z
Learnt from: RajuSudhar
Repo: juspay/neurolink PR: 173
File: src/lib/index.ts:16-16
Timestamp: 2025-09-17T17:55:15.261Z
Learning: In src/lib/types/providers.ts, ProviderConfig was renamed to AIModelProviderConfig to deduplicate type names, as there was an existing ProviderConfig type that better suited the "ProviderConfig" name. This was an intentional breaking change for better type organization.
Applied to files:
src/lib/providers/googleAiStudio.tssrc/lib/providers/googleVertex.tsexamples/structured-output-google-providers.tssrc/lib/models/modelRegistry.tsdocs/reference/provider-comparison.mdsrc/lib/adapters/providerImageAdapter.tsdocs/reference/provider-feature-compatibility.mddocs/sdk/api-reference.mdconfig/models.jsondocs/getting-started/providers/google-ai.mdsrc/lib/constants/enums.tssrc/lib/constants/tokens.ts
📚 Learning: 2025-09-02T13:50:42.770Z
Learnt from: YasmeenOgo
Repo: juspay/neurolink PR: 145
File: src/lib/core/types.ts:0-0
Timestamp: 2025-09-02T13:50:42.770Z
Learning: The APIVersions enum in src/lib/core/types.ts now contains comprehensive API version constants for all major AI providers: Azure OpenAI (latest, stable, legacy), OpenAI (current, beta), Google AI (current, beta), and Anthropic (current). This centralization helps avoid API version drift across the codebase.
Applied to files:
src/lib/providers/googleAiStudio.tssrc/lib/providers/googleVertex.tsexamples/structured-output-google-providers.tssrc/lib/models/modelRegistry.tsdocs/reference/provider-comparison.mdsrc/lib/adapters/providerImageAdapter.tsdocs/reference/provider-feature-compatibility.mddocs/sdk/api-reference.mdconfig/models.jsonsrc/lib/constants/enums.tssrc/lib/constants/tokens.ts
📚 Learning: 2025-12-01T08:39:22.794Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-01T08:39:22.794Z
Learning: Provider implementations must extend a base provider or implement the provider interface and be registered with provider name, factory function, default model, and aliases
Applied to files:
src/lib/providers/googleAiStudio.ts
📚 Learning: 2025-12-01T08:39:22.794Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-01T08:39:22.794Z
Learning: Use MessageBuilder (src/lib/utils/messageBuilder.ts) as the central component for constructing all messages with multimodal support
Applied to files:
test/unit/structured-output.test.tssrc/lib/utils/messageBuilder.ts
📚 Learning: 2025-09-24T06:42:06.088Z
Learnt from: amreetkhuntia
Repo: juspay/neurolink PR: 185
File: src/lib/evaluation/contextBuilder.ts:79-85
Timestamp: 2025-09-24T06:42:06.088Z
Learning: In the NeuroLink codebase, using `(options.prompt || [])` pattern for handling potentially undefined prompt arrays is the preferred approach over extracting to a normalized variable when building conversation history in the ContextBuilder class.
Applied to files:
src/lib/utils/messageBuilder.ts
📚 Learning: 2025-09-24T07:26:41.988Z
Learnt from: amreetkhuntia
Repo: juspay/neurolink PR: 185
File: src/lib/evaluation/prompts.ts:86-101
Timestamp: 2025-09-24T07:26:41.988Z
Learning: In the neurolink codebase, maintainer amreetkhuntia consistently prefers to keep template literal indentation in LLM prompts (including evaluation prompts in src/lib/evaluation/prompts.ts) for readability, even when it results in extra whitespace in the output, as LLMs can parse and understand the content correctly.
Applied to files:
src/lib/utils/messageBuilder.ts
📚 Learning: 2025-09-24T06:43:23.653Z
Learnt from: amreetkhuntia
Repo: juspay/neurolink PR: 185
File: src/lib/evaluation/prompts.ts:59-72
Timestamp: 2025-09-24T06:43:23.653Z
Learning: In the neurolink codebase, maintainer amreetkhuntia prefers to keep template literal indentation in LLM prompts even if it results in technically malformed JSON format, as LLMs can understand and parse it correctly despite formatting issues.
Applied to files:
src/lib/utils/messageBuilder.ts
📚 Learning: 2025-09-01T22:58:39.149Z
Learnt from: sudharsan-juspay
Repo: juspay/neurolink PR: 140
File: src/lib/core/types.ts:198-203
Timestamp: 2025-09-01T22:58:39.149Z
Learning: In src/lib/core/types.ts, StreamOptions (imported from streamTypes.js) and StreamingOptions are intentionally different types with different use cases. StreamingOptions is for unified AI requests with multiple provider configurations, while StreamOptions is for individual streaming operations.
Applied to files:
src/lib/utils/messageBuilder.ts
📚 Learning: 2025-09-17T18:14:34.960Z
Learnt from: RajuSudhar
Repo: juspay/neurolink PR: 173
File: src/lib/types/index.ts:58-62
Timestamp: 2025-09-17T18:14:34.960Z
Learning: RajuSudhar explained that in the Neurolink codebase, there are multiple ProviderConfig types causing inconsistency. One existing ProviderConfig type better suited the "ProviderConfig" name, so they renamed the less-suitable one to AIModelProviderConfig to free up the name. Adding backward compatibility aliases would worsen naming inconsistency rather than help. The remaining duplicates will be systematically deduplicated in the 07-Types-Module.md TODO as part of their phased refactor approach.
Applied to files:
docs/reference/provider-comparison.mdsrc/lib/adapters/providerImageAdapter.tssrc/lib/constants/tokens.ts
📚 Learning: 2025-12-01T08:39:22.794Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-01T08:39:22.794Z
Learning: Update vision capabilities in ProviderImageAdapter when adding multimodal support to new providers
Applied to files:
src/lib/adapters/providerImageAdapter.ts
📚 Learning: 2025-12-01T08:39:22.794Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-01T08:39:22.794Z
Learning: Integrate provider-specific image formatting through ProviderImageAdapter when adding multimodal support to new providers
Applied to files:
src/lib/adapters/providerImageAdapter.ts
🧬 Code graph analysis (5)
src/lib/types/generateTypes.ts (2)
src/lib/types/index.ts (1)
ValidationSchema(14-14)src/lib/types/typeAliases.ts (1)
ValidationSchema(25-25)
src/lib/core/modules/GenerationHandler.ts (3)
examples/interface-usage/registry-interface-demo.js (1)
tools(74-77)src/lib/core/constants.ts (1)
DEFAULT_MAX_STEPS(10-10)src/lib/utils/logger.ts (2)
error(239-241)logger(358-401)
test/continuous-test-suite.ts (1)
test/zod-schema-test-function.ts (1)
testComplexZodSchemaMultiProvider(385-713)
src/lib/utils/messageBuilder.ts (1)
src/lib/config/conversationMemory.ts (1)
STRUCTURED_OUTPUT_INSTRUCTIONS(41-49)
test/zod-schema-test-function.ts (1)
test/fixtures/zod-sample.ts (1)
MetaAdsAnalysisOutputSchema(280-315)
⏰ Context from checks skipped due to timeout of 900000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Agent
🔇 Additional comments (35)
test/fixtures/meta-ads-account-metrics.json (1)
1-25: Well-structured test fixture for Meta Ads metrics.The fixture provides realistic test data with clear period comparison structure. The
accountId: "act_123456789"matches the validation intestComplexZodSchemaMultiProvider(Line 571 intest/zod-schema-test-function.ts).test/continuous-test-suite.ts (2)
33-33: Import addition looks correct.The import uses the
.jsextension which is appropriate for ESM module resolution in Node.js.
2818-2821: Test case properly integrated into the suite.The new
testComplexZodSchemaMultiProvidertest is correctly added to the test array. This aligns with the PR objective to add test coverage for structured output validation across providers.examples/structured-output-test.js (1)
104-104: Model change to Claude 4.5 on Vertex is appropriate.The switch from
gemini-2.5-flashtoclaude-sonnet-4-5@20250929aligns with theVertexModels.CLAUDE_4_5_SONNETdefinition insrc/lib/constants/enums.ts. This model is consistently used across the codebase in test files, examples, and providers, with proper token mappings configured. Using Claude for structured output tests avoids potential Gemini-specific JSON parsing limitations.src/lib/providers/googleAiStudio.ts (1)
55-82: LGTM: Clear and comprehensive structured output limitation documentation.The JSDoc block effectively documents the Google Gemini limitation with proper annotations, error messages, examples, and future support notes. This will help developers understand the constraint and how to work around it.
src/lib/providers/googleVertex.ts (2)
302-337: LGTM: Comprehensive documentation with clear examples.The JSDoc documentation clearly explains the Gemini limitation and provides distinct examples for both Gemini models (requiring disableTools) and Claude models (no limitation). This will help developers choose the right approach.
929-936: No duplicate model entry found.The model suggestions list contains
gemini-2.0-flash-liteonly once at line 1936 within the Google models array. The complete list contains no duplicate entries.src/lib/types/generateTypes.ts (1)
44-96: LGTM: Well-documented public API additions.The new
schemaanddisableToolsproperties are properly typed as optional, maintaining backward compatibility. The extensive JSDoc documentation clearly explains the Google Gemini limitation and provides practical examples for different providers.src/lib/utils/messageBuilder.ts (1)
287-300: LGTM: Clean helper function for structured output detection.The
shouldUseStructuredOutputhelper provides a clear, reusable check for when structured output instructions should be applied. The logic correctly validates both the presence of a schema and the appropriate output format.examples/structured-output-google-providers.ts (2)
136-149: Well-structured example with proper cleanup.Good use of sequential async/await pattern and proper resource cleanup with
dispose(). The example effectively demonstrates the Google provider limitation and workarounds.
90-90: No action needed — the model identifier is valid and current.The model
claude-sonnet-4-5@20250929is a valid Claude Sonnet 4.5 version released on September 29, 2025, and is generally available on Vertex AI across multiple regions. The date suffix follows Vertex AI's standard versioning format and is current as of December 2025.docs/sdk/api-reference.md (1)
595-638: Clear and comprehensive provider limitations documentation.The new section clearly documents the Google Gemini schema+tools limitation with actionable workarounds and a provider support matrix. The examples are helpful for developers encountering this issue.
docs/reference/provider-comparison.md (2)
5-17: Comprehensive provider matrix update.The updated matrix clearly indicates the Tools + Schema limitation for Google providers with a helpful footnote. This is consistent with the API reference documentation.
152-181: Well-structured structured output documentation.The new section clearly categorizes providers by support level and provides a practical workaround code example. The future support note provides helpful context for planning.
src/lib/models/modelRegistry.ts (2)
256-300: LGTM - Claude Sonnet 4.5 model entry.The model entry is well-structured with comprehensive metadata, appropriate capabilities, and reasonable pricing/limits configuration.
244-249: No action needed. All aliases in the model registry are unique—verification confirms 18 total aliases with 18 unique values, meaning there are no conflicts. The aliases "claude-sonnet-latest" (Claude 4.5 Sonnet) and "claude-latest" (Claude 3.5 Sonnet) are distinct strings and will not overwrite each other in the MODEL_ALIASES registry.Likely an incorrect or invalid review comment.
src/lib/core/modules/GenerationHandler.ts (1)
76-78: LGTM!The conditional spreading of
toolsandtoolChoicecorrectly respects thedisableToolsoption and avoids passing an empty tools object to the API.docs/reference/provider-feature-compatibility.md (1)
115-147: LGTM!The new "Structured Output Support Details" section clearly documents the provider-specific behaviors and provides a practical code example for handling the Google providers' tool+schema limitation.
src/lib/adapters/providerImageAdapter.ts (4)
50-65: LGTM!The Gemini 3.x series additions with preview and latest variants are properly structured, and the legacy 1.5 series is preserved for backward compatibility. Based on learnings, this aligns with the pattern of updating vision capabilities when adding multimodal support to new providers.
66-82: LGTM!The Claude 4.5 series additions with dated version suffixes follow the established naming conventions for Anthropic models.
99-149: LGTM!The Vertex AI additions comprehensively cover both Gemini and Claude model families with appropriate versioned (
@) and non-versioned formats for compatibility.
211-252: LGTM!The Bedrock Claude model entries correctly include the ARN-style format (
anthropic.claude-*-v1:0) alongside short names, ensuring proper model matching for AWS Bedrock deployments.test/fixtures/zod-sample.ts (2)
1-38: LGTM!The schema is well-documented with clear field descriptions that serve as AI guidance. The note about Vertex AI compatibility (using descriptions rather than schema validation for length constraints) is a thoughtful accommodation for Google's constrained decoding limitations.
244-274: LGTM!The ActionRoadmapSchema correctly enforces exactly 1 high-priority item with
.length(1)while allowing optional medium and budgetReallocation items with.max(1). This aligns with the validation logic in the test file.test/zod-schema-test-function.ts (3)
460-507: LGTM!The tool usage verification is thorough, checking for required
readFilecalls (at least 2) andgetCurrentTimeusage. The error messages are clear and actionable.
520-540: LGTM!The JSON parsing with markdown fence stripping mirrors the fallback logic in
GenerationHandler.ts, ensuring consistent handling across the codebase.
681-687: LGTM!The rate-limiting delay between provider tests is a good practice to avoid hitting API limits during test execution.
config/models.json (4)
219-233: Gemini 2.0 Flash Lite entry added correctly.Positioned as ultra-fast, cost-optimized variant with appropriate pricing at input: 0.00004, output: 0.00015.
236-291: Bedrock Claude 4.5 entries follow AWS naming conventions.Model IDs use the correct Bedrock format:
anthropic.claude-{variant}-{version}-v1:0.
418-436: Alias and default updates are consistent with new model additions.All referenced models exist in the
modelssection. The semantic aliases (best-reasoning,fastest,cheapest) appropriately point to the new model variants.
6-62: Claude 4.5 model entries verified and well-structured.All model IDs are officially available from Anthropic:
claude-sonnet-4-5-20250929,claude-opus-4-5-20251101, andclaude-haiku-4-5-20251001. Pricing, capabilities, and metadata are consistent with existing entry patterns. Release dates correctly align with the model ID dates.src/lib/constants/tokens.ts (4)
79-91: Claude 4.5 series token limits appropriately increased to 8192.The higher limit for the newer 4.5 series models reflects their enhanced capabilities, while maintaining conservative 4096 defaults for older series.
107-122: Gemini model token limits are consistent.All Gemini variants (3, 2.5, 2.0, 1.5 series) receive 8192 limits with a conservative 4096 default for unknown models.
127-156: Vertex section correctly combines Gemini and Claude models.The Claude models use the Vertex-specific
@notation (e.g.,claude-sonnet-4-5@20250929) which is the correct format for Vertex AI.
160-174: Claude 4.5 Bedrock models are available and correctly configured.AWS Bedrock confirmed availability of all three Claude 4.5 models: Sonnet 4.5 (Sep 29, 2025), Haiku 4.5 (Oct 15, 2025), and Opus 4.5 (Nov 24, 2025). The model IDs follow the correct format with version dates aligned to their release timeline. Additionally, Claude Sonnet 4.5 is available in AWS GovCloud (US) regions.
c645b66 to
53d9c98
Compare
🤖 AI Review & Build Compliance ✅Status: AI analysis complete • Build rules validated • Ready for review 📊 View detailed analysis results🛡️ Analysis Complete
📋 Ready for Merge When
🤖 AI analysis complete - check individual code comments for specific feedback |
53d9c98 to
def02bf
Compare
🤖 AI Review & Build Compliance ✅Status: AI analysis complete • Build rules validated • Ready for review 📊 View detailed analysis results🛡️ Analysis Complete
📋 Ready for Merge When
🤖 AI analysis complete - check individual code comments for specific feedback |
def02bf to
8957f70
Compare
🤖 AI Review & Build Compliance ✅Status: AI analysis complete • Build rules validated • Ready for review 📊 View detailed analysis results🛡️ Analysis Complete
📋 Ready for Merge When
🤖 AI analysis complete - check individual code comments for specific feedback |
|
@coderabbitai full review |
✅ Actions performedFull review triggered. |
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/lib/constants/tokens.ts (1)
79-174: Fix model ID version mismatch for Claude Opus 4.5 in ANTHROPIC and BEDROCK sections.The tokens.ts file has inconsistent Claude Opus 4.5 versions:
- ANTHROPIC section (line 81):
"claude-opus-4-5-20251101"- BEDROCK section (line 162):
"anthropic.claude-opus-4-5-20251101-v1:0"- enums.ts and models.json: both use
"...20251124"Update lines 81 and 162 to use
20251124to match the canonical version in enums.ts and models.json, preventing token limit lookup failures.Additionally, verify that Gemini 3 model variants (
gemini-3-pro-preview,gemini-3-pro-latest) are properly defined in config/models.json beyond the singlegemini-3-pro-previewentry currently present.
♻️ Duplicate comments (6)
src/lib/utils/messageBuilder.ts (1)
652-655: Ensure multimodal callers forward schema/output so structured-output instructions apply there too
buildMultimodalMessagesArraynow mirrors the text-only path and appendsSTRUCTURED_OUTPUT_INSTRUCTIONSwhenshouldUseStructuredOutput(options)is true, which is correct in isolation. This still relies on the caller passing the originalGenerateOptions(withschemaandoutput.format) through unchanged; if a separatemultimodalOptionsobject is constructed upstream that omits those fields, structured-output requests with images/PDFs will miss the JSON-only instructions again. Please confirm that the multimodal call sites are forwardingschemaandoutputintact, or update them if not.test/unit/structured-output.test.ts (1)
108-127: Missing test case: schema + output.format: "text" combination.The test validates output.format: "text" without a schema, but there's no test for when both schema and output.format: "text" are provided together. Per the shouldUseStructuredOutput logic, this combination would NOT add structured output instructions (by design), but this behavior should be explicitly tested.
test/zod-schema-test-function.ts (4)
403-403: Remove redundant JSON-only instructions from the prompt.Lines 403 and 427 include explicit JSON-only instructions (
"ONLY valid JSON","ONLY the JSON object"), but these are now automatically injected viaSTRUCTURED_OUTPUT_INSTRUCTIONSwhen usingschema+output.format: "json". This redundancy may confuse the model or add unnecessary tokens.Also applies to: 427-427
531-535: Remove full JSON output from logs to avoid potential data exposure.Logging the entire parsed output could expose sensitive account metrics or business data. Even for test fixtures, this establishes a pattern that could leak real data if fixtures are changed.
475-475: Remove unused variablehasGetCurrentTime.The variable
hasGetCurrentTimeis declared but never used. The logic at lines 493-507 uses a different variablegetCurrentTimeUsedinstead.Apply this diff:
const toolNames = result.toolExecutions.map((te) => te.name); const hasReadFile = toolNames.includes("readFile"); - const hasGetCurrentTime = toolNames.includes("getCurrentTime");
407-408: Remove confusing internet search instructions from the prompt.Lines 407-408 instruct the model to "search internet, get data from the internet" and "Use the internet search tool also", but the test validates that the model reads data from local files using the
readFiletool (lines 410-412). There is no internet search tool available, and these conflicting instructions may cause test failures or unpredictable behavior.
🧹 Nitpick comments (5)
examples/structured-output-google-providers.ts (2)
47-56: Remove unused result variable.The
resultvariable is never used since this example demonstrates the error case. Theawaitis sufficient to trigger the error.Apply this diff:
try { - const result = await neurolink.generate({ + await neurolink.generate({ input: { text: "Analyze TechCorp as an investment opportunity", },
122-129: Remove unused result variable.Similar to the incorrect usage example, the
resultvariable is never used since this demonstrates an error case.Apply this diff:
try { - const result = await neurolink.generate({ + await neurolink.generate({ input: { text: "Generate complex data" },docs/getting-started/providers/google-vertex.md (1)
730-796: Comprehensive limitation documentation but check for duplication.The Known Limitations section is well-structured and informative, with clear error messages, solutions, and examples. It correctly distinguishes between Gemini-specific limitations and Claude's lack thereof.
However, based on the AI summary, similar or identical content appears in multiple documentation files:
- docs/getting-started/providers/google-ai.md
- docs/sdk/api-reference.md
- docs/reference/provider-comparison.md
- docs/features/structured-output.md
- docs/TROUBLESHOOTING.md
Consider consolidating this content into a single source (e.g., docs/TROUBLESHOOTING.md or a dedicated limitations page) and linking to it from other docs to reduce maintenance burden and prevent documentation drift.
docs/features/structured-output.md (1)
87-118: Consolidate duplicated limitation content.The Google Gemini limitation is described in detail at lines 87-118 and then referenced again at line 125. This creates two sources of truth within the same document.
Additionally, as noted in the google-vertex.md review, this content appears across multiple documentation files, creating a maintenance burden.
Recommendations:
- Remove the detailed section at lines 87-118
- Keep the brief note at line 125 with a link to the comprehensive explanation
- Centralize the detailed limitation documentation in docs/TROUBLESHOOTING.md or a dedicated limitations reference page
- Link to that central documentation from all other docs that mention this limitation
This reduces duplication while keeping important context visible to users.
Also applies to: 125-125
test/fixtures/zod-sample.ts (1)
1-315: MetaAdsAnalysisOutputSchema is well-structured; minor doc vs constraint mismatchThe Meta Ads analysis schema is thorough and well-organized for testing structured output. Note that the file header claims all length constraints are only in descriptions for Vertex compatibility, but
ActionRoadmapSchemastill enforces.length(1)/.max(1)at the type level—fine for tests, just slightly inconsistent with the comment and something to be aware of if this schema is ever reused against live Vertex constrained decoding.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
test/fixtures/meta-ads-campaign-performance.csvis excluded by!**/*.csv
📒 Files selected for processing (25)
config/models.json(5 hunks)docs/TROUBLESHOOTING.md(1 hunks)docs/features/structured-output.md(1 hunks)docs/getting-started/providers/google-ai.md(1 hunks)docs/getting-started/providers/google-vertex.md(1 hunks)docs/reference/provider-comparison.md(2 hunks)docs/reference/provider-feature-compatibility.md(2 hunks)docs/sdk/api-reference.md(1 hunks)examples/structured-output-google-providers.ts(1 hunks)examples/structured-output-test.js(4 hunks)src/lib/adapters/providerImageAdapter.ts(5 hunks)src/lib/config/conversationMemory.ts(1 hunks)src/lib/constants/enums.ts(6 hunks)src/lib/constants/tokens.ts(3 hunks)src/lib/core/modules/GenerationHandler.ts(2 hunks)src/lib/models/modelRegistry.ts(1 hunks)src/lib/providers/googleAiStudio.ts(1 hunks)src/lib/providers/googleVertex.ts(2 hunks)src/lib/types/generateTypes.ts(1 hunks)src/lib/utils/messageBuilder.ts(4 hunks)test/continuous-test-suite.ts(2 hunks)test/fixtures/meta-ads-account-metrics.json(1 hunks)test/fixtures/zod-sample.ts(1 hunks)test/unit/structured-output.test.ts(1 hunks)test/zod-schema-test-function.ts(1 hunks)
🧰 Additional context used
🧠 Learnings (12)
📚 Learning: 2025-09-17T17:55:15.261Z
Learnt from: RajuSudhar
Repo: juspay/neurolink PR: 173
File: src/lib/index.ts:16-16
Timestamp: 2025-09-17T17:55:15.261Z
Learning: In src/lib/types/providers.ts, ProviderConfig was renamed to AIModelProviderConfig to deduplicate type names, as there was an existing ProviderConfig type that better suited the "ProviderConfig" name. This was an intentional breaking change for better type organization.
Applied to files:
src/lib/providers/googleVertex.tsdocs/reference/provider-comparison.mdsrc/lib/providers/googleAiStudio.tsexamples/structured-output-google-providers.tssrc/lib/adapters/providerImageAdapter.tsconfig/models.jsondocs/reference/provider-feature-compatibility.mdsrc/lib/models/modelRegistry.tsdocs/sdk/api-reference.mdsrc/lib/constants/tokens.ts
📚 Learning: 2025-09-02T13:50:42.770Z
Learnt from: YasmeenOgo
Repo: juspay/neurolink PR: 145
File: src/lib/core/types.ts:0-0
Timestamp: 2025-09-02T13:50:42.770Z
Learning: The APIVersions enum in src/lib/core/types.ts now contains comprehensive API version constants for all major AI providers: Azure OpenAI (latest, stable, legacy), OpenAI (current, beta), Google AI (current, beta), and Anthropic (current). This centralization helps avoid API version drift across the codebase.
Applied to files:
src/lib/providers/googleVertex.tsdocs/reference/provider-comparison.mdsrc/lib/providers/googleAiStudio.tssrc/lib/adapters/providerImageAdapter.tsconfig/models.jsonsrc/lib/models/modelRegistry.tssrc/lib/constants/enums.tssrc/lib/constants/tokens.ts
📚 Learning: 2025-09-17T18:14:34.960Z
Learnt from: RajuSudhar
Repo: juspay/neurolink PR: 173
File: src/lib/types/index.ts:58-62
Timestamp: 2025-09-17T18:14:34.960Z
Learning: RajuSudhar explained that in the Neurolink codebase, there are multiple ProviderConfig types causing inconsistency. One existing ProviderConfig type better suited the "ProviderConfig" name, so they renamed the less-suitable one to AIModelProviderConfig to free up the name. Adding backward compatibility aliases would worsen naming inconsistency rather than help. The remaining duplicates will be systematically deduplicated in the 07-Types-Module.md TODO as part of their phased refactor approach.
Applied to files:
docs/reference/provider-comparison.mdsrc/lib/adapters/providerImageAdapter.ts
📚 Learning: 2025-12-01T08:39:22.794Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-01T08:39:22.794Z
Learning: Use MessageBuilder (src/lib/utils/messageBuilder.ts) as the central component for constructing all messages with multimodal support
Applied to files:
test/unit/structured-output.test.tssrc/lib/utils/messageBuilder.ts
📚 Learning: 2025-12-01T08:39:22.794Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-01T08:39:22.794Z
Learning: Provider implementations must extend a base provider or implement the provider interface and be registered with provider name, factory function, default model, and aliases
Applied to files:
src/lib/providers/googleAiStudio.ts
📚 Learning: 2025-09-24T06:42:06.088Z
Learnt from: amreetkhuntia
Repo: juspay/neurolink PR: 185
File: src/lib/evaluation/contextBuilder.ts:79-85
Timestamp: 2025-09-24T06:42:06.088Z
Learning: In the NeuroLink codebase, using `(options.prompt || [])` pattern for handling potentially undefined prompt arrays is the preferred approach over extracting to a normalized variable when building conversation history in the ContextBuilder class.
Applied to files:
src/lib/utils/messageBuilder.tsexamples/structured-output-google-providers.tstest/zod-schema-test-function.ts
📚 Learning: 2025-09-24T07:26:41.988Z
Learnt from: amreetkhuntia
Repo: juspay/neurolink PR: 185
File: src/lib/evaluation/prompts.ts:86-101
Timestamp: 2025-09-24T07:26:41.988Z
Learning: In the neurolink codebase, maintainer amreetkhuntia consistently prefers to keep template literal indentation in LLM prompts (including evaluation prompts in src/lib/evaluation/prompts.ts) for readability, even when it results in extra whitespace in the output, as LLMs can parse and understand the content correctly.
Applied to files:
src/lib/utils/messageBuilder.tstest/zod-schema-test-function.ts
📚 Learning: 2025-09-24T06:43:23.653Z
Learnt from: amreetkhuntia
Repo: juspay/neurolink PR: 185
File: src/lib/evaluation/prompts.ts:59-72
Timestamp: 2025-09-24T06:43:23.653Z
Learning: In the neurolink codebase, maintainer amreetkhuntia prefers to keep template literal indentation in LLM prompts even if it results in technically malformed JSON format, as LLMs can understand and parse it correctly despite formatting issues.
Applied to files:
src/lib/utils/messageBuilder.tstest/zod-schema-test-function.ts
📚 Learning: 2025-09-01T22:58:39.149Z
Learnt from: sudharsan-juspay
Repo: juspay/neurolink PR: 140
File: src/lib/core/types.ts:198-203
Timestamp: 2025-09-01T22:58:39.149Z
Learning: In src/lib/core/types.ts, StreamOptions (imported from streamTypes.js) and StreamingOptions are intentionally different types with different use cases. StreamingOptions is for unified AI requests with multiple provider configurations, while StreamOptions is for individual streaming operations.
Applied to files:
src/lib/utils/messageBuilder.ts
📚 Learning: 2025-12-01T08:39:22.794Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-01T08:39:22.794Z
Learning: Update vision capabilities in ProviderImageAdapter when adding multimodal support to new providers
Applied to files:
src/lib/adapters/providerImageAdapter.ts
📚 Learning: 2025-12-01T08:39:22.794Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-01T08:39:22.794Z
Learning: Integrate provider-specific image formatting through ProviderImageAdapter when adding multimodal support to new providers
Applied to files:
src/lib/adapters/providerImageAdapter.ts
📚 Learning: 2025-09-24T06:41:27.575Z
Learnt from: amreetkhuntia
Repo: juspay/neurolink PR: 185
File: src/lib/evaluation/contextBuilder.ts:53-63
Timestamp: 2025-09-24T06:41:27.575Z
Learning: In the NeuroLink codebase, `LanguageModelV1CallOptions.prompt` is always present and never undefined, so defensive checks are not needed when accessing this property.
Applied to files:
test/zod-schema-test-function.ts
🧬 Code graph analysis (5)
src/lib/types/generateTypes.ts (2)
src/lib/types/index.ts (1)
ValidationSchema(14-14)src/lib/types/typeAliases.ts (1)
ValidationSchema(25-25)
test/unit/structured-output.test.ts (2)
src/lib/types/generateTypes.ts (1)
TextGenerationOptions(232-277)src/lib/config/conversationMemory.ts (1)
STRUCTURED_OUTPUT_INSTRUCTIONS(41-49)
src/lib/utils/messageBuilder.ts (1)
src/lib/config/conversationMemory.ts (1)
STRUCTURED_OUTPUT_INSTRUCTIONS(41-49)
examples/structured-output-google-providers.ts (1)
examples/structured-output-test.js (1)
CompanyAnalysisSchema(26-73)
test/zod-schema-test-function.ts (1)
test/fixtures/zod-sample.ts (1)
MetaAdsAnalysisOutputSchema(280-315)
🔇 Additional comments (21)
examples/structured-output-google-providers.ts (3)
22-39: LGTM - Correct demonstration of Google provider limitations.The example correctly demonstrates the required
disableTools: truefor Google providers with schemas, and the result is properly used for parsing and display.
62-78: LGTM - Demonstrates that other providers work without restrictions.Correctly shows OpenAI doesn't require
disableToolsand uses the result appropriately.
80-95: LGTM - Correctly demonstrates Claude via Vertex has no limitation.Shows that Claude models via Vertex AI support both tools and schemas without restrictions.
test/continuous-test-suite.ts (2)
33-33: LGTM - Clean integration of new test.Import follows existing patterns and correctly references the new Zod schema validation test.
2818-2821: LGTM - Test properly wired into suite.The new multi-provider Zod schema test is correctly added to the test array and will execute as part of the continuous test suite.
src/lib/config/conversationMemory.ts (1)
36-50: LGTM - Clear and comprehensive structured output instructions.The instructions effectively address the issue of conversational filler in JSON responses by explicitly prohibiting common patterns like "Excellent!", "Sure!", markdown wrapping, and extraneous text. The formatting is clear and the requirements are unambiguous.
test/fixtures/meta-ads-account-metrics.json (1)
1-25: LGTM - Well-structured test fixture.The JSON fixture provides realistic ad account metrics with clear field names and appropriate data types. The structure with current and previous periods supports comprehensive testing of schema validation workflows.
src/lib/providers/googleAiStudio.ts (1)
52-82: LGTM - Excellent inline API documentation.The JSDoc block clearly documents the structured output limitation with:
- Clear explanation of the Google API constraint
- Example error message developers will encounter
- Official documentation reference
- Practical code example showing correct usage
- Future capability notes (Gemini 3 Pro Preview)
- Related error mentions
This inline documentation is valuable for developers working directly with the provider class and complements the user-facing documentation.
examples/structured-output-test.js (1)
104-104: Model is confirmed available in Vertex AI GA.
claude-sonnet-4-5@20250929is available on Vertex AI as a generally available model in us-east5, europe-west1, asia-southeast1, and via the global endpoint. No additional verification needed.Also applies to: 139-139, 187-187, 225-225
src/lib/providers/googleVertex.ts (1)
302-337: Gemini structured-output note and model suggestions look consistentThe added JSDoc about Gemini tool+schema limitations and the extended Gemini model suggestions are coherent with the rest of the PR and don’t affect runtime behavior.
Also applies to: 1928-1939
src/lib/types/generateTypes.ts (1)
44-76: GenerateOptions additions for schema/disableTools are well-alignedAdding
schema?: ValidationSchemaanddisableTools?: booleantoGenerateOptionswith provider-specific docs matches the structured-output behavior wired elsewhere (messageBuilder + docs). Just ensure any streaming/legacy paths that buildTextGenerationOptionsalso passschema/output.formatconsistently so the structured-output instructions are applied everywhere.Also applies to: 79-96
docs/getting-started/providers/google-ai.md (1)
875-930: Known Limitations section clearly documents Gemini constraintsThe new section accurately describes the tools+schema conflict and “Too many states for serving” behavior, with concrete examples and aligned guidance (
disableTools: true, alternative providers). Good complement to the SDK/API docs.docs/TROUBLESHOOTING.md (1)
930-1034: Structured-output troubleshooting is consistent and actionableThe new troubleshooting subsection for Gemini schema issues (tools+schema conflict and “Too many states for serving”) matches the provider docs and includes precise error text plus remedies, which should make debugging much easier.
docs/reference/provider-feature-compatibility.md (1)
34-47: Structured-output matrix and details are consistent with implementationThe new column and “Structured Output Support Details” section correctly call out Google’s tools+schema limitation and show the
disableTools: truepattern, while marking other providers as full-support. This matches the SDK behavior and higher-level docs.Also applies to: 48-48, 115-147
docs/sdk/api-reference.md (1)
595-625: API docs for schema limitations match the new options and behaviorThe provider-specific schema limitations (especially for Vertex/Google AI with
disableTools: true) are clearly documented with code samples and a concise support matrix, and they line up with the newGenerateOptionsfields and message-building logic.Also applies to: 627-638
src/lib/utils/messageBuilder.ts (1)
16-19: Structured-output helper and system-prompt injection are reasonable but depend on output.format being setCentralizing the check in
shouldUseStructuredOutput()and appendingSTRUCTURED_OUTPUT_INSTRUCTIONSinbuildMessagesArraylooks good, and usingsystemPrompt.trim()before concatenation keeps the system message clean. The behavior now strictly depends on bothschemabeing set andoutput.formatbeing"json"or"structured", so any legacy code paths that only passschema(withoutoutput) will not get the JSON-only instructions. It’s worth double-checking all structured-output entrypoints (generate/stream) ensureoutput.formatis set appropriately wherever a schema is used and parsing expects strict JSON.Also applies to: 287-300, 325-328
docs/reference/provider-comparison.md (1)
178-180: Update the Future Support section to reflect current Gemini 3 Pro Preview capabilities.The statement about Gemini 3 Pro Preview is factually accurate—the model (released November 18, 2025) does support both function calling and Structured Outputs with JSON Schema support. However, the phrasing "will support both" is now outdated. Update line 180 to reflect this is now released functionality rather than a future capability (e.g., "Gemini 3 Pro Preview now supports both" or move the information to the Full Support section).
Likely an incorrect or invalid review comment.
src/lib/constants/tokens.ts (4)
79-91: LGTM!The Claude 4.5 series additions with 8192 token limits are appropriate for the newer high-capacity models, while maintaining 4096 for the older 3/3.5 series. The model naming follows Anthropic's established pattern.
107-123: LGTM!The Gemini model entries are well-organized by series with clear comments. The consistent 8192 limits for listed models and conservative 4096 default (with the documented rationale for 500 errors) is a sensible approach.
127-156: LGTM!The Vertex section correctly combines both Gemini and Claude model families. The Vertex-specific naming convention using
@for version separation (e.g.,claude-sonnet-4-5@20250929) is properly applied. Token limits are appropriately assigned based on model generation.
159-174: LGTM! Previous model ID mismatch has been resolved.The Bedrock section correctly adds Claude 4.5 series entries with proper ARN format. The previous issue flagged in past reviews regarding the Claude 3.5 Sonnet model ID (
20240620vs20241022) has been addressed—line 167 now correctly usesanthropic.claude-3-5-sonnet-20241022-v1:0to align withconfig/models.json.
| "claude-sonnet-4-5", | ||
| "claude-sonnet-4-5-20250929", | ||
| "claude-opus-4-5", | ||
| "claude-opus-4-5-20251101", |
There was a problem hiding this comment.
Inconsistent date token for Claude Opus 4.5.
Line 71 uses "claude-opus-4-5-20251101" (November 1), but this conflicts with:
enums.tsline 133:CLAUDE_OPUS_4_5 = "claude-opus-4-5-20251124"(Nov 24)modelRegistry.tsline 252:releaseDate: "2025-11-24"(Nov 24)models.jsonline 40:"releaseDate": "2025-11-24"(Nov 24)
Update to "claude-opus-4-5-20251124" to maintain consistency across the codebase.
Apply this diff:
- "claude-opus-4-5-20251101",
+ "claude-opus-4-5-20251124",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "claude-opus-4-5-20251101", | |
| "claude-opus-4-5-20251124", |
🤖 Prompt for AI Agents
In src/lib/adapters/providerImageAdapter.ts around line 71, the Claude Opus 4.5
identifier uses an inconsistent date token "claude-opus-4-5-20251101"; update
this string to "claude-opus-4-5-20251124" so it matches enums.ts (line 133),
modelRegistry.ts (line 252) and models.json (line 40); make the exact
replacement of the identifier value and run a quick search to ensure no other
occurrences use the old 20251101 token.
| "claude-opus-4-5", | ||
| "claude-opus-4.5", | ||
| "anthropic.claude-opus-4-5", | ||
| "anthropic.claude-opus-4-5-20251101-v1:0", |
There was a problem hiding this comment.
Inconsistent date token for Bedrock Claude Opus 4.5.
Line 220 uses "anthropic.claude-opus-4-5-20251101-v1:0" (November 1), but this conflicts with:
enums.tsline 30:CLAUDE_4_5_OPUS = "anthropic.claude-opus-4-5-20251124-v1:0"(Nov 24)
Update to use November 24 for consistency.
Apply this diff:
- "anthropic.claude-opus-4-5-20251101-v1:0",
+ "anthropic.claude-opus-4-5-20251124-v1:0",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "anthropic.claude-opus-4-5-20251101-v1:0", | |
| "anthropic.claude-opus-4-5-20251124-v1:0", |
🤖 Prompt for AI Agents
In src/lib/adapters/providerImageAdapter.ts around line 220, the model token
"anthropic.claude-opus-4-5-20251101-v1:0" uses Nov 1 but enums.ts defines
CLAUDE_4_5_OPUS with Nov 24; replace the string at line 220 with
"anthropic.claude-opus-4-5-20251124-v1:0" (or reference the CLAUDE_4_5_OPUS
constant) so the date token matches enums.ts.
Google Vertex AI providers had confusing behavior with Zod schemas:
- Gemini models failed with "too many states" or "function calling unsupported"
when combining schemas + tools
- Documentation didn't clarify model-specific limitations
- No tests validating schema + tool usage together
**Gemini Models (gemini-2.5-flash, etc.):**
- Cannot combine function calling (tools) with structured output (JSON schema)
- Error: "Function calling with a response mime type: 'application/json' is unsupported"
- Required workaround: `disableTools: true` when using schemas
**Claude Models on Vertex (claude-sonnet-4-5@20250929):**
- Fully support both tools + schemas simultaneously
- No "too many states" errors
- No workarounds needed
- Updated `src/lib/core/modules/GenerationHandler.ts`
- Conditionally include tools only when `shouldUseTools === true`
- Ensures `disableTools: true` properly prevents tools from being passed
Updated docs to clarify Gemini vs Claude differences:
- `docs/getting-started/providers/google-vertex.md` - provider-specific guidance
- `docs/getting-started/providers/google-ai.md` - AI Studio limitations
- `docs/features/structured-output.md` - schema + tools compatibility
- `docs/reference/provider-comparison.md` - feature matrix
- `docs/TROUBLESHOOTING.md` - common errors and solutions
- Provider source code JSDoc comments with examples
Created comprehensive test validating Vertex + Claude with schemas + tools:
- `test/zod-schema-test-function.ts` - multi-provider Zod schema validation
- `test/run-zod-test.ts` - standalone test runner
- `test/fixtures/zod-sample.ts` - production-grade META Ads schema
- `test/fixtures/meta-ads-*.{csv,json}` - realistic test data
- `test/VERTEX_CLAUDE_SCHEMA_FINDINGS.md` - detailed findings document
✅ **ALL VALIDATIONS PASSED**:
- Tool Usage: readFile (2x), getCurrentTime (1x) - all executed successfully
- JSON Parsing: Valid output
- Zod Schema: Complex nested schema validated (exact array lengths, enums, etc.)
- Numeric Constraints: All numeric fields within bounds
- Required Fields: All mandatory fields present
- Data Accuracy: Zero hallucinations, all data from actual files
1. **Vertex + Claude**: Fully supports schema + tools together (confirmed via test)
2. **Vertex + Gemini**: Requires `disableTools: true` when using schemas
3. **7 MCP tools** automatically registered: readFile, writeFile, getCurrentTime, etc.
4. **Complex schemas** work correctly: nested objects, exact lengths, enums, nullish
5. **Tool execution works** with schema validation: Claude used actual file data
- `src/lib/core/modules/GenerationHandler.ts` - respect disableTools flag
- `src/lib/providers/googleVertex.ts` - add JSDoc clarifying limitations
- `src/lib/providers/googleAiStudio.ts` - add JSDoc clarifying limitations
- `docs/getting-started/providers/google-vertex.md`
- `docs/getting-started/providers/google-ai.md`
- `docs/features/structured-output.md`
- `docs/reference/provider-comparison.md`
- `docs/reference/provider-feature-compatibility.md`
- `docs/TROUBLESHOOTING.md`
- `docs/sdk/api-reference.md`
- `test/zod-schema-test-function.ts` (new)
- `test/run-zod-test.ts` (new)
- `test/fixtures/zod-sample.ts` (new)
- `test/fixtures/meta-ads-campaign-performance.csv` (new)
- `test/fixtures/meta-ads-account-metrics.json` (new)
- `test/VERTEX_CLAUDE_SCHEMA_FINDINGS.md` (new)
- `test/continuous-test-suite.ts` (updated)
- `examples/structured-output-google-providers.ts` (new)
Run the validation test:
```bash
npx tsx test/run-zod-test.ts
```
Expected output:
```
✅ VERTEX (claude-sonnet-4-5@20250929) - Overall
All validations passed successfully
✅ Multi-Provider Zod Schema Test
1/1 providers passed all validations
```
```typescript
const sdk = new NeuroLink();
const result = await sdk.generate({
provider: "vertex", // Uses Gemini by default
schema: MySchema, // ❌ Fails with "function calling unsupported"
});
```
```typescript
const sdk = new NeuroLink();
const result = await sdk.generate({
provider: "vertex",
model: "claude-sonnet-4-5@20250929", // ✅ Supports schema + tools
schema: MySchema,
});
```
```typescript
const sdk = new NeuroLink();
const result = await sdk.generate({
provider: "vertex",
model: "gemini-2.5-flash", // Default Gemini model
schema: MySchema,
disableTools: true, // ✅ Required for Gemini
});
```
- Vertex AI Gemini limitation: https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models
- Test findings: `test/VERTEX_CLAUDE_SCHEMA_FINDINGS.md`
- Documentation: `docs/getting-started/providers/google-vertex.md`
8957f70 to
37f0cb1
Compare
🤖 AI Review & Build Compliance ✅Status: AI analysis complete • Build rules validated • Ready for review 📊 View detailed analysis results🛡️ Analysis Complete
📋 Ready for Merge When
🤖 AI analysis complete - check individual code comments for specific feedback |
|
🎉 This PR is included in version 8.5.1 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Pull Request
Description
When using structured output with a Zod schema, AI providers (particularly Vertex) return conversational filler text like "Excellent! Now I have all the information..." before the JSON, causing
NoObjectGeneratedErrorparsing failures.Fix: Append explicit JSON-only instructions to system prompt when structured output is requested.
Type of Change
Related Issues
Changes Made
src/lib/config/conversationMemory.ts: AddedSTRUCTURED_OUTPUT_INSTRUCTIONSconstant with explicit JSON-only output requirementssrc/lib/utils/messageBuilder.ts:shouldUseStructuredOutput()helper to detect whenschema+json/structuredformat are setbuildMessagesArrayandbuildMultimodalMessagesArrayto append instructions to system prompttest/unit/structured-output.test.ts: Added 10 tests validating instruction injection behaviorAI Provider Impact
Component Impact
Testing
Test Environment
Performance Impact
Breaking Changes
None. Additive system prompt enhancement only affects structured output mode.
Screenshots/Demo
N/A
Checklist
Additional Notes
The instructions appended to system prompt:
Warning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
googlechromelabs.github.io/usr/local/bin/node node install.mjs(dns block)https://storage.googleapis.com/chrome-for-testing-public/142.0.7444.175/linux64/chrome-headless-shell-linux64.zip/usr/local/bin/node node install.mjs(http block)If you need me to access, download, or install something from one of these locations, you can either:
Original prompt
This section details on the original issue you should resolve
<issue_title>[Bug]:</issue_title>
<issue_description>### What happened?
neurolink is violating the Zod schema in the structured output by forcing conversational filler (like 'excellent') into the response, resulting in a parsing error.
AI Provider
Multiple providers
Component
SDK
Steps to Reproduce
error.txt
Environment
Relevant log output