feat(workflow): implement comprehensive workflow engine for multi-mod… - #256
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the
WalkthroughAdds a complete Workflow Engine: public types, Zod-backed config, execution core (runner, ensemble executor, judge scorer, conditioner), in-memory registry, metrics/validation utilities, nine predefined workflows, CLI command, examples/tests, and NeuroLink/lib public exports for registering and running workflows. Changes
Sequence Diagram(s)sequenceDiagram
participant Client as Client / CLI
participant NL as NeuroLink
participant WR as WorkflowRunner
participant EE as EnsembleExecutor
participant JS as JudgeScorer
participant RC as ResponseConditioner
participant Prov as AI Provider(s)
Client->>NL: runWorkflow(config | id, options)
NL->>WR: runWorkflow(config, options)
activate WR
WR->>WR: validate config
rect `#e6f3ff`
Note over WR,EE: Execute ensemble (layered or flat)
WR->>EE: executeModels(prompt, config)
activate EE
EE->>Prov: call models (parallel / sequential)
Prov-->>EE: responses (content, timing, usage)
EE-->>WR: EnsembleExecutionResult
deactivate EE
end
rect `#f0fff0`
Note over WR,JS: Score ensemble (if judges configured)
alt judges present
WR->>JS: scoreEnsemble(responses, judges)
activate JS
JS->>Prov: judge evaluations
Prov-->>JS: judge responses (scores, reasoning)
JS-->>WR: JudgeScores / MultiJudgeScores
deactivate JS
else no judges
WR-->>WR: produce neutral scores
end
end
rect `#fff7e6`
Note over WR,RC: Select best & condition
WR->>WR: selectBestResponse(scores)
WR->>RC: conditionResponse(best, allResponses, scores)
activate RC
alt synthesis configured
RC->>Prov: synthesis call
Prov-->>RC: synthesized content
else
RC-->>WR: original content + metadata
end
RC-->>WR: ConditionResult
deactivate RC
end
WR->>WR: aggregate metrics & build WorkflowResult
WR-->>NL: WorkflowResult
deactivate WR
NL-->>Client: WorkflowResult
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
ac14916 to
5e0431d
Compare
5e0431d to
c5b4720
Compare
c5b4720 to
84514b6
Compare
84514b6 to
6353547
Compare
6353547 to
22b9685
Compare
22b9685 to
f33724b
Compare
f33724b to
1e96b8f
Compare
1e96b8f to
1566b85
Compare
|
@arx-optimus-17, I'll conduct a final comprehensive review of the workflow engine implementation. 🎯 Final Review Assessment✅ Major Strengths
🔍 Areas for Final Verification1. Error Handling Robustness
2. Production Readiness
3. API Consistency
4. Edge Cases
5. Backward Compatibility
🎯 Specific Recommendations
📋 Pre-Merge Checklist
🚀 Overall AssessmentThis is a well-architected and comprehensive implementation of a multi-model workflow engine. The code quality is high, type safety is excellent, and the documentation is thorough. Recommendation: ✅ Approve with minor enhancements The implementation is production-ready for the testing/evaluation phase as documented. The main consideration is ensuring teams understand the response conditioner is currently a stub and planning the rollout timeline for phase 2 features. 🎯 Post-Merge Priorities
Let me trigger a final comprehensive review to catch any remaining implementation issues. ✅ Actions performedFull review triggered. |
There was a problem hiding this comment.
Actionable comments posted: 8
♻️ Duplicate comments (9)
docs/WORKFLOW-ENGINE-LLD.md (1)
880-885: Bug in code example:finalResultis undefined.This issue was flagged in a previous review. The code example on line 883 references
finalResult.metadata, butfinalResultis never defined in theexecutemethod example. This should be corrected.src/lib/neurolink.ts (1)
1700-2055: Workflow integration: use typed errors and safer provider/model defaults.There are a few consistency and robustness gaps in the workflow paths:
generateWithWorkflowandstreamWithWorkflowthrow plainErrorfor “workflow not found” / “either workflow or workflowConfig must be provided” instead of using the SDK’s typed errors viaErrorFactory(or dedicated workflow error helpers).runWorkflowalso throws a plainErrorwhen a workflow ID is not found.- In
generateWithWorkflow,provider/modelfall back toworkflowConfig.models[0]?.provider/.modelwhenselectedResponseis absent. Workflows that rely solely onmodelGroupsor have an emptymodelsarray will yieldundefinedhere, andworkflow.selectedModelwill become strings like"undefined-undefined", which is awkward for consumers and can violate type expectations.Consider:
- Introducing workflow-specific constructors in
ErrorFactory(e.g.,workflowNotFound(id),invalidWorkflowConfig(reason)) and using them in all these branches, instead ofnew Error(...).- Defaulting
provider/modelandselectedModelto safe string values such as"unknown"when neitherselectedResponsenor a concretemodels[0]entry exists, soGenerateResult.workflow.*fields are always well-formed.Also applies to: 2057-2235, 6302-6472
src/lib/workflow/core/judgeScorer.ts (1)
597-614:calculateConsensusLevelstill risks-Infinitywhen nobestResponseis set.If all
JudgeScoresobjects havebestResponseundefined/empty,modeCountsremains empty andMath.max(...Array.from(modeCounts.values()))returns-Infinity, yielding a consensus of-Infinity / judgeResults.length.Add a guard before computing
maxCount:
- If
modeCounts.size === 0, return0(or another neutral value) to indicate “no consensus” rather than propagating-Infinity.Suggested patch
const bestResponses = judgeResults.map((r) => r.bestResponse); const modeCounts = new Map<string, number>(); bestResponses.forEach((response) => { if (response) { modeCounts.set(response, (modeCounts.get(response) || 0) + 1); } }); - const maxCount = Math.max(...Array.from(modeCounts.values())); - return maxCount / judgeResults.length; + // If no judge produced a bestResponse, treat consensus as zero + if (modeCounts.size === 0) { + return 0; + } + + const maxCount = Math.max(...Array.from(modeCounts.values())); + return maxCount / judgeResults.length;src/lib/workflow/core/responseConditioner.ts (1)
141-215: Synthesis path should use typed errors and avoid hardcoded provider/model defaults.Two concerns in
synthesizeImprovedResponse:
- The empty-result case throws a raw
Error("Synthesis model returned empty response"). Insrc/lib/**, new errors should be created viaErrorFactory(or a workflow-specific typed error) for consistent error typing and observability.synthesisProvider/synthesisModeldefault to"azure"/"gpt-4o"whenconfig.synthesisModelis not provided. This couples conditioning to a specific provider/model and can be surprising, especially given the Azure provider already has its own defaults. You already have a metadata-only fallback whenconfig.synthesisModelis absent; relying on that and requiring an explicitsynthesisModelwhen synthesis is desired would keep behavior clearer.Suggested direction:
- Import
ErrorFactory(or useWorkflowError) and throw a typed conditioning error when synthesis returns no content.- Drop the
"azure"/"gpt-4o"fallbacks and either:
- Require
config.synthesisModelto be present for synthesis (otherwise useaddMetadataOnly), or- Delegate to provider-level defaults by passing
config.synthesisModel?.provider/.modelthrough unchanged.Optionally, in
addModelAttribution, you may want to skip the “Evaluation Score: 0/100” footer when no judge scores are available instead of defaulting to 0.src/lib/workflow/core/ensembleExecutor.ts (2)
52-64: Retry logic defined in config but not implemented.
ExecutionConfigdefinesretries,retryDelay, andretryableErrors, butexecuteModeldoesn't implement retry behavior. Failed model executions are returned immediately without retry attempts.This was flagged in a previous review and remains unaddressed.
346-352: Unused_executionConfigparameter inexecuteModelGroups.The underscore prefix indicates intentional non-use, but this config contains important settings like
minResponsesand retry behavior that should be passed to layer execution.This was flagged in a previous review.
src/lib/workflow/config.ts (3)
109-125:ConditioningConfigSchemamissingsynthesisModelfield.The
ConditioningConfiginterface intypes.ts(lines 167-172) includes asynthesisModelfield, but the Zod schema doesn't validate it. This could allow invalid configurations to pass validation.This was flagged in a previous review.
384-404:createWorkflowConfigomitsmodelGroupsand prompt defaults.The function doesn't include
modelGroups,defaultSystemPrompt, ordefaultJudgePromptfrom the partial input, so these fields will be silently dropped.This was flagged in a previous review.
487-498: Cost estimation ignoresmodelGroups.
estimateWorkflowCostusesconfig.models.lengthbut workflows may usemodelGroupsinstead. ThegetAllModelshelper already exists for this purpose.This was flagged in a previous review.
🧹 Nitpick comments (14)
src/lib/utils/pdfProcessor.ts (2)
267-269: UseErrorFactoryfor typed errors.Per coding guidelines, errors should be created using
ErrorFactoryrather than plainErrorobjects.🔎 Proposed fix using ErrorFactory
First, add the import at the top of the file:
+import { ErrorFactory } from "../errors/index.js"; import type { FileProcessingResult, PDFProviderConfig,Then update the error:
- throw new Error( - `Invalid format: "${format}". Supported formats: "png", "jpeg".`, - ); + throw ErrorFactory.createValidationError( + `Invalid format: "${format}". Supported formats: "png", "jpeg".`, + );As per coding guidelines, use
ErrorFactoryfor creating typed errors across the SDK.
254-380: Wrap async operations withwithTimeoututility.Per coding guidelines for
src/lib/utils/*.ts, async operations should be wrapped with thewithTimeoututility for timeout handling and graceful degradation. The function performs multiple external library calls (pdfjs-dist, canvas) that could hang or take excessive time, especially for large or complex PDFs.Consider wrapping the main conversion logic with timeout protection:
import { withTimeout } from './timeout.js'; // adjust path as needed static async convertPDFToImages( pdfBuffer: Buffer, options?: { maxPages?: number; scale?: number; format?: "png" | "jpeg"; quality?: number; timeout?: number; // Add timeout option }, ): Promise<Array<{ buffer: Buffer; pageNumber: number }>> { const timeout = options?.timeout || 30000; // Default 30s return withTimeout( async () => { // existing conversion logic }, timeout, 'PDF to image conversion' ); }As per coding guidelines, wrap async operations with
withTimeoututility for timeout handling and graceful degradation.src/lib/workflow/workflows/fallbackWorkflow.ts (1)
50-56: Document the placeholder models pattern.Both workflows include a placeholder
modelsarray alongsidemodelGroups. While the comment notes this is "required by schema," this pattern could confuse consumers who might not understand the precedence relationship.Consider adding a JSDoc comment to the exported constants explaining the precedence:
/** * Fast-Fallback Workflow Configuration * * Uses layer-based execution with sequential groups: * ... * * @remarks * The `models` array is a schema-required placeholder. * Execution uses `modelGroups`, which takes precedence. */ export const FAST_FALLBACK_WORKFLOW: WorkflowConfig = { // ...Also applies to: 164-169
src/lib/workflow/PROMPT-EXAMPLES.ts (1)
12-37: Add explicit type annotations to example workflow configurations.The example workflows are exported without type annotations, which reduces type safety and IDE support. Consider adding explicit
WorkflowConfigtyping to ensure the examples conform to the expected schema.🔎 Proposed fix
+import type { WorkflowConfig } from "./types.js"; + -const simpleWorkflow = { +const simpleWorkflow: Partial<WorkflowConfig> = { id: "simple-ensemble", name: "Simple Ensemble with Workflow Defaults", type: "ensemble",Apply similar typing to
mixedWorkflow,advancedWorkflow, andmultiJudgeWorkflow.src/lib/workflow/workflows/consensusWorkflow.ts (1)
129-137: Consider adding validation for empty system prompts.The
createConsensus3WithPromptfunction accepts any string, including empty strings. Consider adding basic validation or trimming.🔎 Proposed fix
export function createConsensus3WithPrompt( systemPrompt: string, ): WorkflowConfig { + const trimmedPrompt = systemPrompt.trim(); + if (!trimmedPrompt) { + throw new Error("System prompt cannot be empty"); + } return { ...CONSENSUS_3_WORKFLOW, id: `consensus-3-custom-${Date.now()}`, - defaultSystemPrompt: systemPrompt, + defaultSystemPrompt: trimmedPrompt, }; }src/lib/types/generateTypes.ts (1)
300-304: Consider aligningjudgeScores.selectedModelwithselectedModelfield.There's potential redundancy between
judgeScores.selectedModel(line 303) and the top-levelselectedModel(line 305). Consider documenting the distinction or consolidating.src/lib/workflow/core/types/registryTypes.ts (1)
11-16: Consider consolidatingWorkflowMetadatawithRegistryEntry.
WorkflowMetadata(lines 49-53) has the same fields as the metadata portion ofRegistryEntry(lines 11-16). Consider havingRegistryEntryextend or useWorkflowMetadatato reduce duplication.🔎 Proposed fix
+/** + * Workflow metadata + */ +export interface WorkflowMetadata { + registeredAt: string; + lastUsed?: string; + usageCount: number; +} + /** * Registry entry with metadata (internal) */ -export interface RegistryEntry { +export interface RegistryEntry extends WorkflowMetadata { config: WorkflowConfig; - registeredAt: string; - lastUsed?: string; - usageCount: number; } - -/** - * Workflow metadata - */ -export interface WorkflowMetadata { - registeredAt: string; - lastUsed?: string; - usageCount: number; -}Also applies to: 49-53
src/lib/workflow/core/types/judgeTypes.ts (1)
31-35: Consider makingerrorfield more specific.The
errorfield is typed asWorkflowError, but scoring failures might not always produce a fullWorkflowError. Consider usingWorkflowError | Error | stringor a simpler error representation.🔎 Proposed fix
export interface ScoreResult { scores: JudgeScores | MultiJudgeScores; judgeTime: number; - error?: WorkflowError; + error?: WorkflowError | Error; }src/lib/workflow/utils/types/metricsTypes.ts (1)
36-41: Consider adding workflow IDs toWorkflowComparison.The comparison result references
workflow1andworkflow2but doesn't include their IDs, making it harder to understand the comparison context without external tracking.🔎 Proposed fix
export interface WorkflowComparison { + workflow1Id: string; + workflow2Id: string; workflow1: SummaryStats; workflow2: SummaryStats; winner: "workflow1" | "workflow2" | "tie"; reasoning: string; }src/cli/commands/workflow.ts (2)
35-46: Consider the impact of clearing all workflows on every CLI invocation.Calling
neurolink.clearWorkflows()at the start of every workflow command will remove any custom workflows that may have been registered programmatically. While this ensures a clean slate for CLI operations, it could be unexpected behavior if the SDK instance is shared or if workflows were registered elsewhere in the application lifecycle.Consider one of these approaches:
- Document this behavior clearly in the CLI help text
- Only clear and re-register if the registry is empty or stale
- Use a separate registry instance for CLI operations
55-361: Method exceeds recommended line limit.The
createWorkflowCommandstatic method is 307 lines long, exceeding the 300-line threshold flagged by static analysis. While the code is well-organized with clear sections (list handler, info handler, run handler), extracting these handlers into separate private methods would improve maintainability and testability.Suggested refactoring approach
Consider extracting the handler logic:
static createWorkflowCommand(): CommandModule<{}, WorkflowCommandArgs> { return { command: "workflow", describe: "Run multi-model workflows with judge-based evaluation", builder: (yargs) => { // ... builder configuration (keep as is) }, handler: async (args) => { try { registerPredefinedWorkflows(); if (args.list) { await this.handleList(); return; } if (args.info && args.workflow) { await this.handleInfo(args.workflow); return; } if (args.prompt && args.workflow) { await this.handleRun(args); return; } } catch (error) { // ... error handling } }, }; } private static async handleList(): Promise<void> { /* ... */ } private static async handleInfo(workflowId: string): Promise<void> { /* ... */ } private static async handleRun(args: WorkflowCommandArgs): Promise<void> { /* ... */ }src/lib/workflow/core/workflowRegistry.ts (2)
126-144: Side effect in getter:getWorkflowmutates registry state.Calling
getWorkflowincrementsusageCountand updateslastUsed, which may be unexpected for a "get" operation. Consider separating read-only retrieval from usage tracking, or renaming togetAndTrackWorkflow.🔎 Proposed refactor
+/** + * Get workflow configuration by ID (read-only, no tracking) + */ +export function peekWorkflow(workflowId: string): WorkflowConfig | undefined { + return workflowRegistry.get(workflowId)?.config; +} + /** - * Get workflow configuration by ID + * Get workflow configuration by ID and track usage */ export function getWorkflow(workflowId: string): WorkflowConfig | undefined {
160-194: Pagination offset not applied whenlimitis undefined.When
limitis not provided, theoffsetparameter is silently ignored. Users might expectoffsetto work independently for skipping results.🔎 Proposed fix
// Apply pagination - if (limit !== undefined) { - workflows = workflows.slice(offset, offset + limit); - } + if (limit !== undefined || offset > 0) { + workflows = workflows.slice(offset, limit !== undefined ? offset + limit : undefined); + }src/lib/workflow/types.ts (1)
364-391:MultiJudgeScoreshas potentially confusing duplicate field semantics.Fields like
scoresandrankingare documented as "points to"averageScoresandaggregatedRankingrespectively (lines 384-385). This creates ambiguity about whether they're separate copies or actual references. Consider using getters or documenting the relationship more explicitly.Verify that consumers understand
scoresis the same asaverageScoresand ensure no code expects them to differ.
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (46)
WORKFLOW-ENGINE-COMPLETE.mdWORKFLOW-ENGINE-IMPLEMENTATION-GUIDE.mdWORKFLOW-INTEGRATION-COMPLETE.mdWORKFLOW-INTEGRATION-REQUIREMENTS.mddocs/WORKFLOW-ENGINE-HLD.mddocs/WORKFLOW-ENGINE-LLD.mddocs/index.mdexamples/workflow-integration-example.tsmemory-bank/workflow-engine-implementation.mdmkdocs.ymlpackage.jsonscripts/build-validations.cjssrc/cli/commands/workflow.tssrc/cli/index.tssrc/lib/index.tssrc/lib/neurolink.tssrc/lib/types/generateTypes.tssrc/lib/types/streamTypes.tssrc/lib/utils/pdfProcessor.tssrc/lib/workflow/LAYER-EXAMPLES.tssrc/lib/workflow/PROMPT-EXAMPLES.tssrc/lib/workflow/__tests__/workflow.test.tssrc/lib/workflow/config.tssrc/lib/workflow/core/ensembleExecutor.tssrc/lib/workflow/core/judgeScorer.tssrc/lib/workflow/core/responseConditioner.tssrc/lib/workflow/core/types/conditionerTypes.tssrc/lib/workflow/core/types/ensembleTypes.tssrc/lib/workflow/core/types/index.tssrc/lib/workflow/core/types/judgeTypes.tssrc/lib/workflow/core/types/layerTypes.tssrc/lib/workflow/core/types/registryTypes.tssrc/lib/workflow/core/workflowRegistry.tssrc/lib/workflow/core/workflowRunner.tssrc/lib/workflow/index.tssrc/lib/workflow/types.tssrc/lib/workflow/utils/types/index.tssrc/lib/workflow/utils/types/metricsTypes.tssrc/lib/workflow/utils/types/validationTypes.tssrc/lib/workflow/utils/workflowMetrics.tssrc/lib/workflow/utils/workflowValidation.tssrc/lib/workflow/workflows/adaptiveWorkflow.tssrc/lib/workflow/workflows/consensusWorkflow.tssrc/lib/workflow/workflows/fallbackWorkflow.tssrc/lib/workflow/workflows/multiJudgeWorkflow.tstest/unit/tts-audio-output.test.ts
🧰 Additional context used
📓 Path-based instructions (6)
src/lib/types/*.ts
📄 CodeRabbit inference engine (CLAUDE.md)
Organize TypeScript types by domain (providers, generation, streaming, MCP, etc.) to avoid circular dependencies
Files:
src/lib/types/generateTypes.tssrc/lib/types/streamTypes.ts
src/lib/**/*.ts
📄 CodeRabbit inference engine (CLAUDE.md)
Use
ErrorFactoryfor creating typed errors across the SDK
Files:
src/lib/types/generateTypes.tssrc/lib/workflow/core/types/ensembleTypes.tssrc/lib/workflow/utils/types/index.tssrc/lib/workflow/core/types/layerTypes.tssrc/lib/workflow/utils/types/validationTypes.tssrc/lib/workflow/LAYER-EXAMPLES.tssrc/lib/workflow/core/workflowRunner.tssrc/lib/workflow/core/types/conditionerTypes.tssrc/lib/workflow/PROMPT-EXAMPLES.tssrc/lib/workflow/workflows/consensusWorkflow.tssrc/lib/workflow/utils/types/metricsTypes.tssrc/lib/workflow/core/types/registryTypes.tssrc/lib/workflow/utils/workflowMetrics.tssrc/lib/workflow/core/responseConditioner.tssrc/lib/workflow/workflows/adaptiveWorkflow.tssrc/lib/workflow/core/types/judgeTypes.tssrc/lib/workflow/core/judgeScorer.tssrc/lib/workflow/workflows/multiJudgeWorkflow.tssrc/lib/workflow/core/workflowRegistry.tssrc/lib/workflow/utils/workflowValidation.tssrc/lib/utils/pdfProcessor.tssrc/lib/workflow/__tests__/workflow.test.tssrc/lib/workflow/core/ensembleExecutor.tssrc/lib/workflow/workflows/fallbackWorkflow.tssrc/lib/neurolink.tssrc/lib/index.tssrc/lib/types/streamTypes.tssrc/lib/workflow/core/types/index.tssrc/lib/workflow/config.tssrc/lib/workflow/index.tssrc/lib/workflow/types.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Maintain strict TypeScript type checking across all SDK modules
Files:
src/lib/types/generateTypes.tsexamples/workflow-integration-example.tssrc/lib/workflow/core/types/ensembleTypes.tssrc/lib/workflow/utils/types/index.tssrc/lib/workflow/core/types/layerTypes.tstest/unit/tts-audio-output.test.tssrc/lib/workflow/utils/types/validationTypes.tssrc/lib/workflow/LAYER-EXAMPLES.tssrc/lib/workflow/core/workflowRunner.tssrc/lib/workflow/core/types/conditionerTypes.tssrc/lib/workflow/PROMPT-EXAMPLES.tssrc/lib/workflow/workflows/consensusWorkflow.tssrc/lib/workflow/utils/types/metricsTypes.tssrc/lib/workflow/core/types/registryTypes.tssrc/cli/commands/workflow.tssrc/lib/workflow/utils/workflowMetrics.tssrc/lib/workflow/core/responseConditioner.tssrc/lib/workflow/workflows/adaptiveWorkflow.tssrc/lib/workflow/core/types/judgeTypes.tssrc/lib/workflow/core/judgeScorer.tssrc/lib/workflow/workflows/multiJudgeWorkflow.tssrc/lib/workflow/core/workflowRegistry.tssrc/lib/workflow/utils/workflowValidation.tssrc/lib/utils/pdfProcessor.tssrc/lib/workflow/__tests__/workflow.test.tssrc/lib/workflow/core/ensembleExecutor.tssrc/lib/workflow/workflows/fallbackWorkflow.tssrc/lib/neurolink.tssrc/lib/index.tssrc/lib/types/streamTypes.tssrc/lib/workflow/core/types/index.tssrc/cli/index.tssrc/lib/workflow/config.tssrc/lib/workflow/index.tssrc/lib/workflow/types.ts
src/cli/commands/*.ts
📄 CodeRabbit inference engine (CLAUDE.md)
Use yargs command modules with
CommandFactorypattern for all CLI commands
Files:
src/cli/commands/workflow.ts
src/lib/utils/*.ts
📄 CodeRabbit inference engine (CLAUDE.md)
Wrap async operations with
withTimeoututility for timeout handling and graceful degradation
Files:
src/lib/utils/pdfProcessor.ts
src/lib/utils/pdfProcessor.ts
📄 CodeRabbit inference engine (CLAUDE.md)
Process PDFs with
PDFProcessorfor native document support and structured content extraction
Files:
src/lib/utils/pdfProcessor.ts
🧠 Learnings (27)
📚 Learning: 2025-12-29T06:57:39.349Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-29T06:57:39.349Z
Learning: Applies to src/lib/types/*.ts : Organize TypeScript types by domain (providers, generation, streaming, MCP, etc.) to avoid circular dependencies
Applied to files:
src/lib/workflow/core/types/ensembleTypes.tssrc/lib/workflow/utils/types/index.tssrc/lib/workflow/utils/types/validationTypes.tssrc/lib/workflow/utils/types/metricsTypes.tssrc/lib/workflow/core/types/registryTypes.tssrc/lib/neurolink.tssrc/lib/workflow/core/types/index.tssrc/lib/workflow/types.ts
📚 Learning: 2025-11-17T13:53:20.209Z
Learnt from: vigneshJuspay
Repo: juspay/neurolink PR: 237
File: memory-bank/tts-provider-implementation-plan.md:92-106
Timestamp: 2025-11-17T13:53:20.209Z
Learning: In PR 237's TTS modality implementation approach, TTS functionality uses GOOGLE_AI_API_KEY (not GOOGLE_TTS_API_KEY) when using the google-ai provider. TTS is implemented as an output modality that leverages the existing google-ai provider authentication.
Applied to files:
test/unit/tts-audio-output.test.ts
📚 Learning: 2025-11-11T14:02:21.868Z
Learnt from: vigneshJuspay
Repo: juspay/neurolink PR: 214
File: src/lib/index.ts:205-233
Timestamp: 2025-11-11T14:02:21.868Z
Learning: In the NeuroLink TTS SDK (src/lib/tts/), use GOOGLE_TTS_API_KEY environment variable specifically for Google Cloud Text-to-Speech access. GOOGLE_AI_API_KEY does not provide TTS access and should not be used for TTS functionality. The keys are intentionally kept separate for better access control and separation of concerns.
Applied to files:
test/unit/tts-audio-output.test.ts
📚 Learning: 2025-12-18T15:13:28.435Z
Learnt from: vigneshJuspay
Repo: juspay/neurolink PR: 693
File: src/lib/core/baseProvider.ts:490-517
Timestamp: 2025-12-18T15:13:28.435Z
Learning: In juspay/neurolink TTS integration (PR #693), when options.provider is "auto" and passed to TTSProcessor.synthesize, it will fail automatically during handler lookup since only concrete providers ("google-ai", "vertex") are registered as TTS handlers. No explicit validation against "auto" is needed—the implicit failure at handler registration lookup is by design.
Applied to files:
test/unit/tts-audio-output.test.tssrc/lib/workflow/core/responseConditioner.ts
📚 Learning: 2025-12-29T06:57:39.349Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-29T06:57:39.349Z
Learning: Run `pnpm run check` to validate TypeScript types before committing changes
Applied to files:
scripts/build-validations.cjs
📚 Learning: 2025-12-29T06:57:39.349Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-29T06:57:39.349Z
Learning: Applies to **/*.{ts,tsx} : Maintain strict TypeScript type checking across all SDK modules
Applied to files:
scripts/build-validations.cjssrc/lib/workflow/core/types/index.ts
📚 Learning: 2025-09-17T18:14:34.960Z
Learnt from: RajuSudhar
Repo: juspay/neurolink PR: 173
File: src/lib/types/index.ts:58-62
Timestamp: 2025-09-17T18:14:34.960Z
Learning: RajuSudhar explained that in the Neurolink codebase, there are multiple ProviderConfig types causing inconsistency. One existing ProviderConfig type better suited the "ProviderConfig" name, so they renamed the less-suitable one to AIModelProviderConfig to free up the name. Adding backward compatibility aliases would worsen naming inconsistency rather than help. The remaining duplicates will be systematically deduplicated in the 07-Types-Module.md TODO as part of their phased refactor approach.
Applied to files:
src/lib/workflow/LAYER-EXAMPLES.tsdocs/index.mdsrc/lib/workflow/core/responseConditioner.ts
📚 Learning: 2025-09-17T17:55:15.261Z
Learnt from: RajuSudhar
Repo: juspay/neurolink PR: 173
File: src/lib/index.ts:16-16
Timestamp: 2025-09-17T17:55:15.261Z
Learning: In src/lib/types/providers.ts, ProviderConfig was renamed to AIModelProviderConfig to deduplicate type names, as there was an existing ProviderConfig type that better suited the "ProviderConfig" name. This was an intentional breaking change for better type organization.
Applied to files:
src/lib/workflow/LAYER-EXAMPLES.tsdocs/index.mdsrc/lib/workflow/core/responseConditioner.tssrc/lib/neurolink.tssrc/lib/workflow/core/types/index.tssrc/lib/workflow/types.ts
📚 Learning: 2025-12-29T06:57:39.349Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-29T06:57:39.349Z
Learning: Build with `pnpm run build` and test CLI with `pnpm run build:cli && pnpm run cli` before final validation
Applied to files:
package.jsonsrc/cli/index.ts
📚 Learning: 2025-12-29T06:57:39.349Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-29T06:57:39.349Z
Learning: Run relevant test suites with `pnpm test` to validate changes before building
Applied to files:
package.json
📚 Learning: 2025-12-29T06:57:39.349Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-29T06:57:39.349Z
Learning: Applies to src/lib/types/index.ts : Add new AI providers to the `AIProviderName` enum in `src/lib/types/index.ts`
Applied to files:
src/lib/workflow/core/types/registryTypes.tssrc/lib/neurolink.tssrc/lib/workflow/core/types/index.ts
📚 Learning: 2025-12-29T06:57:39.349Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-29T06:57:39.349Z
Learning: Applies to src/cli/commands/*.ts : Use yargs command modules with `CommandFactory` pattern for all CLI commands
Applied to files:
src/cli/commands/workflow.tssrc/cli/index.ts
📚 Learning: 2025-12-29T06:57:39.349Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-29T06:57:39.349Z
Learning: Applies to src/cli/factories/commandFactory.ts : Update CLI provider choices in `src/cli/factories/commandFactory.ts` when adding new providers
Applied to files:
src/cli/commands/workflow.tssrc/cli/index.ts
📚 Learning: 2025-09-28T21:08:19.655Z
Learnt from: RajuSudhar
Repo: juspay/neurolink PR: 174
File: todos/refactor/03-providers-module.md:3-6
Timestamp: 2025-09-28T21:08:19.655Z
Learning: RajuSudhar mentioned that remaining interfaces in the SageMaker module (30+ interfaces across 5 files) will be resolved in a different PR, but no specific PR was found. The providers module refactor was marked COMPLETED prematurely while significant SageMaker interface conversion work remains pending.
Applied to files:
docs/index.md
📚 Learning: 2025-09-01T14:12:14.227Z
Learnt from: swaroopvarma1
Repo: juspay/neurolink PR: 141
File: docs/REAL-TIME-SPEECH-AGENTS.md:124-149
Timestamp: 2025-09-01T14:12:14.227Z
Learning: In the NeuroLink Speech-to-Speech agent system, the team prefers simple void-returning APIs (sendAudioFrame, sendText, flush) over Promise-based backpressure mechanisms, prioritizing ease of use and implementation simplicity for real-time speech processing.
Applied to files:
docs/index.md
📚 Learning: 2025-09-02T13:50:42.770Z
Learnt from: YasmeenOgo
Repo: juspay/neurolink PR: 145
File: src/lib/core/types.ts:0-0
Timestamp: 2025-09-02T13:50:42.770Z
Learning: The APIVersions enum in src/lib/core/types.ts now contains comprehensive API version constants for all major AI providers: Azure OpenAI (latest, stable, legacy), OpenAI (current, beta), Google AI (current, beta), and Anthropic (current). This centralization helps avoid API version drift across the codebase.
Applied to files:
docs/index.mdsrc/lib/neurolink.ts
📚 Learning: 2025-12-29T06:57:39.349Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-29T06:57:39.349Z
Learning: Applies to src/lib/mcp/toolRegistry.ts : Register all external tools and MCP servers with `MCPToolRegistry` for availability to AI models
Applied to files:
src/lib/workflow/core/workflowRegistry.tssrc/lib/neurolink.ts
📚 Learning: 2025-12-29T06:57:39.349Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-29T06:57:39.349Z
Learning: Applies to src/lib/utils/pdfProcessor.ts : Process PDFs with `PDFProcessor` for native document support and structured content extraction
Applied to files:
src/lib/utils/pdfProcessor.ts
📚 Learning: 2025-12-29T06:57:39.349Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-29T06:57:39.349Z
Learning: Applies to src/lib/**/*.ts : Use `ErrorFactory` for creating typed errors across the SDK
Applied to files:
src/lib/neurolink.ts
📚 Learning: 2025-09-01T22:58:39.149Z
Learnt from: sudharsan-juspay
Repo: juspay/neurolink PR: 140
File: src/lib/core/types.ts:198-203
Timestamp: 2025-09-01T22:58:39.149Z
Learning: In src/lib/core/types.ts, StreamOptions (imported from streamTypes.js) and StreamingOptions are intentionally different types with different use cases. StreamingOptions is for unified AI requests with multiple provider configurations, while StreamOptions is for individual streaming operations.
Applied to files:
src/lib/neurolink.tssrc/lib/types/streamTypes.ts
📚 Learning: 2025-12-29T06:57:39.349Z
Learnt from: CR
Repo: juspay/neurolink PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-12-29T06:57:39.349Z
Learning: Applies to src/lib/utils/transformationUtils.ts : Use `transformToolExecutions()` utility to convert tool results for provider API compatibility
Applied to files:
src/lib/neurolink.ts
📚 Learning: 2025-11-04T22:14:18.719Z
Learnt from: RajuSudhar
Repo: juspay/neurolink PR: 0
File: :0-0
Timestamp: 2025-11-04T22:14:18.719Z
Learning: In the juspay/neurolink repository, all new type definitions must be placed in src/lib/types/. New type definitions outside this directory should be flagged and blocked in code reviews.
Applied to files:
src/lib/neurolink.tssrc/lib/workflow/core/types/index.ts
📚 Learning: 2025-09-24T06:42:06.088Z
Learnt from: amreetkhuntia
Repo: juspay/neurolink PR: 185
File: src/lib/evaluation/contextBuilder.ts:79-85
Timestamp: 2025-09-24T06:42:06.088Z
Learning: In the NeuroLink codebase, using `(options.prompt || [])` pattern for handling potentially undefined prompt arrays is the preferred approach over extracting to a normalized variable when building conversation history in the ContextBuilder class.
Applied to files:
src/lib/neurolink.ts
📚 Learning: 2025-12-15T18:35:37.783Z
Learnt from: vigneshJuspay
Repo: juspay/neurolink PR: 0
File: :0-0
Timestamp: 2025-12-15T18:35:37.783Z
Learning: In juspay/neurolink TTS implementation (PR #691), the new StreamChunk discriminated union type introduced in TTS-019 will be integrated with StreamResult.stream during the actual TTS streaming implementation PR (TTS-020/TTS-021), not in the type-definition PR. This phased approach keeps type updates and implementation changes atomic.
Applied to files:
src/lib/types/streamTypes.ts
📚 Learning: 2025-09-01T06:15:59.759Z
Learnt from: amreetkhuntia
Repo: juspay/neurolink PR: 133
File: src/lib/core/types.ts:208-210
Timestamp: 2025-09-01T06:15:59.759Z
Learning: The middleware?: MiddlewareFactoryOptions field is already present in both TextGenerationOptions and StreamOptions interfaces in the neurolink codebase.
Applied to files:
src/lib/types/streamTypes.ts
📚 Learning: 2025-09-28T21:00:08.243Z
Learnt from: RajuSudhar
Repo: juspay/neurolink PR: 174
File: src/lib/mcp/contracts/mcpContract.ts:0-0
Timestamp: 2025-09-28T21:00:08.243Z
Learning: The src/lib/mcp/contracts/mcpContract.ts file was completely removed during the MCP types refactor in PR #174, with its types moved to centralized modules like src/lib/types/mcpTypes.ts and src/lib/types/index.ts.
Applied to files:
src/lib/workflow/core/types/index.ts
📚 Learning: 2025-11-05T20:31:04.103Z
Learnt from: RajuSudhar
Repo: juspay/neurolink PR: 227
File: src/lib/utils/redis.ts:15-15
Timestamp: 2025-11-05T20:31:04.103Z
Learning: In the juspay/neurolink repository, type centralization rules (requiring types in src/lib/types/) do not apply to private, non-exported utility types used within a single file. Simple readability helpers like `type RedisClient = ReturnType<typeof createClient>` should remain in their implementation file when they are not exported and only used locally. Only exported types shared across modules, business/domain types, and public API types require centralization.
Applied to files:
src/lib/workflow/core/types/index.ts
🧬 Code graph analysis (21)
examples/workflow-integration-example.ts (3)
src/lib/workflow/workflows/consensusWorkflow.ts (1)
CONSENSUS_3_WORKFLOW(44-108)src/lib/workflow/workflows/multiJudgeWorkflow.ts (1)
MULTI_JUDGE_5_WORKFLOW(50-154)src/lib/workflow/workflows/adaptiveWorkflow.ts (1)
QUALITY_MAX_WORKFLOW(44-170)
src/lib/workflow/core/types/ensembleTypes.ts (1)
src/lib/workflow/types.ts (4)
ModelConfig(97-118)ExecutionConfig(193-219)EnsembleResponse(305-331)WorkflowError(507-523)
src/lib/workflow/core/types/layerTypes.ts (1)
src/lib/workflow/types.ts (2)
EnsembleResponse(305-331)ModelGroup(40-60)
test/unit/tts-audio-output.test.ts (2)
src/lib/types/ttsTypes.ts (1)
TTSResult(75-97)src/lib/utils/ttsProcessor.ts (2)
TTSError(30-51)TTS_ERROR_CODES(18-25)
src/lib/workflow/utils/types/validationTypes.ts (2)
src/lib/workflow/utils/types/index.ts (1)
ValidationIssues(11-11)src/lib/workflow/types.ts (2)
WorkflowValidationError(475-480)WorkflowValidationWarning(485-490)
src/lib/workflow/LAYER-EXAMPLES.ts (1)
src/lib/workflow/types.ts (1)
WorkflowConfig(65-92)
src/lib/workflow/core/workflowRunner.ts (3)
src/lib/workflow/config.ts (3)
usesModelGroups(281-283)PLACEHOLDER_PROVIDER(32-32)PLACEHOLDER_MODEL(33-33)src/lib/workflow/core/types/ensembleTypes.ts (1)
EnsembleExecutionResult(29-35)src/lib/workflow/core/types/judgeTypes.ts (1)
ScoreResult(31-35)
src/lib/workflow/utils/types/metricsTypes.ts (1)
src/lib/workflow/utils/types/index.ts (3)
WorkflowExecutionMetrics(9-9)SummaryStats(7-7)WorkflowComparison(8-8)
src/lib/workflow/core/types/registryTypes.ts (1)
src/lib/workflow/types.ts (2)
WorkflowConfig(65-92)WorkflowValidationResult(466-470)
src/lib/workflow/utils/workflowMetrics.ts (2)
src/lib/workflow/utils/types/metricsTypes.ts (3)
WorkflowExecutionMetrics(9-19)SummaryStats(24-31)WorkflowComparison(36-41)src/lib/workflow/types.ts (2)
WorkflowResult(256-300)EnsembleResponse(305-331)
src/lib/workflow/core/responseConditioner.ts (3)
src/lib/workflow/core/types/conditionerTypes.ts (2)
ConditionOptions(16-23)ConditionResult(28-40)src/lib/workflow/types.ts (4)
ConditioningConfig(157-188)EnsembleResponse(305-331)JudgeScores(337-359)MultiJudgeScores(364-391)src/lib/index.ts (3)
EnsembleResponse(309-309)JudgeScores(310-310)MultiJudgeScores(311-311)
src/lib/workflow/workflows/adaptiveWorkflow.ts (3)
src/lib/workflow/types.ts (1)
WorkflowConfig(65-92)src/lib/workflow/config.ts (1)
WORKFLOW_CREATION_DATE(36-36)src/lib/utils/logger.ts (1)
logger(358-401)
src/lib/workflow/core/types/judgeTypes.ts (1)
src/lib/workflow/types.ts (5)
JudgeConfig(124-151)EnsembleResponse(305-331)JudgeScores(337-359)MultiJudgeScores(364-391)WorkflowError(507-523)
src/lib/workflow/workflows/multiJudgeWorkflow.ts (4)
src/lib/index.ts (4)
MULTI_JUDGE_5_WORKFLOW(344-344)WorkflowConfig(304-304)MULTI_JUDGE_3_WORKFLOW(345-345)createMultiJudgeWorkflow(346-346)src/lib/workflow/index.ts (4)
MULTI_JUDGE_5_WORKFLOW(113-113)WorkflowConfig(31-31)MULTI_JUDGE_3_WORKFLOW(112-112)createMultiJudgeWorkflow(111-111)src/lib/workflow/types.ts (1)
WorkflowConfig(65-92)src/lib/workflow/config.ts (1)
WORKFLOW_CREATION_DATE(36-36)
src/lib/workflow/core/workflowRegistry.ts (2)
src/lib/workflow/core/types/registryTypes.ts (6)
RegistryEntry(11-16)RegisterOptions(21-24)RegisterResult(29-34)ListOptions(39-44)WorkflowMetadata(49-53)RegistryStats(58-67)src/lib/workflow/types.ts (1)
WorkflowConfig(65-92)
src/lib/workflow/utils/workflowValidation.ts (3)
src/lib/workflow/types.ts (6)
WorkflowConfig(65-92)WorkflowValidationResult(466-470)WorkflowValidationError(475-480)WorkflowValidationWarning(485-490)ModelConfig(97-118)JudgeConfig(124-151)src/lib/workflow/config.ts (4)
hasJudge(461-463)getAllJudges(470-478)MIN_SCORE(25-25)MAX_SCORE(26-26)src/lib/workflow/utils/types/validationTypes.ts (1)
ValidationIssues(14-17)
src/lib/workflow/core/ensembleExecutor.ts (1)
src/lib/workflow/core/types/index.ts (2)
ExecuteEnsembleOptions(9-9)EnsembleExecutionResult(8-8)
src/lib/types/streamTypes.ts (1)
src/lib/types/sdkTypes.ts (1)
AudioChunk(27-27)
src/cli/index.ts (2)
src/cli/parser.ts (1)
initializeCliParser(12-198)src/cli/commands/workflow.ts (1)
WorkflowCommandFactory(51-362)
src/lib/workflow/config.ts (3)
src/lib/index.ts (6)
AIProviderName(38-38)DEFAULT_SCORE_SCALE(369-369)WorkflowConfig(304-304)ModelConfig(306-306)ModelGroup(308-308)JudgeConfig(307-307)src/lib/workflow/index.ts (13)
WorkflowConfigSchema(59-59)ConditioningConfig(16-16)DEFAULT_EXECUTION_CONFIG(54-54)ExecutionConfig(19-19)DEFAULT_SCORE_SCALE(165-165)usesModelGroups(58-58)WorkflowConfig(31-31)getAllModels(56-56)ModelConfig(24-24)getModelGroups(57-57)ModelGroup(25-25)DEFAULT_JUDGE_CONFIG(55-55)JudgeConfig(21-21)src/lib/workflow/types.ts (6)
ConditioningConfig(157-188)ExecutionConfig(193-219)WorkflowConfig(65-92)ModelConfig(97-118)ModelGroup(40-60)JudgeConfig(124-151)
src/lib/workflow/index.ts (1)
src/lib/index.ts (1)
WORKFLOW_ENGINE_VERSION(368-368)
🪛 GitHub Check: 🛡️ Code Quality & Security Gate
src/cli/commands/workflow.ts
[warning] 55-55:
Static method 'createWorkflowCommand' has too many lines (307). Maximum allowed is 300
🪛 GitHub Check: test (20)
src/cli/commands/workflow.ts
[warning] 55-55:
Static method 'createWorkflowCommand' has too many lines (307). Maximum allowed is 300
🪛 LanguageTool
docs/WORKFLOW-ENGINE-HLD.md
[style] ~4-~4: Some style guides suggest that commas should set off the year in a month-day-year date.
Context: ...Version**: 1.0 Date: November 28, 2025 Status: Design Phase Author...
(MISSING_COMMA_AFTER_YEAR)
[grammar] ~12-~12: Ensure spelling is correct
Context: ...les and judge-based scoring. It extends Neurolink's existing provider abstraction to suppor...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
[grammar] ~408-~408: Ensure spelling is correct
Context: ...? Integration Points ### With Existing Neurolink Infrastructure #### 1. AIProviderFacto...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
docs/WORKFLOW-ENGINE-LLD.md
[style] ~4-~4: Some style guides suggest that commas should set off the year in a month-day-year date.
Context: ...Version**: 1.0 Date: November 28, 2025 Status: Design Phase Author...
(MISSING_COMMA_AFTER_YEAR)
| - **LiteLLM Integration** – Access 100+ AI models from all major providers through unified interface. → [Setup Guide](LITELLM-INTEGRATION.md) | ||
| - **SageMaker Integration** – Deploy and use custom trained models on AWS infrastructure. → [Setup Guide](SAGEMAKER-INTEGRATION.md) | ||
| - **SageMaker Integration** – Deploy and use custom-trained models on AWS infrastructure. → [Setup Guide](SAGEMAKER-INTEGRATION.md) | ||
| - **OpenRouter Integration** – Access 300+ models from OpenAI, Anthropic, Google, Meta, and more through a single unified API. → [Setup Guide](getting-started/providers/openrouter.md) |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
OpenRouter API current number of models available 2025
💡 Result:
As of December 30, 2025, OpenRouter’s public documentation/catalog lists “400+” models available via its API. [1][2]
Sources:
- OpenRouter documentation — Models overview (shows “Explore and browse 400+ models”). [1]
- Independent review referencing OpenRouter’s public catalog (reports catalog size ~500+ in 2025). [2]
Update OpenRouter model count from outdated figure.
Line 33 claims "Access 300+ models" but OpenRouter's current documentation lists "400+" models available as of December 2025. Update both line 33 and line 70 to reflect the current count (400+) to avoid outdated marketing claims.
🤖 Prompt for AI Agents
In docs/index.md around lines 33 and 70, update the OpenRouter model count from
"300+ models" to "400+ models" to reflect the current December 2025 figure; edit
both lines so the text reads "Access 400+ models from OpenAI, Anthropic, Google,
Meta, and more through a single unified API. → [Setup
Guide](getting-started/providers/openrouter.md)" (or the equivalent phrase used
at line 70) ensuring the same updated count appears in both places for
consistency.
| | **OpenAI Compatible** | Any OpenAI-compatible endpoint | Varies | ✅ Full | ✅ Production | [Setup Guide](getting-started/provider-setup.md#openai-compatible) | | ||
|
|
||
| **[📖 Provider Comparison Guide](reference/provider-comparison.md)** - Detailed feature matrix and selection criteria | ||
| **[🔬 Provider Feature Compatibility](reference/provider-feature-compatibility.md)** - Test-based compatibility reference for all 19 features across 11 providers |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, check if docs/index.md exists and read the relevant lines
if [ -f "docs/index.md" ]; then
echo "=== docs/index.md (lines 55-85) ==="
sed -n '55,85p' docs/index.md
else
echo "docs/index.md not found"
fiRepository: juspay/neurolink
Length of output: 3351
🏁 Script executed:
# Check if the provider-feature-compatibility.md file exists
if [ -f "docs/reference/provider-feature-compatibility.md" ]; then
echo "=== docs/reference/provider-feature-compatibility.md (first 100 lines) ==="
head -100 docs/reference/provider-feature-compatibility.md
else
echo "docs/reference/provider-feature-compatibility.md not found"
fiRepository: juspay/neurolink
Length of output: 4861
Correct provider count discrepancy in feature compatibility reference.
Line 78 references "11 providers" tested, but the main provider table lists 13 providers. The discrepancy is valid—only 11 of the 13 are actually included in the test matrix. However, the untested providers are OpenRouter and OpenAI Compatible, not LiteLLM (which is tested). Update the reference or clarify which providers are intentionally excluded from the compatibility testing.
🤖 Prompt for AI Agents
In docs/index.md around line 78, the provider count and exclusion list are
inconsistent: the text says "11 providers" but the main table has 13; update the
sentence to explicitly state which two providers are excluded from the
compatibility tests (OpenRouter and OpenAI Compatible) or reword to "13
providers (11 included in compatibility tests; OpenRouter and OpenAI Compatible
excluded)"; ensure the phrasing clearly identifies OpenRouter and OpenAI
Compatible as the intentionally untested providers.
| ### Input Validation | ||
|
|
||
| ````typescript | ||
| // Sanitize all user inputs before passing to models | ||
| function sanitizeInput(input: string): string { | ||
| // Remove potential prompt injection attempts | ||
| return input | ||
| .replace(/```[^`]*```/g, "") // Remove code blocks | ||
| .replace(/<script[^>]*>.*?<\/script>/gi, "") // Remove scripts | ||
| .trim(); | ||
| } | ||
| ```` |
There was a problem hiding this comment.
Input sanitization example is overly simplistic and may have issues.
The sanitizeInput function in the security section removes all code blocks, which could break legitimate technical content. Also, the regex for removing code blocks is incomplete (backticks can appear in other contexts). Consider documenting this as a placeholder that needs production-hardening.
🔎 Suggested improvement
// Sanitize all user inputs before passing to models
+// NOTE: This is a simplified example. Production implementations should use
+// established sanitization libraries and consider the specific use case.
function sanitizeInput(input: string): string {
- // Remove potential prompt injection attempts
- return input
- .replace(/```[^`]*```/g, "") // Remove code blocks
- .replace(/<script[^>]*>.*?<\/script>/gi, "") // Remove scripts
- .trim();
+ // Consider using established sanitization libraries
+ // This example is for documentation purposes only
+ return input.trim();
}🤖 Prompt for AI Agents
In docs/WORKFLOW-ENGINE-LLD.md around lines 1773-1784, the provided
sanitizeInput implementation is overly simplistic and unsafe for production (it
bluntly strips code blocks with an incomplete regex and may break legitimate
content); mark this function as a documentation-only placeholder, revert removal
of code blocks, change the example to simply return input.trim() with a clear
comment stating "placeholder — use a vetted sanitization/escaping library in
production", and add a short note recommending specific libraries or frameworks
(e.g., DOMPurify/OWASP Java HTML Sanitizer or server-side validators) and to add
tests/validation rules tailored to expected input types.
| // Basic usage info | ||
| usage: workflowResult.usage | ||
| ? { | ||
| input: workflowResult.usage.totalInputTokens, | ||
| output: workflowResult.usage.totalOutputTokens, | ||
| total: workflowResult.usage.totalTokens, | ||
| } | ||
| : undefined, | ||
|
|
||
| // Performance | ||
| responseTime: workflowResult.totalTime, | ||
|
|
||
| // Workflow-specific data | ||
| workflow: { | ||
| originalResponse: | ||
| workflowResult.originalContent || workflowResult.content, // Original unmodified best response | ||
| processedResponse: workflowResult.content, // After conditioning (with metadata) | ||
| ensembleResponses: workflowResult.ensembleResponses.map((r) => ({ | ||
| provider: r.provider, | ||
| model: r.model, | ||
| content: r.content, | ||
| responseTime: r.responseTime, | ||
| status: r.status, | ||
| error: r.error, | ||
| })), | ||
| judgeScores: workflowResult.judgeScores | ||
| ? { | ||
| scores: workflowResult.judgeScores.scores, | ||
| reasoning: workflowResult.reasoning, | ||
| selectedModel: `${workflowResult.selectedResponse?.provider}-${workflowResult.selectedResponse?.model}`, | ||
| } | ||
| : undefined, | ||
| selectedModel: `${workflowResult.selectedResponse?.provider}-${workflowResult.selectedResponse?.model}`, | ||
| metrics: { | ||
| totalTime: workflowResult.totalTime, | ||
| ensembleTime: workflowResult.ensembleTime, | ||
| judgeTime: workflowResult.judgeTime, | ||
| conditioningTime: workflowResult.conditioningTime, | ||
| }, | ||
| workflowId: workflowResult.workflow, | ||
| workflowName: workflowResult.workflowName, | ||
| }, | ||
| }; | ||
|
|
||
| logger.debug("[NeuroLink] Workflow generation complete", { | ||
| workflowId: workflowResult.workflow, | ||
| selectedModel: generateResult.workflow?.selectedModel, | ||
| score: workflowResult.score, | ||
| totalTime: workflowResult.totalTime, | ||
| }); | ||
|
|
||
| return generateResult; | ||
| } |
There was a problem hiding this comment.
Streaming with workflows: audio-only inputs can crash and metadata may misrepresent the selected model.
Two issues in the streaming workflow path:
streamWithWorkflowassumesoptions.input.textis a non-empty string (logging and passing it intorunWorkflowWithStreaming), butvalidateStreamInputallows audio-only streams. Callingstream({ input: { audio: ... }, workflow: ... })will pass validation and then hit asubstring/string operation onundefined, causing a runtime error.streamWithWorkflowinitializesStreamResult.provider/modelfromworkflowConfig.models[0]and never updates them fromfinalResult.selectedResponse. For workflows that select a different model at judge time or only definemodelGroups, this metadata will be inaccurate orundefined, whileworkflow.selectedModelalready falls back to"unknown"when necessary.
Recommended adjustments:
- In the workflow-branch of
stream, enforce text-based input (e.g., throw a typed error viaErrorFactoryifoptions.input.textis missing/empty whenworkflow/workflowConfigis set) to avoid audio-only misuse until workflows explicitly support audio. - After
finalResultis available, updatestreamResult.providerandstreamResult.modelfromresult.selectedResponse(with"unknown"fallbacks) so top-level metadata matches the actually selected model.
Also applies to: 2057-2140, 3003-3015
🤖 Prompt for AI Agents
In src/lib/neurolink.ts around lines 2003-2055 (also apply same fix to 2057-2140
and 3003-3015): the streaming workflow path assumes options.input.text exists
and initializes StreamResult.provider/model from workflowConfig.models[0], which
breaks for audio-only inputs and when the judge selects a different model. Fix
by validating that when a workflow/workflowConfig is provided the input contains
non-empty text (throw a typed error via ErrorFactory if options.input.text is
missing/empty) to block audio-only usage until workflows support audio, and
after the finalResult (finalResult.selectedResponse) is available update
streamResult.provider and streamResult.model from finalResult.selectedResponse
(falling back to "unknown" if missing) so top-level metadata matches the actual
selected model.
| it("should register all predefined workflows", () => { | ||
| registerWorkflow(CONSENSUS_3_WORKFLOW); | ||
| registerWorkflow(CONSENSUS_3_FAST_WORKFLOW); | ||
| registerWorkflow(FAST_FALLBACK_WORKFLOW); | ||
| registerWorkflow(AGGRESSIVE_FALLBACK_WORKFLOW); | ||
| registerWorkflow(MULTI_JUDGE_5_WORKFLOW); | ||
| registerWorkflow(MULTI_JUDGE_3_WORKFLOW); | ||
| registerWorkflow(QUALITY_MAX_WORKFLOW); | ||
| registerWorkflow(SPEED_FIRST_WORKFLOW); | ||
| registerWorkflow(BALANCED_ADAPTIVE_WORKFLOW); | ||
|
|
||
| const list = listWorkflows(); | ||
| // Some workflows may fail validation and not be registered | ||
| expect(list.length).toBeGreaterThan(0); | ||
| const ids = list.map((w) => w.id); | ||
| expect(ids).toContain("consensus-3"); | ||
| expect(ids).toContain("multi-judge-5"); | ||
| }); |
There was a problem hiding this comment.
Weak assertion reduces test reliability.
The vague expectation expect(list.length).toBeGreaterThan(0) combined with the comment "Some workflows may fail validation and not be registered" makes this test non-deterministic and masks potential registration failures.
🔎 Suggested improvement
Be explicit about expected registration outcomes:
- const list = listWorkflows();
- // Some workflows may fail validation and not be registered
- expect(list.length).toBeGreaterThan(0);
+ const list = listWorkflows();
+ // All predefined workflows should register successfully
+ expect(list.length).toBe(9);
const ids = list.map((w) => w.id);
expect(ids).toContain("consensus-3");
expect(ids).toContain("multi-judge-5");
+ // Add explicit checks for all 9 workflow IDsIf some workflows are expected to fail validation, document which ones and why, or fix the workflows to pass validation.
🤖 Prompt for AI Agents
In src/lib/workflow/__tests__/workflow.test.ts around lines 139 to 156, the test
uses a weak non-deterministic assertion expect(list.length).toBeGreaterThan(0)
which can mask registration failures; update the test to assert explicit,
deterministic outcomes by either (a) listing which workflows are expected to
successfully register and asserting list.length equals that exact number and
that each expected id is present, or (b) if some workflows are known to fail
validation, add comments documenting which ones and replace the loose length
check with assertions that only the known-valid workflow ids are present (and
assert the length equals the number of those known-valid ids) so the test
reliably fails when a workflow unexpectedly stops registering.
| export function importRegistry( | ||
| json: string, | ||
| options: RegisterOptions = {}, | ||
| ): RegisterResult[] { | ||
| try { | ||
| const workflows = JSON.parse(json) as WorkflowConfig[]; | ||
| const results: RegisterResult[] = []; | ||
|
|
||
| workflows.forEach((config) => { | ||
| const result = registerWorkflow(config, options); | ||
| results.push(result); | ||
| }); | ||
|
|
||
| logger.info(`[${functionTag}] Registry import completed`, { | ||
| total: workflows.length, | ||
| successful: results.filter((r) => r.success).length, | ||
| }); | ||
|
|
||
| return results; | ||
| } catch (error) { | ||
| logger.error(`[${functionTag}] Registry import failed`, { | ||
| error: (error as Error).message, | ||
| }); | ||
| return [ | ||
| { | ||
| success: false, | ||
| workflowId: "import-error", | ||
| error: `Import failed: ${(error as Error).message}`, | ||
| }, | ||
| ]; | ||
| } |
There was a problem hiding this comment.
Unsafe JSON parsing in importRegistry may accept malformed data.
JSON.parse output is cast directly to WorkflowConfig[] without validation. Malformed JSON could pass through if validateBeforeRegister is disabled. Consider validating the array structure before iteration.
🔎 Proposed fix
export function importRegistry(
json: string,
options: RegisterOptions = {},
): RegisterResult[] {
try {
- const workflows = JSON.parse(json) as WorkflowConfig[];
+ const parsed = JSON.parse(json);
+ if (!Array.isArray(parsed)) {
+ throw new Error("Expected an array of workflow configurations");
+ }
+ const workflows = parsed as WorkflowConfig[];
const results: RegisterResult[] = [];📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| export function importRegistry( | |
| json: string, | |
| options: RegisterOptions = {}, | |
| ): RegisterResult[] { | |
| try { | |
| const workflows = JSON.parse(json) as WorkflowConfig[]; | |
| const results: RegisterResult[] = []; | |
| workflows.forEach((config) => { | |
| const result = registerWorkflow(config, options); | |
| results.push(result); | |
| }); | |
| logger.info(`[${functionTag}] Registry import completed`, { | |
| total: workflows.length, | |
| successful: results.filter((r) => r.success).length, | |
| }); | |
| return results; | |
| } catch (error) { | |
| logger.error(`[${functionTag}] Registry import failed`, { | |
| error: (error as Error).message, | |
| }); | |
| return [ | |
| { | |
| success: false, | |
| workflowId: "import-error", | |
| error: `Import failed: ${(error as Error).message}`, | |
| }, | |
| ]; | |
| } | |
| export function importRegistry( | |
| json: string, | |
| options: RegisterOptions = {}, | |
| ): RegisterResult[] { | |
| try { | |
| const parsed = JSON.parse(json); | |
| if (!Array.isArray(parsed)) { | |
| throw ErrorFactory.createError("INVALID_REGISTRY_FORMAT", "Expected an array of workflow configurations"); | |
| } | |
| const workflows = parsed as WorkflowConfig[]; | |
| const results: RegisterResult[] = []; | |
| workflows.forEach((config) => { | |
| const result = registerWorkflow(config, options); | |
| results.push(result); | |
| }); | |
| logger.info(`[${functionTag}] Registry import completed`, { | |
| total: workflows.length, | |
| successful: results.filter((r) => r.success).length, | |
| }); | |
| return results; | |
| } catch (error) { | |
| logger.error(`[${functionTag}] Registry import failed`, { | |
| error: (error as Error).message, | |
| }); | |
| return [ | |
| { | |
| success: false, | |
| workflowId: "import-error", | |
| error: `Import failed: ${(error as Error).message}`, | |
| }, | |
| ]; | |
| } |
🤖 Prompt for AI Agents
In src/lib/workflow/core/workflowRegistry.ts around lines 348 to 378, the code
directly casts JSON.parse(json) to WorkflowConfig[] and iterates, which can
accept malformed data; change parsing to validate the parsed value is an array
and that each item is an object with required WorkflowConfig fields (e.g.,
id/name/steps or whatever the WorkflowConfig type requires) before calling
registerWorkflow; if validation fails, log a descriptive error and return a
failed RegisterResult instead of proceeding; implement a small validation helper
(or reuse existing schema/validator) to check types/required properties and skip
or reject invalid entries so malformed JSON or incorrect shapes do not get
registered.
| ### Unit Tests (`src/workflow/__tests__/workflow.test.ts`) | ||
|
|
||
| - ✅ Predefined workflow loading (9 workflows) | ||
| - ✅ Workflow registry operations (CRUD) | ||
| - ✅ Configuration structure validation | ||
| - ✅ Score scale consistency | ||
| - ✅ Unique ID enforcement | ||
| - ✅ Models vs ModelGroups exclusivity |
There was a problem hiding this comment.
Test file path inconsistency.
The path src/workflow/__tests__/workflow.test.ts should be src/lib/workflow/__tests__/workflow.test.ts based on the actual file structure shown at line 237.
🔎 Proposed fix
-### Unit Tests (`src/workflow/__tests__/workflow.test.ts`)
+### Unit Tests (`src/lib/workflow/__tests__/workflow.test.ts`)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ### Unit Tests (`src/workflow/__tests__/workflow.test.ts`) | |
| - ✅ Predefined workflow loading (9 workflows) | |
| - ✅ Workflow registry operations (CRUD) | |
| - ✅ Configuration structure validation | |
| - ✅ Score scale consistency | |
| - ✅ Unique ID enforcement | |
| - ✅ Models vs ModelGroups exclusivity | |
| ### Unit Tests (`src/lib/workflow/__tests__/workflow.test.ts`) | |
| - ✅ Predefined workflow loading (9 workflows) | |
| - ✅ Workflow registry operations (CRUD) | |
| - ✅ Configuration structure validation | |
| - ✅ Score scale consistency | |
| - ✅ Unique ID enforcement | |
| - ✅ Models vs ModelGroups exclusivity |
🤖 Prompt for AI Agents
In WORKFLOW-ENGINE-COMPLETE.md around lines 163 to 170, the unit test file path
is incorrect — it lists src/workflow/__tests__/workflow.test.ts but the actual
file lives at src/lib/workflow/__tests__/workflow.test.ts; update the
documentation to use the correct path (replace src/workflow/... with
src/lib/workflow/...) so the README matches the repository structure everywhere
and verify other references to this test path in the document.
| src/workflow/ | ||
| ├── types.ts # Core type definitions | ||
| ├── config.ts # Zod validation schemas | ||
| ├── index.ts # Public API exports | ||
| ├── LAYER-EXAMPLES.ts # Execution pattern examples | ||
| ├── core/ | ||
| │ ├── ensembleExecutor.ts # Model execution engine | ||
| │ ├── judgeScorer.ts # Judge evaluation | ||
| │ ├── responseConditioner.ts # Stub conditioner | ||
| │ ├── workflowRegistry.ts # Registry operations | ||
| │ ├── workflowRunner.ts # Main orchestrator | ||
| │ └── types/ # Core type definitions | ||
| │ ├── ensembleTypes.ts | ||
| │ ├── judgeTypes.ts | ||
| │ ├── conditionerTypes.ts | ||
| │ ├── registryTypes.ts | ||
| │ ├── layerTypes.ts | ||
| │ └── index.ts | ||
| ├── utils/ | ||
| │ ├── workflowValidation.ts # Validation logic | ||
| │ ├── workflowMetrics.ts # Metrics calculation | ||
| │ └── types/ # Utility type definitions | ||
| │ ├── validationTypes.ts | ||
| │ ├── metricsTypes.ts | ||
| │ └── index.ts | ||
| ├── workflows/ | ||
| │ ├── consensusWorkflow.ts # Consensus workflows (2) | ||
| │ ├── fallbackWorkflow.ts # Fallback workflows (2) | ||
| │ ├── multiJudgeWorkflow.ts # Multi-judge workflows (2) | ||
| │ └── adaptiveWorkflow.ts # Adaptive workflows (3) | ||
| └── __tests__/ | ||
| └── workflow.test.ts # Unit tests (244 lines) | ||
|
|
||
| examples/ | ||
| └── workflow-integration-example.ts # Integration example (130 lines) | ||
|
|
||
| lib/ | ||
| ├── index.ts # Main SDK exports (includes workflow) | ||
| └── neurolink.ts # NeuroLink class (with workflow methods) | ||
| ``` |
There was a problem hiding this comment.
File structure paths missing lib/ prefix.
The file tree shows src/workflow/ but the actual implementation is under src/lib/workflow/. This could confuse developers navigating the codebase.
🔎 Proposed fix
-src/workflow/
+src/lib/workflow/
├── types.ts # Core type definitions
...🤖 Prompt for AI Agents
In WORKFLOW-ENGINE-COMPLETE.md around lines 206 to 245, the displayed file tree
lists paths starting with src/workflow/ but the actual implementation lives
under src/lib/workflow/; update the tree entries to use the correct lib-prefixed
paths (e.g., src/lib/workflow/, src/lib/workflow/core/, src/lib/workflow/utils/,
src/lib/workflow/workflows/, src/lib/workflow/__tests__, and adjust any example
or lib references accordingly) so the documentation matches the repository
layout.
af99383 to
677dcd1
Compare
| ├── lib/ | ||
| │ ├── workflow/ | ||
| │ │ ├── index.ts # Public API exports (60 lines) | ||
| │ │ ├── types.ts # Type definitions (250 lines) |
There was a problem hiding this comment.
Types to be moved to the common types folder.
| ``` | ||
| workflow/ | ||
| ├── index.ts # Public API exports | ||
| ├── types.ts # Core workflow types |
There was a problem hiding this comment.
Types to be moved to the common types folder.
| ``` | ||
| ┌────────────────────────────────────────────────────────────┐ | ||
| │ 1. USER REQUEST │ | ||
| │ neuro.generateWorkflow({ │ |
There was a problem hiding this comment.
You cannot add any new functions. Only generate and stream can be added and extended with required inputs if there is any input requirement.
| ### WorkflowConfig | ||
|
|
||
| ```typescript | ||
| interface WorkflowConfig { |
There was a problem hiding this comment.
No interfaces, only types can be created.
| ### ModelConfig | ||
|
|
||
| ```typescript | ||
| interface ModelConfig { |
There was a problem hiding this comment.
No interfaces, only types can be created.
| }); | ||
|
|
||
| // Execute custom workflow | ||
| const customResult = await neuro.generateWorkflow({ |
There was a problem hiding this comment.
use generate and stream only as exposed commands
| /** | ||
| * Complete workflow configuration | ||
| */ | ||
| export interface WorkflowConfig { |
| console.log("Running 3 models in parallel with judge evaluation...\n"); | ||
|
|
||
| try { | ||
| const result1 = await neurolink.runWorkflow(CONSENSUS_3_WORKFLOW, { |
There was a problem hiding this comment.
stream and generate to be exposed only
677dcd1 to
117e80a
Compare
| ### **CLI Commands** (`src/cli/commands/workflow.ts`) | ||
| ```bash | ||
| # List all available workflows | ||
| neurolink workflow --list |
| ### **CLI Commands** (`src/cli/commands/workflow.ts`) | ||
| ```bash | ||
| # List all available workflows | ||
| neurolink workflow --list |
There was a problem hiding this comment.
same for all workflow commands
| @@ -0,0 +1,308 @@ | |||
| # Workflow Engine - Implementation Complete ✅ | |||
There was a problem hiding this comment.
do we need all of these documents? they should either be in correct location in doc or removed. or move to memory bank.
also need to add in readme q1 2026 items and an entry for this new feature we are adding
| @@ -0,0 +1,523 @@ | |||
| /** | |||
| @@ -0,0 +1,17 @@ | |||
| /** | |||
There was a problem hiding this comment.
move all to common types fodler
2d1b39d to
d950ed9
Compare
12af40f to
d34e6fa
Compare
…el orchestration - Add core workflow engine with ensemble executor, judge scorer, and response conditioner - Implement 9 predefined workflows (consensus, multi-judge, fallback, adaptive) - Add workflow registry for centralized workflow management - Integrate 5 workflow methods into NeuroLink SDK class - Add CLI workflow command with list, info, and execution support - Implement workflow validation and metrics tracking - Add comprehensive TypeScript types for all workflow components - Export all workflow functionality from lib/index.ts - Remove redundant .d.ts files (auto-generated during build) - Fix import paths after moving to src/lib/workflow/ - Update memory bank documentation with implementation details
d34e6fa to
fbb41bb
Compare
|
🎉 This PR is included in version 9.4.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
…el orchestration
Pull Request
Description
Type of Change
Related Issues
Changes Made
AI Provider Impact
Component Impact
Testing
Test Environment
Performance Impact
Breaking Changes
Screenshots/Demo
Checklist
Additional Notes
Summary by CodeRabbit
New Features
Public API
CLI
Documentation
Tests
✏️ Tip: You can customize this high-level summary in your review settings.