Skip to content

feat(multimodal): add comprehensive PDF file support with native docu… #211

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
murdore merged 1 commit into from
Oct 12, 2025
Merged

feat(multimodal): add comprehensive PDF file support with native docu… #211

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 12 additions & 9 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Extracted from production systems at Juspay and battle-tested at enterprise scal
## What's New (Q4 2025)

- **CSV File Support** – Attach CSV files to prompts for AI-powered data analysis with auto-detection. → [CSV Guide](docs/features/multimodal-chat.md#csv-file-support)
- **PDF File Support** – Process PDF documents with native visual analysis for Vertex AI, Anthropic, Bedrock, AI Studio. → [PDF Guide](docs/features/pdf-support.md)
- **LiteLLM Integration** – Access 100+ AI models from all major providers through unified interface. → [Setup Guide](docs/LITELLM-INTEGRATION.md)
- **SageMaker Integration** – Deploy and use custom trained models on AWS infrastructure. → [Setup Guide](docs/SAGEMAKER-INTEGRATION.md)
- **Human-in-the-loop workflows** – Pause generation for user approval/input before tool execution. → [HITL Guide](docs/features/hitl.md)
Expand Down Expand Up @@ -266,9 +267,11 @@ const result = await neurolink.generate({
text: "Create a comprehensive analysis",
files: [
"./sales_data.csv", // Auto-detected as CSV
"examples/data/invoice.pdf", // Auto-detected as PDF
"./diagrams/architecture.png", // Auto-detected as image
],
},
provider: "vertex", // PDF-capable provider (see docs/features/pdf-support.md)
enableEvaluation: true,
region: "us-east-1",
});
Expand All @@ -281,15 +284,15 @@ Full command and API breakdown lives in [`docs/cli/commands.md`](docs/cli/comman

## Platform Capabilities at a Glance

| Capability | Highlights |
| ------------------------ | -------------------------------------------------------------------------------------------------------- |
| **Provider unification** | 12+ providers with automatic fallback, cost-aware routing, provider orchestration (Q3). |
| **Multimodal pipeline** | Stream images + CSV data across providers with local/remote assets. Auto-detection for mixed file types. |
| **Quality & governance** | Auto-evaluation engine (Q3), guardrails middleware (Q4), HITL workflows (Q4), audit logging. |
| **Memory & context** | Conversation memory, Mem0 integration, Redis history export (Q4), context summarization (Q4). |
| **CLI tooling** | Loop sessions (Q3), setup wizard, config validation, Redis auto-detect, JSON output. |
| **Enterprise ops** | Proxy support, regional routing (Q3), telemetry hooks, configuration management. |
| **Tool ecosystem** | MCP auto discovery, LiteLLM hub access, SageMaker custom deployment, web search. |
| Capability | Highlights |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------ |
| **Provider unification** | 12+ providers with automatic fallback, cost-aware routing, provider orchestration (Q3). |
| **Multimodal pipeline** | Stream images + CSV data + PDF documents across providers with local/remote assets. Auto-detection for mixed file types. |
| **Quality & governance** | Auto-evaluation engine (Q3), guardrails middleware (Q4), HITL workflows (Q4), audit logging. |
| **Memory & context** | Conversation memory, Mem0 integration, Redis history export (Q4), context summarization (Q4). |
| **CLI tooling** | Loop sessions (Q3), setup wizard, config validation, Redis auto-detect, JSON output. |
| **Enterprise ops** | Proxy support, regional routing (Q3), telemetry hooks, configuration management. |
| **Tool ecosystem** | MCP auto discovery, LiteLLM hub access, SageMaker custom deployment, web search. |

## Documentation Map

Expand Down
19 changes: 10 additions & 9 deletions docs/features/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ Comprehensive guides for all NeuroLink features organized by category. Each guid
| ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------- |
| :material-image-text: **[Multimodal Chat Experiences](multimodal-chat.md)** | Stream text and images together with automatic provider fallbacks and format conversion. |
| :material-table-large: **[CSV File Support](csv-support.md)** | Process CSV files for data analysis with automatic format conversion. Works with all providers. |
| :material-file-pdf-box: **[PDF File Support](pdf-support.md)** | Process PDF documents for visual analysis and content extraction. Native provider support. |
| :material-chart-line: **[Auto Evaluation Engine](auto-evaluation.md)** | Automated quality scoring and metrics export for AI response validation using LLM-as-judge. |
| :material-console: **[CLI Loop Sessions](cli-loop-sessions.md)** | Persistent interactive mode with conversation memory and session state for prompt engineering. |
| :material-earth: **[Regional Streaming Controls](regional-streaming.md)** | Region-specific model deployment and routing for compliance and latency optimization. |
Expand All @@ -38,15 +39,15 @@ Comprehensive guides for all NeuroLink features organized by category. Each guid

## Platform Capabilities at a Glance

| Category | Features | Documentation |
| ------------------------ | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| **Provider unification** | 12+ providers with automatic failover, cost-aware routing, provider orchestration (Q3) | [Provider Setup](../getting-started/provider-setup.md) |
| **Multimodal pipeline** | Stream images + CSV data across providers with local/remote assets. Auto-detection for mixed file types. | [Multimodal Guide](multimodal-chat.md), [CSV Support](csv-support.md) |
| **Quality & governance** | Auto-evaluation engine (Q3), guardrails middleware (Q4), HITL workflows (Q4), audit logging | [Auto Evaluation](auto-evaluation.md), [Guardrails](guardrails.md), [HITL](hitl.md) |
| **Memory & context** | Conversation memory, Mem0 integration, Redis history export (Q4), context summarization (Q4) | [Conversation Memory](../CONVERSATION-MEMORY.md), [Redis Export](conversation-history.md) |
| **CLI tooling** | Loop sessions (Q3), setup wizard, config validation, Redis auto-detect, JSON output | [CLI Loop](cli-loop-sessions.md), [CLI Commands](../cli/commands.md) |
| **Enterprise ops** | Proxy support, regional routing (Q3), telemetry hooks, configuration management | [Enterprise Proxy](../ENTERPRISE-PROXY-SETUP.md), [Telemetry](../TELEMETRY-GUIDE.md) |
| **Tool ecosystem** | MCP auto discovery, LiteLLM hub access, SageMaker custom deployment, web search | [MCP Integration](../advanced/mcp-integration.md), [MCP Catalog](../guides/mcp/server-catalog.md) |
| Category | Features | Documentation |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- |
| **Provider unification** | 12+ providers with automatic failover, cost-aware routing, provider orchestration (Q3) | [Provider Setup](../getting-started/provider-setup.md) |
| **Multimodal pipeline** | Stream images + CSV data + PDF documents across providers with local/remote assets. Auto-detection for mixed file types. | [Multimodal Guide](multimodal-chat.md), [CSV Support](csv-support.md), [PDF Support](pdf-support.md) |
| **Quality & governance** | Auto-evaluation engine (Q3), guardrails middleware (Q4), HITL workflows (Q4), audit logging | [Auto Evaluation](auto-evaluation.md), [Guardrails](guardrails.md), [HITL](hitl.md) |
| **Memory & context** | Conversation memory, Mem0 integration, Redis history export (Q4), context summarization (Q4) | [Conversation Memory](../CONVERSATION-MEMORY.md), [Redis Export](conversation-history.md) |
| **CLI tooling** | Loop sessions (Q3), setup wizard, config validation, Redis auto-detect, JSON output | [CLI Loop](cli-loop-sessions.md), [CLI Commands](../cli/commands.md) |
| **Enterprise ops** | Proxy support, regional routing (Q3), telemetry hooks, configuration management | [Enterprise Proxy](../ENTERPRISE-PROXY-SETUP.md), [Telemetry](../TELEMETRY-GUIDE.md) |
| **Tool ecosystem** | MCP auto discovery, LiteLLM hub access, SageMaker custom deployment, web search | [MCP Integration](../advanced/mcp-integration.md), [MCP Catalog](../guides/mcp/server-catalog.md) |

---

Expand Down
87 changes: 87 additions & 0 deletions docs/features/multimodal-chat.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,93 @@ await neurolink.generate({
- Combine CSV with visualization images for comprehensive analysis
- Works with ALL providers (not just vision-capable models)

## PDF File Support

### Quick Start

```bash
# Auto-detect PDF files
npx @juspay/neurolink generate "Summarize this report" \
--file ./financial-report.pdf \
--provider vertex

# Explicit PDF processing
npx @juspay/neurolink generate "Extract key terms" \
--pdf ./contract.pdf \
--provider anthropic

# Multiple PDFs
npx @juspay/neurolink generate "Compare these documents" \
--pdf ./version1.pdf \
--pdf ./version2.pdf \
--provider vertex
```

### SDK Usage

```typescript
// Auto-detect (recommended)
await neurolink.generate({
input: {
text: "Analyze this document",
files: ["./report.pdf", "./data.csv"],
},
provider: "vertex",
});

// Explicit PDF
await neurolink.generate({
input: {
text: "Compare Q1 and Q2 reports",
pdfFiles: ["./q1-report.pdf", "./q2-report.pdf"],
},
provider: "anthropic",
});

// Streaming with PDF
const stream = await neurolink.stream({
input: {
text: "Summarize this contract",
pdfFiles: ["./contract.pdf"],
},
provider: "vertex",
});
```

### Supported Providers

| Provider | Max Size | Max Pages | Notes |
| --------------------- | -------- | --------- | ------------------------------- |
| **Google Vertex AI** | 5 MB | 100 | `gemini-1.5-pro` recommended |
| **Anthropic** | 5 MB | 100 | `claude-3-5-sonnet` recommended |
| **AWS Bedrock** | 5 MB | 100 | Requires AWS credentials |
| **Google AI Studio** | 2000 MB | 100 | Best for large files |
| **OpenAI** | 10 MB | 100 | `gpt-4o`, `gpt-4o-mini`, `o1` |
| **Azure OpenAI** | 10 MB | 100 | Uses OpenAI Files API |
| **LiteLLM** | 10 MB | 100 | Depends on upstream model |
| **OpenAI Compatible** | 10 MB | 100 | Depends on upstream model |
| **Mistral** | 10 MB | 100 | Native PDF support |
| **Hugging Face** | 10 MB | 100 | Native PDF support |

**Not supported:** Ollama

### Best Practices

- **Choose the right provider**: Use Vertex AI or Anthropic for best results
- **Check file size**: Most providers limit to 5MB, AI Studio supports up to 2GB
- **Use streaming**: For large documents, streaming gives faster initial results
- **Combine with other files**: Mix PDF with CSV data and images for comprehensive analysis
- **Be specific in prompts**: "Extract all monetary values" vs "Tell me about this PDF"

### Token Usage

PDFs consume significant tokens:

- **Text-only mode**: ~1,000 tokens per 3 pages
- **Visual mode**: ~7,000 tokens per 3 pages

Set appropriate `maxTokens` for PDF analysis (recommended: 2000-8000 tokens).

## Troubleshooting

| Symptom | Action |
Expand Down
Loading
Loading