Fix/csv metadata detection

[y-naaz]

Pull Request

Description

What does this PR do?

Enhances CSV metadata detection with comprehensive data type detection, column statistics, and data quality analysis. Previously, CSV processing only returned basic metadata (columnCount, rowCount, hasHeader). Now returns rich per-column analysis including detected data types, null counts, value ranges, unique counts, date formats, and data quality warnings.

Related Issues

Fixes #365

Type of Change

Please select the type of change:

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)
• Performance improvement
• Test coverage improvement
• Build/CI configuration
• Other (please describe):

Motivation and Context

Why is this change needed? What problem does it solve?

• Problem: Lines 127-130 in csvProcessor.ts only returned basic metadata (columnCount, rowCount, hasHeader). No data type detection, no null/empty detection, no value ranges, no duplicate detection, no column name validation.
• Impact: Users couldn't analyze CSV data quality, detect data types for schema inference, or identify problematic columns before processing.
• Solution: Enhanced metadata with comprehensive column analysis including type detection, statistics, and quality warnings.

Changes Made

What specific changes were made?

• Added CSVColumnDataType type with 11 data types: string, number, integer, float, boolean, date, datetime, email, url, empty, mixed
• Added CSVDataQualityWarning type for structured warnings with severity levels
• Added CSVColumnMetadata type for rich per-column metadata
• Implemented data type detection patterns for integers, floats, booleans, dates, emails, URLs
• Added date format detection (ISO8601, MM/DD/YYYY, DD-MM-YYYY, DD.MM.YYYY, YYYY/MM/DD)
• Added column name validation (empty names, special characters, leading numbers, excessive length)
• Added per-column statistics: nullCount, uniqueCount, minValue, maxValue, avgValue, sampleValues
• Added data quality warnings: high_null_rate, invalid_name, mixed_types, duplicates, empty_values
• Added overall data quality score calculation (0-100)
• Extended FileProcessingResult.metadata with new CSV-specific fields
• Enhanced both raw and structured format outputs to include new metadata

Breaking Changes

Does this PR introduce breaking changes?

• No breaking changes
• Yes, breaking changes (describe below)

New metadata fields are additive. Existing code consuming CSV metadata will continue to work unchanged.

Testing

How has this been tested?

Please describe the tests you ran and their results:

• Unit tests added/updated
• Integration tests added/updated
• E2E tests pass
• Manual testing completed
• Tested with multiple providers: [list providers]
• Tested on multiple platforms: [list platforms]

Test Coverage

• All new code is covered by tests
• Existing tests pass
• Coverage percentage maintained or improved

Manual Testing Steps

Provide steps for manual testing:

1. Process a CSV file with mixed data types
2. Verify columnMetadata array is populated with correct type detection
3. Process a CSV with empty/null values and verify nullCount and dataQualityWarnings
4. Process a CSV with date columns and verify dateFormat detection
5. Check dataQualityScore reflects the overall data quality

Code Quality

Have you followed code quality standards?

• Code follows the project's style guidelines (ESLint passes)
• Code is properly formatted (Prettier applied)
• Self-review of code completed
• No console.log statements (using logger instead)
• No hardcoded API keys or secrets
• TypeScript strict mode compliance
• Proper error handling implemented
• TODO/FIXME comments reference issues

Documentation

Have you updated documentation?

• JSDoc comments added/updated for public APIs
• README.md updated (if needed)
• Documentation in /docs updated (if needed)
• Code examples added/updated (if needed)
• CHANGELOG.md updated (if applicable)
• Migration guide provided (if breaking changes)

Commit Message Format

Does your commit follow semantic commit conventions?

• Commit message follows format: type(scope): description
• Valid type used: feat, fix, docs, style, refactor, test, chore, build, ci, perf, revert
• Scope specified (e.g., providers, cli, docs, middleware)

Example: fix(csv-processor): enhance metadata detection with data types and quality analysis

Dependencies

Does this PR add, update, or remove dependencies?

• No dependency changes
• Dependencies added (list below)
• Dependencies updated (list below)
• Dependencies removed (list below)

Performance Impact

Does this change affect performance?

• No performance impact
• Performance improved (provide metrics)
• Performance degraded (justify why acceptable)

Minor performance impact due to additional column analysis. Mitigated by:

• Analysis limited to first 500 rows for raw format
• Type detection uses efficient regex patterns
• Statistics calculated in single pass through data

Security Considerations

Are there any security implications?

• No security implications
• Security review needed
• Security vulnerability fixed

Deployment Notes

Special deployment instructions?

• No special deployment steps
• Requires environment variable changes (list below)
• Requires database migration
• Requires Redis schema update
• Other (describe below)

Screenshots / Videos

N/A - No UI changes

Reviewer Checklist

For reviewers:

• Code follows project style and conventions
• Changes are well-documented
• Tests provide adequate coverage
• No obvious performance issues
• No security vulnerabilities introduced
• Breaking changes are properly documented
• Documentation is clear and accurate

Additional Notes

Any additional information for reviewers:
Part of "Phase 2: High Priority" milestone.
Labels: type:bug, priority:high, component:csv-processor, modality:csv
Estimated effort: 4h

---

Pre-submission Checklist

Before submitting, ensure you have:

• Read and followed the Contributing Guidelines
• Verified all automated pre-commit checks pass
• Tested changes locally with pnpm test
• Built the project successfully with pnpm build
• Run pnpm run validate:all and all checks pass
• Reviewed your own code for obvious issues
• Ensured commit messages follow semantic format
• Updated relevant documentation
• Added tests for new functionality
• Checked that CI/CD pipeline passes (after creating PR)

Thank you for contributing to NeuroLink!
New metadata structure returned:

{
  columnMetadata: [
    {
      name: "age",
      index: 0,
      detectedType: "integer",
      typeConfidence: 95,
      nullCount: 2,
      uniqueCount: 45,
      sampleValues: ["25", "30", "42"],
      minValue: 18,
      maxValue: 65,
      avgValue: 34.5
    }
  ],
  dataQualityWarnings: [
    {
      column: "email",
      type: "high_null_rate",
      message: "Column has 35% empty/null values",
      severity: "warning",
      affectedRows: 35
    }
  ],
  dataQualityScore: 82,
  hasHeaders: true,
  detectedDelimiter: ","
}




<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

* **New Features**
  * CSV import now provides per-column analysis with detected data types, sample values, null/unique counts, min/max/avg where applicable, and column name issues.
  * Adds data quality warnings and an overall data quality score to help spot problematic columns.
  * Automatic header and delimiter detection improved and surfaced in CSV metadata.

* **Tests**
  * Added extensive tests covering column profiling, quality warnings, scoring, and header detection.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

• 🔍 Trigger a full review

Walkthrough

Adds CSV profiling: per-column metadata, data-quality warnings, a quality score, header/delimiter detection, and related types; csvProcessor computes and attaches these fields into FileProcessingResult.metadata and tests validate the behavior.

Changes

Cohort / File(s)	Summary
Type Definitions src/lib/types/fileTypes.ts	Added CSVColumnDataType, CSVColumnMetadata, CSVDataQualityWarning types; extended FileProcessingResult.metadata with columnMetadata, dataQualityWarnings, dataQualityScore, hasHeaders, and detectedDelimiter.
CSV Processing Implementation src/lib/utils/csvProcessor.ts	Added type-detection utilities, column analysis pipeline (analyzeColumn, determineColumnType, generateDataQualityWarnings, calculateDataQualityScore, analyzeColumns, detectHasHeaders) and integrated results (column metadata, warnings, score, hasHeaders, detectedDelimiter) into processor outputs.
Unit Tests test/unit/utils/csvProcessor.test.ts	Added extensive tests for column type detection, stats (min/max/avg/null/unique), sample values, date formats, name issues, data-quality warnings, quality scoring, header detection, and metadata inclusion across CSV processing paths.

Sequence Diagram(s)

(omitted)

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• feat(csv): Add file extension field to CSV metadata #642: Modifies CSV metadata types and csvProcessor outputs; overlaps with adding profiling fields to FileProcessingResult.metadata.
• feat(csv): add sampleDataFormat option for CSV metadata #616: Related changes to csvProcessor and sampleData/metadata propagation; likely touches similar CSV output paths.
• OFFICE-006: Add TypeScript type definitions for office document formats #676: Updates fileTypes.ts and expands format-specific metadata; intersects with added type declarations here.

Suggested reviewers

• murdore
• itz-PrathamMittal
• coder-dodo

Poem

🐰 I hopped through columns, sniffed each name,
Found dates, numbers, and a mixed-type game.
Warnings I nibbled, scores I spun,
Metadata bundled — now the profiling's done! ✨

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Fix/csv metadata detection' directly corresponds to the main change: comprehensive CSV metadata detection with per-column analysis, data quality warnings, and quality scoring.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

This PR enhances CSV processing by adding rich per-column metadata and data quality analysis, and also introduces configurable enforcement of PDF page limits.

Changes:

• Add CSV data-type detection, column statistics, and data quality scoring/warnings, and include them in FileProcessingResult.metadata for both raw and structured CSV formats.
• Extend shared file type definitions with CSV column metadata types and PDF options (including enforceLimits).
• Update PDFProcessor to throw on page-limit violations by default, with an option to bypass limits while logging a warning.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
src/lib/utils/csvProcessor.ts	Implements column-level type detection, statistics, and quality scoring, and wires these into CSV processing outputs and logging.
src/lib/types/fileTypes.ts	Adds CSV metadata and warning types, augments FileProcessingResult metadata, and extends PDFProcessorOptions with the enforceLimits flag.
src/lib/utils/pdfProcessor.ts	Changes page-limit handling to an error-by-default behavior with an opt-out via enforceLimits, and improves the error message with actionable guidance.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[y-naaz]

@coderabbitai trigger full review and check if changes are according to tagged issue or not

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Get PR details including linked issues
gh pr view 802 --repo juspay/neurolink --json number,title,body,closingIssuesReferences

# Also check for issue references in the PR body more explicitly
gh pr view 802 --repo juspay/neurolink --json body -q '.body'

Length of output: 17731

---

🏁 Script executed:

#!/bin/bash
# Search for issues related to CSV metadata detection
gh issue list --repo juspay/neurolink --search "CSV metadata" --state all --json number,title,body,state,labels --limit 20

# Also search for issues mentioning csvProcessor or weak metadata
gh issue list --repo juspay/neurolink --search "csvProcessor metadata weak" --state all --json number,title,body,state,labels --limit 10

Length of output: 34589

---

@y-naaz I've analyzed PR #802 against the related issue. Here's my assessment:

Issue Identification

The PR body contains a placeholder Fixes #(issue number for "CSV metadata detection is weak"), but I found the actual issue:

Issue #365: "CSV-005: Weak Metadata Detection"

• Status: Open
• Priority: High
• Labels: type:bug, priority:high, component:csv-processor, modality:csv
• Estimated effort: 4h

Alignment Analysis

The PR fully addresses all requirements from issue #365:

Issue Requirement	PR Implementation	Status
Data type detection per column	11 types: string, number, integer, float, boolean, date, datetime, email, url, empty, mixed	✅ Implemented
Null/empty counts	nullCount field in CSVColumnMetadata	✅ Implemented
Value ranges for numeric columns	minValue, maxValue, avgValue fields	✅ Implemented
Unique value counts	uniqueCount field	✅ Implemented
Date format detection	5 formats: ISO8601, MM/DD/YYYY, DD-MM-YYYY, DD.MM.YYYY, YYYY/MM/DD	✅ Implemented
Column name validation	Checks for empty names, special chars, leading numbers, excessive length	✅ Implemented
Sample values per column	sampleValues array	✅ Implemented
Data quality warnings	5 warning types with severity levels and affected row counts	✅ Implemented

Beyond Requirements

The PR includes additional enhancements not explicitly requested:

• Email and URL type detection
• Type confidence scoring (typeConfidence 0-100)
• Overall data quality score (dataQualityScore 0-100)
• Header detection (hasHeaders)
• Delimiter detection (detectedDelimiter)

Recommendation

Action needed: Update the PR description to properly link issue #365 by changing:

Fixes #(issue number for "CSV metadata detection is weak")

to:

Fixes `#365`

This will automatically close issue #365 when the PR is merged.

✅ Actions performed

Full review triggered.

[y-naaz]

@coderabbitai done , I have added the issue

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

@y-naaz Thank you for updating the PR description with the issue reference! The PR will now automatically close issue #365 when merged.

The full review is complete based on the previous analysis. Your implementation comprehensively addresses all requirements from issue #365 and includes valuable additional enhancements. The changes look good to proceed! 🎉

[y-naaz]

@murdore please review and merge

[github-actions]

🎉 This PR is included in version 9.1.1 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀