feat: Advanced Web Crawling with Domain Configuration

[leex279]

📋 Summary

This PR implements Advanced Web Crawling with comprehensive domain filtering and pattern matching configuration. Users can now precisely control which domains and URLs are crawled with advanced filtering rules.

Closes #546

✨ Features

Advanced Domain Configuration UI

• Collapsible Configuration Panel - Clean toggle interface for advanced options
• Domain Allowlist - Specify exactly which domains to crawl
• Domain Blocklist - Exclude specific domains from crawling scope
• URL Pattern Matching - Include/exclude patterns with regex support
• Real-time Validation - Validate domains and patterns before crawling
• Dynamic Management - Add/remove rules with visual feedback

Enhanced Crawling Engine

• CrawlConfig Integration - Structured filtering configuration
• Server-side Filtering - Backend application of domain rules
• Pattern Matching - Support for complex URL filtering logic
• Performance Optimization - Skip filtered content during discovery
• Progress Integration - Status updates for filtering operations

API Enhancements

• Enhanced /api/knowledge-items/crawl-v2 - Accepts crawl_config parameter
• CrawlConfig Interface - Type-safe domain filtering configuration
• Validation Layer - Backend validation of filtering rules
• Backwards Compatibility - Falls back to regular crawl endpoint when no config

🔧 Technical Implementation

Frontend Components

• Advanced Configuration Panel - Collapsible domain filtering UI
• Domain Input Management - Add/remove domains with validation
• Pattern Configuration - Include/exclude pattern inputs with regex support
• Configuration State - Persistent filtering rules during crawl session

Backend Integration

• CrawlConfig Processing - Parse and validate domain filtering rules
• Crawling Service Enhancement - Apply filters during URL discovery
• Performance Optimization - Early filtering reduces crawling overhead
• Progress Reporting - Status updates include filtering information

🎯 Advanced Crawling Flow

1. URL Input - User enters website URL to crawl
2. Configuration - Configure domain filters and patterns in advanced panel
3. Validation - Backend validates URL and filtering rules
4. Enhanced Crawling - Apply filters during URL discovery process
5. Filtered Results - Only matching content gets processed and stored
6. Progress Tracking - Real-time updates with filtering status

🔍 Use Cases

Multi-domain Documentation Sites

• Crawl docs.example.com but exclude community.example.com
• Include API reference pages but exclude marketing content
• Focus on specific documentation sections

Large Website Optimization

• Exclude known irrelevant domains (ads, tracking, CDNs)
• Target specific subdirectories with pattern matching
• Reduce crawling time and improve content quality

Pattern-based Filtering

• Include only /docs/ and /api/ paths
• Exclude /blog/ and /news/ sections
• Target specific file types or page structures

🧪 Testing Features

• ✅ Domain filtering UI works correctly
• ✅ Pattern matching validates input
• ✅ Backend applies filters during crawling
• ✅ Configuration persists through crawl session
• ✅ Enhanced crawl endpoint accepts crawl_config
• ✅ Backwards compatibility with regular crawling
• ✅ Performance improvement with filtering enabled
• ✅ TypeScript compilation passes

---

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing Touches

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feature/advanced-crawling-domain-filtering

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore or @coderabbit ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.