Implement robots.txt compliance for web crawler (Issue #275)#846

Closed

leex279 wants to merge 3 commits intomainfrom

feature/respect-robots-txt-275

Commits on Nov 7, 2025

Implement robots.txt compliance for web crawler

leex279
and
claude
committed
Add robots.txt validation to recursive crawling strategy

leex279
and
claude
committed

Commits on Nov 8, 2025

Add validation for crawler config environment variables

leex279
and
claude
committed