Skip to content

fix(web): parallelize firecrawl extract fanout - #47611

Open
LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:fix-47599-firecrawl-extract-parallel
Open

fix(web): parallelize firecrawl extract fanout#47611
LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:fix-47599-firecrawl-extract-parallel

Conversation

@LeonSGP43

Copy link
Copy Markdown
Contributor

Summary

  • parallelize Firecrawl multi-URL extract calls instead of scraping each URL serially
  • keep per-URL timeouts, policy checks, and result ordering intact
  • add a regression test proving two slow URLs no longer serialize

Closes #47599

Testing

  • python3 -m pytest tests/plugins/web/test_web_search_provider_plugins.py -q
  • uv run --frozen ruff check plugins/web/firecrawl/provider.py tests/plugins/web/test_web_search_provider_plugins.py
  • git diff --check

@alt-glitch alt-glitch added type/bug Something isn't working comp/plugins Plugin system and bundled plugins tool/web Web search and extraction P3 Low — cosmetic, nice to have labels Jun 17, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused parallelization work. The current-main premise is confirmed: plugins/web/firecrawl/provider.py:458-496 still serially awaits each Firecrawl scrape.

Problems

  • This branch predates current main's redirect-SSRF guard (2e12401ed; plugins/web/firecrawl/provider.py:529-547). That guard still uses results.append(...); continue, but this refactor removes results in favor of extract_one(). Preserve the guard by converting it to return {...} during salvage, or unsafe redirected URLs will hit an undefined results reference.
  • The new timing test patches the client and website policy but not is_safe_url. The provider calls it at plugins/web/firecrawl/provider.py:530, and tools/url_safety.py:384-388 performs DNS resolution. Patch it to return True so the test remains hermetic.

Suggested changes

  • Convert the current redirect-SSRF branch to the same per-task return shape as the other early exits.
  • Stub is_safe_url in the regression test.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data tool/web Web search and extraction type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: web_extract took minutes to finish

3 participants