Skip to content

feat(search): add Firecrawl search provider - #8814

Merged
diegosouzapw merged 3 commits into
cheaper-inference:release/v3.8.49from
allanvb:feat/firecrawl-search-provider
Jul 28, 2026
Merged

diegosouzapw merged 3 commits into
cheaper-inference:release/v3.8.49from
allanvb:feat/firecrawl-search-provider

Conversation

@allanvb

@allanvb allanvb commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds Firecrawl as a /v1/search provider under the same provider id as web fetch: firecrawl.

One connection, one API key, two surfaces:

  • POST /v1/web/fetch -> existing Firecrawl scrape/extract path
  • POST /v1/search -> Firecrawl POST /v2/search with sources: ["web"] | ["news"]

Why

Firecrawl is commonly used for web search and extraction, but search support for Firecrawl was never implemented. Now this is possible by reusing already existing connections.

Implementation

  • Registered firecrawl in the open-sse search registry (POST https://api.firecrawl.dev/v2/search, Bearer auth, web + news).
  • Added open-sse/handlers/search/firecrawlSearch.ts for request build + v2 response normalization (data.web / data.news only; legacy v1 envelopes ignored).
  • Wired build/normalize into open-sse/handlers/search.ts.
  • Accepted firecrawl in v1SearchSchema for POST /v1/search.
  • Dashboard: single Search card with serviceKinds: ["webSearch", "webFetch"].
  • Removed the specialty apikey duplicate.
  • Honors FIRECRAWL_BASE_URL self-host root the same way fetch does.

Validation

  • Focused tests: node --import tsx/esm --test tests/unit/firecrawl-search.test.ts tests/unit/search-route.test.ts tests/unit/search-registry.test.ts tests/unit/providers-page-utils.test.ts tests/integration/search-providers-catalog.test.ts — 101 passed
  • Typecheck: npm run typecheck:core
  • npm run lint
  • New automated tests in this PR

Coverage Notes

Change is mostly in open-sse search registry/handler plus small dashboard catalog/schema wiring under src/.

The focused tests cover registry shape, schema accept/reject, handleSearch request body (sources web|news), v2 bucket normalization, self-host base URL, route credential resolve on firecrawl, catalog search+fetch presence, and dashboard category resolve. Full coverage reporting remains with CI.

Reviewer Notes

No migration and no feature flag.

Intentional product choice: unified id firecrawl, not firecrawl-search. True dual runtime id; not a search-only split with credential fallback.

Fetch pipeline is unchanged in behavior. Search reuses the same connection id and API key.

stryker.conf.json one-line add is unrelated to Firecrawl; it lists tests/unit/isLocalStreamLifecycleError-abort-shape.test.ts (covers circuitBreaker.ts, from #8699) so the mutation tap drift gate does not fail this PR.

Manual check: add Firecrawl API key under id firecrawl -> POST /v1/search with provider: "firecrawl" -> same key works for POST /v1/web/fetch -> dashboard shows one Firecrawl card (search + fetch).

Introduce the Firecrawl search provider to handle `POST /v1/search` requests via Firecrawl /v2/search API (sources: web|news)
@allanvb
allanvb requested a review from diegosouzapw as a code owner July 27, 2026 23:24
@allanvb
allanvb force-pushed the feat/firecrawl-search-provider branch from ab9d1fe to 054a8f8 Compare July 27, 2026 23:57
@diegosouzapw
diegosouzapw merged commit 6a59c09 into cheaper-inference:release/v3.8.49 Jul 28, 2026
5 checks passed
@diegosouzapw diegosouzapw mentioned this pull request Jul 28, 2026
HouMinXi pushed a commit to HouMinXi/OmniRoute that referenced this pull request Aug 2, 2026
* feat(search): add Firecrawl search provider

Introduce the Firecrawl search provider to handle `POST /v1/search` requests via Firecrawl /v2/search API (sources: web|news)

* chore(changelog): add changelog on firecrawl search provider support

* fix(search): satisfy file-size freeze and APIKEY count

---------

Co-authored-by: allanvb <allanvb@users.noreply.github.com>
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
* feat(search): add Firecrawl search provider

Introduce the Firecrawl search provider to handle `POST /v1/search` requests via Firecrawl /v2/search API (sources: web|news)

* chore(changelog): add changelog on firecrawl search provider support

* fix(search): satisfy file-size freeze and APIKEY count

---------

Co-authored-by: allanvb <allanvb@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants