fix: Firecrawl web_fetch 500 with include_metadata=true

[ponkcore]

Problem

Every web_fetch call routed through the Firecrawl provider fails with HTTP 500 (or 502) iff include_metadata=true is set. The crash is URL-agnostic — it reproduces on trivial pages like https://example.com as well as on complex JS-heavy pages. Without include_metadata the same URLs succeed. Other providers (Tavily) handle include_metadata=true correctly.

Steps to reproduce

1. Call omniroute_web_fetch with any URL, format=markdown, include_metadata=true, provider="firecrawl".
2. Observe 500 error:

500: {"success":false,"code":"UNKNOWN_ERROR",
  "error":"An unexpected error occurred...",
  "exception ID":"019ef0c0-886b-70ea-8b41-500da228f385"}

Test matrix

URL	format	include_metadata	provider	Result
example.com	markdown	true	firecrawl	500
example.com	html	true	firecrawl	502
github.com/...	markdown	true	firecrawl	500
news.ycombinator.com	markdown	true	firecrawl	500
example.com	markdown	false	firecrawl	200 OK
example.com	html	false	firecrawl	200 OK
example.com	markdown	true	tavily-search	200 OK

Root Cause

In open-sse/executors/firecrawl-fetch.ts, when includeMetadata is true, the code sends:

requestBody.includeTags = ["title", "description", "og:title", "og:description"];

Firecrawl's includeTags parameter is a content filter — it expects HTML tags or CSS selectors (e.g. "title", "meta", ".content"). The values "description", "og:title", and "og:description" are not valid CSS selectors and cause Firecrawl's internal parser to crash with HTTP 500.

Critically, this parameter is unnecessary — Firecrawl returns metadata (title, description, OpenGraph tags, etc.) automatically in every scrape response via response.data.metadata. The response parsing code in this same file already reads it correctly:

const rawMeta = scraped.metadata as Record<string, unknown> | null | undefined;
const metadata = includeMetadata
  ? {
      title: rawMeta?.title != null ? String(rawMeta.title) : null,
      description: rawMeta?.description != null ? String(rawMeta.description) : null,
    }
  : null;

The includeMetadata flag should only control whether the metadata object is surfaced in OmniRoute's response — not send invalid parameters to Firecrawl.

Fix

Remove the includeTags assignment block entirely. Metadata is already extracted from response.data.metadata in the response parsing code.

Verification

• include_metadata=true + firecrawl → 500 (before fix, confirmed across 10+ calls)
• include_metadata=false + firecrawl → 200 OK (unaffected)
• include_metadata=true + tavily → 200 OK (unaffected, different API endpoint)
• Firecrawl API docs confirm: includeTags = "Tags to include in the output" (content filter), metadata = automatic response field

Environment

OmniRoute v3.8.33, consistent across 10+ calls from same session.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!