fix(web): re-check Firecrawl final URLs for SSRF

[zapabob]

Summary

• Re-check Firecrawl metadata.sourceURL final URLs with is_safe_url() before returning scraped content.
• Preserve the existing website blocklist redirect check after the SSRF guard.
• Add regression coverage for a Firecrawl redirect/final URL pointing at the cloud metadata endpoint.

Security impact

web_extract already rejects unsafe input URLs before dispatching to a backend. Firecrawl can report a different final URL after scrape/redirect via metadata.sourceURL; this patch blocks that final target before any scraped content is returned to the agent.

Verification

• uv run --extra dev ruff check plugins/web/firecrawl/provider.py tests/tools/test_website_policy.py
• uv run --extra dev python -X utf8 -m pytest tests/tools/test_website_policy.py::TestWebToolPolicy::test_web_extract_blocks_firecrawl_unsafe_final_url tests/tools/test_website_policy.py::TestWebToolPolicy::test_web_extract_blocks_redirected_final_url -q --timeout-method=thread
• uv run --extra dev python -X utf8 -m pytest tests/tools/test_website_policy.py -q --timeout-method=thread
• uv run --extra dev ruff check .

[tonydwb]

Code Review Summary

Verdict: Approved ⚡

✅ What's Good

• Good SSRF defense-in-depth: Firecrawl can report a different final URL after scrape/redirect via metadata.sourceURL. This patch properly re-checks that final URL with is_safe_url() before returning scraped content — closing a real SSRF bypass vector.
• Clean integration: The new check sits naturally alongside the existing check_website_access redirect check. The continue after the block prevents any content leakage.
• Well-structured test: The test verifies that the unsafe final URL is blocked, that the website policy check is NOT called for unsafe final URLs, and that scraped content is wiped. The mock setup is clean and focused.
• No scope creep: 2 files changed, 64 additions, 0 deletions — tightly scoped to the security concern.
• Existing test fix: The test monkeypatch for is_safe_url was added to the redirected-final-url test to prevent false failures, showing good awareness of test interactions.

Summary

A tightly scoped, well-tested security fix. No issues found.

---

Reviewed by Hermes Agent

[egilewski]

Recommendation: rebase or resolve the current-main conflict before merge. The security fix itself validated in focused checks.

Checked current main 6f6eb871d83415fe2980f3483cc41a435ba22196 against PR head 85714c46cc513ca84364692c3ba9eb8e3fcf1f56. With a mocked Firecrawl scrape returning metadata.sourceURL = "http://169.254.169.254/latest/meta-data/", current main returned "metadata credentials" from that unsafe final URL. The patched provider blocks that final URL before returning content, with content == "" and Blocked: URL targets a private or internal network address.

Validation:

• python -m pytest -o addopts='' -p no:cacheprovider tests/tools/test_website_policy.py::TestWebToolPolicy::test_web_extract_blocks_firecrawl_unsafe_final_url tests/tools/test_website_policy.py::TestWebToolPolicy::test_web_extract_blocks_redirected_final_url -q --timeout-method=thread -> 2 passed
• python -m pytest -o addopts='' -p no:cacheprovider tests/tools/test_website_policy.py -q --timeout-method=thread -> 21 passed
• coderabbit review --plain --base 6f6eb871d83415fe2980f3483cc41a435ba22196 --type uncommitted -> No findings

GitHub currently reports mergeable=CONFLICTING / mergeStateStatus=DIRTY, so this should be refreshed against current main before merging.

Signed: GPT-5.5-xhigh in Codex

[teknium1]

Merged via PR #56143. Your commit was cherry-picked onto current main with your authorship preserved in git log (commit 2e12401). I added one small follow-up commit adapting the new test to the current web_extract_tool signature (main removed the use_llm_processing kwarg and moved the input SSRF gate to async_is_safe_url). Thanks for the security fix.