fix: harden Rust MediaFetcher agains redirect bypass and DNS rebinding

[zhongdaor-nv]

Overview:

As title

Similar fix in rust side as #8282

Details:

• RFC-defined blocked IP ranges (RFC 1918 / 5735 / 6598 / 6890 / 4193 / 4291)
• Cloud-metadata / k8s hostname blocklist (metadata.google.internal, kubernetes.default.svc, localhost, …)
• Case-insensitive hostname matching; trailing-dot FQDN normalized
• New allow_private_ips escape hatch for on-prem / local-dev
• Async check_if_url_allowed_with_dns resolves hostnames and rejects
  any that resolve to blocked IPs
• Custom BlocklistResolver attached to the shared reqwest::Client via ClientBuilder::dns_resolver — filters blocked IPs out of DNS answers before reqwest ever sees them, closing the DNS-rebinding gap that the Python side left open (because httpx has no clean equivalent of reqwest's dns_resolver hook)
• redirect::Policy::custom revalidates the URL-level policy on every redirect hop, capped

Where should the reviewer start?

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes GitHub issue: #xxx

Summary by CodeRabbit

Release Notes

• New Features

  • Added configurable option to allow private IP addresses when fetching media content.

• Security

  • Enhanced media URL validation to prevent SSRF attacks by blocking private IP ranges during media fetching.
  • Improved DNS resolution handling with security-focused filtering for media requests.

[coderabbitai]

Walkthrough

The changes add SSRF (Server-Side Request Forgery) protection for media URL fetching by introducing IP/hostname blocklisting, DNS filtering via a custom resolver, and an allow_private_ips configuration flag. The ipnet crate dependency is added to the workspace and lib crate, with corresponding implementation in the media loader module and Python bindings.

Changes

Cohort / File(s)	Summary
Workspace and Crate Dependencies Cargo.toml, lib/llm/Cargo.toml	Add ipnet crate (version 2) as a workspace-managed dependency for IP address/CIDR range handling.
Core SSRF Hardening Implementation lib/llm/src/preprocessor/media/loader.rs	Introduce IP/hostname blocklisting for private/internal ranges, custom reqwest DNS resolver that filters blocked IPs, configurable allow_private_ips flag on MediaFetcher, synchronous and async URL validation methods with DNS resolution, custom redirect policy enforcement, and comprehensive test cases for blocked ranges and hostname variants.
Python Bindings lib/bindings/python/rust/llm/preprocessor.rs	Add allow_private_ips(&mut self, allow: bool) method to the MediaFetcher Python binding to expose the SSRF hardening configuration.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~35 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 65.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly describes the main security hardening change: protecting MediaFetcher against redirect bypass and DNS rebinding attacks.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description covers the Overview, Details, and Where to start sections with sufficient information about SSRF hardening changes, blocked IP ranges, custom resolver, and redirect policy implementation.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@lib/llm/src/preprocessor/media/loader.rs`:
- Around line 135-152: The flag allow_private_ips currently disables both the
RFC-range IP check and the hostname blocklist (BLOCKED_HOSTS) in the loader URL
guard, which is surprising and dangerous; either (A) rename the flag to
something explicit like allow_internal_targets/allow_private_targets and update
its field doc-comment and any Python binding to reflect that it disables both
host and IP checks, or (B) split into two flags (allow_private_ips and
allow_blocked_hosts) and change the guard to check each independently using
is_blocked_ip and BLOCKED_HOSTS; at minimum add a /// doc comment on the
allow_private_ips field explaining it disables both blocklists and update any
bindings/docs to match.
- Line 23: The current hardcoded MAX_REDIRECTS constant in loader.rs is set to 3
which is lower than reqwest's default and can break valid multi-hop flows;
update the constant (MAX_REDIRECTS) to a higher value (e.g., 10 to match
reqwest's default) and ensure any code that relies on MAX_REDIRECTS (redirect
handling logic and calls to check_if_url_allowed or BlocklistResolver) continues
to re-validate each hop so security posture is unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: eb834529-7bad-4b46-ae95-aa6bfea7dfe8

📥 Commits

Reviewing files that changed from the base of the PR and between 6ba534a and 4853a0f.

⛔ Files ignored due to path filters (3)

• Cargo.lock is excluded by !**/*.lock
• lib/bindings/kvbm/Cargo.lock is excluded by !**/*.lock
• lib/bindings/python/Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (4)

• Cargo.toml
• lib/bindings/python/rust/llm/preprocessor.rs
• lib/llm/Cargo.toml
• lib/llm/src/preprocessor/media/loader.rs