[parabala]

• Add Web scraper site-specific configuration for specific sites that require special handling, due to anti-bot detection, dynamic content loading, or navigation patterns

Summary by CodeRabbit

• New Features

  • Site-specific scraper configuration to tailor crawling behavior per domain.
  • Runtime hostname-based matching to apply per-site settings.

• Improvements

  • Better anti-bot handling via randomized user agents, managed browser usage, and safer Chromium launch options.
  • Cache-bypass option to force fresh fetches when needed.

• Chores

  • Added an environment-configurable setting to supply site configuration.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6a3f7ef8-21b9-4aa0-a0c7-14fbfb3c49ac

📥 Commits

Reviewing files that changed from the base of the PR and between c266499 and 4338224.

📒 Files selected for processing (1)

• backend/app/services/web_scraper/scraper_config.py

🚧 Files skipped from review as they are similar to previous changes (1)

• backend/app/services/web_scraper/scraper_config.py

---

📝 Walkthrough

Walkthrough

Adds a new WEB_SCRAPER_SITE_CONFIG setting and loader, exposes get_site_config(), and applies per-site runtime scrape settings plus stronger browser options when building crawl run configurations.

Changes

Cohort / File(s)	Summary
Environment / Settings backend/.env.example, backend/app/core/config.py	Added WEB_SCRAPER_SITE_CONFIG to example env and to Settings (defaults to "{}").
Site config loader backend/app/services/web_scraper/scraper_config.py	New module that reads/parses WEB_SCRAPER_SITE_CONFIG, normalizes keys (lowercase, sorted by length), handles JSON errors/warnings, and exposes cached get_site_config().
Scraper runtime backend/app/services/web_scraper/scraper_service.py	_build_run_config now accepts url, forces cache_mode=CacheMode.BYPASS, matches hostname against site config and merges per-site settings; crawler BrowserConfig updated with user_agent_mode="random", use_managed_browser=True, and Chromium args --no-sandbox, --disable-dev-shm-usage. Logging updated.

Sequence Diagram

sequenceDiagram
    participant Env as Environment (WEB_SCRAPER_SITE_CONFIG)
    participant Config as scraper_config.get_site_config()
    participant Scraper as Scraper Service
    participant Crawler as Crawl Run Config / Browser

    Env->>Config: provide JSON string
    Config->>Config: parse, normalize, cache site_config
    Scraper->>Config: get_site_config() for target URL
    Config-->>Scraper: return matching site settings (or {})
    Scraper->>Crawler: build run config (merge site settings, set cache_mode=BYPASS, set browser args)
    Crawler-->>Scraper: final run configuration

Loading

Estimated Code Review Effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐰 I hop through keys and JSON bright,

I parse and sort by day and night.
I nudge the browser, sly and spry—
The crawl begins beneath my sky.

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding configuration for specific sites in the web scraper that require special handling.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

backend/app/services/web_scraper/scraper_service.py (1)

358-364: ⚠️ Potential issue | 🔴 Critical

Duplicate dictionary key remove_overlay_elements causes conflicting behavior.

The key is defined twice: first as False (line 361) with a comment explaining it's "disabled to avoid navigation conflicts", then immediately overwritten to True (line 363). This contradicts the comment and the second assignment silently wins. Decide which value is intended and remove the duplicate.

🐛 Proposed fix (if True is intended)

         config_kwargs = {
             "wait_until": "domcontentloaded",  # Faster than networkidle, avoids timeout
             "page_timeout": WEB_SCRAPER_TIMEOUT,
-            "remove_overlay_elements": False,  # Disabled to avoid navigation conflicts
             "cache_mode": CacheMode.BYPASS,  # Force refresh, bypassing the local cache
             "remove_overlay_elements": True,  # Automatically remove mask layers
         }

🐛 Proposed fix (if False is intended)

         config_kwargs = {
             "wait_until": "domcontentloaded",  # Faster than networkidle, avoids timeout
             "page_timeout": WEB_SCRAPER_TIMEOUT,
             "remove_overlay_elements": False,  # Disabled to avoid navigation conflicts
             "cache_mode": CacheMode.BYPASS,  # Force refresh, bypassing the local cache
-            "remove_overlay_elements": True,  # Automatically remove mask layers
         }

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@backend/app/services/web_scraper/scraper_service.py` around lines 358 - 364,
The config_kwargs dict in scraper_service.py defines the duplicate key
remove_overlay_elements twice (first False, then True) causing the second value
to silently override the first; decide the intended behavior and remove the
unwanted entry so only one remove_overlay_elements is present in config_kwargs,
and update its comment to reflect the chosen behavior (e.g., keep
remove_overlay_elements: False with comment "disabled to avoid navigation
conflicts" or keep True with comment "automatically remove mask layers"); locate
the dict by the symbol config_kwargs and edit the duplicate
remove_overlay_elements entries accordingly.

🧹 Nitpick comments (2)

backend/app/services/web_scraper/scraper_config.py (1)

18-24: Add logging when JSON parsing fails and include return type hint.

Silent failure on invalid JSON makes debugging difficult. Per coding guidelines, type hints are required.

♻️ Proposed fix

+import logging
+
+logger = logging.getLogger(__name__)
+
-def _load_site_config():
+def _load_site_config() -> dict:
     """Load site configuration from environment variable."""
     config_str = os.environ.get("WEB_SCRAPER_SITE_CONFIG", "{}")
     try:
         return json.loads(config_str)
     except json.JSONDecodeError:
+        logger.warning(
+            "Invalid JSON in WEB_SCRAPER_SITE_CONFIG, using empty config"
+        )
         return {}

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@backend/app/services/web_scraper/scraper_config.py` around lines 18 - 24, The
_load_site_config function should declare its return type and log JSON parse
failures; change the signature to include a type hint (e.g., def
_load_site_config() -> Dict[str, Any]:), ensure you have typing imports (Dict,
Any), and in the except json.JSONDecodeError as e block call the module logger
(or logging.getLogger(__name__)) to log a clear error like "Failed to parse
WEB_SCRAPER_SITE_CONFIG" with the exception details before returning {} so
failures are visible in logs.

backend/app/services/web_scraper/scraper_service.py (1)

366-372: Substring domain matching may apply config to unintended URLs.

Using domain in url will match "example.com" in URLs like https://notexample.com/ or https://example.com.malicious.site/. Consider matching against the parsed hostname instead.

♻️ Proposed fix using hostname matching

+from urllib.parse import urlparse
+
         # Apply site-specific configuration if URL matches known sites
         if url:
+            parsed_url = urlparse(url)
+            hostname = parsed_url.netloc.lower()
             for domain, site_config in WEB_SCRAPER_SITE_CONFIG.items():
-                if domain in url:
+                domain_lower = domain.lower()
+                if hostname == domain_lower or hostname.endswith("." + domain_lower):
                     logger.info(f"Applying site-specific config for {domain}: {url}")
                     config_kwargs.update(site_config)
                     break

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@backend/app/services/web_scraper/scraper_service.py` around lines 366 - 372,
The current substring check ("domain in url") can misapply site configs; instead
parse the URL hostname (e.g., via urllib.parse.urlparse(url).hostname),
normalize to lowercase, strip any port, and then match the domain against the
hostname using either exact equality or subdomain matching (hostname == domain
or hostname.endswith("." + domain)); replace the "if domain in url" condition in
the loop that iterates WEB_SCRAPER_SITE_CONFIG and continues to call logger.info
and config_kwargs.update(site_config) when a match is found.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@backend/app/services/web_scraper/scraper_service.py`:
- Around line 358-364: The config_kwargs dict in scraper_service.py defines the
duplicate key remove_overlay_elements twice (first False, then True) causing the
second value to silently override the first; decide the intended behavior and
remove the unwanted entry so only one remove_overlay_elements is present in
config_kwargs, and update its comment to reflect the chosen behavior (e.g., keep
remove_overlay_elements: False with comment "disabled to avoid navigation
conflicts" or keep True with comment "automatically remove mask layers"); locate
the dict by the symbol config_kwargs and edit the duplicate
remove_overlay_elements entries accordingly.

---

Nitpick comments:
In `@backend/app/services/web_scraper/scraper_config.py`:
- Around line 18-24: The _load_site_config function should declare its return
type and log JSON parse failures; change the signature to include a type hint
(e.g., def _load_site_config() -> Dict[str, Any]:), ensure you have typing
imports (Dict, Any), and in the except json.JSONDecodeError as e block call the
module logger (or logging.getLogger(__name__)) to log a clear error like "Failed
to parse WEB_SCRAPER_SITE_CONFIG" with the exception details before returning {}
so failures are visible in logs.

In `@backend/app/services/web_scraper/scraper_service.py`:
- Around line 366-372: The current substring check ("domain in url") can
misapply site configs; instead parse the URL hostname (e.g., via
urllib.parse.urlparse(url).hostname), normalize to lowercase, strip any port,
and then match the domain against the hostname using either exact equality or
subdomain matching (hostname == domain or hostname.endswith("." + domain));
replace the "if domain in url" condition in the loop that iterates
WEB_SCRAPER_SITE_CONFIG and continues to call logger.info and
config_kwargs.update(site_config) when a match is found.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8a7aefcd-f649-40c9-9935-299a1867b18a

📥 Commits

Reviewing files that changed from the base of the PR and between 348974a and a91911e.

📒 Files selected for processing (3)

• backend/.env.example
• backend/app/services/web_scraper/scraper_config.py
• backend/app/services/web_scraper/scraper_service.py

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

backend/app/services/web_scraper/scraper_service.py (1)

358-364: ⚠️ Potential issue | 🟠 Major

Remove the duplicated remove_overlay_elements entry.

Line 361 never takes effect because Line 363 overwrites the same key. That means this block always enables overlay removal, while the preceding comment says the opposite, and Ruff will flag the duplicate literal.

🧹 Proposed fix

         config_kwargs = {
             "wait_until": "domcontentloaded",  # Faster than networkidle, avoids timeout
             "page_timeout": WEB_SCRAPER_TIMEOUT,
-            "remove_overlay_elements": False,  # Disabled to avoid navigation conflicts
             "cache_mode": CacheMode.BYPASS,  # Force refresh, bypassing the local cache
-            "remove_overlay_elements": True,  # Automatically remove mask layers
+            "remove_overlay_elements": True,  # Default; override per-site when needed
         }

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@backend/app/services/web_scraper/scraper_service.py` around lines 358 - 364,
Duplicate key in config_kwargs causes remove_overlay_elements to be overwritten;
open scraper_service.py and edit the config_kwargs dict (used where
WEB_SCRAPER_TIMEOUT and CacheMode are referenced) to remove the duplicated
"remove_overlay_elements" entry and ensure the remaining setting matches the
intended behavior described by the comment (either keep False and delete the
later True, or vice versa), then adjust the comment so it accurately reflects
the final value.

🧹 Nitpick comments (1)

backend/app/services/web_scraper/scraper_service.py (1)

335-337: Trace the new per-site config decision path.

This code now changes crawler behavior based on url, matched site config, and proxy, but none of those choices will be visible in telemetry when a scrape fails. Please trace the touched methods and set span attributes for the target host, matched config key, and proxy mode.

As per coding guidelines, "Use @trace_async(span_name, tracer_name, extract_attributes) decorator to trace entire async functions in OpenTelemetry", "Use @trace_sync(span_name, tracer_name, extract_attributes) decorator to trace entire sync functions in OpenTelemetry", and "Use set_span_attribute(key, value) to set attribute on current OpenTelemetry span".

Also applies to: 339-378

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@backend/app/services/web_scraper/scraper_service.py` around lines 335 - 337,
The scrape call pathway that builds run_config and calls crawler.arun should
record the target host, the matched per-site config key, and the proxy mode on
the active OpenTelemetry span: decorate the async function that contains the
shown block (the function that calls _build_run_config and await crawler.arun)
with `@trace_async`(span_name, tracer_name, extract_attributes) and/or add the
decorator to any sibling async functions in the 339-378 region; before calling
_build_run_config / crawler.arun use set_span_attribute("scrape.target_host",
url_host), set_span_attribute("scrape.matched_config_key",
matched_config_key_or_null) and set_span_attribute("scrape.proxy_mode",
proxy_url and "proxy" or "none") so those values appear in telemetry; ensure you
derive url_host and matched_config_key from the same logic that
_build_run_config uses so the span attributes match the actual runtime
decisions.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@backend/app/services/web_scraper/scraper_service.py`:
- Around line 366-372: The current loop checks domain substrings against the
full url which can produce false matches; instead parse the hostname from url
(e.g., via urllib.parse.urlparse(url).hostname), sort WEB_SCRAPER_SITE_CONFIG
keys by length descending to prefer longest matches, then perform exact equality
or suffix matching against that hostname, update config_kwargs with the matching
site_config, and change the logger.info to reference the hostname and chosen
domain (keep the existing break after a match). Ensure this uses the parsed
hostname variable wherever domain matching occurs and handles a None hostname
safely.

---

Outside diff comments:
In `@backend/app/services/web_scraper/scraper_service.py`:
- Around line 358-364: Duplicate key in config_kwargs causes
remove_overlay_elements to be overwritten; open scraper_service.py and edit the
config_kwargs dict (used where WEB_SCRAPER_TIMEOUT and CacheMode are referenced)
to remove the duplicated "remove_overlay_elements" entry and ensure the
remaining setting matches the intended behavior described by the comment (either
keep False and delete the later True, or vice versa), then adjust the comment so
it accurately reflects the final value.

---

Nitpick comments:
In `@backend/app/services/web_scraper/scraper_service.py`:
- Around line 335-337: The scrape call pathway that builds run_config and calls
crawler.arun should record the target host, the matched per-site config key, and
the proxy mode on the active OpenTelemetry span: decorate the async function
that contains the shown block (the function that calls _build_run_config and
await crawler.arun) with `@trace_async`(span_name, tracer_name,
extract_attributes) and/or add the decorator to any sibling async functions in
the 339-378 region; before calling _build_run_config / crawler.arun use
set_span_attribute("scrape.target_host", url_host),
set_span_attribute("scrape.matched_config_key", matched_config_key_or_null) and
set_span_attribute("scrape.proxy_mode", proxy_url and "proxy" or "none") so
those values appear in telemetry; ensure you derive url_host and
matched_config_key from the same logic that _build_run_config uses so the span
attributes match the actual runtime decisions.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 12fbb5a8-b6c8-48a6-b5e6-caf516c3aac1

📥 Commits

Reviewing files that changed from the base of the PR and between a91911e and e212a75.

📒 Files selected for processing (3)

• backend/.env.example
• backend/app/services/web_scraper/scraper_config.py
• backend/app/services/web_scraper/scraper_service.py

✅ Files skipped from review due to trivial changes (2)

• backend/.env.example
• backend/app/services/web_scraper/scraper_config.py

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

backend/app/services/web_scraper/scraper_config.py (1)

73-77: Narrow the exception handler to avoid catching unrelated exceptions.

Catching bare Exception can mask programming errors (e.g., TypeError, AttributeError) that should propagate. Since the try block primarily accesses settings and performs dict operations, consider catching only the expected exceptions.

Proposed fix

     except json.JSONDecodeError as e:
         logger.error(f"[WebScraperConfig] Failed to parse WEB_SCRAPER_SITE_CONFIG: {e}")
         return {}
-    except Exception as e:
-        logger.error(
-            f"[WebScraperConfig] Error loading WEB_SCRAPER_SITE_CONFIG from settings: {e}"
-        )
-        return {}
+    except (TypeError, KeyError, AttributeError) as e:
+        logger.error(
+            f"[WebScraperConfig] Error loading WEB_SCRAPER_SITE_CONFIG from settings: {e}"
+        )
+        return {}

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@backend/app/services/web_scraper/scraper_config.py` around lines 73 - 77,
Replace the broad "except Exception as e" around the WEB_SCRAPER_SITE_CONFIG
loading with a narrow exception tuple that reflects expected failures when
reading settings and manipulating dicts (e.g., except (AttributeError, KeyError,
TypeError, ValueError) as e), so that programming errors still propagate; keep
the existing logger.error call but ensure it logs the captured exception
variable (e) for context when these expected exceptions occur.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@backend/app/services/web_scraper/scraper_config.py`:
- Around line 36-38: The current info-level log in WebScraperConfig that prints
the raw config_value risks exposing secrets; change it to not log the config
contents at INFO level—remove or replace the logger.info(f"[WebScraperConfig]
Loading WEB_SCRAPER_SITE_CONFIG from settings {config_value}") call with a
non-sensitive message (e.g., a simple "loaded WEB_SCRAPER_SITE_CONFIG" at INFO),
and if needed for debugging use logger.debug to log only non-sensitive
identifiers such as domain keys (e.g., via list(config.keys())) while keeping
any secret fields omitted.
- Around line 23-77: The functions _load_site_config and get_site_config lack
return type annotations; add explicit types (e.g., -> Dict[str, Any] or ->
Mapping[str, Any]) to both function signatures and ensure typing imports (Dict,
Any or Mapping) are present at top of the module; update the docstring/returns
only if needed to match the chosen type and keep behavior unchanged (normalize
config returns an empty dict on error so Dict[str, Any] is appropriate).

---

Nitpick comments:
In `@backend/app/services/web_scraper/scraper_config.py`:
- Around line 73-77: Replace the broad "except Exception as e" around the
WEB_SCRAPER_SITE_CONFIG loading with a narrow exception tuple that reflects
expected failures when reading settings and manipulating dicts (e.g., except
(AttributeError, KeyError, TypeError, ValueError) as e), so that programming
errors still propagate; keep the existing logger.error call but ensure it logs
the captured exception variable (e) for context when these expected exceptions
occur.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 366ad16d-c1c9-403b-b536-6d16ae2b7b7a

📥 Commits

Reviewing files that changed from the base of the PR and between e212a75 and c266499.

📒 Files selected for processing (3)

• backend/app/core/config.py
• backend/app/services/web_scraper/scraper_config.py
• backend/app/services/web_scraper/scraper_service.py

✅ Files skipped from review due to trivial changes (1)

• backend/app/core/config.py

🚧 Files skipped from review as they are similar to previous changes (1)

• backend/app/services/web_scraper/scraper_service.py