fix(discovery): Detect soft 404s via content-type validation

[steveant]

Summary

• Added content-type validation to _check_url_exists() in the discovery service
• Detects soft 404s where servers return HTTP 200 with text/html for missing files (like llms.txt)
• Logs informative message when soft 404 is detected and returns False to allow crawler fallback

Problem

When crawling sites like platform.claude.com, the discovery service incorrectly detected llms.txt as existing because the server returned HTTP 200 with HTML content (a redirect/error page) instead of a proper 404 status.

This caused the crawler to attempt parsing HTML as plain text, leading to failed ingestion.

Solution

Added validation that checks the response Content-Type header against expected types:

• .txt/.md files must return text/plain or text/markdown (not text/html)
• .xml files must return text/xml or application/xml variants

When text/html is received for these file types, it's treated as a soft 404 and logged accordingly.

Changes

• Added EXPECTED_CONTENT_TYPES class constant mapping file extensions to valid content types
• Extended _check_url_exists() to validate content-type after successful status check
• Added informative logging for soft 404 detection

Test plan

• Tested crawling platform.claude.com - correctly falls back to HTML crawling
• Verified llms.txt soft 404 is logged: Soft 404 detected: ... returned text/html instead of ['text/plain', ...]
• Normal sites with valid llms.txt files continue to work

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Enhanced URL validation to detect soft-404 errors through content-type verification for text and XML files.
  • Improved redirect security by validating destination scheme and hostname, blocking unsafe redirects.
  • Expanded error handling and logging for invalid URL formats and blocked redirect targets.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

Enhanced the discovery service's URL validation to include content-type verification for text and XML files, enabling soft-404 detection. Added redirect handling with scheme and hostname validation to block unsafe redirects. Expanded error handling and logging throughout the validation process.

Changes

Cohort / File(s)

Summary

Discovery Service URL Validation Enhancement python/src/server/services/crawling/discovery_service.py

Added EXPECTED_CONTENT_TYPES class attribute mapping file extensions (.txt, .md, .xml) to valid content-type lists. Enhanced _check_url_exists() method with content-type validation for soft-404 detection (treating text/html as soft-404 for these extensions). Implemented redirect target validation for scheme and hostname safety. Expanded error handling and logging for invalid URL formats, blocked schemes, and unsafe redirects. Updated method docstring.

Sequence Diagram(s)

sequenceDiagram
    actor Client
    participant DiscoveryService
    participant URLValidator
    participant RemoteServer

    Client->>DiscoveryService: check_url_exists(url)
    DiscoveryService->>URLValidator: validate URL format & scheme
    alt Invalid URL or blocked scheme
        URLValidator-->>DiscoveryService: log & return error
        DiscoveryService-->>Client: false
    else Valid URL
        DiscoveryService->>RemoteServer: HEAD/GET request
        RemoteServer-->>DiscoveryService: response + headers
        alt Redirect detected
            DiscoveryService->>URLValidator: validate redirect target<br/>(scheme & hostname)
            alt Unsafe redirect
                URLValidator-->>DiscoveryService: log & return error
                DiscoveryService-->>Client: false
            else Safe redirect
                DiscoveryService->>RemoteServer: follow to final destination
                RemoteServer-->>DiscoveryService: final response
                DiscoveryService->>DiscoveryService: check content-type
            end
        else No redirect
            DiscoveryService->>DiscoveryService: check content-type
        end
        alt File extension in EXPECTED_CONTENT_TYPES
            alt Content-type matches expected
                DiscoveryService-->>Client: true
            else Content-type mismatch<br/>(including text/html)
                DiscoveryService-->>Client: false (soft-404)
            end
        else Unknown extension
            DiscoveryService-->>Client: true
        end
    end

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

• Security implications: Redirect validation logic requires careful review to ensure all unsafe redirect scenarios are properly blocked
• Content-type mapping: Verify the EXPECTED_CONTENT_TYPES mapping is comprehensive and correct for all targeted file types
• Edge cases: Review handling of edge cases (malformed responses, missing headers, multiple redirects, mixed-case content-types)
• Integration impact: Assess how soft-404 detection affects existing discovery workflows and service behavior

Poem

🐰✨ A rabbit hops through URLs with care,
Content-types validated fair and square,
Redirects checked for safety's sake,
Soft-404s the service can now make,
Discovery springs forward, strong and true! 🌱

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: adding content-type validation to detect soft 404s in the discovery service.
Description check	✅ Passed	The description covers all critical template sections: Summary, Changes Made (mapped to 'Changes'), Type of Change (bug fix), and Test plan. Some optional sections like Affected Services and Checklist are not filled, but the core content is comprehensive and complete.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

Tip

📝 Customizable high-level summaries are now available in beta!

You can now customize how CodeRabbit generates the high-level summary in your pull requests — including its content, structure, tone, and formatting.

• Provide your own instructions using the high_level_summary_instructions setting.
• Format the summary however you like (bullet lists, tables, multi-section layouts, contributor stats, etc.).
• Use high_level_summary_in_walkthrough to move the summary from the description to the walkthrough section.

Example instruction:

"Divide the high-level summary into five sections:

1. 📝 Description — Summarize the main change in 50–60 words, explaining what was done.
2. 📓 References — List relevant issues, discussions, documentation, or related PRs.
3. 📦 Dependencies & Requirements — Mention any new/updated dependencies, environment variable changes, or configuration updates.
4. 📊 Contributor Summary — Include a Markdown table showing contributions:
   | Contributor | Lines Added | Lines Removed | Files Changed |
5. ✔️ Additional Notes — Add any extra reviewer context.
   Keep each section concise (under 200 words) and use bullet or numbered lists for clarity."

Note: This feature is currently in beta for Pro-tier users, and pricing will be announced later.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

python/src/server/services/crawling/discovery_service.py (1)

293-298: Consider moving class constant to the top of the class with other constants.

For consistency, EXPECTED_CONTENT_TYPES could be placed near MAX_RESPONSE_SIZE, DISCOVERY_PRIORITY, and FILE_EXTENSIONS at the top of the class (around line 54-76).

Also, some servers use application/x-markdown for Markdown files—consider adding it:

 EXPECTED_CONTENT_TYPES = {
-    '.txt': ['text/plain', 'text/markdown', 'text/x-markdown'],
-    '.md': ['text/plain', 'text/markdown', 'text/x-markdown'],
+    '.txt': ['text/plain', 'text/markdown', 'text/x-markdown', 'application/x-markdown'],
+    '.md': ['text/plain', 'text/markdown', 'text/x-markdown', 'application/x-markdown'],
     '.xml': ['text/xml', 'application/xml', 'application/rss+xml', 'application/atom+xml'],
 }

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9bb1683 and 33abb60.

📒 Files selected for processing (1)

• python/src/server/services/crawling/discovery_service.py (2 hunks)

🔇 Additional comments (2)

python/src/server/services/crawling/discovery_service.py (2)

300-311: LGTM!

The docstring updates accurately describe the new content-type validation behavior and the updated return semantics.

---

369-390: Solid implementation of soft 404 detection.

The content-type validation logic correctly:

• Parses the content-type header, stripping charset parameters
• Only blocks text/html responses for text/XML files (avoiding false positives)
• Logs a debug message for other unexpected content types without blocking

One edge case: if the server returns no Content-Type header, content_type becomes '' (falsy), so the if content_type and content_type not in valid_types check passes and the URL is considered valid. This lenient behavior seems reasonable for maximum compatibility, but worth confirming it's intentional.

[steveant]

Superseded by #928 which consolidates this fix along with other improvements.