feat: Document Intelligence Layer — HTML/HTM file support + unified document pipeline

[chanmeimei]

Summary

Adds a Document Intelligence Layer (agent/document_processing/) that enables Hermes to understand uploaded HTML files and other document types through a unified normalization pipeline.

Problem

Currently, .html and .htm files sent via Telegram (and other platforms) are rejected with "Unsupported document type". Users sending API documentation, web pages, or exported HTML files cannot have Hermes read and understand them.

Changes

A. File type whitelist (gateway/platforms/base.py)

• Added .html and .htm to SUPPORTED_DOCUMENT_TYPES

B. New module: agent/document_processing/ (6 files)

File	Purpose
types.py	DocumentResult — canonical output format for all parsers
html_parser.py	BeautifulSoup HTML parser (strips script/style/noscript, extracts title/text/links)
url_fetcher.py	HTTP fetcher with SSRF protection, 10s timeout, 5MB limit
normalizer.py	Builds standardized DocumentResult from parser output
router.py	Single entry-point — dispatches by file extension
__init__.py	Public API exports

C. Telegram gateway refactor (gateway/platforms/telegram.py)

• Document content injection now delegates to document_processing.router.process_document() instead of inline UTF-8 decode
• Graceful fallback to raw text injection if the module fails
• Supports ALL text-based extensions, not just .md/.txt

D. Dependencies (pyproject.toml)

• Added beautifulsoup4>=4.13.0,<5 to core dependencies
• Fallback regex parser works without bs4 if needed

Security

• SSRF protection: blocks localhost, 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, link-local, IPv6 private
• URL fetch timeout: 10 seconds
• Max HTML size: 5 MB (streaming enforcement)
• script/style/noscript tags fully removed from parsed output

Architecture

Gateway platforms only handle receiving files — all understanding logic lives in agent/document_processing/. Adding PDF/DOCX/XLSX parsers in the future requires only adding a new parser file + one router branch. No gateway code changes needed.

Tests

21 test cases covering all 7 acceptance scenarios:

1. ✅ .html upload → body text extraction
2. ✅ .htm upload → body text extraction
3. ✅ URL fetch (example.com) → title + body
4. ✅ Unsupported file → clear error message
5. ✅ SSRF: localhost/private IPs blocked
6. ✅ Oversized HTML handling
7. ✅ script/style/noscript removal

21 passed in 6.07s

Standard Output Format

All documents normalize to:

{
  "source_type": "telegram_file | url | local_file",
  "document_type": "html | pdf | txt | md | json | csv",
  "title": "",
  "text": "",
  "links": [],
  "metadata": { "filename": "", "url": "", "mime_type": "", "size": 0 }
}

[alt-glitch]

Overlaps with #12702 and #12009 — both add .html/.htm to the document allowlist. This PR is broader (adds full document_processing module), but the gateway allowlist change is the same.

[teknium1]

Thanks for addressing HTML attachments. The original rejection is no longer present on current main: 4314d451c accepts every authorized inbound file type, and the active Telegram plugin includes .html/.htm in its text-injection set (gateway/platforms/base.py:1370-1383; plugins/platforms/telegram/adapter.py:7969-8002).

Problems

• agent/document_processing/url_fetcher.py:79-87 checks only the initial hostname and then follows redirects via requests.get(..., allow_redirects=True). A public redirect can therefore reach a private target. Current main documents this exact SSRF class in 500c2b1e4.
• The PR edits gateway/platforms/telegram.py, but that adapter was moved to plugins/platforms/telegram/adapter.py in 736ffb3bc; the PR is currently conflicting.

Suggested changes

• If HTML cleanup remains desired, port a narrow parser path to the active plugin adapter and cover .html and .htm there.
• Route any URL retrieval through the repository's redirect-safe URL validation pattern rather than the proposed standalone requests fetcher.

Automated hermes-sweeper review.

[teknium1]

Blocking: _check_ssrf() validates only the original hostname, while allow_redirects=True can follow a public URL to a private target without revalidation. Current main's 500c2b1e4 fixes this exact SSRF redirect class; use an equivalent per-redirect guard before adding this fetch path.