Skip to content

feat(web): content pipeline for web::fetch (static crawl parity, Phase 1) #384

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andersonleal merged 2 commits into from
Jul 1, 2026
Merged

feat(web): content pipeline for web::fetch (static crawl parity, Phase 1) #384

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
18224d3
feat(web): content pipeline for web::fetch (static crawl parity, Phas…
andersonleal Jul 1, 2026
8560d09
fix(web): address PR review — anchor link density, HTML truncation
andersonleal Jul 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion web/Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions web/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ tl = { package = "astral-tl", version = "0.7" }
ipnet = "2"
base64 = "0.22"
bytes = "1"
rust-stemmers = "1"
url = "2"

[dev-dependencies]
serde_json = "1"
Expand Down
11 changes: 10 additions & 1 deletion web/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,11 @@ default:
| `body` | — | raw string body; ignored on GET/HEAD |
| `response_format` | `"text"` | `"text"` \| `"base64"` (binary) \| `"json"` (also parses into `json`) |
| `format` | — | page-reading mode: `"markdown"` \| `"text"` \| `"html"` (see below) |
| `content_filter` | — | `{ type: "pruning"\|"bm25", query?, threshold?, threshold_type?, min_word_threshold? }`. Page mode only. When set, `body` holds the **filtered** content (pruning default threshold 0.48; bm25 default 1.0; `query` falls back to page metadata). |
| `target_elements` | — | CSS selectors (tl subset: tag/`.class`/`#id`); restrict rendered content to these regions |
| `excluded_tags` | — | tag/selectors to drop before rendering (e.g. `nav`, `footer`) |
| `include_links` | `false` | adds `links: { internal, external }` (absolute URLs, classified by host) |
| `include_media` | `false` | adds `media: { images, videos, audios }` (absolute URLs) |
| `timeout_ms` | `default_timeout_ms` | clamped DOWN to `max_timeout_ms` |
| `max_bytes` | `max_response_bytes` (`default_response_bytes` in `format` mode) | over-cap body is truncated, not errored |
| `follow_redirects` | `true` | each hop re-checked against the SSRF blocklist |
Expand All @@ -100,7 +105,9 @@ default:
"bytes_truncated": false, // true when body hit max_bytes (NOT an error)
"redirect_chain": ["https://…/a"], // omitted when no redirects
"content_type": "text/html", // page-reading mode only
"transformed": "markdown" // only when an HTML transform actually ran
"transformed": "markdown", // only when an HTML transform actually ran
"links": { "internal": [{ "href": "…", "text": "…" }], "external": [ … ] }, // include_links
"media": { "images": [{ "src": "…", "alt": "…" }], "videos": [ … ], "audios": [ … ] }, // include_media
}
```

Expand Down Expand Up @@ -149,6 +156,8 @@ ignored.
- **`format: "text"`** — `text/html` → plain text.
- **`format: "html"`** — raw HTML.

When `content_filter` is set, `body` is the filtered output (there is no separate field) — feed it straight to a model. (If filtering would empty the page, or the page is too large or deeply nested to transform, `body` falls back to the unfiltered content.)

In page-reading mode the request goes out with a browser User-Agent +
format-matched `Accept` header, and retries once with the honest UA on a
Cloudflare `cf-mitigated: challenge` response. The HTML→markdown/text
Expand Down
14 changes: 14 additions & 0 deletions web/skills/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,20 @@ One exception: in page-reading mode (`format` set), an `image/*` response return
| `max_bytes` | 5 MiB (256 KiB in `format` mode) | raw fetches default to the 5 MiB ceiling; page-reading mode (`format` set) uses a context-safe 256 KiB. Pass an explicit value to override (up to the 5 MiB ceiling). Over-cap body is truncated, not errored |
| `follow_redirects` | `true` | each hop re-checked against the SSRF blocklist |

# Content pipeline

Four request controls for page-reading mode (`format` set):

| Field | Effect |
|---|---|
| `content_filter` | `{ type: "pruning"\|"bm25", query?, threshold?, threshold_type?, min_word_threshold? }` — filters `body` to signal content. Pruning default threshold 0.48; BM25 default 1.0. `query` falls back to page `<title>`/`<meta description>` if omitted. |
| `target_elements` | Restrict body to matching regions. Selector subset: tag, `.class`, `#id` — other forms are silently ignored. |
| `excluded_tags` | Drop matching elements before rendering (e.g. `["nav", "footer"]`). |
| `include_links` | `true` → adds `links: { internal, external }` arrays (absolute URLs, classified by host). |
| `include_media` | `true` → adds `media: { images, videos, audios }` arrays (absolute URLs). |

When `content_filter` is set, `body` **is** the filtered output — no separate field. Feed it straight to a model.

# Response

**Success** (`ok: true`) — returned for *any* completed response, 2xx through 5xx:
Expand Down
Loading
Loading