feat(web): migrate web worker from TypeScript to Rust

web/Cargo.toml

@@ -0,0 +1,45 @@
+[workspace]
+
+[package]
+name = "web"
+version = "1.0.0"
+edition = "2021"
+publish = false
+
+[[bin]]
+name = "iii-web"
+path = "src/main.rs"
+
+[lib]
+name = "web"
+path = "src/lib.rs"
+
+[dependencies]
+iii-sdk = "=0.19.2"
+arc-swap = "1"
+tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "signal", "time", "net"] }
+serde = { version = "1", features = ["derive"] }
+serde_json = "1"
+serde_yaml = "0.9"
+anyhow = "1"
+thiserror = "2"
+tracing = "0.1"
+tracing-subscriber = { version = "0.3", features = ["fmt", "env-filter"] }
+clap = { version = "4", features = ["derive"] }
+async-trait = "0.1"
+schemars = "0.8"
+futures = "0.3"
+reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json", "stream", "http2"] }
+htmd = "0.5"
+tl = { package = "astral-tl", version = "0.7" }
+ipnet = "2"
+base64 = "0.22"
+bytes = "1"
+
+[dev-dependencies]
+serde_json = "1"
+tempfile = "3"
+wiremock = "0.6"
+tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "signal", "time", "net"] }
+futures = "0.3"
+bytes = "1"

web/README.md

@@ -0,0 +1,201 @@
+# web
+
+Outbound HTTP(S) client on the [iii engine](https://github.com/iii-hq/iii)
+bus. A single bus function, `web::fetch`, performs a network request and
+returns a parsed `{ ok, status, headers, body }` envelope with enforced
+size/timeout caps and server-side SSRF protection. It is the network
+counterpart to the `shell` worker: use `web::fetch` instead of
+`shell::exec` with `curl`/`wget` for any HTTP request.
+
+The worker is a faithful Rust port of the original TypeScript `web::fetch`
+implementation — same request fields and success/error/image envelopes, so
+existing callers and the harness consumer are unaffected.
+
+## Table of contents
+
+1. [Install](#install)
+2. [Configuration](#configuration)
+3. [The `web::fetch` function](#the-webfetch-function)
+4. [Reading the result](#reading-the-result)
+5. [Page-reading mode](#page-reading-mode)
+6. [SSRF guard](#ssrf-guard)
+7. [Local development & testing](#local-development--testing)
+
+---
+
+## Install
+
+```bash
+iii worker add web
+```
+
+`iii worker add` fetches the binary (`iii-web`), writes a config block into
+`~/.iii/config.yaml`, and the engine starts the worker on the next
+`iii start`.
+
+---
+
+## Configuration
+
+The worker reads its operational ceilings from the `configuration` worker
+under the `web` key. It registers a schema + seed on boot, fetches the
+authoritative value, and hot-reloads on change. All keys are optional and
+fall back to the defaults below.
+
+```yaml
+web:
+  default_timeout_ms: 30000      # per-request timeout when caller omits timeout_ms
+  max_timeout_ms: 120000         # ceiling; caller timeout_ms is clamped DOWN to this
+  default_response_bytes: 262144 # default body cap in page-reading mode (256 KiB)
+  max_response_bytes: 5242880    # absolute body ceiling (5 MiB); caller max_bytes clamped to this
+  max_transform_bytes: 1048576   # HTML→markdown/text runs only on bodies ≤ this (1 MiB)
+  max_redirects: 5               # hops before too_many_redirects
+  user_agent: 'iii-harness/0.1 (+web::fetch)'
+  allow_loopback: true           # set false in prod so 127.0.0.0/8, ::1, localhost are blocked
+```
+
+A copy of these defaults lives in [`config.yaml.example`](./config.yaml.example).
+
+---
+
+## The `web::fetch` function
+
+Minimal call — a URL is the only required field; everything else has a
+default:
+
+```jsonc
+// agent_trigger { function: "web::fetch", payload: { "url": "https://api.github.com/zen" } }
+// → { "ok": true, "status": 200, "body": "Design for failure.", ... }
+```
+
+### Request fields
+
+| Field | Default | Notes |
+|---|---|---|
+| `url` *(required)* | — | absolute `http(s)://` |
+| `method` | `GET` | case-insensitive; GET HEAD POST PUT PATCH DELETE OPTIONS |
+| `headers` | `{}` | object; keys as written |
+| `json` | — | structured payload; auto-stringified + sets `content-type: application/json`. **Wins over `body`.** |
+| `body` | — | raw string body; ignored on GET/HEAD |
+| `response_format` | `"text"` | `"text"` \| `"base64"` (binary) \| `"json"` (also parses into `json`) |
+| `format` | — | page-reading mode: `"markdown"` \| `"text"` \| `"html"` (see below) |
+| `timeout_ms` | `default_timeout_ms` | clamped DOWN to `max_timeout_ms` |
+| `max_bytes` | `max_response_bytes` (`default_response_bytes` in `format` mode) | over-cap body is truncated, not errored |
+| `follow_redirects` | `true` | each hop re-checked against the SSRF blocklist |
+
+### Response
+
+**Success** (`ok: true`) — returned for *any* completed response, 2xx through 5xx:
+
+```jsonc
+{
+  "ok": true,
+  "status": 200,
+  "status_text": "OK",
+  "headers": { "content-type": "application/json", ... },  // keys LOWER-CASED
+  "body": "<utf8 text | base64 when response_format=base64>",
+  "json": { ... },            // only when response_format="json" and parse succeeded
+  "parse_error": "…",         // only when response_format="json" and JSON.parse failed
+  "response_format": "json",
+  "bytes_truncated": false,   // true when body hit max_bytes (NOT an error)
+  "redirect_chain": ["https://…/a"],  // omitted when no redirects
+  "content_type": "text/html",  // page-reading mode only
+  "transformed": "markdown"     // only when an HTML transform actually ran
+}
+```
+
+**Failure** (`ok: false`) — the fetch never produced a response (no `status`):
+
+```jsonc
+{ "ok": false, "error": "blocked_host", "message": "…" }
+```
+
+| `error` | Cause | Fix |
+|---|---|---|
+| `invalid_payload` | Payload failed schema (bad `method`, wrong types) | Correct the named fields |
+| `invalid_url` | `url` isn't a parseable absolute `http(s)://` URL | Pass a full absolute URL incl. scheme |
+| `blocked_host` | Target resolves to a private / link-local / cloud-metadata IP | Don't target internal/metadata hosts; for dev loopback set `allow_loopback` |
+| `timeout` | Slower than `timeout_ms` | Raise `timeout_ms` (up to ceiling) or shrink work via `max_bytes` |
+| `too_many_redirects` | More than `max_redirects` hops | Use the final URL, or `follow_redirects: false` and read the `location` header |
+| `transport_error` | Connection refused/reset, TLS/DNS failure, or unparseable redirect `Location` | Check host/port/cert; retry if transient |
+
+There is **no `too_large` error** — oversize responses come back `ok: true`
+with `bytes_truncated: true`.
+
+---
+
+## Reading the result
+
+**`ok` tells you if the request COMPLETED, not whether the server was happy.**
+
+- A 404 or 500 is a **successful fetch** → `ok: true`, `status: 404`. Branch on `status` for HTTP outcomes.
+- `ok: false` means the request never produced a response. Branch on `error`.
+
+```jsonc
+if (!r.ok)              → fetch failed; look at r.error
+else if (r.status>=400) → server returned an HTTP error; look at r.status / r.body
+else                    → success; use r.body or r.json
+```
+
+---
+
+## Page-reading mode
+
+`response_format` (transport encoding) and `format` (page-reading) are
+orthogonal — set one, not both. When `format` is set, `response_format` is
+ignored.
+
+- **`format: "markdown"`** — `text/html` → Markdown (the right default for reading pages; far fewer tokens than raw HTML).
+- **`format: "text"`** — `text/html` → plain text.
+- **`format: "html"`** — raw HTML.
+
+In page-reading mode the request goes out with a browser User-Agent +
+format-matched `Accept` header, and retries once with the honest UA on a
+Cloudflare `cf-mitigated: challenge` response. The HTML→markdown/text
+transform runs only on bodies ≤ `max_transform_bytes`; larger pages come
+back raw with `transformed` unset.
+
+**Images:** a viewable `image/*` response (jpeg/png/gif/webp, 2xx,
+complete, non-empty) returns the actual image plus a one-line text summary
+instead of the JSON envelope. Anything else falls back to the normal
+envelope with `response_format: "base64"`.
+
+---
+
+## SSRF guard
+
+DNS is resolved **once**, every resolved address is checked, then the
+request is dialed at the **validated IP** (pinned `lookup` + TLS
+`servername`) — so the IP that passed the check is the IP connected to (no
+DNS-rebind window). On each 3xx the `Location` is re-resolved and
+re-validated before following, and `Authorization` / `Cookie` /
+`Proxy-Authorization` are stripped when the redirect changes host or
+downgrades `https → http`.
+
+Blocked **unconditionally** → `error: "blocked_host"`:
+
+- Private RFC1918 (`10/8`, `172.16/12`, `192.168/16`)
+- Link-local incl. **cloud metadata** `169.254.169.254`
+- IPv6 ULA (`fc00::/7`), link-local (`fe80::/10`), multicast
+- `::ffff:`-mapped IPv4 in both dotted (`::ffff:169.254.169.254`) and hex (`::ffff:a9fe:a9fe`) forms
+
+Loopback (`127.0.0.0/8`, `::1`, `localhost`) is **allowed by default** for
+dev convenience; operators set `web.allow_loopback: false` for prod, after
+which loopback returns `blocked_host`.
+
+---
+
+## Local development & testing
+
+```bash
+cargo build                  # build the iii-web binary
+cargo test                   # unit + wiremock integration + adversarial SSRF tests
+cargo clippy -- -D warnings  # lints
+cargo fmt --check            # formatting
+```
+
+The agent-facing authoring guide lives in
+[`skills/index.md`](./skills/index.md) (served at runtime via
+`directory::skills::get`). For the exact, authoritative field types, call
+`engine::functions::info { function_id: "web::fetch" }` — the live schema
+wins if it ever disagrees with this document.

web/build.rs

@@ -0,0 +1,6 @@
+fn main() {
+    println!(
+        "cargo:rustc-env=TARGET={}",
+        std::env::var("TARGET").unwrap()
+    );
+}

web/config.yaml.example

@@ -0,0 +1,9 @@
+web:
+  default_timeout_ms: 30000
+  max_timeout_ms: 120000
+  default_response_bytes: 262144
+  max_response_bytes: 5242880
+  max_transform_bytes: 1048576
+  max_redirects: 5
+  user_agent: 'iii-harness/0.1 (+web::fetch)'
+  allow_loopback: true

web/iii.worker.yaml

@@ -0,0 +1,10 @@
+iii: v1
+name: web
+language: rust
+deploy: binary
+manifest: Cargo.toml
+bin: iii-web
+description: Outbound HTTP client on the iii bus (web::fetch).
+
+dependencies:
+  configuration: "^0.19.0"