(MOT-4501) feat(browser): host the scrapling surface natively as browser::scrapling::* - #838
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedToo many files! This PR contains 374 files, which is 274 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. Usage-priced reviews support at most 300 files. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (9)
📒 Files selected for processing (374)
You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
skill-check — worker0 verified, 61 skipped (no docs/).
Four for four. Nicely done. |
…ing::* Ports the scrapling surface from the Python worker to native Rust inside the browser worker: 19 browser::* functions (10 parse ops + fetch, dynamic/stealthy fetch, screenshot-url, sessions, crawl). Parse surface (css, xpath, find, find-by-text/regex, find-similar, describe, extract, to-markdown, regex): - dom layer with lxml-aligned text/blank-text semantics on a vendored xmloxide parser; vendored cssselect (parsel pseudo-elements) and rustpython sre_engine (Python-regex parity) - detached ::text/::attr() descendant idiom; xpath subset lexer/parser/evaluator with recursion cap; generated-selector port; difflib-parity find-similar scoring; markdownify-parity to-markdown - parse ops run via spawn_blocking so a pathological regex cannot stall the shared runtime - schemas pinned to read-only Python goldens (locked oracle environment under oracle/), differential behavior-fixture harness, e2e harness + CI workflow Fetch surface: - safe mode (default): reqwest with per-hop SSRF re-validation and socket pinning, redirect cookie replay (host-scoped), clamped timeouts/redirects/retry delays, total-budget enforcement inside an attempt - browser tiers on a private pipe-based CDP client (no chromiumoxide): per-command 180s ceiling, recoverable Lagged handling, any-session child-target routing (OOPIF/worker resume), target close on every error path, bounded Cloudflare solve - egress gate: local HTTP/CONNECT proxy pinning Chromium's traffic to SSRF-validated addresses; plain-HTTP requests forced Connection:close so a keep-alive client cannot reuse a pinned socket cross-host - SSRF blocklist covers NAT64/6to4/IPv4-compatible v6 embeds - compat mode (feature scrapling-compat, Tier-1 Linux only): vendored curl_impersonate for fingerprint-faithful fetches; exact Chrome pin applies to compat only — safe mode accepts any system Chromium - sessions registry with pending-slot leak guard, insertion-order pruning, bounded shutdown; session/fetch/crawl policy lists aligned so compat-only options are refused consistently Configuration and console: - scrapling.* config block: security_mode, chromium_executable, allow_loopback, defaults, concurrency/session caps, adaptive storage quota, inject_guidance - inject_guidance hot-applies via the shared iii-config-client BindingSlot: console flips bind/unbind the pre-generate guidance hook live (guidance carried statically via metadata.inject_prompt) - custom config form gains a Scraping section with the guidance toggle; scrapling function-trigger renderers for the console chat - chromiumoxide::handler demoted to error by default: its bindings lag the system Chromium, and protocol-skew frames it drops anyway were WARN-spamming per page load (operator RUST_LOG mentioning chromiumoxide always wins) The standalone scrapling-native crate is deleted; its release wiring went with .github/release-workers.yaml (removed upstream in #765).
andersonleal
force-pushed
the
feat/scrapling-native
branch
from
August 19, 2026 13:28
e87f175 to
00d33c4
Compare
…2e suite Adds 18 e2e cases (27 -> 45) so each hardening fix from the scrapling-native pass is exercised over the real bus: SSRF v6-embedded-v4 forms, egress-gate Connection forcing + 403 denial page, max_redirects refusal/clamp, redirect Set-Cookie replay with hostname scoping, repeated-header flattening, duration clamps, compat-only and real_chrome refusals, the Cloudflare solve deadline, dedicated-worker resume (child-target routing), limit coercion, describe kind:null, PI stripping, markdownify parity, session pending-slot rollback, the inject_guidance hot-apply flip, safe-mode config-proxy omission, and crawl IDN normalization. The harness now registers the harness::hook::pre-generate trigger type (as a real agent-harness stack would) so the guidance binding activates out of the engine's pending map and its bind/unbind is observable through engine::registered-triggers::list. The local origin server gains redirect, multi-cookie, header-echo, fake-Cloudflare, dedicated-worker, and IDN-link endpoints; the three configuration-mutating cases restore the browser entry in finally. Features with no wire-observable behavior stay unit-test-only and are documented in the cases-hardening.ts header. Default sentinel timeout raised to 240s for the added browser-tier cases.
CI resolves localhost to ::1 while the origin server listened only on 127.0.0.1, so the hostname-scoping hop failed with a transport error on the runner. The harness now starts an IPv6-loopback twin server and the redirect targets its IP literal directly — no resolver in the path on any machine.
Three independent failures, none introduced by the e2e work: - Compat certification (x86_64): the rebase onto main picked up scrapling/ worker changes (configuration.py, main.py, guidance.py, metadata) made after oracle/manifest.json froze the source fingerprint. Re-freeze the manifest's source section and prove parse parity is unchanged: 19 schema goldens + 135 behavior fixtures byte-identical, 30k parser differential cases and the HTTP differential suite green against the locked oracle. verify_oracle.py now reads the recorded version from scrapling/pyproject instead of a hardcode the CI version bot silently stales. - Compat certification (aarch64): fetch_chromium_artifacts.sh used ln -sfn onto a path the restored rust-cache can materialize as a real directory, which ln refuses to overwrite. rm the destinations first (rerun-safe). - browser: rust lint + test: the generic per-worker gate runs --all-features, but browser's scrapling-compat feature links a pinned curl-impersonate artifact and its tests hard-require the frozen Chrome 148 — both fetched only by the dedicated compat-certification job, which already runs the identical fmt/clippy/test --all-features gate on both Tier-1 targets. The generic gate now covers browser's default feature set (clippy --all-targets and the full 206-test suite, verified locally); the workflow-contract strings in test_rust_ci_workflows.py still hold.
…cle diffs The arm64 archives are Playwright chromium builds that both unpack into chrome-linux/ (the x64 Chrome-for-Testing zips have distinct roots), so the headless-shell find matched nothing and the pw symlink step ran on an empty variable — that empty expansion, not a cache restore, was the real cause of both aarch64 fetch failures. The script now uses the merged directory (which is exactly Playwright's arm64 layout: chrome-linux/chrome + chrome-linux/headless_shell) and fails loudly if neither layout is found. Verified against the real pinned arm64 archives locally. verify_oracle.py now prints which leaf paths disagree before exiting; two CI round-trips have already been spent on an opaque 'differs' with no detail.
…both Tier-1 Chromium builds Two more compat-certification root causes, both surfaced by the new diff printer: - PR CI checks out the MERGE commit, so fingerprinting all of scrapling/** meant any main-side metadata churn (here: #834 touching iii.worker.yaml dependency ranges) broke every open PR's freeze without affecting parse behavior. The source fingerprint now covers scrapling/src only — exactly what gen_goldens.py imports (its dependencies come from oracle/requirements.lock, not worker metadata) — and the version label, which the CI bot bumps every merge, is recorded as provenance but excluded from the parser-runtime comparison. scrapling/src is byte-identical between this branch and main, so the freeze now holds on the merge ref. - The aarch64 frozen Chromium is the Playwright build 1223, which reports 148.0.7778.0, not the x64 Chrome-for-Testing 148.0.7778.96 — the certified version pin is now the set of both frozen Tier-1 builds, shared between certify_chromium and the certified session test.
… with The worker consumes html (parser + entities), tree, serial, xpath, and the parser core (which needs validation::dtd); everything else in the vendored fork was upstream dead weight: the WHATWG html5 parser, CSS engine, SAX/reader streaming APIs, RelaxNG/XSD/Schematron validators, XInclude, catalogs, serde/async integration, the FFI layer, the xmllint CLI, benches, and their examples. Removing them cuts 34.7k lines from the vendored tree (the PR's largest single block) with zero behavior surface: the fork's remaining 702 unit tests + 40 doctests pass, strict clippy is clean, and the browser suite incl. the 135 Python-parity fixtures is green against the trimmed fork. The fork README and lib.rs now document exactly what was removed and that upstream is the restore source.
strip drops DWARF debug info; thin LTO + one codegen unit let the optimizer dead-strip and deduplicate across crate boundaries. The tuned binary lands below main's stock build (24.1 MiB) despite the +11.7 MiB scrapling surface. No feature, dependency, or runtime behavior changes — the 45-case e2e suite passes against the LTO-built artifact. Cost: a few extra minutes per release build (dev profile untouched).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The scrapling surface, ported from the Python worker to native Rust inside the browser worker: 19
browser::*functions — the 10 parse ops (css,xpath,find,find-by-text,find-by-regex,find-similar,describe,extract,to-markdown,regex) plusfetch,dynamic-fetch,stealthy-fetch,screenshot-url, thesession-*quartet, andcrawl. The standalone scrapling-native crate is deleted.Parse surface
tests/golden/schemas, read-only, regenerated only byscripts/gen_goldens.pyunder the locked oracle environment inoracle/), and a differential behavior-fixture harness (tests/golden/behavior, ~170 fixtures) against frozen scrapling 0.4.9.xmloxide(lxml-aligned DOM/text/blank-text rules),cssselect(parsel pseudo-elements, generated-selector algorithm),rustpython-sre_engine(Python-regex parity incl. lookbehind/conditionals/atomic groups). Ports carry NOTICE/LICENSE files.spawn_blockingso a catastrophic-backtracking regex cannot stall the shared runtime.Fetch surface
Laggedhandling, any-session child-target routing (OOPIFs/workers resume), target close on every error path, deadline-bounded Cloudflare solve. Chromium egresses through a local HTTP/CONNECT gate that pins traffic to SSRF-validated addresses; plain-HTTP requests are forcedConnection: closeso a keep-alive client can't reuse a pinned socket cross-host.scrapling-compatfeature, Tier-1 Linux only): vendoredcurl_impersonatefor fingerprint-faithful fetches; the exact Chrome pin applies to compat only — safe mode accepts any system Chromium.Configuration and console
browser.scrapling.*config block:security_mode,chromium_executable,allow_loopback, fetch defaults, concurrency/session caps, adaptive storage quota,inject_guidance.inject_guidancehot-applies via the sharediii-config-clientBindingSlot(same pattern as fp/web/workflow): console flips bind/unbind the pre-generate guidance hook live, and the guidance travels statically viametadata.inject_promptso the harness never RPCs per generate step. The custom config form gains a Scraping section with the toggle.chromiumoxide::handleris demoted toerrorby default: its protocol bindings lag the system Chromium and it WARN-spams "WS Invalid message" for frames it drops anyway; an operatorRUST_LOGmentioning chromiumoxide always wins (src/logging.rs, tested).Verification
cargo fmt+clippy --all-targetsclean; 206 lib tests + schema/behavior/CDP/compat integration suites green (default features);--features scrapling-compatcompiles (its one gated test needs the certified Chrome 148 binary, present in CI). Browser UI: 27 vitest tests green,tsc --noEmitclean. e2e harness + CI workflow included (browser-scrapling-e2e.yml).E2E coverage
45 harness cases against a live engine, the release worker binary, and the frozen Chromium (
tests/e2e/,HARNESS_DONE: PASS 45/45locally and in CI). All 19 functions are exercised, and every wire-observable hardening fix has a dedicated case incases-hardening.ts:Connection: closeat the origin,Proxy-Connection/keep-alive stripped, origin-form rewrite, and the 403browser egress deniedpage for blocked addressesmax_redirects: safe-mode refusal of-1and the clamp to 100 against a live redirect loopSet-Cookiereplay on same-host hops, with hostname-only scoping proven across two loopback origins (127.0.0.1vs[::1])", "; duration/retry_delayclamps instead of panicshttp3,verify:false,proxies,proxy_auth,stealthy_headers,impersonate,dns_over_https) andreal_chromeon every browser tier, session-open, and per-page in crawlwaitForDebuggerOnStart(the child-target routing fix)limitfloat/string coercion,describe kind:null-> XPath, processing-instruction stripping, markdownify fixes (NBSP-preserving collapse, empty href/title, list spacing,ol start)session limit reached (8)fires exactly at the cap, closing frees slotsinject_guidancehot-apply observed end-to-end: the harness registers theharness::hook::pre-generatetrigger type as a stand-in agent harness, then watches the binding appear/disappear viaengine::registered-triggers::listacross config flipsallowed_domainspunycode-normalize before matchingThe three configuration-mutating cases restore the
browserconfiguration entry infinally. Internals with no wire-observable behavior (chromiumoxide log filter, CDP 180s ceiling and Lagged/broadcast handling, close-on-error target leaks,insertion_orderpruning, OOPIF iframe HTML) stay unit-test-certified and are documented in thecases-hardening.tsheader. A max-effort review pass ran over the branch; all 32 verified findings (SSRF gaps, hang/panic paths, session leaks, Python-parity drift) are fixed in this diff.Fixes MOT-4501