(MOT-4463) fix(workers): allow shared dependency minor updates - #834
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
skill-check — worker0 verified, 61 skipped (no docs/).
Four for four. Nicely done. |
ytallo
changed the title
ytallo
changed the title
ytallo
marked this pull request as ready for review
August 19, 2026 16:11
andersonleal
added a commit
that referenced
this pull request
Aug 19, 2026
…both Tier-1 Chromium builds Two more compat-certification root causes, both surfaced by the new diff printer: - PR CI checks out the MERGE commit, so fingerprinting all of scrapling/** meant any main-side metadata churn (here: #834 touching iii.worker.yaml dependency ranges) broke every open PR's freeze without affecting parse behavior. The source fingerprint now covers scrapling/src only — exactly what gen_goldens.py imports (its dependencies come from oracle/requirements.lock, not worker metadata) — and the version label, which the CI bot bumps every merge, is recorded as provenance but excluded from the parser-runtime comparison. scrapling/src is byte-identical between this branch and main, so the freeze now holds on the merge ref. - The aarch64 frozen Chromium is the Playwright build 1223, which reports 148.0.7778.0, not the x64 Chrome-for-Testing 148.0.7778.96 — the certified version pin is now the set of both frozen Tier-1 builds, shared between certify_chromium and the certified session test.
andersonleal
added a commit
that referenced
this pull request
Aug 19, 2026
…ser::scrapling::* (#838) * feat(browser): host the scrapling surface natively as browser::scrapling::* Ports the scrapling surface from the Python worker to native Rust inside the browser worker: 19 browser::* functions (10 parse ops + fetch, dynamic/stealthy fetch, screenshot-url, sessions, crawl). Parse surface (css, xpath, find, find-by-text/regex, find-similar, describe, extract, to-markdown, regex): - dom layer with lxml-aligned text/blank-text semantics on a vendored xmloxide parser; vendored cssselect (parsel pseudo-elements) and rustpython sre_engine (Python-regex parity) - detached ::text/::attr() descendant idiom; xpath subset lexer/parser/evaluator with recursion cap; generated-selector port; difflib-parity find-similar scoring; markdownify-parity to-markdown - parse ops run via spawn_blocking so a pathological regex cannot stall the shared runtime - schemas pinned to read-only Python goldens (locked oracle environment under oracle/), differential behavior-fixture harness, e2e harness + CI workflow Fetch surface: - safe mode (default): reqwest with per-hop SSRF re-validation and socket pinning, redirect cookie replay (host-scoped), clamped timeouts/redirects/retry delays, total-budget enforcement inside an attempt - browser tiers on a private pipe-based CDP client (no chromiumoxide): per-command 180s ceiling, recoverable Lagged handling, any-session child-target routing (OOPIF/worker resume), target close on every error path, bounded Cloudflare solve - egress gate: local HTTP/CONNECT proxy pinning Chromium's traffic to SSRF-validated addresses; plain-HTTP requests forced Connection:close so a keep-alive client cannot reuse a pinned socket cross-host - SSRF blocklist covers NAT64/6to4/IPv4-compatible v6 embeds - compat mode (feature scrapling-compat, Tier-1 Linux only): vendored curl_impersonate for fingerprint-faithful fetches; exact Chrome pin applies to compat only — safe mode accepts any system Chromium - sessions registry with pending-slot leak guard, insertion-order pruning, bounded shutdown; session/fetch/crawl policy lists aligned so compat-only options are refused consistently Configuration and console: - scrapling.* config block: security_mode, chromium_executable, allow_loopback, defaults, concurrency/session caps, adaptive storage quota, inject_guidance - inject_guidance hot-applies via the shared iii-config-client BindingSlot: console flips bind/unbind the pre-generate guidance hook live (guidance carried statically via metadata.inject_prompt) - custom config form gains a Scraping section with the guidance toggle; scrapling function-trigger renderers for the console chat - chromiumoxide::handler demoted to error by default: its bindings lag the system Chromium, and protocol-skew frames it drops anyway were WARN-spamming per page load (operator RUST_LOG mentioning chromiumoxide always wins) The standalone scrapling-native crate is deleted; its release wiring went with .github/release-workers.yaml (removed upstream in #765). * test(browser): cover every wire-observable hardening feature in the e2e suite Adds 18 e2e cases (27 -> 45) so each hardening fix from the scrapling-native pass is exercised over the real bus: SSRF v6-embedded-v4 forms, egress-gate Connection forcing + 403 denial page, max_redirects refusal/clamp, redirect Set-Cookie replay with hostname scoping, repeated-header flattening, duration clamps, compat-only and real_chrome refusals, the Cloudflare solve deadline, dedicated-worker resume (child-target routing), limit coercion, describe kind:null, PI stripping, markdownify parity, session pending-slot rollback, the inject_guidance hot-apply flip, safe-mode config-proxy omission, and crawl IDN normalization. The harness now registers the harness::hook::pre-generate trigger type (as a real agent-harness stack would) so the guidance binding activates out of the engine's pending map and its bind/unbind is observable through engine::registered-triggers::list. The local origin server gains redirect, multi-cookie, header-echo, fake-Cloudflare, dedicated-worker, and IDN-link endpoints; the three configuration-mutating cases restore the browser entry in finally. Features with no wire-observable behavior stay unit-test-only and are documented in the cases-hardening.ts header. Default sentinel timeout raised to 240s for the added browser-tier cases. * fix(browser): pin the hop-x scoping case to [::1] instead of localhost CI resolves localhost to ::1 while the origin server listened only on 127.0.0.1, so the hostname-scoping hop failed with a transport error on the runner. The harness now starts an IPv6-loopback twin server and the redirect targets its IP literal directly — no resolver in the path on any machine. * fix(browser): heal the three red CI gates Three independent failures, none introduced by the e2e work: - Compat certification (x86_64): the rebase onto main picked up scrapling/ worker changes (configuration.py, main.py, guidance.py, metadata) made after oracle/manifest.json froze the source fingerprint. Re-freeze the manifest's source section and prove parse parity is unchanged: 19 schema goldens + 135 behavior fixtures byte-identical, 30k parser differential cases and the HTTP differential suite green against the locked oracle. verify_oracle.py now reads the recorded version from scrapling/pyproject instead of a hardcode the CI version bot silently stales. - Compat certification (aarch64): fetch_chromium_artifacts.sh used ln -sfn onto a path the restored rust-cache can materialize as a real directory, which ln refuses to overwrite. rm the destinations first (rerun-safe). - browser: rust lint + test: the generic per-worker gate runs --all-features, but browser's scrapling-compat feature links a pinned curl-impersonate artifact and its tests hard-require the frozen Chrome 148 — both fetched only by the dedicated compat-certification job, which already runs the identical fmt/clippy/test --all-features gate on both Tier-1 targets. The generic gate now covers browser's default feature set (clippy --all-targets and the full 206-test suite, verified locally); the workflow-contract strings in test_rust_ci_workflows.py still hold. * fix(browser): recognize the merged arm64 chromium layout and name oracle diffs The arm64 archives are Playwright chromium builds that both unpack into chrome-linux/ (the x64 Chrome-for-Testing zips have distinct roots), so the headless-shell find matched nothing and the pw symlink step ran on an empty variable — that empty expansion, not a cache restore, was the real cause of both aarch64 fetch failures. The script now uses the merged directory (which is exactly Playwright's arm64 layout: chrome-linux/chrome + chrome-linux/headless_shell) and fails loudly if neither layout is found. Verified against the real pinned arm64 archives locally. verify_oracle.py now prints which leaf paths disagree before exiting; two CI round-trips have already been spent on an opaque 'differs' with no detail. * fix(browser): scope the oracle freeze to executed source and certify both Tier-1 Chromium builds Two more compat-certification root causes, both surfaced by the new diff printer: - PR CI checks out the MERGE commit, so fingerprinting all of scrapling/** meant any main-side metadata churn (here: #834 touching iii.worker.yaml dependency ranges) broke every open PR's freeze without affecting parse behavior. The source fingerprint now covers scrapling/src only — exactly what gen_goldens.py imports (its dependencies come from oracle/requirements.lock, not worker metadata) — and the version label, which the CI bot bumps every merge, is recorded as provenance but excluded from the parser-runtime comparison. scrapling/src is byte-identical between this branch and main, so the freeze now holds on the merge ref. - The aarch64 frozen Chromium is the Playwright build 1223, which reports 148.0.7778.0, not the x64 Chrome-for-Testing 148.0.7778.96 — the certified version pin is now the set of both frozen Tier-1 builds, shared between certify_chromium and the certified session test. * chore(browser): trim the xmloxide fork to the slice the worker parses with The worker consumes html (parser + entities), tree, serial, xpath, and the parser core (which needs validation::dtd); everything else in the vendored fork was upstream dead weight: the WHATWG html5 parser, CSS engine, SAX/reader streaming APIs, RelaxNG/XSD/Schematron validators, XInclude, catalogs, serde/async integration, the FFI layer, the xmllint CLI, benches, and their examples. Removing them cuts 34.7k lines from the vendored tree (the PR's largest single block) with zero behavior surface: the fork's remaining 702 unit tests + 40 doctests pass, strict clippy is clean, and the browser suite incl. the 135 Python-parity fixtures is green against the trimmed fork. The fork README and lib.rs now document exactly what was removed and that upstream is the restore source. * perf(browser): tune the release profile — 35.8 MiB -> 23.0 MiB (-36%) strip drops DWARF debug info; thin LTO + one codegen unit let the optimizer dead-strip and deduplicate across crate boundaries. The tuned binary lands below main's stock build (24.1 MiB) despite the +11.7 MiB scrapling surface. No feature, dependency, or runtime behavior changes — the 45-case e2e suite passes against the LTO-built artifact. Cost: a few extra minutes per release build (dev profile untouched).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
0.xfor sharedstateandconfigurationdependencies across all consumers^1.0.0forllm-router, where caret already allows compatible minor updatesWhy
A fully specified caret range on a
0.xdependency stays within that minor line, preventing workers from resolving later compatible minor releases. The0.xwildcard is already supported by both iii and the Registry, so those manifests can express the intended compatibility without changing the worker resolver. Forllm-router,^1.0.0already spans stable1.xreleases and remains unchanged.0.xintentionally accepts every stable pre-1.0 release; the Registry continues to select the highest compatible published version.Validation
pytest -q .github/scripts/tests— 199 passed, 3 subtests passed0.xwildcard and^1.0.0caret boundsgit diff --checkFixes MOT-4463