Repository navigation
feat(scraping): browser automation as a job factory, with zero new dependencies - #140
Merged
Merged
Conversation
…pendencies
`scrape()` is the fourth factory over an existing primitive, after `llm()`, `agent()` and
`backfill()`. A scrape has an input schema, needs a required idempotency key, a tenant, a
retry policy, a timeout and concurrency — and, decisively, `step.run` checkpoints, because
recovery must resume at the broken page rather than restart the run. That is a `job`, so
this returns one. No ninth primitive.
**puppeteer-core is not a dependency.** The launcher is injected — `localBrowser({ launcher })`
— and the CDP port is declared structurally, the same shape `s3Driver({ client })` already
uses. The app owns the browser binary it had to own anyway.
Puppeteer over Playwright is not only preference: Playwright's `connectOverCDP` cannot
upgrade the WebSocket under Bun (oven-sh/bun#9911), which forced a real repo into a
two-runtime monorepo. Verified live before committing to it — Bun 1.3.14, puppeteer-core
25.8.0, headless Chrome 150, both `launch()` and `connect({ browserWSEndpoint })`.
**Remote CDP attach is the primary path**, not an afterthought: production creates a
stealth browser elsewhere and attaches. `close()` stops both halves, or the remote bills
forever.
**Hybrid browser + HTTP.** Drive the browser through login and 2FA, then hit the site's own
JSON endpoints for the bulk. `http` is session-bound — the browser's cookies, the same
proxy, the same `allowHosts`, the same timeout and signal — and both legs replay from one
fixture format, because a hybrid path with two fixture stories is the untestable path.
**Sessions: acquire, persist, reuse, validate, burn.** The burn is the non-obvious half:
anti-bot cookies get poisoned, so a retry that reloads a flagged profile re-trips the same
block every time. An authentication failure is terminal and never retried — a site that
locks an account after three attempts turns a retrying framework into the thing that
destroys the user's account.
**`expect: { minRows, maxDrop }`** is the alarm for the worst failure a scraper has: the run
that succeeds and returns nothing, and stays green for weeks. The collapsed run is not
recorded, so the baseline cannot follow a collapse downward.
Not shipped, deliberately: stealth payloads and captcha. Two teams reached that
independently, one having removed an injected script because the injection was itself
detectable — a framework-shipped payload is a shared fingerprint handed to every user.
Testing: `fakePage` is the default under `bun test`, so Chrome is not required; an
unrecorded fixture request throws rather than reaching the network; and `driver-parity.test.ts`
runs one suite across fake, fixture and the real driver's code path, so the fake cannot
drift. `mock.module` is banned in the package's own CLAUDE.md, with the observed
cross-file leak that motivated it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J7WVaWYBVFBCdtHrVJHD5n
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 25 minutes Limit details: You’ve used the included review currently available. Your 70 included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yml Review profile: ASSERTIVE Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (67)
Comment |
This was referenced Aug 19, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
@ultimat3/scrapingat tier 5. Designed from an audit of ~12,000 lines of real production scraping code across five repos, not from first principles.scrape()is a job factory — the fourth, afterllm(),agent()andbackfill()A scrape has an input schema, needs a required idempotency key (re-logging into a bank after a worker kill is the exact bug), a tenant, a retry policy, a timeout and concurrency — and decisively,
step.runcheckpoints, because recovery must resume at the broken page rather than restart the run. That is ajob. No ninth primitive.Tier 5 is the lowest its real imports allow:
core/schema(0),storage(1, artifacts),jobs(3, it returns aJobHandle),ai(4, the recovery seam). Same argumentCLAUDE.mdmakes fordbat tier 1, run upward.Zero new dependencies
puppeteer-coreis not a dependency. The launcher is injected —localBrowser({ launcher })— and the CDP port is declared structurally, the same shapes3Driver({ client })already uses. The app owns the browser binary it had to own anyway.Puppeteer over Playwright is not merely preference: Playwright's
connectOverCDPcannot upgrade the WebSocket under Bun (oven-sh/bun#9911), which forced a real repo in this workspace into a two-runtime monorepo — disqualifying for a Bun-only framework. Verified live before committing to the design: Bun 1.3.14, puppeteer-core 25.8.0, headless Chrome 150, bothlaunch()andconnect({ browserWSEndpoint })including the WebSocket upgrade.What the audit changed about the design
close()must stop both halves or the remote session bills foreverhttpis session-bound: the browser's cookies, the same proxy (a different exit IP mid-session is itself an anti-bot trip), the sameallowHosts, timeout and signalexpect: { minRows, maxDrop }Not shipped, deliberately
Stealth payloads and captcha. Two teams reached this independently — one moved stealth into a Chromium fork on purpose, another removed an injected script because the injection was itself detectable. A framework-shipped stealth payload is a shared fingerprint handed to every user.
No plugin API, per axiom 8. Extensibility is the
ScrapeDriverseam (implementable from public exports alone) and wrappingscrape()— primitives are functions returning values.Testing
fakePageis the default underbun test: Chrome is not required to run the suite.driver-parity.test.tsruns one suite across fake, fixture and the real driver's code path (over an injected fake CDP browser, nevermock.module), so the fake cannot drift from the real one.mock.moduleis banned in the package's ownCLAUDE.md, with the observed cross-file leak that motivated it.AuthContexthad nosecrets, so a login body had no way to reach the credential it is supposed to type. Fixed in source, not by weakening the example.Note on local verification
bun run verifyis green on this branch except for one test —scripts/verify.test.ts's four-full-repo-scan test — which times out under an 8-worker shard on a machine currently at load average 15 from unrelated processes. It costs 10.9s in isolation and has passed CI's 30s budget on the three PRs merged ahead of this one. Every other step and all 1239 other tests pass. CI is the uncontended gate here.🤖 Generated with Claude Code
https://claude.ai/code/session_01J7WVaWYBVFBCdtHrVJHD5n
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.