(MOT-3889) fix(scrapling): move dependency bootstrap to scripts.install - #434
Conversation
Folding install into scripts.start broke on the engine's boot script:
`exec {run_cmd}` binds exec to the FIRST command of a compound string,
so pip ran, exited, and the VM silently tore down before scrapling
install or the worker ever ran (crashloop with logs ending at pip).
The engine now allows scripts.install for bundle workers (same
publisher-shell trust context as scripts.start, run once via the
/var/.iii-prepared marker), so split the manifest back into
install/start:
- scrapling/iii.worker.yaml: install runs pip + browser download once;
start is a single command again
- validate_worker.py: stop rejecting scripts.install for bundles
(scripts.setup and non-preset runtime.base_image still rejected)
- _publish-registry.yml: bundle smoke-boot runs scripts.install
synchronously before scripts.start, mirroring the engine
- _bundle.yml: comments updated
Verified locally: fresh install registers in ~90s and scrapling::fetch
returns a live response. Requires the engine changes on iii PR #1937.
Fixes MOT-3889
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 20 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
skill-check — worker0 verified, 35 skipped (no docs/).
Four for four. Nicely done. |
1 participant
Why
Follow-up to #433. Folding dependency bootstrap into
scripts.startbroke at runtime: the engine's boot script emittedexec {run_cmd}, andexecbinds to the first command of a compound string — soexec pip install -e . && scrapling install && python -m src.mainreplaced the shell with pip, which exited after installing, and the VM silently tore down. The worker crash-looped with logs ending at pip success;scrapling installand the worker itself never ran.What
The engine now allows
scripts.installfor bundle workers (it executes in the same publisher-shell trust context asscripts.start, and the boot script's/var/.iii-preparedmarker already gives run-once semantics), so this restores the clean split:scrapling/iii.worker.yaml—install: "pip install -e . && scrapling install"(runs once, first boot),start: "python -m src.main"(single command, every boot)validate_worker.py— stop rejectingscripts.installfor bundles;scripts.setupand non-presetruntime.base_imagestill rejected_publish-registry.yml— bundle smoke-boot runsscripts.installsynchronously beforescripts.start, mirroring the engine_bundle.yml— comments updatedDepends on
Engine changes on iii-hq/iii#1937:
scripts.installand presetruntime.base_imagerefsscripts.startvia quotedsh -c(the root-cause fix for the silent teardown)Verified
Fresh local install with the patched engine: provision (pip + browser download, fully visible in
iii worker logs) → worker registered in ~90s →scrapling::fetch {"url":"https://example.com"}returned a live response. Subsequent boots skip install via the prepared marker.After merge: cut scrapling v0.2.2 so the registry bundle carries the split manifest.
Fixes MOT-3889