fix(OMN-15134): self-heal workspace-reset hook against root-owned debris

[jonahgabriel]

Summary

Fixes OMN-15134: root-owned .venv artifacts under the omninode-deploy-runner Actions job workspace jammed the built-in "Set up runner" job-started cleanup hook (run 30171892373), blocking the OMN-14900 stability deploy hop before any workflow-defined step ran:

rm: cannot remove '.../.venv/lib/python3.12/site-packages/pydantic/__pycache__/__init__.cpython-312.pyc': Permission denied
##[error]Process completed with exit code 1.

Root cause (verified live, not inferred)

This image's ENTRYPOINT legitimately starts as root (docker-socket GID fix, OMNI_HOME chown, operator-env copy) before gosu-dropping to runner for every job step — so a bare docker exec omninode-deploy-runner ... without -u runner silently inherits root. PR #2446's own body cites the exact command that did this, describing it as a "read-only" verification probe:

uv run python from the repo root: pydantic 2.13.4 importable

uv run is not read-only — it transparently runs uv sync when no .venv exists, creating one. Run via a bare docker exec (no -u runner), this created the .venv as root. Confirmed live:

• 18 root-owned paths under the job workspace .venv, all timestamped 18:56
• /root/.cache/uv/interpreter-v4/...msgpack populated (root's own uv cache — direct proof uv executed with HOME=/root)
• write-visibility probe proved the job workspace is not bind-mounted from host (a file written inside the container is invisible on the host and vice versa) — ruling out the docker-outside-of-docker path-mismatch class entirely; this is a same-container out-of-band-root-exec bug, not a DooD mount bug

The workspace-reset hook itself (runner-job-started.sh) correctly runs as the unprivileged runner user (same identity every job step executes as) and correctly could not remove root-owned files — it just gave no diagnostic and no recovery path, wedging every subsequent job on this single-runner label until an operator hand-ran a root docker exec rm -rf.

Fix

Harden runner-job-started.sh to fail loud, then self-heal:

1. Plain rm -rf (unchanged happy path).
2. On failure: log the exact rm error + a find -not -user $(id -u) listing of the offending paths.
3. Attempt a narrowly scoped sudo -n rm -rf fallback — one command, one argument pattern (/bin/rm -rf -- <path under this runner's own _work tree>), added as a NOPASSWD sudoers rule in the Dockerfile. Never a general root shell.
4. If sudo is unavailable or the fallback also fails, exit 1 with an explicit manual-remediation command — never silently succeeds or hangs.

Rejected alternatives

• (a) Flip the image's default USER to runner — rejected. The ENTRYPOINT's root-phase init (docker-socket GID fix, OMNI_HOME chown, operator-env copy) genuinely needs root before it drops privilege via gosu. Making the image default non-root either breaks that init, or (if compose's user: root overrides it back for the compose-managed lifecycle) doesn't change what docker exec defaults to anyway — docker inspect --format '{{.Config.User}}' reflects the container's effective configured user regardless of whether it came from the image default or a runtime override, so a split default wouldn't have closed the ad hoc docker exec vector that actually produced this incident.
• (c) Forbid .venv in the bind-mounted job workspace — N/A. Verified live: this job workspace is not bind-mounted from host at all (see write-visibility probe above), so there's no host-path mismatch to fix here.

Verification (live, .201, read-only-then-scoped-mutation per standing rule)

• Cleaned the 18 confirmed root-owned paths + /root/.cache/uv on the live omninode-deploy-runner container (count_after=0, re-verified via find -user root).
• Copied the hardened hook into the live container and reproduced the exact RED condition (simulated root-owned debris via docker exec -u root) against the real Ubuntu 22.04 / GNU-realpath environment — confirmed the hook fails loud with the offending path and a clear "sudo not installed" diagnostic (pre-image-rebuild, no sudoers rule live yet), and exits non-zero with the manual-remediation instruction. Confirmed the happy path (workspace owned by runner) still resets clean.
• Added tests/scripts/test_runner_job_started_root_owned_debris.py: happy path, undeletable-debris fail-loud + fallback-unavailable path, and the pre-existing path-confinement regression guard. Skips on non-GNU-realpath hosts (this repo's macOS .200 gate host); runs for real on Linux CI.

Gates run on .200 (patch-transfer + sha256 verify, never edited in place): ruff format --check, ruff check, mypy, full pre-commit run (shellcheck included) — all clean.

Follow-up (tracked separately, not in this PR)

The image itself still needs rebuilding + the running container recreating to pick up the sudoers rule — tracked as the deploy step for this same ticket (OMN-15134), performed after this PR merges.

Refs: OMN-15134, OMN-14900 (parent stability deploy hop), OMN-15131 / PR #2446 (the same-session probe that produced the debris)

Evidence-Ticket: OMN-15134
Evidence-Source: OCC#4865

Evidence-Commit: aa73c9a6f5129ec1d768289d82877ed33e392974

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 34 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4c4dd8b9-5116-47f6-98ca-c6e29d4008b9

📥 Commits

Reviewing files that changed from the base of the PR and between ead076a and 2e8b868.

📒 Files selected for processing (3)

• docker/runners/Dockerfile
• docker/runners/runner-job-started.sh
• tests/scripts/test_runner_job_started_root_owned_debris.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch jonah/omn-15134-runner-workspace-reset-root-owned-debris

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

⚠️ Hostile Reviewer — DEGRADED (informational)

Blocking findings (critical): 0
Total findings: 0
Models succeeded: none

Note: All reviewer models failed or were unavailable. Degraded results are informational during the pilot phase (OMN-8468/OMN-8524) and do not block merge. Error: all review endpoints [192.168.86.201:8000 192.168.86.201:8001 ] unreachable — preflight short-circuit (no models available)

---

Gate semantics (pilot phase)

Verdict	Meaning	Blocks merge?
passed	No critical findings	No
blocked	CRITICAL findings found	Yes
degraded	All models unavailable (infra)	No (pilot)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8468/OMN-8524)