Skip to content

fix(bootstrap): lite venv is CUDA-free — secrets paths stop pulling torch - #2961

Merged
POWERFULMOVES merged 5 commits into
mainfrom
fix/lite-bootstrap-no-cuda
Sep 5, 2026
Merged

POWERFULMOVES merged 5 commits into
mainfrom
fix/lite-bootstrap-no-cuda

Conversation

@POWERFULMOVES

Copy link
Copy Markdown
Owner

What

env-bootstrap-lite installs requirements-lite.txt on every funnel/preflight path — and that file carried sentence-transformers + scikit-learn, dragging torch and the multi-GB nvidia wheel stack into secrets materialization. Measured 2026-09-05: both secrets-funnel-from-prod and secrets-funnel-sync-from-bundle timed out mid-CUDA-wheel-download on a node at 98% disk (25G free).

  • Core lite (PyYAML, rich, jsonschema, psutil, numpy) — verified: 15MB venv, seconds, zero torch
  • Heavy tier split to requirements-lite-embeddings.txt (the actual embedding-decode consumers), opt-in via --with-embeddings, CPU-wheel-index documented
  • chit_backend.py heavy imports made defensive (matching chit_decoder.py's existing pattern) — imports clean without the tier

Testing

  • Fresh venv from the new core file: 15MB, all four core imports OK, import torch fails (as intended)
  • import chit_backend in a core-only venv: clean, SentenceTransformer is None and KMeans is None
  • Funnel path (chit-manifest-sync → bootstrap) no longer resolves any nvidia-* wheel

Warning

Generated with Crush

…orch

env-bootstrap-lite installs requirements-lite.txt on every funnel and
preflight path, and that file carried sentence-transformers and
scikit-learn — torch and the multi-GB nvidia wheel stack (cufft 204MB,
nvshmem, nvrtc...) on every secrets-materialization run. Measured
2026-09-05: both secrets-funnel-from-prod and funnel-sync-from-bundle
timed out mid-wheel-download on a node at 98% disk; a fresh lite venv
cost multi-GB where the funnel path needs 15MB.

- requirements-lite.txt: core only (PyYAML, rich, jsonschema, psutil,
  numpy) — verified: 15MB venv, seconds to install, zero torch
- requirements-lite-embeddings.txt: the heavy tier for the actual
  embedding-decode tools (chit_backend, catalog_lensing_engine,
  chit_decoder_mm), opt-in via --with-embeddings; header notes the CPU
  wheel index preference
- chit_backend.py: sentence-transformers/sklearn imports made defensive
  (chit_decoder.py already was) — imports cleanly without the tier,
  fails only at real use
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@coderabbitai

coderabbitai Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)
  • coderabbit

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Team

Run ID: c434e0af-d3f6-4931-a1b1-186ecde4386e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added chit-aware Touches a CHIT-aware service — control-body signature gate applies docs Documentation governance AGNOTE register / agent definitions / damage-control hooks labels Sep 5, 2026
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

🔏 CHIT-aware change — control-body routing

This PR touches a CHIT-aware service (ports 8086/8087/8103/8106/8113/9224 surface). Before a Control-Body ACK:

  • the diff (or PR body) should reference a CHIT signature trail — chit.signed.v1, a CHIT: trailer, or a make -C pmoves sign-trail entry;
  • the chit-pr-audit-agent (.claude/agents/chit-pr-audit-agent.md) is the gate that grants/refuses ACK eligibility;
  • signing quickstart: pmoves/docs/AGENTS/CHIT_SIGNING_TUTORIAL.md.

Advisory routing only — the blocking contract check is CHIT Contract.

…id-flight

Operator catch on #2961: the CUDA-wheel disaster was a symptom; the bug is
that chit-manifest-{register,sync,check} invoked env-bootstrap-lite
UNCONDITIONALLY mid-run, so a missing venv was discovered by INSTALLING
(hundreds of MB) in the middle of a secrets materialization — on a full
disk, inside timeouts, with no way to know before you started.

- bootstrap_light_env.py --check: precheck ONLY — verifies the venv
  python exists and every requirement's import resolves THROUGH the venv
  (subprocess probe, not host importlib); never creates, never installs;
  exit 3 with the exact remediation command
- make env-bootstrap-check (preflight.mk) — the gate target
- the three chit-manifest-* entries in codex.mk now call the gate
  instead of the bootstrap

Verified both paths: absent venv -> Error 3 with remediation before any
network; real venv -> "precheck OK" and the chit step proceeds.
@POWERFULMOVES

Copy link
Copy Markdown
Owner Author

Added per operator review — the real bug was upstream of the wheels: the funnel steps (chit-manifest-{register,sync,check}) invoked env-bootstrap-lite unconditionally MID-RUN, so a missing venv was "discovered" by installing hundreds of MB inside a secrets run — on a full disk, inside timeouts, no way to know before starting.

Now: bootstrap_light_env.py --check (precheck ONLY — venv present + every requirement imports through the venv python, subprocess-probed; never creates, never installs; exit 3 with the exact remediation command) behind make env-bootstrap-check, wired as the gate for all three chit-manifest entries. Verified both directions: absent venv fails before any network; real venv → precheck OK and the step proceeds.

(Also on this thread: docker prune on SPARK reclaimed 0B — the space genuinely lives in the ~54G of verified plain copies awaiting operator sign, not in image bloat.)

….gitignore, count both bootstrap targets in the recursion guard

Pair-review fixes for the two red CI jobs (5090, Windows node):

- pmoves/.venv-pmoves was committed as a symlink (mode 120000) pointing at a
  B850 home path. On CI it dangles, so lgtm_marker_check raises
  FileNotFoundError (suppression-marker-check exit 3) and
  test_this_repository_is_clean fails. On a Windows checkout it materializes
  as a 60-byte FILE named .venv-pmoves, which collides with the real venv
  directory and would silently demote CODEX_PY to the bare interpreter,
  the node-local-state failure class. Removed; .gitignore gains the bare
  path so a symlink cannot slip past the trailing-slash directory rule.
- tests/make/test_args_no_leak_to_submake: the funnel entry points now
  recurse into env-bootstrap-check (no ARGS consumer) instead of
  env-bootstrap-lite, so the "call sites are actually present" guard counts
  both targets (>=4 lite all ARGS-cleared, >=3 check, >=7 combined).

32 passed, 1 skipped across the two affected test files; lgtm_marker_check
exits 0.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017g8jC7dupS2ubafo6zPQY6
@POWERFULMOVES

Copy link
Copy Markdown
Owner Author

Pair-review from the 5090 (Windows node), pushed 2430ce6 onto this branch:

  • pmoves/.venv-pmoves was committed as a symlink (mode 120000, pointing at a B850 home path). On CI it dangles, so lgtm_marker_check raises FileNotFoundError: suppression-marker-check exit 3 and test_this_repository_is_clean red. On this Windows checkout it materialized as a 60-byte file named .venv-pmoves, which collides with the real venv directory and would silently demote CODEX_PY to the bare interpreter, the same node-local-state class this PR fixes elsewhere. Removed; .gitignore now also ignores the bare path so a symlink cannot slip past the trailing-slash rule.
  • test_env_bootstrap_lite_recursions_are_actually_present expected ≥7 env-bootstrap-lite recursions; this PR correctly retargets three funnel entry points to env-bootstrap-check, leaving 4. The guard now counts both targets (≥4 lite all ARGS-cleared, ≥3 check, ≥7 combined) so it still proves real call sites.

Both affected test files: 32 passed / 1 skipped; lgtm_marker_check exit 0. The conftest ImportErrors (uvicorn/structlog) in the ratchet log are baselined ERRORs, not new. Design note, not a blocker: with chit-manifest-* gated by env-bootstrap-check, a fresh clone's first secrets-funnel now fails fast with exit 3 until make env-bootstrap-lite has run once. Intended per the PR, but bring-up docs should say so.

@POWERFULMOVES

Copy link
Copy Markdown
Owner Author

Verified on the 5090 (Windows, uv): fresh venv from the new core requirements-lite.txt → install 1s (cache-warm), 53 MB, import yaml, rich, jsonschema, psutil, numpy OK, import torch → ModuleNotFoundError, 0 nvidia/torch dirs in site-packages; import chit_backend in that core-only venv is clean with SentenceTransformer is None and KMeans is None. Matches the B850 measurement, so the split holds on a second, differently-shaped node. Updating the branch onto main now for a fresh CI run.

@POWERFULMOVES
POWERFULMOVES merged commit 4a97fb9 into main Sep 5, 2026
29 checks passed
@POWERFULMOVES
POWERFULMOVES deleted the fix/lite-bootstrap-no-cuda branch September 5, 2026 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chit-aware Touches a CHIT-aware service — control-body signature gate applies docs Documentation governance AGNOTE register / agent definitions / damage-control hooks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant