Make the llama wrapper quoting test pass on SIP-enabled Macs (wrapper unaffected)

[seathatflowsinourveins]

Summary

tests.test_adoption_bootstrap_macos.LlamaWrapperQuotingTests fails on a real Mac with System Integrity Protection (SIP) enabled and passes on GitHub's hosted macos-15 runner. The bug is in the test harness. The generated bin/llama-server wrapper works under SIP. This PR changes only the test and re-hashes it in manifests/evidence.json.

Root cause

• The test's stand-in llama-server was a #!/bin/sh script that printed $0 and $DYLD_LIBRARY_PATH.
• With SIP enabled, macOS purges every DYLD_* variable when it launches a SIP-protected binary. /bin/sh is one (ls -lO /bin/sh shows restricted).
• The wrapper sets and exports DYLD_LIBRARY_PATH after its own /bin/sh has started, then execs its target:
  • With the real binary, the target is an unprotected Mach-O, so the value arrives.
  • With the stand-in, the target's image is /bin/sh again, so the value is purged and the stand-in printed an empty line.

Reproduced at the untouched head 4911baf (the report was against 8eacd8b). Path shortened:

AssertionError: Lists differ: ['<root>/tools/llama-cpp-b1/llama-server', ''] != ['<root>/tools/llama-cpp-b1/llama-server', '<root>/tools/llama-cpp-b1']

Each exec hop was probed on macOS 26.5.1 (25F80) arm64 with csrutil status reporting enabled. The native probe was compiled and ad-hoc signed; PLAIN_PROBE is a control variable:

Hop	DYLD_LIBRARY_PATH	PLAIN_PROBE
E1: value handed into /bin/sh	purged	kept
E2: set and exported inside /bin/sh, then exec a native binary (the wrapper's shape)	kept	kept
E3: set and exported inside /bin/sh, then exec a #!/bin/sh script (the old stand-in's shape)	purged	kept
E4: Python parent runs the #!/bin/sh script directly	purged	kept
E5: Python parent runs the native binary directly	kept	kept

Why CI passed: on main at 4911baf, the validate-macos job (job 107565829718, label macos-15) passed "Gate on the adoption test modules" with the old stand-in. So that runner did not purge DYLD_* for /bin/sh. This is consistent with GitHub running hosted macOS images with SIP disabled, which actions/runner-images#8162 documents for macos-13. csrutil status was not observed on the macos-15 runner itself.

The product wrapper works under SIP (unchanged)

I downloaded the pinned asset llama-b11057-bin-macos-arm64.tar.gz (11,178,107 bytes); its sha256 matched the pin. Its llama-server is a Mach-O arm64 binary, signed flags=0x20002(adhoc,linker-signed) with no hardened runtime and no entitlements. It loads @rpath/libllama*.dylib and @rpath/libggml*.dylib and has LC_RPATH @loader_path.

• Install and run: fetch, find_one and install_llama_cpp were extracted verbatim from the script. The archive was pre-placed, so fetch's checksum short-circuit ran and curl never did. The install root contained $(touch PWNED), backticks, quotes and $HOME.
  • <root>/bin/llama-server --version printed version: 0.4.1-dev (build 11057, commit 59657a613) and exited 0.
  • sh <wrapper> --version gave the same result.
  • No PWNED or PWNED2 file was created.
• DYLD_LIBRARY_PATH alone loads the dylibs: a copy of the binary with no dylibs beside it aborts with dyld: Library not loaded: @rpath/libllama-server-impl.dylib. The same copy runs when started by the wrapper's own three lines, evaluated by /bin/sh.
• launchd is unaffected: com.native-stack.llama-embed runs ${ECO_ROOT}/bin/llama-server directly and sets no DYLD_* of its own.
• Observations, not acted on:
  • For b11057, @loader_path resolves the dylibs even through a bare symlink from another directory. The wrapper's DYLD_LIBRARY_PATH is therefore a fallback for this build, and the script comment saying a symlink would fail does not hold for it.
  • Under SIP, a caller's own DYLD_LIBRARY_PATH never reaches the wrapper. The ${DYLD_LIBRARY_PATH:+:…} append only takes effect on hosts without SIP.

Test change

• Stand-in: it is now a script whose shebang is a symlink to sys.executable, the test's own interpreter. Like the real binary, that interpreter is not SIP-protected.
• Control run: the stand-in is first run directly with DYLD_LIBRARY_PATH=control and must print that value. If the interpreter drops it (a SIP-protected or hardened-runtime interpreter), the test skips only the DYLD_LIBRARY_PATH assertion, and only after the PWNED, exit-code and exec-target assertions have run.
• Unset input: the wrapper now runs with DYLD_LIBRARY_PATH unset instead of empty. A POSIX shell keeps exporting a variable it inherited, even after reassignment. With "" inherited, a wrapper missing its export line still passed on Linux and on the SIP-disabled runners. With the variable unset, every host needs the export line. On this host, a plain variable showed the rule: inherited empty then reassigned, the child sees x; unset then assigned, the child sees nothing.
• Unchanged checks: the same hostile install root, the same PWNED/PWNED2 checks, and the same exact exec target and DYLD_LIBRARY_PATH value.

Mutation checks on this SIP-enabled Mac, each run against a scratch copy of the script (none committed):

Script or interpreter change	New test
unchanged	passes (not skipped)
export DYLD_LIBRARY_PATH removed	fails: the value is empty
llama_prefix="$prefix" (double quotes)	fails: $(...) executed
unquoted exec target	fails: exec of a truncated path
stand-in interpreter = SIP-protected /usr/bin/python3	the control run detects the drop; skipped with a reason after the other assertions

Limitation: I could not run the missing-export mutant against a shell without SIP. This host has no non-Apple shell, and a copied /bin/bash is killed at launch. This PR's Linux and macos-15 CI runs exercise the new test there.

Checks (this host: macOS 26.5.1 arm64, Python 3.14.7)

• Target test: python3 -m unittest -v tests.test_adoption_bootstrap_macos.LlamaWrapperQuotingTests passes (not skipped). Before this change it failed.
• Adoption test modules:
  • Ran test_adoption_bootstrap, test_adoption_bootstrap_macos, test_adoption_contract, test_adoption_docs_consistency, test_adoption_guarded_runners, test_adoption_launchd and test_adoption_status.
  • Also ran the CI gate's test_render_config, test_workflow_hardening and test_workflow_security_coverage, plus test_evidence_manifest.
  • Result: 379 tests with skipped=41, plus 6 errors. All 6 were EmbedAcceptanceScriptTests hitting a PermissionError when binding 127.0.0.1 inside the command sandbox. That class re-run outside the sandbox: 8 tests OK.
• Validation: python3 scripts/validate.py passed (5228 hashed files, 143 receipts).
• Shellcheck 0.11.0 (-S style):
  • adoption/bootstrap-macos.sh and adoption/bootstrap-linux.sh are clean.
  • A wrapper generated for an ordinary install path is clean.
  • For the hostile test path, shellcheck reports SC2089, SC2090 and SC2016 on the single-quoted literal. That is the intended literal-data behavior.
• Evidence: manifests/evidence.json was re-hashed with host_receipts.register_file, for tests/test_adoption_bootstrap_macos.py only.

Evidence class: local integration on one SIP-enabled Mac, plus CI job metadata. No launchd service, embedding acceptance or model run, and no receipt added.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[seathatflowsinourveins]

Closing as superseded by #228, which carries this PR's commit 3fd39cd unchanged (cherry-picked with -x) and adds the missing npm skip for the symlinked-prefix test. Landing #228 instead. (native-agent-stack-9a, foundation PR queue)