From 2d33dbd5bae0b82a7aca821f9cd8dfbb6deec966 Mon Sep 17 00:00:00 2001 From: seathatflowsinourveins <234074349+seathatflowsinourveins@users.noreply.github.com> Date: Sun, 27 Sep 2026 11:21:31 -0400 Subject: [PATCH 1/3] Mac-native model hosting on mac-coordinator-64gb-20260925 (host request #379) The memory/RAG model set this 64 GB Mac hosts natively under coordinator load, measured 2026-09-27, with the proposed host-roles change and a dated decision record: - pinned RAG embedder (llama.cpp b11057 Metal, embeddinggemma-300M-Q8_0, port 8232): embed acceptance passes, cosine 0.99966, all layers on MTL0; - Nemotron-3-Embed-1B: no supported Mac runtime (Ministral3Model is not registered in llama.cpp's converter at b11146); - Qwen3.8-27B UD-Q4_K_M on Metal: llama-bench pp512 371.5 / tg128 15.9; li26 (descriptive, 360 real 8-K filings) micro-F1 0.9909 for C0 on b11057 and b11146 and C2 (MTP) on b11146, identical predictions, 15.7 / 15.8 / 23.9 tok/s; - worst case with the memory stack's Ollama models: 53% free, no swap growth, 0 memorystatus kills; one swap event when a Next.js build overlapped the full set (peak 13.9 GB, no kills); - reranker-gate timing: Qwen3.8-27B misses ai-memory's 20 s gate at 8K on llama.cpp (27-28 s) and is borderline on Ollama MLX (18.5 s); the production 9B, Qwen3-Reranker-0.6B and GPT-6 via OmniRoute meet it. The host value file stays private (every path key carries personal paths). Pin drift is recorded, not fixed. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017d27CMzkbLezdLv7vAStPu --- adoption/host-roles.json | 7 +- adoption/platforms/macos-arm64.md | 19 +- .../2026-09-27-mac-native-model-hosting.md | 79 +++ .../mac-model-hosting-20260927/README.md | 249 +++++++ .../mac-model-hosting-20260927/capacity.json | 104 +++ .../coresidency.json | 284 ++++++++ .../embed-acceptance.json | 54 ++ .../mac-model-hosting-20260927/gate.json | 606 ++++++++++++++++++ .../li26-mac-descriptive.json | 120 ++++ .../li26-window-samples.json | 39 ++ .../li26_mac_descriptive.py | 177 +++++ .../swap-event.json | 56 ++ 12 files changed, 1789 insertions(+), 5 deletions(-) create mode 100644 docs/decisions/2026-09-27-mac-native-model-hosting.md create mode 100644 evidence/artifacts/mac-model-hosting-20260927/README.md create mode 100644 evidence/artifacts/mac-model-hosting-20260927/capacity.json create mode 100644 evidence/artifacts/mac-model-hosting-20260927/coresidency.json create mode 100644 evidence/artifacts/mac-model-hosting-20260927/embed-acceptance.json create mode 100644 evidence/artifacts/mac-model-hosting-20260927/gate.json create mode 100644 evidence/artifacts/mac-model-hosting-20260927/li26-mac-descriptive.json create mode 100644 evidence/artifacts/mac-model-hosting-20260927/li26-window-samples.json create mode 100644 evidence/artifacts/mac-model-hosting-20260927/li26_mac_descriptive.py create mode 100644 evidence/artifacts/mac-model-hosting-20260927/swap-event.json diff --git a/adoption/host-roles.json b/adoption/host-roles.json index b350d2dbc..5a9639f30 100644 --- a/adoption/host-roles.json +++ b/adoption/host-roles.json @@ -24,10 +24,13 @@ "request_label": "host:mac-coordinator", "owns": [ "macos-acceptance", - "coordination" + "coordination", + "model-hosting", + "memory-e2e", + "rag-e2e" ], "credentials": [], - "note": "The replacement coordinator Mac." + "note": "The replacement coordinator Mac. It also owns model-hosting, memory-e2e and rag-e2e for its own native memory/RAG foundation layer (user decision 2026-09-27, request #379; measured basis in docs/decisions/2026-09-27-mac-native-model-hosting.md). GPU runtime qualification, cross-host model qualification and independent review stay with the workstation. Routing still follows only the request label." } }, "peers": [ diff --git a/adoption/platforms/macos-arm64.md b/adoption/platforms/macos-arm64.md index fac9f2b68..7aed47418 100644 --- a/adoption/platforms/macos-arm64.md +++ b/adoption/platforms/macos-arm64.md @@ -110,7 +110,14 @@ plans for is the labelled projection `macos-arm64-64gb-projected` in [`adoption/hardware-profiles.json`](../hardware-profiles.json) (about 38 GB of unified memory as the generation budget, `full` semantic-RAG tier, both drawn from one shared pool); for this page's embedding choice it follows the 48 GB -rules. None of these sizes has a real qualification run yet. +rules. The projections themselves have no qualification run. The measured +64 GB host `mac-coordinator-64gb-20260925` has had one since 2026-09-27. The +pinned RAG embedder, the memory stack's Ollama models and a 27B-class 4-bit +generation model ran together under coordinator load with no swap growth and +no memory kills. A large Node production build running alongside that set did +push the host into swap (peak about 14 GB, no kills), so the shared-pool +warning still applies to work beyond that set +([#379 evidence](../../evidence/artifacts/mac-model-hosting-20260927/README.md)). ## Prerequisites @@ -707,8 +714,14 @@ currently loaded from that same destination path, rename into place, confirmed loaded from its own destination path, or not loaded at all with a file present to clean up). On the hosted runner (run `35875188590`, "What a hosted run proves" above) `launchd-agents.sh` bootstrapped and booted out the -`qdrant` and `llama-embed` agents; `ai-memory` has not run, and none of the -three has run on a Mac workstation. +`qdrant` and `llama-embed` agents; `ai-memory` has not run there. On +2026-09-27 the `llama-embed` agent began running on the coordinator Mac +`mac-coordinator-64gb-20260925`. It passes the embedding acceptance there, +with all layers on Metal ([#379 evidence](../../evidence/artifacts/mac-model-hosting-20260927/README.md)). +That host's pre-existing `local.agent-ecosystem.qdrant` and +`local.agent-ecosystem.ai-memory` agents serve the other two roles. The +`qdrant` and `ai-memory` templates therefore still have not run on a Mac +workstation. **2026-09-23 decision: brew-services semantics, no backup or reconcile.** diff --git a/docs/decisions/2026-09-27-mac-native-model-hosting.md b/docs/decisions/2026-09-27-mac-native-model-hosting.md new file mode 100644 index 000000000..4b12371e8 --- /dev/null +++ b/docs/decisions/2026-09-27-mac-native-model-hosting.md @@ -0,0 +1,79 @@ +# Decision: the 64 GB coordinator Mac hosts its own memory/RAG models natively (2026-09-27) + +**Decided by:** the user, 2026-09-27, in host request +[#379](https://github.com/seathatflowsinourveins/native-agent-stack/issues/379): memory/RAG is a native +foundation layer on every host, not a service borrowed from the workstation. This record gives the +measured basis on `mac-coordinator-64gb-20260925` and the proposed role change. It was written by the Mac +coordinator session on branch `claude/mac-model-hosting-20260927` (base `origin/main@8bb52541`). + +**Scope:** [`adoption/host-roles.json`](../../adoption/host-roles.json) (the proposal below) and the +evidence in +[`evidence/artifacts/mac-model-hosting-20260927/`](../../evidence/artifacts/mac-model-hosting-20260927/README.md) +with its host receipt. Lane: `lane:foundation`. Nothing here changes a pin, a production service or the +memory-layer decision: retrieval quality belongs to the S3 preregistration (#390). + +## Context + +Measured on this Mac (Apple M5 Pro, 64 GB unified memory, macOS 27.0) with a normal coordinator session +active. The full numbers and their evidence classes are in the artifact README. + +- **RAG embedder.** The pinned profile's `com.native-stack.llama-embed` LaunchAgent (llama.cpp b11057 + `--embedding`, embeddinggemma-300M-Q8_0, port 8232) passes + `tools/adoption/embed_acceptance.py` against the Linux reference: 768 dimensions, cosine 0.99966. All 25 + layers run on Metal (`MTL0`), per llama-server's own log. +- **Memory stack.** ai-memory, Ollama (qwen3-embedding:4b, qwen3.5-9b-64k) and Qdrant already ran here as + `local.agent-ecosystem.*` agents. No second Qdrant or ai-memory was started. +- **Generation.** `unsloth/Qwen3.8-27B-GGUF@4ca7207` UD-Q4_K_M (16.46 GB, 27.8B dense) runs fully on + Metal. + - llama-bench: pp512 371.5, pp4096 341.7, tg128 15.9 tokens/s. + - Serving at a 32K context: peak RSS 22.9 GiB, swap unchanged, 0 memorystatus kills. + - li26 on 360 real 8-K filings, descriptive: micro-F1 0.9909 and JSON-valid 1.0 for C0 on b11057 and b11146 + and for C2 with MTP drafting on b11146. Predictions are identical on all 360 filings. Median decode is + 15.7, 15.8 and 23.9 tokens/s respectively, peak RSS stays at or under 17.6 GiB, and there are 0 memory + kills. +- **Worst case.** The C2 profile at a 32K context on b11214 (19.4 GiB peak RSS) and ai-memory's two Ollama + models (12.5 GiB) were resident and driven at once, next to the RAG embedder and Qdrant. System free + memory was 53%, swap did not grow, there were 0 kills, and every request succeeded. +- **The one swap event.** Swap grew only once, peaking at 13.9 GB for about 40 s with no kills. That was + when an OmniRoute Next.js production build and ai-memory's model load overlapped a li26 run. +- **Workstation embedder.** Nemotron-3-Embed-1B has no supported Mac runtime. Its `Ministral3Model` is not + registered in llama.cpp's converter at b11146, and NVIDIA publishes no GGUF or MLX artifact. + +## Decision + +1. **This Mac hosts, natively and together:** + - the pinned RAG embedder (embeddinggemma-300M on llama.cpp Metal, port 8232); + - the memory stack's own models (Ollama qwen3-embedding:4b and qwen3.5-9b-64k); + - Qdrant; + - one 27B-class 4-bit generation model on llama.cpp Metal. + The measured budget for this co-resident set is a generation server of up to 19.4 GiB RSS at a 32K context, plus 12.5 GiB of memory-stack models, the embedder and Qdrant, with 53% of system memory still free, no swap growth and no memorystatus kills. +2. **`adoption/host-roles.json`:** `mac-coordinator` also owns `model-hosting`, `memory-e2e` and `rag-e2e` + for its own foundation layer. GPU runtime qualification, cross-host model qualification and independent + review stay with the workstation. Routing is unchanged, because `scripts/host_requests.py` routes on the + request label alone. `owns` records what the host accepts. +3. **What exceeds this Mac** (the trigger for a larger one): a 27B-class LLM as ai-memory's reranker at 8K tokens, which runs 27-28 s on llama.cpp Metal and 18.5 s median on Ollama's MLX engine against the 20 s gate. Also a heavy build overlapping the full model set, which pushed the host into swap (peak 13.9 GB, no kills). A model with no supported Mac runtime, such as Nemotron-3-Embed today, is a runtime gap, not a memory limit. The 30B-class challengers would fit the 38 GB generation allowance alone, which is projected, not measured + +## Alternatives considered + +- **Borrow the workstation's models over the network.** Rejected by the user's decision, and by the + memory-stack rule "never depend on another host's localhost service". +- **Serve the workstation's embedder (Nemotron-3-Embed-1B) on the Mac.** It has no supported Mac runtime + (step 2). Community conversions are not upstream and are untested for this model's pooling. +- **Ollama's MLX engine for generation.** This Mac's Ollama 0.34.4 already holds nvfp4 builds of + Qwen3.8-27B. The frozen li26 harness cannot score them, because `eval_arm.py` checks llama.cpp's + `/props`. Only its latency was measured, for the reranker gate: at about 7.5K prompt tokens it answered in 18.5 s median against llama.cpp Metal's 27.9 s, 1.5× faster at prefill but still borderline against the 20 s gate (1 of 3 calls over) +- **MLX-LM or LM Studio.** Considered in the macOS page's embedding decision and not activated. There is + still no measured comparison on this project's workload. +- **Re-pin llama.cpp to b11146 on macOS for MTP drafting.** Recorded as a finding with this host's b11146 + runs as native evidence, but not done here. A pin change is its own reviewed change. + +## Overturn conditions + +This decision is revisited when any one of these is measured on this Mac: + +- swap growth or a memorystatus kill under normal coordinator load with the co-resident set above; +- a required model set whose weights and KV cache exceed the measured headroom; +- the chosen memory reranker LLM missing ai-memory's 20 s gate at 4-8K tokens (more than 5% fallback) + while a larger machine would meet it; +- a Mac-native Nemotron-3-Embed (llama.cpp registration or an NVIDIA GGUF/MLX) that beats embeddinggemma + on this project's own retrieval comparison, with the headroom to serve it. diff --git a/evidence/artifacts/mac-model-hosting-20260927/README.md b/evidence/artifacts/mac-model-hosting-20260927/README.md new file mode 100644 index 000000000..e1f9be996 --- /dev/null +++ b/evidence/artifacts/mac-model-hosting-20260927/README.md @@ -0,0 +1,249 @@ +# Mac-native model hosting on `mac-coordinator-64gb-20260925` (host request #379) + +Measured on 2026-09-27 on the 64 GB coordinator Mac (Apple M5 Pro, macOS 27.0, `arm64`). A normal +coordinator session was active throughout: Claude Code with its subagents, the ai-memory, Ollama and +Qdrant services, and ordinary desktop applications. The request is +[#379](https://github.com/seathatflowsinourveins/native-agent-stack/issues/379). The decision it supports is +[`docs/decisions/2026-09-27-mac-native-model-hosting.md`](../../../docs/decisions/2026-09-27-mac-native-model-hosting.md). + +## What this Mac hosts natively, and what exceeds it + +**Hosted natively, measured together under coordinator load:** +- the pinned RAG embedder (embeddinggemma-300M-Q8_0 on llama.cpp Metal, port 8232), which passes acceptance + with all layers on Metal; +- the memory stack's own models (Ollama `qwen3-embedding:4b` and `qwen3.5-9b-64k`), with the 9B inside + ai-memory's 20 s reranker gate at 8K tokens (5.5 s); +- Qdrant; +- one 27B-class 4-bit generation model, Qwen3.8-27B UD-Q4_K_M on llama.cpp Metal. On li26 it matches the + workstation's quality (micro-F1 0.9909) at 15.7 tokens/s, or 23.9 tokens/s with MTP drafting; +- together, with the generation server at a 32K context (19.4 GiB peak RSS): 53% system memory free, no swap + growth, 0 memorystatus kills; +- GPT-6 through this Mac's own OmniRoute gateway, for S3's LLM roles: 4.7-9.6 s on rerank-shaped prompts. + +**What exceeds this Mac:** each item is the trigger for a larger machine. +- **A 27B-class reranker LLM at 8K tokens.** Qwen3.8-27B answers in 27-28 s on llama.cpp Metal and 18.5 s + median (20.7 s max) on Ollama's MLX engine, against ai-memory's 20 s gate. It fits in memory but not in time. +- **A heavy build overlapping the full model set.** A Next.js production build pushed the host into swap (peak + 13.9 GB, no kills). The model set alone never swapped. +- **Not a memory limit:** Nemotron-3-Embed has no supported Mac runtime at all. +- **A projection, not measured:** the 64 GB profile's generation allowance is about 38 GB (64 × 0.6). It holds + the measured 19.4 GiB comfortably. The 30B-class challengers in the verified sweep (granite-4.2-30b, about + 27.8 GB, and Nemotron-3.5-Lightning, about 27 GB at 32K) would fit that allowance alone, but with less + headroom once the memory stack's 12.5 GiB is resident. + +## Step 1: the pinned macOS local RAG profile + +**Guard first.** Before any launchd step, `launchctl list | grep -E 'agent-ecosystem|native-stack'` showed +four pre-existing agents: `local.agent-ecosystem.ai-memory`, `.ollama` and `.qdrant` running, and +`.maintenance` stopped with last exit 1. No second Qdrant or ai-memory was started. The one agent this +request added is `com.native-stack.llama-embed`, rendered from +[`adoption/launchd/com.native-stack.llama-embed.plist.template`](../../../adoption/launchd/com.native-stack.llama-embed.plist.template). + +| Port (loopback) | Served by | Origin | +| --- | --- | --- | +| 6333 | Qdrant 1.19.1, `local.agent-ecosystem.qdrant` | pre-existing (#253); its use receipt is `mac-coordinator-64gb-20260925--qdrant--use--20260925` | +| 8232 | llama.cpp b11057 `llama-server --embedding`, embeddinggemma-300M-Q8_0, `com.native-stack.llama-embed` | added for this request | +| 11434 | Ollama 0.34.4, `local.agent-ecosystem.ollama` | pre-existing | +| 49374 | ai-memory, `local.agent-ecosystem.ai-memory` | pre-existing | +| 16333 | nothing | the template's `com.native-stack.qdrant` port; not started, because the pre-existing Qdrant serves this host | + +**Embedding acceptance** ([`embed-acceptance.json`](embed-acceptance.json)), with the page's own command: + +```sh +python3 tools/adoption/embed_acceptance.py http://127.0.0.1:8232 \ + evidence/artifacts/macos-embed-reference-20260923/macos-embed-reference-20260923.json +``` + +It passed: 768 dimensions, cosine 0.99966 against the Linux reference (threshold 0.99). The served file is +`ggml-org/embeddinggemma-300M-GGUF@0f741b5a` `embeddinggemma-300M-Q8_0.gguf`, sha256 `b5ce9d77…0d63`. + +**Metal or CPU**, from llama-server's own startup log (the acceptance script cannot tell): each of the three +model loads in the log reports `using device MTL0 (Apple M5 Pro)` and `offloaded 25/25 layers to GPU`, and +no layer line names the CPU. The layers run on Metal. + +## Step 2: Nemotron-3-Embed-1B has no supported Mac runtime + +NVIDIA publishes no Mac-native form of `nvidia/Nemotron-3-Embed-1B-BF16@c0c9fea9`, and the supported +llama.cpp converter does not accept its architecture. Nothing was downloaded or converted. + +- Its `config.json` declares `architectures: ["Ministral3Model"]`, `is_causal: false`, hidden size 2048. + Its base model is `mistralai/Ministral-3-3B-Instruct-2512`. +- NVIDIA's four `Nemotron-3-Embed` repositories carry no `gguf`, `mlx` or `onnx` tag. +- llama.cpp registers only `Mistral3ForConditionalGeneration` and `Ministral3ForCausalLM` + (`conversion/mistral3.py` lines 14-16 at v0.5.0 and at b11146 `7fe450e1`). At b11146 the bare + `Ministral3Model` makes `conversion/__init__.py` raise and `convert_hf_to_gguf.py` exit 1 with + "not supported". +- Community GGUF and MLX conversions exist. They are not upstream, the request excludes self-conversion, + and their fidelity to the average-pooled bidirectional model is untested. + +This reopens when llama.cpp registers `Ministral3Model` or NVIDIA publishes a GGUF or MLX artifact. + +## Step 3: capacity under coordinator load + +**Model.** `unsloth/Qwen3.8-27B-GGUF@4ca720788d1e01f1bff70c033e0d0028fd02e502`, +`Qwen3.8-27B-UD-Q4_K_M.gguf`, 16,464,440,224 bytes, sha256 `322e194f…3482`. It is 27.8B dense at 4 bits, +the smallest file of the request's 32B class. It is also the li26 production generation model, so the Mac +runs can be compared with the workstation's li26 results. + +### Upstream throughput and serving ([`capacity.json`](capacity.json)) + +| Measure | Result | Evidence class | +| --- | --- | --- | +| `llama-bench -p 512,4096 -n 128 -ngl 99 -fa on -r 3` (b11057, Metal) | pp512 371.5 ± 1.1, pp4096 341.7 ± 15.3, tg128 15.9 ± 0.6 tokens/s | `native_proven` | +| llama-server, 32,768-token unified context (4 default slots), 9 requests, `cache_prompt: false` | median time to first token 4.67 s at a median 1,603 prompt tokens; 15.1 s at 5,189 tokens; median decode 15.8 tokens/s | `local_integration` (the prompts are this repository's own docs) | +| Server peak RSS | 22,941 MiB | measured, 72 one-second samples | +| System free memory (`memory_pressure -Q`) | 89% before, 61% loaded, 89% after | measured | +| Swap | 16.81 MB used before, during and after (no change) | measured | +| Jetsam | 0 memorystatus kills | measured: 66 log lines matched, 63 were runningboardd "Ignoring jetsam update", 2 runningboard diagnostics, 1 the `log show` query itself | + +The memory stack's own Ollama models were idle-unloaded during this probe. The worst case with them loaded +is measured below. + +### li26 on real SEC filings, descriptive ([`li26-mac-descriptive.json`](li26-mac-descriptive.json)) + +The frozen li26 protocol +([`blueprints/convergence-practice/local-inference-latest-20260926`](../../../blueprints/convergence-practice/local-inference-latest-20260926/)) +ran unchanged through `eval_arm.py` on the frozen inputs (360 SEC 8-K filings of 2020-03-02, acquired on +this host with EdgarTools 5.58.0 under the private SEC contact; inputs sha256 `2a5c8c9b…`). This Mac is not +the preregistered host: every number below describes this host and none is a parity or replacement +result. `metrics.json`'s `profile` field repeats the plan's declaration for each arm. What actually ran +here is full Metal offload (`--gpu-layers 99`) with the plan's other serving values, and C2 without +`--n-cpu-ffn`. Evidence class: `native_proven` for "this host served this model on 360 real filings". + +| Run | Build | micro-F1 | JSON-valid | Exact match | Median decode | Median time to first token | Server peak RSS | Errors / memory kills | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| C0 | b11057 (the macOS pin) | 0.9909 | 1.0 | 0.975 | 15.73 tokens/s | 5.61 s | 16,300 MiB | 0 / 0 | +| C0 | b11146 (the plan's runtime) | 0.9909 | 1.0 | 0.975 | 15.80 tokens/s | 5.49 s | 16,530 MiB | 0 / 0 | +| C2, MTP drafting (`--spec-type draft-mtp --spec-draft-n-max 3`) | b11146 | 0.9909 | 1.0 | 0.975 | 23.94 tokens/s | 5.80 s | 17,606 MiB | 0 / 0 | + +- **C2 vs C0, same build:** identical predictions on all 360 filings (paired bootstrap difference 0.0, + 95% interval [0.0, 0.0]), 1.52× the median decode speed. All four of the plan's criteria pass, but only as a + description of this host. +- **b11146 vs b11057, same model and flags:** identical predictions on all 360 filings, 1.004× decode. The + runtime update changes no output here. +- **For comparison only, not parity:** the workstation's frozen li26 run (RTX 4090, with partial offload for C0) + recorded micro-F1 0.9909 for C0 and C2, and median decode 5.25 (C0) and 26.28 (C2) tokens/s + ([`results/decision.json`](../../../blueprints/convergence-practice/local-inference-latest-20260926/results/decision.json)). +- **The memory stack was live throughout:** [`li26-window-samples.json`](li26-window-samples.json) has 118 + minute samples taken while the li26 servers ran. + - In 6 of them, ai-memory's Ollama models (`qwen3-embedding:4b`, `qwen3.5-9b-64k`) were resident, up to + 14.95 GiB. + - System free memory ranged from 34% to 63%. + +The b11057 macOS pin cannot run C2: its `llama-server` has no `--spec-type` option. C0 and C2 therefore +also ran on the plan's own runtime, official b11146 (`llama-b11146-bin-macos-arm64.tar.gz`, GitHub asset +digest sha256 `1ad3f9ef…5711`, verified before use). It was unpacked into an isolated trial directory, +outside the staging and production prefixes. The co-residency and gate probes ran on the newest official build, +b11214 (`llama-b11214-bin-macos-arm64.tar.gz`, GitHub asset digest sha256 `ea650e93…8219`, verified before use), +in the same isolated trial directory. + +### Worst-case co-residency ([`coresidency.json`](coresidency.json)) + +All model servers were resident at once and driven concurrently for 96 s under coordinator load: +- the li26 C2 profile of Qwen3.8-27B at a 32,768-token context on the newest llama.cpp, b11214; +- the memory stack's own Ollama models, `qwen3-embedding:4b` (4.07 GiB) and `qwen3.5-9b-64k` at a 64K + context (8.38-9.1 GiB); +- the RAG embedder on port 8232, and Qdrant. + +| Measure | Result | +| --- | --- | +| 27B generation server peak RSS | 19,436 MiB | +| System free memory | 70% (memory stack resident), 53% (all resident), 54% (after requests), 68% (27B server stopped) | +| Swap during the probe | no growth (4,977 MB before, 4,953 MB after; it was already raised by the event below) | +| Memorystatus kills | 0 | +| 27B requests | 4, 0 errors, median 5,556 prompt tokens, median time to first token 21.0 s, median decode 16.2 tokens/s, 177 of 275 drafted tokens accepted | +| 9B memory-LLM requests (Ollama) | 4, 0 errors, median 7.9 tokens/s while sharing the GPU | +| 4B embedding batches (Ollama) | 5, 0 errors, 40 vectors of 2,560 dimensions, median 4.5 s per batch of 8 | + +### The one swap event ([`swap-event.json`](swap-event.json)) + +Swap stayed at 16.81 MB through both C0 runs and the first 16 minutes of C2. At 10:37 it jumped to a peak of +13,934 MB and was back under 6 GB within 40 s. It then drained slowly to about 5 GB. Three loads overlapped in +that minute: +- the li26 C2 server, at 16.9 GiB RSS; +- ai-memory loading its two Ollama models (12.5 GiB), which were absent at 10:36:56 and present at 10:37:56; +- an OmniRoute Next.js production build (`npm run build:release`, between 10:34:40 and 10:38:22 by file birth + times) for the GPT-6 gateway being installed on this host. + +There were no memorystatus kills, and li26 quality was unaffected (0 errors, the same predictions as C0). The +model set on its own never swapped. A heavy build running alongside the full model set did. + +### Reranker-gate timing for S3 section 5 ([`gate.json`](gate.json)) + +ai-memory's C4 reranker must answer within 20 s per call at 4-8K-token prompts. These rerank-shaped prompts +were a query plus numbered 600-character passages taken from this repository's own docs; the answers were not +scored. Each local configuration got one warm-up call, then three calls at each size, with one model resident at +a time. The llama.cpp configurations ran on the newest build, b11214. + +| Configuration | ~4.4K tokens, median (max) | ~7.5K tokens, median (max) | Over 20 s at ~7.5K | +| --- | --- | --- | --- | +| Qwen3.8-27B UD-Q4_K_M, llama.cpp Metal | 17.6 s (18.0) | 27.9 s (30.0) | 3 of 3 | +| The same with MTP drafting (li26 C2 profile) | 16.9 s (17.4) | 27.1 s (29.1) | 3 of 3 | +| Qwen3.8-27B nvfp4, Ollama 0.34.4 MLX engine | 11.4 s (12.5) | 18.5 s (20.7) | 1 of 3 | +| Production `qwen3.5-9b-64k`, Ollama MLX engine | 3.3 s (3.3) | 5.5 s (5.8) | 0 | +| `ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF@a02f48bb`, llama.cpp `--rerank` (27 and 54 passages) | 0.7 s (0.8) | 1.4 s (1.4) | 0 | +| GPT-6 astra / sol / luna through this Mac's OmniRoute gateway (one 4K and two 8K calls each) | 6.6 / 9.3 / 9.6 s | 5.1 / 7.3 / 6.8 s | 0 | + +- **Qwen3.8-27B misses the gate on llama.cpp.** It leads the memory-LLM sweep on AA-LCR, but prompt processing + dominates at 8K, so on this Mac it cannot meet the gate through llama.cpp. MTP drafting speeds decoding, not + prefill. +- **Ollama's MLX build is 1.5× faster at prefill,** which is still borderline at 8K. +- **Three options clear the gate here:** the production 9B, a dedicated cross-encoder (about 20× faster than an + LLM reranker), and GPT-6 through the gateway, which S3 uses for its LLM roles. + +## Step 5 guards and findings + +**Pin drift, recorded and not fixed** (the request's guard: no re-pin without native evidence): + +- **llama.cpp:** the macOS pin b11057 cannot run the workstation's production li26 C2 profile, because it + has no MTP drafting (`--spec-type`). b11146 can, and ran it here. Upstream's newest build is b11214 + (2026-09-27). +- **codex:** the macOS pin is 0.155.1 and Linux runs 0.157.1. This Mac runs the ChatGPT-bundled + 0.158.0-alpha.2.1. +- **ai-memory:** the macOS pin is 2.3.2 and the hook template names 2.4.1. The running build, `19b6429` + (a local build reporting 2.4.0), is 46 commits ahead of and 62 behind official v2.4.1, and official + v2.4.0 is its ancestor. +- **Embeddings:** the memory stack embeds with Ollama `qwen3-embedding:4b` (2,560 dimensions, Q4_K_M, + through Ollama's bundled llama-server). It consolidates and reranks with `qwen3.5-9b-64k` (nvfp4 + safetensors on Ollama's MLX engine). The pinned RAG profile is llama.cpp embeddinggemma-300M + (768 dimensions) on port 8232. These are two consumers with two embedding spaces, and both run. + +**Other findings.** + +- `local.agent-ecosystem.maintenance` last exited 1; it predates this request. +- The Hindsight memory trial cannot use its embedded Postgres on a Mac without Homebrew. The latest + `pg0-embedded` 0.15.2 ships a macOS Postgres 18.1.0 linked to `/opt/homebrew/opt/openssl@3`. `initdb` + runs `postgres -V` through `popen` and `/bin/sh`, so macOS strips any `DYLD_*` fallback. + - The trial resolved this with the macOS profile's own Postgres choice: Apple Container 1.4.1 (the + Apple-signed, notarized installer, sha256-verified, installed on 2026-09-27) running the official + `pgvector/pgvector:0.8.6-pg18-trixie` image, pinned by digest. + - Hindsight 0.10.1 starts against it and both health endpoints return 200. + - Apple Container's loopback `--publish` proxy dropped Hindsight's migration connections, while the + container's host-only address worked. + - This concerns the memory trials, not this request. + +**Host value file.** `adoption/hosts/mac-coordinator-64gb-20260925.json` was created for +`tools/adoption/render_config.py` from the measured profile and stays private. Its path keys (`HOME`, +`ECO_ROOT`, `PROJECT_ROOT`, `HOST_PATH`, `CODE_INDEX_PATH`, `EMBED_MODEL_PATH`) all carry this host's +personal paths, so under the request's own rule it is not committed. + +## What is not claimed + +- No li26 parity with the workstation, and no replacement or serving-profile decision for this host. +- No retrieval-quality comparison of embedders or memory systems. Those belong to the S3 preregistration + (#390), which runs only after it freezes. +- The b11146 runtime is a trial install. It is not a new macOS pin. + +## Files + +| File | What it is | +| --- | --- | +| `README.md` | this summary | +| `embed-acceptance.json` | the acceptance output, the model, the Metal lines from the startup log, and the launchd state before step 1 | +| `capacity.json` | llama-bench results, serving timings, memory snapshots and the jetsam classification | +| `li26-mac-descriptive.json` | the li26 descriptive summary, re-derivable with `li26_mac_descriptive.py --check` from the private run state | +| `li26_mac_descriptive.py` | scores the private runs with the frozen `analyze.py` functions | +| `coresidency.json` | the worst-case co-residency probe | +| `gate.json` | reranker-gate timing per candidate model and runtime, including GPT-6 through this Mac's OmniRoute gateway | +| `swap-event.json` | the one swap event: timeline, what was co-resident, and the OmniRoute build window from file birth times | +| `li26-window-samples.json` | minute samples of co-resident Ollama models, free memory and swap while the li26 runs were active | diff --git a/evidence/artifacts/mac-model-hosting-20260927/capacity.json b/evidence/artifacts/mac-model-hosting-20260927/capacity.json new file mode 100644 index 000000000..4e549e6f9 --- /dev/null +++ b/evidence/artifacts/mac-model-hosting-20260927/capacity.json @@ -0,0 +1,104 @@ +{ + "co_resident_during_probe": "this Claude Code session and its subagents, the com.native-stack.llama-embed LaunchAgent (embeddinggemma-300M on port 8232), Qdrant (local.agent-ecosystem.qdrant, port 6333), ai-memory and Ollama services (their models were idle-unloaded at the time), and ordinary desktop applications", + "hardware": "Apple M5 Pro, 64 GB unified memory", + "host_id": "mac-coordinator-64gb-20260925", + "jetsam": { + "log_show_self_reference": 1, + "matched_lines": 66, + "memorystatus_kill_events": 0, + "other_runningboard_diagnostics": 2, + "runningboard_ignoring_jetsam_update": 63 + }, + "kind": "mac_capacity_probe", + "llama_bench": { + "argv": "llama-bench -m -p 512,4096 -n 128 -ngl 99 -fa on -r 3 -o json", + "evidence_class": "native_proven", + "flash_attn": true, + "model_params": 27320697856, + "n_gpu_layers": 99, + "tests": [ + { + "n_gen": 0, + "n_prompt": 512, + "repetitions": 3, + "stddev": 1.11, + "tokens_per_second": 371.52 + }, + { + "n_gen": 0, + "n_prompt": 4096, + "repetitions": 3, + "stddev": 15.34, + "tokens_per_second": 341.68 + }, + { + "n_gen": 128, + "n_prompt": 0, + "repetitions": 3, + "stddev": 0.62, + "tokens_per_second": 15.9 + } + ], + "threads": 6 + }, + "llama_server": { + "argv": "llama-server -m -c 32768 -ngl 99 -fa on --host 127.0.0.1 --port 18080 (server defaults: 4 slots sharing one unified 32768-token context)", + "cache_prompt": false, + "evidence_class": "local_integration", + "long_prompt": { + "predicted_ms": 435.11, + "predicted_n": 7, + "predicted_per_second": 13.79, + "prompt_ms": 15122.6, + "prompt_n": 5189, + "prompt_per_second": 343.13 + }, + "median_decode_tokens_per_second": 15.84, + "median_prompt_tokens_short": 1603.0, + "median_ttft_ms_short": 4672, + "peak_rss_mib": 22941, + "requests": 9, + "rss_samples": 72, + "why_local_integration": "the prompts are this repository's own docs/token-session-handbook.md chunks behind the li26 prompt header, not a production workload" + }, + "memory": { + "after_bench": { + "free_percent": 71, + "swap_used": "16.81M" + }, + "after_requests": { + "free_percent": 61, + "swap_used": "16.81M" + }, + "after_stop": { + "free_percent": 89, + "swap_used": "16.81M" + }, + "baseline": { + "free_percent": 89, + "swap_used": "16.81M" + }, + "server_loaded": { + "free_percent": 61, + "swap_used": "16.81M" + } + }, + "model": { + "bytes": 16464440224, + "filename": "Qwen3.8-27B-UD-Q4_K_M.gguf", + "repository": "unsloth/Qwen3.8-27B-GGUF", + "revision": "4ca720788d1e01f1bff70c033e0d0028fd02e502", + "sha256": "322e194ff79741c7baa497c240f677f54b201b0efab44ca8e50f122b39123482" + }, + "runtime": { + "backends": "MTL,BLAS", + "commit": "59657a613", + "gpu": "Apple M5 Pro", + "llama_cpp_build": 11057, + "source": "manifest-pinned macOS bootstrap install (adoption/pins-macos-arm64.json)" + }, + "window": { + "end_local": "2026-09-27 08:41:27", + "start_local": "2026-09-27 08:38:53" + } +} diff --git a/evidence/artifacts/mac-model-hosting-20260927/coresidency.json b/evidence/artifacts/mac-model-hosting-20260927/coresidency.json new file mode 100644 index 000000000..3f15a894c --- /dev/null +++ b/evidence/artifacts/mac-model-hosting-20260927/coresidency.json @@ -0,0 +1,284 @@ +{ + "generation_server_peak_rss_mib": 19436, + "host_id": "mac-coordinator-64gb-20260925", + "jetsam": { + "log_show_self_reference": 1, + "matched_lines": 1, + "memorystatus_kill_events": 0, + "other_runningboard_diagnostics": 0, + "runningboard_ignoring_jetsam_update": 0 + }, + "kind": "mac_coresidency_probe", + "load": { + "embed4b": { + "batches": 5, + "dims": [ + 2560 + ], + "errors": 0, + "median_batch_ms": 4524, + "vectors": 40 + }, + "gen27b": { + "calls": 4, + "draft_accepted_of_drafted": [ + 177, + 275 + ], + "errors": 0, + "median_decode_tokens_per_second": 16.19, + "median_prompt_tokens": 5555.5, + "median_ttft_ms": 20988 + }, + "gen9b": { + "calls": 4, + "errors": 0, + "median_eval_tokens_per_second": 7.945 + }, + "wall_seconds": 95.7 + }, + "memory": { + "after_requests": { + "free_percent": 54, + "ollama_resident": [ + { + "gib": 9.1, + "name": "qwen3.5-9b-64k:latest" + }, + { + "gib": 4.07, + "name": "qwen3-embedding:4b" + } + ], + "swap_used": "4953.12M" + }, + "after_stop": { + "free_percent": 68, + "ollama_resident": [ + { + "gib": 9.1, + "name": "qwen3.5-9b-64k:latest" + }, + { + "gib": 4.07, + "name": "qwen3-embedding:4b" + } + ], + "swap_used": "4953.12M" + }, + "all_resident": { + "free_percent": 53, + "ollama_resident": [ + { + "gib": 8.38, + "name": "qwen3.5-9b-64k:latest" + }, + { + "gib": 4.07, + "name": "qwen3-embedding:4b" + } + ], + "swap_used": "4977.12M" + }, + "memory_stack_resident": { + "free_percent": 70, + "ollama_resident": [ + { + "gib": 8.38, + "name": "qwen3.5-9b-64k:latest" + }, + { + "gib": 4.07, + "name": "qwen3-embedding:4b" + } + ], + "swap_used": "4977.12M" + } + }, + "requests_exit": "0", + "resident_set": { + "coordinator": "this Claude Code session with subagents, ai-memory, and ordinary desktop applications", + "generation": "llama.cpp b11146 llama-server, Qwen3.8-27B UD-Q4_K_M, 32768-token context, one slot, full Metal offload, --spec-type draft-mtp --spec-draft-n-max 3", + "memory_stack": "Ollama 0.34.4 qwen3-embedding:4b and qwen3.5-9b-64k (production models, requested with keep_alive 10m, never pulled or changed)", + "rag": "com.native-stack.llama-embed (embeddinggemma-300M, port 8232) and Qdrant (port 6333), unchanged" + }, + "server_healthy": true, + "top_processes": { + "after_requests": [ + { + "process": "llama-server", + "rss_mib": 19436 + }, + { + "process": "ollama", + "rss_mib": 7994 + }, + { + "process": "llama-server", + "rss_mib": 3445 + }, + { + "process": "MTLCompilerService", + "rss_mib": 1016 + }, + { + "process": "Google", + "rss_mib": 714 + }, + { + "process": "claude", + "rss_mib": 666 + }, + { + "process": "Google", + "rss_mib": 564 + }, + { + "process": "omniroute", + "rss_mib": 515 + }, + { + "process": "Google", + "rss_mib": 455 + }, + { + "process": "Google", + "rss_mib": 400 + } + ], + "after_stop": [ + { + "process": "ollama", + "rss_mib": 7994 + }, + { + "process": "llama-server", + "rss_mib": 3445 + }, + { + "process": "MTLCompilerService", + "rss_mib": 1016 + }, + { + "process": "Google", + "rss_mib": 714 + }, + { + "process": "claude", + "rss_mib": 666 + }, + { + "process": "Google", + "rss_mib": 564 + }, + { + "process": "omniroute", + "rss_mib": 515 + }, + { + "process": "Google", + "rss_mib": 455 + }, + { + "process": "Google", + "rss_mib": 400 + }, + { + "process": "Google", + "rss_mib": 399 + } + ], + "all_resident": [ + { + "process": "llama-server", + "rss_mib": 18710 + }, + { + "process": "ollama", + "rss_mib": 7987 + }, + { + "process": "llama-server", + "rss_mib": 3079 + }, + { + "process": "MTLCompilerService", + "rss_mib": 1013 + }, + { + "process": "Google", + "rss_mib": 762 + }, + { + "process": "claude", + "rss_mib": 644 + }, + { + "process": "Google", + "rss_mib": 599 + }, + { + "process": "omniroute", + "rss_mib": 575 + }, + { + "process": "Google", + "rss_mib": 477 + }, + { + "process": "Google", + "rss_mib": 449 + } + ], + "memory_stack_resident": [ + { + "process": "ollama", + "rss_mib": 7988 + }, + { + "process": "llama-server", + "rss_mib": 3084 + }, + { + "process": "Google", + "rss_mib": 771 + }, + { + "process": "claude", + "rss_mib": 724 + }, + { + "process": "omniroute", + "rss_mib": 626 + }, + { + "process": "Google", + "rss_mib": 599 + }, + { + "process": "Google", + "rss_mib": 477 + }, + { + "process": "Google", + "rss_mib": 465 + }, + { + "process": "Google", + "rss_mib": 450 + }, + { + "process": "Google", + "rss_mib": 397 + } + ] + }, + "warm_http": { + "embed4b": "200", + "gen9b": "200" + }, + "window_local": [ + "2026-09-27 11:06:40", + "2026-09-27 11:08:25" + ] +} diff --git a/evidence/artifacts/mac-model-hosting-20260927/embed-acceptance.json b/evidence/artifacts/mac-model-hosting-20260927/embed-acceptance.json new file mode 100644 index 000000000..27e4d97db --- /dev/null +++ b/evidence/artifacts/mac-model-hosting-20260927/embed-acceptance.json @@ -0,0 +1,54 @@ +{ + "backend_from_startup_log": { + "device_lines": [ + "0.13.206.256 I cmn common_param: - MTL0 : Apple M5 Pro (53084 MiB, 53083 MiB free)", + "0.13.276.651 I llama_prepare_model_devices: using device MTL0 (Apple M5 Pro) (unknown id) - 53083 MiB free", + "0.13.328.764 I load_tensors: offloaded 25/25 layers to GPU" + ], + "layer_lines_assigned_to_cpu": 0, + "model_loads_in_log": 3, + "offload_lines": [ + "offloaded 25/25 layers to GPU" + ] + }, + "command": "python3 tools/adoption/embed_acceptance.py http://127.0.0.1:8232 evidence/artifacts/macos-embed-reference-20260923/macos-embed-reference-20260923.json", + "host_id": "mac-coordinator-64gb-20260925", + "kind": "mac_embed_acceptance", + "launchd_before_step_1": [ + { + "label": "local.agent-ecosystem.ai-memory", + "last_exit": 0, + "running": true + }, + { + "label": "local.agent-ecosystem.maintenance", + "last_exit": 1, + "running": false + }, + { + "label": "local.agent-ecosystem.ollama", + "last_exit": 0, + "running": true + }, + { + "label": "local.agent-ecosystem.qdrant", + "last_exit": 0, + "running": true + } + ], + "model": { + "filename": "embeddinggemma-300M-Q8_0.gguf", + "repository": "ggml-org/embeddinggemma-300M-GGUF", + "revision": "0f741b5a6585bd53aeb15cd1372c56f2a0f65e12", + "sha256": "b5ce9d77a3fc4b3b39ccb5643c36777911cc4eb46a66962eadfa3f5f60490d63" + }, + "result": { + "cosine": 0.999661109106148, + "dimension_actual": 768, + "dimension_expected": 768, + "passed": true, + "threshold": 0.99, + "url": "http://127.0.0.1:8232/v1/embeddings" + }, + "service": "com.native-stack.llama-embed LaunchAgent (llama.cpp b11057 llama-server --embedding, port 8232)" +} diff --git a/evidence/artifacts/mac-model-hosting-20260927/gate.json b/evidence/artifacts/mac-model-hosting-20260927/gate.json new file mode 100644 index 000000000..167dbcc6f --- /dev/null +++ b/evidence/artifacts/mac-model-hosting-20260927/gate.json @@ -0,0 +1,606 @@ +{ + "configs": [ + { + "argv_extra": "(none)", + "calls": [ + { + "decode_tps": 14.59, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 12613, + "prompt_tokens": 4178, + "size": "4k", + "wall_s": 16.94 + }, + { + "decode_tps": 15.59, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 13583, + "prompt_tokens": 4403, + "size": "4k", + "wall_s": 17.63 + }, + { + "decode_tps": 15.49, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 13945, + "prompt_tokens": 4485, + "size": "4k", + "wall_s": 18.02 + }, + { + "decode_tps": 15.29, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 23155, + "prompt_tokens": 7363, + "size": "8k", + "wall_s": 27.28 + }, + { + "decode_tps": 14.14, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 23395, + "prompt_tokens": 7485, + "size": "8k", + "wall_s": 27.86 + }, + { + "decode_tps": 15.02, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 25755, + "prompt_tokens": 8161, + "size": "8k", + "wall_s": 29.96 + } + ], + "label": "gate-qwen38-c0", + "model": "unsloth/Qwen3.8-27B-GGUF@4ca7207 UD-Q4_K_M", + "resident_after": [], + "runtime": "llama.cpp b11214 Metal (newest build)", + "startup_failure": null, + "summary": { + "4k": { + "calls": 3, + "errors": 0, + "max_wall_s": 18.02, + "median_prompt_tokens": 4403, + "median_wall_s": 17.63, + "over_gate": 0 + }, + "8k": { + "calls": 3, + "errors": 0, + "max_wall_s": 29.96, + "median_prompt_tokens": 7485, + "median_wall_s": 27.86, + "over_gate": 3 + } + } + }, + { + "argv_extra": "--spec-type draft-mtp --spec-draft-n-max 3", + "calls": [ + { + "decode_tps": 20.78, + "draft_accepted": 43, + "draft_n": 60, + "error": null, + "prompt_ms": 13058, + "prompt_tokens": 4178, + "size": "4k", + "wall_s": 16.09 + }, + { + "decode_tps": 17.61, + "draft_accepted": 40, + "draft_n": 68, + "error": null, + "prompt_ms": 13857, + "prompt_tokens": 4403, + "size": "4k", + "wall_s": 17.44 + }, + { + "decode_tps": 22.03, + "draft_accepted": 44, + "draft_n": 55, + "error": null, + "prompt_ms": 14001, + "prompt_tokens": 4485, + "size": "4k", + "wall_s": 16.87 + }, + { + "decode_tps": 20.96, + "draft_accepted": 43, + "draft_n": 57, + "error": null, + "prompt_ms": 23480, + "prompt_tokens": 7363, + "size": "8k", + "wall_s": 26.49 + }, + { + "decode_tps": 21.67, + "draft_accepted": 44, + "draft_n": 55, + "error": null, + "prompt_ms": 24133, + "prompt_tokens": 7485, + "size": "8k", + "wall_s": 27.05 + }, + { + "decode_tps": 22.3, + "draft_accepted": 45, + "draft_n": 54, + "error": null, + "prompt_ms": 26252, + "prompt_tokens": 8161, + "size": "8k", + "wall_s": 29.08 + } + ], + "label": "gate-qwen38-c2", + "model": "unsloth/Qwen3.8-27B-GGUF@4ca7207 UD-Q4_K_M", + "resident_after": [], + "runtime": "llama.cpp b11214 Metal (newest build)", + "startup_failure": null, + "summary": { + "4k": { + "calls": 3, + "errors": 0, + "max_wall_s": 17.44, + "median_prompt_tokens": 4403, + "median_wall_s": 16.87, + "over_gate": 0 + }, + "8k": { + "calls": 3, + "errors": 0, + "max_wall_s": 29.08, + "median_prompt_tokens": 7485, + "median_wall_s": 27.05, + "over_gate": 3 + } + } + }, + { + "argv_extra": null, + "calls": [ + { + "decode_tps": 22.24, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 9258, + "prompt_tokens": 4190, + "size": "4k", + "wall_s": 10.78 + }, + { + "decode_tps": 35.88, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 10731, + "prompt_tokens": 4415, + "size": "4k", + "wall_s": 12.54 + }, + { + "decode_tps": 29.59, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 10725, + "prompt_tokens": 4496, + "size": "4k", + "wall_s": 11.42 + }, + { + "decode_tps": 24.8, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 17768, + "prompt_tokens": 7375, + "size": "8k", + "wall_s": 18.47 + }, + { + "decode_tps": 4.37, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 17930, + "prompt_tokens": 7496, + "size": "8k", + "wall_s": 18.18 + }, + { + "decode_tps": 28.42, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 19413, + "prompt_tokens": 8172, + "size": "8k", + "wall_s": 20.74 + } + ], + "label": "gate-qwen38-ollama-mlx", + "model": "qwen3.8:27b-mlx", + "resident_after": [ + { + "ctx": 16384, + "gib": 23.05, + "name": "qwen3.8:27b-mlx" + } + ], + "runtime": "Ollama 0.34.4 (MLX engine, nvfp4 safetensors)", + "startup_failure": null, + "summary": { + "4k": { + "calls": 3, + "errors": 0, + "max_wall_s": 12.54, + "median_prompt_tokens": 4415, + "median_wall_s": 11.42, + "over_gate": 0 + }, + "8k": { + "calls": 3, + "errors": 0, + "max_wall_s": 20.74, + "median_prompt_tokens": 7496, + "median_wall_s": 18.47, + "over_gate": 1 + } + } + }, + { + "argv_extra": null, + "calls": [ + { + "decode_tps": 40.52, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 2741, + "prompt_tokens": 4190, + "size": "4k", + "wall_s": 3.17 + }, + { + "decode_tps": 40.99, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 2709, + "prompt_tokens": 4415, + "size": "4k", + "wall_s": 3.33 + }, + { + "decode_tps": 40.81, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 2820, + "prompt_tokens": 4496, + "size": "4k", + "wall_s": 3.34 + }, + { + "decode_tps": 40.54, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 4959, + "prompt_tokens": 7375, + "size": "8k", + "wall_s": 5.49 + }, + { + "decode_tps": 40.39, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 4832, + "prompt_tokens": 7496, + "size": "8k", + "wall_s": 5.37 + }, + { + "decode_tps": 39.28, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": 5223, + "prompt_tokens": 8172, + "size": "8k", + "wall_s": 5.77 + } + ], + "label": "gate-production-qwen35-9b", + "model": "qwen3.5-9b-64k", + "resident_after": [ + { + "ctx": 65536, + "gib": 11.14, + "name": "qwen3.5-9b-64k:latest" + } + ], + "runtime": "Ollama 0.34.4 (MLX engine, nvfp4 safetensors)", + "startup_failure": null, + "summary": { + "4k": { + "calls": 3, + "errors": 0, + "max_wall_s": 3.34, + "median_prompt_tokens": 4415, + "median_wall_s": 3.33, + "over_gate": 0 + }, + "8k": { + "calls": 3, + "errors": 0, + "max_wall_s": 5.77, + "median_prompt_tokens": 7496, + "median_wall_s": 5.49, + "over_gate": 0 + } + } + }, + { + "argv_extra": "--rerank --ctx-size 32768 --parallel 8 --gpu-layers 99", + "calls": [ + { + "decode_tps": null, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": null, + "prompt_tokens": null, + "size": "4k", + "wall_s": 0.8 + }, + { + "decode_tps": null, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": null, + "prompt_tokens": null, + "size": "4k", + "wall_s": 0.72 + }, + { + "decode_tps": null, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": null, + "prompt_tokens": null, + "size": "4k", + "wall_s": 0.71 + }, + { + "decode_tps": null, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": null, + "prompt_tokens": null, + "size": "8k", + "wall_s": 1.39 + }, + { + "decode_tps": null, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": null, + "prompt_tokens": null, + "size": "8k", + "wall_s": 1.34 + }, + { + "decode_tps": null, + "draft_accepted": null, + "draft_n": null, + "error": null, + "prompt_ms": null, + "prompt_tokens": null, + "size": "8k", + "wall_s": 1.4 + } + ], + "label": "gate-qwen3-reranker-0.6b", + "model": "ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF@a02f48bb qwen3-reranker-0.6b-q8_0.gguf", + "resident_after": [], + "runtime": "llama.cpp b11214 Metal (newest build), --rerank", + "startup_failure": null, + "summary": { + "4k": { + "calls": 3, + "documents": 27, + "errors": 0, + "max_wall_s": 0.8, + "median_wall_s": 0.72, + "over_gate": 0 + }, + "8k": { + "calls": 3, + "documents": 54, + "errors": 0, + "max_wall_s": 1.4, + "median_wall_s": 1.39, + "over_gate": 0 + } + } + } + ], + "finished_utc": "2026-09-27T15:15:37Z", + "gate_seconds": 20.0, + "gpt6_via_mac_omniroute": { + "calls": [ + { + "completion_tokens": 283, + "http": 200, + "model": "cx/gpt-6-luna", + "prompt_tokens": 3925, + "served_model": "gpt-6-luna", + "size": "4k", + "wall_s": 9.59 + }, + { + "completion_tokens": 108, + "http": 200, + "model": "cx/gpt-6-luna", + "prompt_tokens": 7184, + "served_model": "gpt-6-luna", + "size": "8k", + "wall_s": 5.72 + }, + { + "completion_tokens": 225, + "http": 200, + "model": "cx/gpt-6-luna", + "prompt_tokens": 7755, + "served_model": "gpt-6-luna", + "size": "8k", + "wall_s": 7.87 + }, + { + "completion_tokens": 69, + "http": 200, + "model": "cx/gpt-6-sol", + "prompt_tokens": 3925, + "served_model": "gpt-6-sol", + "size": "4k", + "wall_s": 9.27 + }, + { + "completion_tokens": 172, + "http": 200, + "model": "cx/gpt-6-sol", + "prompt_tokens": 7184, + "served_model": "gpt-6-sol", + "size": "8k", + "wall_s": 8.76 + }, + { + "completion_tokens": 65, + "http": 200, + "model": "cx/gpt-6-sol", + "prompt_tokens": 7755, + "served_model": "gpt-6-sol", + "size": "8k", + "wall_s": 5.81 + }, + { + "completion_tokens": 105, + "http": 200, + "model": "cx/gpt-6-astra", + "prompt_tokens": 3925, + "served_model": "gpt-6-astra", + "size": "4k", + "wall_s": 6.58 + }, + { + "completion_tokens": 80, + "http": 200, + "model": "cx/gpt-6-astra", + "prompt_tokens": 7184, + "served_model": "gpt-6-astra", + "size": "8k", + "wall_s": 5.52 + }, + { + "completion_tokens": 59, + "http": 200, + "model": "cx/gpt-6-astra", + "prompt_tokens": 7755, + "served_model": "gpt-6-astra", + "size": "8k", + "wall_s": 4.69 + } + ], + "note": "network and provider latency on 2026-09-27; the answers were not scored", + "prompts": "the same rerank-shaped prompt builder as the local configurations; one 4k and two 8k calls per model, max_tokens 64", + "runtime": "this Mac's OmniRoute 3.8.51 gateway (loopback, keyless), two signed-in Codex accounts, codex providerConcurrency 4", + "summary": { + "cx/gpt-6-astra": { + "4k": { + "calls": 1, + "errors": 0, + "max_wall_s": 6.58, + "median_prompt_tokens": 3925, + "median_wall_s": 6.58, + "over_gate": 0 + }, + "8k": { + "calls": 2, + "errors": 0, + "max_wall_s": 5.52, + "median_prompt_tokens": 7469.5, + "median_wall_s": 5.11, + "over_gate": 0 + } + }, + "cx/gpt-6-luna": { + "4k": { + "calls": 1, + "errors": 0, + "max_wall_s": 9.59, + "median_prompt_tokens": 3925, + "median_wall_s": 9.59, + "over_gate": 0 + }, + "8k": { + "calls": 2, + "errors": 0, + "max_wall_s": 7.87, + "median_prompt_tokens": 7469.5, + "median_wall_s": 6.79, + "over_gate": 0 + } + }, + "cx/gpt-6-sol": { + "4k": { + "calls": 1, + "errors": 0, + "max_wall_s": 9.27, + "median_prompt_tokens": 3925, + "median_wall_s": 9.27, + "over_gate": 0 + }, + "8k": { + "calls": 2, + "errors": 0, + "max_wall_s": 8.76, + "median_prompt_tokens": 7469.5, + "median_wall_s": 7.29, + "over_gate": 0 + } + } + } + }, + "host_id": "mac-coordinator-64gb-20260925", + "kind": "mac_reranker_gate_timing", + "prompt_source": "this repository's docs, rerank-shaped (query + numbered 600-char passages); answers not scored", + "started_utc": "2026-09-27T15:08:34Z" +} diff --git a/evidence/artifacts/mac-model-hosting-20260927/li26-mac-descriptive.json b/evidence/artifacts/mac-model-hosting-20260927/li26-mac-descriptive.json new file mode 100644 index 000000000..371bd28ce --- /dev/null +++ b/evidence/artifacts/mac-model-hosting-20260927/li26-mac-descriptive.json @@ -0,0 +1,120 @@ +{ + "host_id": "mac-coordinator-64gb-20260925", + "kind": "li26_mac_metal_descriptive", + "model": "unsloth/Qwen3.8-27B-GGUF@4ca720788d1e01f1bff70c033e0d0028fd02e502 Qwen3.8-27B-UD-Q4_K_M.gguf", + "pairs": [ + { + "arm": "C2@b11146", + "ci95": [ + 0.0, + 0.0 + ], + "control": "C0@b11146", + "criteria_descriptive": { + "faster_median_decode": true, + "json_valid_rate": true, + "noninferior_micro_f1": true, + "run_clean": true + }, + "decode_speedup": 1.515, + "delta_micro_f1": 0.0, + "identical_predictions": 360, + "isolates": "MTP drafting on the same build and flags", + "n": 360 + }, + { + "arm": "C0@b11146", + "ci95": [ + 0.0, + 0.0 + ], + "control": "C0@b11057", + "criteria_descriptive": { + "faster_median_decode": true, + "json_valid_rate": true, + "noninferior_micro_f1": true, + "run_clean": true + }, + "decode_speedup": 1.004, + "delta_micro_f1": 0.0, + "identical_predictions": 360, + "isolates": "runtime build b11057 -> b11146 with the same model and flags", + "n": 360 + } + ], + "parity": false, + "plan_sha256": "b90db46f20624895089cee5bbce27b0c8df2f3c783b1873cea5920a0f8e18b95", + "profile_note": "metrics.json's profile field repeats the plan's declaration; argv is what ran here", + "prompt_sha256": "faf9273a6807e982154667a8654ec2cff637cf57c8d85c34093e615876ae1d58", + "runs": { + "C0@b11057": { + "argv": "llama-server --model Qwen3.8-27B-UD-Q4_K_M.gguf --alias li26-c0 --ctx-size 8192 --parallel 1 --gpu-layers 99 --fit off --threads 6 --threads-batch 6 --batch-size 512 --ubatch-size 128 --cache-ram 0 --no-webui --offline", + "build_info": "b11057-59657a613", + "errors": 0, + "events": [], + "exact_match_rate": 0.975, + "json_valid_rate": 1.0, + "macro_codes": 12, + "macro_f1": 0.9867, + "median_decode_tokens_per_second": 15.73, + "median_ttft_ms": 5609, + "memory_kills": 0, + "micro_f1": 0.9909, + "n": 360, + "not_attempted": 0, + "runtime": "b11057 (59657a613), the macOS pin", + "server_peak_rss_mib": 16300, + "status": "completed", + "window_local": [ + "2026-09-27 08:49:28", + "2026-09-27 09:36:07" + ] + }, + "C0@b11146": { + "argv": "llama-server --model Qwen3.8-27B-UD-Q4_K_M.gguf --alias li26-c0 --ctx-size 8192 --parallel 1 --gpu-layers 99 --fit off --threads 6 --threads-batch 6 --batch-size 512 --ubatch-size 128 --cache-ram 0 --no-webui --offline", + "build_info": "b11146-7fe450e19", + "errors": 0, + "events": [], + "exact_match_rate": 0.975, + "json_valid_rate": 1.0, + "macro_codes": 12, + "macro_f1": 0.9867, + "median_decode_tokens_per_second": 15.8, + "median_ttft_ms": 5485, + "memory_kills": 0, + "micro_f1": 0.9909, + "n": 360, + "not_attempted": 0, + "runtime": "b11146 (7fe450e19), the plan's runtime", + "server_peak_rss_mib": 16530, + "status": "completed", + "window_local": [ + "2026-09-27 09:36:20", + "2026-09-27 10:21:01" + ] + }, + "C2@b11146": { + "argv": "llama-server --model Qwen3.8-27B-UD-Q4_K_M.gguf --alias li26-c2 --ctx-size 8192 --parallel 1 --gpu-layers 99 --fit off --threads 6 --threads-batch 6 --batch-size 512 --ubatch-size 128 --cache-ram 0 --no-webui --offline --spec-type draft-mtp --spec-draft-n-max 3", + "build_info": "b11146-7fe450e19", + "errors": 0, + "events": [], + "exact_match_rate": 0.975, + "json_valid_rate": 1.0, + "macro_codes": 12, + "macro_f1": 0.9867, + "median_decode_tokens_per_second": 23.94, + "median_ttft_ms": 5802, + "memory_kills": 0, + "micro_f1": 0.9909, + "n": 360, + "not_attempted": 0, + "runtime": "b11146 (7fe450e19), the plan's runtime", + "server_peak_rss_mib": 17606, + "status": "completed", + "window_local": [ + "2026-09-27 10:21:01", + "2026-09-27 11:06:32" + ] + } + } +} diff --git a/evidence/artifacts/mac-model-hosting-20260927/li26-window-samples.json b/evidence/artifacts/mac-model-hosting-20260927/li26-window-samples.json new file mode 100644 index 000000000..3eb01ef51 --- /dev/null +++ b/evidence/artifacts/mac-model-hosting-20260927/li26-window-samples.json @@ -0,0 +1,39 @@ +{ + "free_percent_max": 63, + "free_percent_min": 34, + "interval_seconds": 60, + "kind": "mac_li26_window_memory", + "max_ollama_resident_gib": 14.950000000000001, + "note": "sampled every 60 s while the li26 runs were active; ai-memory loads and unloads its Ollama models on its own schedule", + "ollama_models_seen": [ + "qwen3-embedding:4b", + "qwen3.5-9b-64k:latest" + ], + "samples": 118, + "samples_with_ollama_models_resident": 6, + "swap_used_values": [ + "16.81M", + "4993.12M", + "5009.19M", + "5017.19M", + "5033.19M", + "5137.19M", + "5438.81M", + "5630.81M", + "5638.81M", + "5646.81M", + "5654.81M", + "5662.81M", + "5670.81M", + "5678.81M", + "5710.81M", + "5742.81M", + "5750.81M", + "5766.81M", + "5774.81M", + "5782.88M", + "5798.88M", + "5838.88M", + "7647.81M" + ] +} diff --git a/evidence/artifacts/mac-model-hosting-20260927/li26_mac_descriptive.py b/evidence/artifacts/mac-model-hosting-20260927/li26_mac_descriptive.py new file mode 100644 index 000000000..49b3ba663 --- /dev/null +++ b/evidence/artifacts/mac-model-hosting-20260927/li26_mac_descriptive.py @@ -0,0 +1,177 @@ +#!/usr/bin/env python3 +"""Descriptive li26 summary of this Mac's Metal runs (host request #379). + +Scores each run's private eval/metrics.json with the frozen blueprint's own functions +(analyze.py: counts, filing_scores, micro_f1, macro_f1, paired_bootstrap with the plan's +resamples, seed and alpha). The Mac is not the preregistered host, so this is description, +never a replacement decision or a parity claim. The output holds no document text, no +per-filing data and no paths. + + python3 -B li26_mac_descriptive.py --runs DIR # print the summary JSON + python3 -B li26_mac_descriptive.py --runs DIR --check F # re-derive and compare with F + +metrics.json's "profile" field repeats the plan's declaration for the arm; ARGV below is +what actually ran on this Mac (full Metal offload; C2 without --n-cpu-ffn). +""" +from __future__ import annotations + +import argparse +import hashlib +import importlib.util +import json +import re +import statistics +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[3] +BLUEPRINT = ROOT / "blueprints/convergence-practice/local-inference-latest-20260926" +COMMON = ("--ctx-size 8192 --parallel 1 --gpu-layers 99 --fit off --threads 6 --threads-batch 6 " + "--batch-size 512 --ubatch-size 128 --cache-ram 0 --no-webui --offline") +RUNS = { # label -> (run directory name, llama.cpp build, extra flags) + "C0@b11057": ("c0-metal-20260927", "b11057 (59657a613), the macOS pin", ""), + "C0@b11146": ("c0-metal-b11146-20260927", "b11146 (7fe450e19), the plan's runtime", ""), + "C2@b11146": ("c2-metal-b11146-20260927", "b11146 (7fe450e19), the plan's runtime", + "--spec-type draft-mtp --spec-draft-n-max 3"), +} +PAIRS = ( + ("C0@b11146", "C2@b11146", "MTP drafting on the same build and flags"), + ("C0@b11057", "C0@b11146", "runtime build b11057 -> b11146 with the same model and flags"), +) + +_spec = importlib.util.spec_from_file_location("li26_analyze", BLUEPRINT / "analyze.py") +analyze = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(analyze) + + +def meta(run_dir: Path) -> dict: + out = {} + for line in (run_dir / "meta.txt").read_text().splitlines(): + head, sep, rest = line.partition(" ") + if head in ("start", "end") and sep: + out[head] = rest + elif "=" in line: + key, _, value = line.partition("=") + out[key.strip()] = value.strip() + return out + + +def memory_kills(run_dir: Path) -> int: + """memorystatus kill lines in the run window, excluding the `log show` command's own entry.""" + path = run_dir / "jetsam.log" + if not path.exists(): + return -1 + return sum(1 for line in path.read_text().splitlines() + if re.search(r"memorystatus_kill|killing", line, re.I) and "log run noninteractively" not in line) + + +def peak_rss_mib(run_dir: Path): + peak = None + path = run_dir / "memory-samples.txt" + if path.exists(): + for line in path.read_text().splitlines(): + parts = line.split() + if len(parts) >= 2 and parts[1].isdigit(): + peak = max(peak or 0, int(parts[1]) // 1024) + return peak + + +def summary(metrics: dict, run_dir: Path, label: str) -> dict: + filings = metrics["filings"] + n = len(filings) + triples = [analyze.counts(f["gold"], f["predicted"]) for f in filings] + scores = [analyze.filing_scores(f["gold"], f["predicted"]) for f in filings] + macro, per_code = analyze.macro_f1([f["gold"] for f in filings], [f["predicted"] for f in filings]) + speeds = [f["predicted_per_second"] for f in filings if f.get("http_status") == 200 + and (f.get("predicted_n") or 0) >= 1 and isinstance(f.get("predicted_per_second"), (int, float))] + ttft = [f["prompt_ms"] for f in filings if f.get("http_status") == 200 and isinstance(f.get("prompt_ms"), (int, float))] + m = meta(run_dir) + _, build, extra = RUNS[label] + return { + "argv": f"llama-server --model Qwen3.8-27B-UD-Q4_K_M.gguf --alias li26-{label[:2].lower()} {COMMON} {extra}".strip(), + "runtime": build, "build_info": (metrics.get("server") or {}).get("build_info"), + "n": n, "status": metrics["status"], "events": metrics["events"], + "micro_f1": round(analyze.micro_f1(triples), 4), + "macro_f1": round(macro, 4) if macro is not None else None, "macro_codes": len(per_code), + "exact_match_rate": round(sum(s["exact"] for s in scores) / n, 4) if n else None, + "json_valid_rate": round(sum(f["parse_status"] == "valid" for f in filings) / n, 4) if n else 0.0, + "errors": sum(f.get("error") not in (None, "not_attempted") for f in filings), + "not_attempted": sum(f.get("error") == "not_attempted" for f in filings), + "median_decode_tokens_per_second": round(statistics.median(speeds), 2) if speeds else None, + "median_ttft_ms": round(statistics.median(ttft)) if ttft else None, + "server_peak_rss_mib": peak_rss_mib(run_dir), "memory_kills": memory_kills(run_dir), + "window_local": [m.get("start"), m.get("end")], + } + + +def derive(runs_dir: Path) -> dict: + raw = (BLUEPRINT / "plan.json").read_bytes() + plan = json.loads(raw) + rule = plan["decision_rule"] + boot = rule["bootstrap"] + out = {"kind": "li26_mac_metal_descriptive", "host_id": "mac-coordinator-64gb-20260925", "parity": False, + "plan_sha256": hashlib.sha256(raw).hexdigest(), "prompt_sha256": plan["prompt_sha256"], + "model": "unsloth/Qwen3.8-27B-GGUF@4ca720788d1e01f1bff70c033e0d0028fd02e502 Qwen3.8-27B-UD-Q4_K_M.gguf", + "profile_note": "metrics.json's profile field repeats the plan's declaration; argv is what ran here", + "runs": {}, "pairs": []} + loaded = {} + for label, (name, _, _) in RUNS.items(): + path = runs_dir / name / "eval" / "metrics.json" + if not path.exists(): + out["runs"][label] = {"state": "missing"} + continue + metrics = json.loads(path.read_text()) + if metrics.get("plan_sha256") != out["plan_sha256"] or metrics.get("prompt_sha256") != plan["prompt_sha256"]: + raise SystemExit(f"{label}: different plan or prompt") + loaded[label] = metrics + out["runs"][label] = summary(metrics, runs_dir / name, label) + for control, arm, isolates in PAIRS: + if control not in loaded or arm not in loaded: + out["pairs"].append({"control": control, "arm": arm, "state": "missing"}) + continue + c, a = loaded[control]["filings"], loaded[arm]["filings"] + if [(f["accession"], tuple(f["gold"])) for f in c] != [(f["accession"], tuple(f["gold"])) for f in a]: + raise SystemExit(f"{arm} is not paired with {control}") + b = analyze.paired_bootstrap([analyze.counts(f["gold"], f["predicted"]) for f in c], + [analyze.counts(f["gold"], f["predicted"]) for f in a], + boot["resamples"], boot["seed"], boot["alpha"]) + rc, ra = out["runs"][control], out["runs"][arm] + out["pairs"].append({ + "control": control, "arm": arm, "isolates": isolates, "n": len(c), + "delta_micro_f1": round(b["point"], 4), "ci95": [round(b["lower"], 4), round(b["upper"], 4)], + "identical_predictions": sum(sorted(x["predicted"] or []) == sorted(y["predicted"] or []) + and x["parse_status"] == y["parse_status"] for x, y in zip(c, a)), + "decode_speedup": round(ra["median_decode_tokens_per_second"] / rc["median_decode_tokens_per_second"], 3), + "criteria_descriptive": { + "noninferior_micro_f1": b["lower"] >= rule["noninferiority_margin"], + "faster_median_decode": ra["median_decode_tokens_per_second"] > rc["median_decode_tokens_per_second"], + "json_valid_rate": ra["json_valid_rate"] >= rule["json_valid_min"], + "run_clean": ra["status"] == "completed" and not ra["events"] and ra["errors"] == 0 + and ra["not_attempted"] == 0 and ra["memory_kills"] == 0, + }, + }) + return out + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--runs", type=Path, required=True, help="the private mac-runs state directory") + parser.add_argument("--check", type=Path, help="committed summary to compare with") + args = parser.parse_args() + derived = json.dumps(derive(args.runs), indent=2, sort_keys=True) + "\n" + if args.check: + same = args.check.read_text() == derived + d = json.loads(derived) + print(json.dumps({"matches_committed": same, "sha256": hashlib.sha256(derived.encode()).hexdigest(), + "runs": {k: {x: v.get(x) for x in ("micro_f1", "json_valid_rate", "median_decode_tokens_per_second", + "median_ttft_ms", "errors", "memory_kills")} + for k, v in d["runs"].items()}, + "pairs": [{x: p.get(x) for x in ("control", "arm", "delta_micro_f1", "ci95", "identical_predictions", + "decode_speedup")} for p in d["pairs"]]}, sort_keys=True)) + return 0 if same else 1 + sys.stdout.write(derived) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/evidence/artifacts/mac-model-hosting-20260927/swap-event.json b/evidence/artifacts/mac-model-hosting-20260927/swap-event.json new file mode 100644 index 000000000..a1d86d90f --- /dev/null +++ b/evidence/artifacts/mac-model-hosting-20260927/swap-event.json @@ -0,0 +1,56 @@ +{ + "at_c2_end": { + "local": "11:06:29", + "swap_mb": 4977.12 + }, + "below_6gb_by": "10:38:18", + "co_resident_at_spike": { + "apple_container": "one 2 GiB VM (the Hindsight trial Postgres), running since 10:15 local", + "li26_c2_server_rss_mib": 16879, + "minute_samples_around_first_rise": { + "after": { + "free": "34%", + "local": "10:37:56", + "ollama": [ + { + "ctx": 4096, + "name": "qwen3-embedding:4b", + "size_gib": 4.07 + }, + { + "ctx": 65536, + "name": "qwen3.5-9b-64k:latest", + "size_gib": 8.48 + } + ], + "swap_used": "7647.81M" + }, + "before": { + "free": "37%", + "local": "10:36:56", + "ollama": [], + "swap_used": "16.81M" + } + }, + "omniroute_build": { + "build_release_output_written_local": "10:38:22", + "note": "upstream `npm run build:release` (a Next.js production build) ran between these two times, from file birth times", + "npm_ci_finished_local": "10:34:40" + } + }, + "date": "2026-09-27", + "first_rise": { + "local": "10:37:06", + "swap_mb": 4731.31 + }, + "host_id": "mac-coordinator-64gb-20260925", + "kind": "mac_swap_event", + "li26_c2_effect": "none on quality: 0 errors, predictions identical to C0 on all 360 filings", + "memory_kills_in_c2_window": 0, + "peak": { + "local": "10:37:38", + "swap_mb": 13933.56 + }, + "reading": "the model set alone did not swap (16.81 MB through two full li26 runs and the first 16 minutes of C2); swap grew only while a large Node production build overlapped the full model set, then drained slowly and did not grow during the worst-case co-residency probe", + "swap_before_mb": 16.81 +} From 867fd4bf6e0662c1f5890ff1d8c0b9ad7d6a8f04 Mon Sep 17 00:00:00 2001 From: seathatflowsinourveins <234074349+seathatflowsinourveins@users.noreply.github.com> Date: Sun, 27 Sep 2026 11:23:25 -0400 Subject: [PATCH 2/3] #379: llama-cpp use receipt on mac-coordinator-64gb-20260925, evidence registered host_receipts.py record at the published revision b16b84b8: six live commands (launchd and ports, the embed server's identity, embed acceptance, the Metal device check, a llama-bench of Qwen3.8-27B on Metal, and the li26 descriptive re-derivation), all exit 0, result pass, with three qualified_models entries. manifests/evidence.json from main with this branch's files re-registered; validate.py passed (7343 hashed files); host_receipts.py validate 176 passed. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017d27CMzkbLezdLv7vAStPu --- ...gb-20260925--llama-cpp--use--20260927.json | 122 ++++++++++++++++++ manifests/evidence.json | 59 ++++++++- 2 files changed, 179 insertions(+), 2 deletions(-) create mode 100644 evidence/hosts/mac-coordinator-64gb-20260925/mac-coordinator-64gb-20260925--llama-cpp--use--20260927.json diff --git a/evidence/hosts/mac-coordinator-64gb-20260925/mac-coordinator-64gb-20260925--llama-cpp--use--20260927.json b/evidence/hosts/mac-coordinator-64gb-20260925/mac-coordinator-64gb-20260925--llama-cpp--use--20260927.json new file mode 100644 index 000000000..0751ba70e --- /dev/null +++ b/evidence/hosts/mac-coordinator-64gb-20260925/mac-coordinator-64gb-20260925--llama-cpp--use--20260927.json @@ -0,0 +1,122 @@ +{ + "schema_version": 1, + "id": "mac-coordinator-64gb-20260925--llama-cpp--use--20260927", + "kind": "host_acceptance", + "host": { + "host_id": "mac-coordinator-64gb-20260925", + "platform_id": "macos-arm64", + "os": "macos", + "architecture": "arm64", + "second_physical_machine": true, + "hardware_profile_ref": "mac-coordinator-64gb-20260925" + }, + "catalog_revision": "b16b84b821f75a8c900557b763a2cdbef62e8f4e", + "recorded_by": { + "identity_sha256": "91719ef8aa813c0bf50b26570098baa57e4c043d7d23d4cc5c26ce7c17710954", + "model": "claude-opus-5-5" + }, + "component_id": "llama-cpp", + "stage": "use", + "commands": [ + { + "cmd": "launchctl list | awk '$3 ~ /agent-ecosystem|native-stack/ {print ($1==\"-\"?\"stopped\":\"running\"), \"last_exit=\" $2, $3}' && for p in 6333 8232 11434 49374 16333; do printf \"port %s: \" \"$p\"; lsof -nP -iTCP:\"$p\" -sTCP:LISTEN 2>/dev/null | awk 'NR==2{print $1; f=1} END{if(!f) print \"none\"}'; done && curl -fsS -m 5 http://127.0.0.1:8232/health && echo", + "exit": 0, + "duration_s": 0.335, + "output_sha256": "5a44eb414069f492fa8c63591169c1e60b8b223b8102cb0d45323b96a6a96d45", + "output_excerpt": "running last_exit=0 local.agent-ecosystem.ai-memory\nstopped last_exit=1 local.agent-ecosystem.maintenance\nrunning last_exit=0 local.agent-ecosystem.ollama\nrunning last_exit=0 local.agent-ecosystem.qdrant\nrunning last_exit=0 com.native-stack.omniroute\nrunning last_exit=0 com.native-stack.llama-embed\nport 6333: qdrant\nport 8232: llama-ser\nport 11434: ollama\nport 49374: ai-memory\nport 16333: none\n{\"s" + }, + { + "cmd": "P=\"$HOME/Library/LaunchAgents/com.native-stack.llama-embed.plist\" && plutil -extract ProgramArguments json -o - \"$P\" | python3 -c 'import json,os,sys; a=json.load(sys.stdin); print(\"argv:\", \" \".join(os.path.basename(x) if x.startswith(\"/\") else x for x in a))' && M=$(plutil -extract ProgramArguments.5 raw -o - \"$P\") && stat -f \"%z bytes\" \"$M\" && shasum -a 256 \"$M\" | cut -d\" \" -f1 && \"$(plutil -extract ProgramArguments.0 raw -o - \"$P\")\" --version 2>&1 | head -1", + "exit": 0, + "duration_s": 0.709, + "output_sha256": "b81f7ab8026a4286e2b332ba2eecfc564681d91d534c50de0cf015fb87017e53", + "output_excerpt": "argv: llama-server --embedding --port 8232 -m embeddinggemma-300M-Q8_0.gguf\n333590944 bytes\nb5ce9d77a3fc4b3b39ccb5643c36777911cc4eb46a66962eadfa3f5f60490d63\n0.00.000.202 I srv llama_server: initializing ...\n" + }, + { + "cmd": "python3 tools/adoption/embed_acceptance.py http://127.0.0.1:8232 evidence/artifacts/macos-embed-reference-20260923/macos-embed-reference-20260923.json", + "exit": 0, + "duration_s": 0.12, + "output_sha256": "2fb3cf7549659183986fc0f6d8c9740ec0d25fed7758b49c3e4ce394a20ab375", + "output_excerpt": "{\n \"passed\": true,\n \"dimension_expected\": 768,\n \"dimension_actual\": 768,\n \"cosine\": 0.999661109106148,\n \"threshold\": 0.99,\n \"url\": \"http://127.0.0.1:8232/v1/embeddings\"\n}\n" + }, + { + "cmd": "P=\"$HOME/Library/LaunchAgents/com.native-stack.llama-embed.plist\" && S=$(plutil -extract ProgramArguments.0 raw -o - \"$P\") && M=$(plutil -extract ProgramArguments.5 raw -o - \"$P\") && D=$(mktemp -d \"${TMPDIR:-/tmp}/mac379-metal.XXXXXX\") && { \"$S\" --embedding --port 18233 -m \"$M\" -lv 4 > \"$D/log\" 2>&1 & pid=$!; for i in $(seq 1 60); do curl -s -m 2 http://127.0.0.1:18233/health | grep -q ok && break; sleep 0.5; done; kill $pid; wait $pid 2>/dev/null; grep -E \"using device MTL0|offloaded [0-9]+/[0-9]+ layers|assigned to device CPU\" \"$D/log\" | sed -E \"s/^[0-9.]+ [A-Z] +//\" | sort | uniq -c; rm -rf \"$D\"; }", + "exit": 0, + "duration_s": 0.568, + "output_sha256": "4d2a1e385ca58133c2e5bc9a3c84e7e66cd49eb608c8f27b47f5e245fa919bd7", + "output_excerpt": " 1 llama_prepare_model_devices: using device MTL0 (Apple M5 Pro) (unknown id) - 53083 MiB free\n 1 load_tensors: offloaded 25/25 layers to GPU\n" + }, + { + "cmd": "M=\"$HOME/.local/share/codex-ecosystem-staging/state/models/qwen3.8-27b-ud-q4km-4ca72078/Qwen3.8-27B-UD-Q4_K_M.gguf\" && stat -f \"%z bytes\" \"$M\" && shasum -a 256 \"$M\" | cut -d\" \" -f1 && \"$HOME/.local/share/codex-ecosystem-staging/tools/llama-cpp-b11057/llama-bench\" -m \"$M\" -p 512 -n 128 -ngl 99 -fa on -r 1 -o json 2>/dev/null | python3 -c 'import json,sys; [print(\"build\", e[\"build_number\"], e[\"build_commit\"], e[\"backends\"], e[\"gpu_info\"], \"ngl\", e[\"n_gpu_layers\"], \"pp%d/tg%d\" % (e[\"n_prompt\"], e[\"n_gen\"]), round(e[\"avg_ts\"], 1), \"tok/s\") for e in json.load(sys.stdin)]'", + "exit": 0, + "duration_s": 39.859, + "output_sha256": "e9ca313cf4167017a710331d80e559b55362a050878bfa74af05a265d6adde9c", + "output_excerpt": "16464440224 bytes\n322e194ff79741c7baa497c240f677f54b201b0efab44ca8e50f122b39123482\nbuild 11057 59657a613 MTL,BLAS Apple M5 Pro ngl 99 pp512/tg0 370.4 tok/s\nbuild 11057 59657a613 MTL,BLAS Apple M5 Pro ngl 99 pp0/tg128 16.1 tok/s\n" + }, + { + "cmd": "python3 -B evidence/artifacts/mac-model-hosting-20260927/li26_mac_descriptive.py --runs \"$HOME/.local/state/native-agent-stack/local-inference-latest-20260926/mac-runs\" --check evidence/artifacts/mac-model-hosting-20260927/li26-mac-descriptive.json", + "exit": 0, + "duration_s": 0.732, + "output_sha256": "00b71872252282a35a08eff93f9cc3c04cb9a3b5ef3c1e91669c6abd3af3040c", + "output_excerpt": "{\"matches_committed\": true, \"pairs\": [{\"arm\": \"C2@b11146\", \"ci95\": [0.0, 0.0], \"control\": \"C0@b11146\", \"decode_speedup\": 1.515, \"delta_micro_f1\": 0.0, \"identical_predictions\": 360}, {\"arm\": \"C0@b11146\", \"ci95\": [0.0, 0.0], \"control\": \"C0@b11057\", \"decode_speedup\": 1.004, \"delta_micro_f1\": 0.0, \"identical_predictions\": 360}], \"runs\": {\"C0@b11057\": {\"errors\": 0, \"json_valid_rate\": 1.0, \"median_decod" + } + ], + "tool_versions": { + "llama-cpp": "b11057 (0.4.1-dev)" + }, + "observed_at_utc": "2026-09-27T15:22:06Z", + "result": "pass", + "claim": "On host mac-coordinator-64gb-20260925 (macOS 27.0 arm64, Apple M5 Pro, 64 GB unified memory) the pinned RAG profile ran: the com.native-stack.llama-embed LaunchAgent (llama.cpp b11057) served embeddinggemma-300M-Q8_0 on 127.0.0.1:8232 beside the pre-existing Qdrant (6333), Ollama (11434) and ai-memory (49374) agents, with no second Qdrant or ai-memory and nothing on 16333 (first command). Its launchd argv, served file size and sha256, and binary version are shown by the second command. tools/adoption/embed_acceptance.py passed against the Linux reference (third command), and a short-lived instance with the same argv at -lv 4 reported MTL0 with all 25 layers offloaded (fourth command). The b11057 llama-bench ran Qwen3.8-27B UD-Q4_K_M @4ca72078 (16,464,440,224 bytes, sha256 322e194f...3482) fully on Metal (fifth command). The sixth command re-derived the committed li26 descriptive summary (evidence/artifacts/mac-model-hosting-20260927/li26-mac-descriptive.json) from the retained private runs with the frozen analyze.py functions and reports whether it matches.", + "limitations": [ + "This Mac is not the li26 preregistered host. The li26 figures describe this host and are neither a parity result nor a replacement decision; the sixth command re-derives the committed summary from the retained private runs and does not run li26 again.", + "The capacity probe's llama-bench figures are upstream-tool measurements (native_proven). Its llama-server serving figures used this repository's own docs as prompts, so they are local_integration, recorded in capacity.json.", + "Builds b11146 and b11214 are isolated trial installs used for the plan's runtime (li26) and for the newest-build probes. The macOS pin stays b11057; re-pinning is its own reviewed change.", + "The worst-case co-residency probe requested the memory stack's own Ollama models with keep_alive and drove all three model servers at once. It did not replay ai-memory's real consolidation traffic.", + "Embedding acceptance checks one fixed request against the Linux reference vector. It is not a retrieval-quality comparison; that belongs to the S3 preregistration (#390)." + ], + "evidence_class": "native_proven", + "reviews": [ + { + "kind": "self", + "ref": "scripts/host_receipts.py record", + "verdict": "agree", + "at_utc": "2026-09-27T15:22:06Z", + "reviewer": { + "identity_sha256": "91719ef8aa813c0bf50b26570098baa57e4c043d7d23d4cc5c26ce7c17710954", + "model": "claude-opus-5-5" + } + } + ], + "layer_refs": [ + { + "catalog": "foundation", + "layer_id": "observation-inference" + } + ], + "qualified_models": [ + { + "model_id": "ggml-org/embeddinggemma-300M-GGUF", + "revision": "0f741b5a6585bd53aeb15cd1372c56f2a0f65e12", + "runtime": "llama.cpp llama-server --embedding on Metal (LaunchAgent com.native-stack.llama-embed, 127.0.0.1:8232), file embeddinggemma-300M-Q8_0.gguf", + "runtime_version": "b11057 (0.4.1-dev)", + "bars": "tools/adoption/embed_acceptance.py against evidence/artifacts/macos-embed-reference-20260923: 768 dimensions and cosine >= 0.99 (0.99966 measured); all 25 layers offloaded to MTL0 per llama-server's own log. This receipt's third and fourth commands.", + "result": "pass" + }, + { + "model_id": "unsloth/Qwen3.8-27B-GGUF", + "revision": "4ca720788d1e01f1bff70c033e0d0028fd02e502", + "runtime": "llama.cpp llama-server on Metal, file Qwen3.8-27B-UD-Q4_K_M.gguf; li26 serving values with --gpu-layers 99, 8192 context, one slot (the li26 C0 profile, full offload)", + "runtime_version": "b11057 (0.4.1-dev)", + "bars": "li26 frozen protocol run descriptively on this host (360 SEC 8-K filings of 2020-03-02, greedy non-thinking): micro-F1 0.9909, JSON-valid 1.0, 0 errors, 0 memory kills, median decode 15.73 tokens/s. Not the preregistered host; no parity or replacement claim. The sixth command re-derives it from the retained private runs.", + "result": "pass" + }, + { + "model_id": "unsloth/Qwen3.8-27B-GGUF", + "revision": "4ca720788d1e01f1bff70c033e0d0028fd02e502", + "runtime": "llama.cpp llama-server on Metal, the same file; the li26 C2 drafting profile (--spec-type draft-mtp --spec-draft-n-max 3) with --gpu-layers 99, 8192 context, one slot", + "runtime_version": "b11146 (0.5.0-dev), an isolated trial install of the official release asset", + "bars": "The li26 frozen rule applied descriptively to this host's own C0/C2 pair on b11146: paired bootstrap 95% lower bound 0.0 >= -0.02 (identical predictions on all 360 filings), median decode 23.94 vs 15.80 tokens/s, JSON-valid 1.0, no errors or memory kills. The sixth command re-derives it.", + "result": "pass" + } + ] +} diff --git a/manifests/evidence.json b/manifests/evidence.json index 1acdf8f04..faa50e40d 100644 --- a/manifests/evidence.json +++ b/manifests/evidence.json @@ -3667,8 +3667,8 @@ }, { "path": "adoption/platforms/macos-arm64.md", - "sha256": "74d41a5bace078700b8fb5b141db0c8b65b47ec1f2bf41bbe040fa0fced9b64f", - "bytes": 55691 + "sha256": "e89a6b1fabc92a1c5cf6eef22d4a8f9ee845c405af6d4d7f3e20b2105bfc935d", + "bytes": 56642 }, { "path": "adoption/pre-login-observation.json", @@ -13851,6 +13851,11 @@ "sha256": "e9bf99f9905121292e1c6a96b8b59e12127ddd1f74b9ab8240cb8497e8c19443", "bytes": 8203 }, + { + "path": "docs/decisions/2026-09-27-mac-native-model-hosting.md", + "sha256": "56c682d70dbd95d9499e4197fca4c2c85157873ebd6c95cf4c7a8740e633f726", + "bytes": 6256 + }, { "path": "docs/decisions/2026-09-27-mac-single-writer-staged.md", "sha256": "517441575c63c19ff252f6fb53eca2e09a5c910e565aabde45da221e6922247f", @@ -30641,6 +30646,51 @@ "sha256": "eea0ecfdfd6c022208f0c812dd634c7ba6fff4b0455e875dce017d051ba4ab73", "bytes": 4691 }, + { + "path": "evidence/artifacts/mac-model-hosting-20260927/README.md", + "sha256": "cb36851fee45d7ca7698d7676cee3535dbd487bc8a30a34324974eef2b401de6", + "bytes": 17438 + }, + { + "path": "evidence/artifacts/mac-model-hosting-20260927/capacity.json", + "sha256": "1eb91f3aa06e3138ff49259964203c3f240630f8d270dbeddcc37512e29415ff", + "bytes": 3205 + }, + { + "path": "evidence/artifacts/mac-model-hosting-20260927/coresidency.json", + "sha256": "7aec6d2595e9945928c9395b98be84bdcd960a6294efe5a7346527b44cc80984", + "bytes": 5809 + }, + { + "path": "evidence/artifacts/mac-model-hosting-20260927/embed-acceptance.json", + "sha256": "a5b64858d49ff54c0ecca6e8f581e6248aa6a95b02719fcfb284c4d6a9ecfc9d", + "bytes": 1768 + }, + { + "path": "evidence/artifacts/mac-model-hosting-20260927/gate.json", + "sha256": "9944f8d614543e0743e4effc97dc8804ed79e32893500058df386053c3851581", + "bytes": 15361 + }, + { + "path": "evidence/artifacts/mac-model-hosting-20260927/li26-mac-descriptive.json", + "sha256": "586cde01ad338a2f435571e00e2b161ca20d97ffc141b948bb88dab1a8d469fc", + "bytes": 4046 + }, + { + "path": "evidence/artifacts/mac-model-hosting-20260927/li26-window-samples.json", + "sha256": "d5b1794bc28f8909fbbbd3a0a7495ce9208b586a7e6f8663ab96022481a44a0a", + "bytes": 838 + }, + { + "path": "evidence/artifacts/mac-model-hosting-20260927/li26_mac_descriptive.py", + "sha256": "935893d8beeee746716869b95d3a1c67ee595336b77e191751273763eb822ead", + "bytes": 9460 + }, + { + "path": "evidence/artifacts/mac-model-hosting-20260927/swap-event.json", + "sha256": "1f8a103c20d063e941523b95baa5820ed3f3a8b835ebf1e1efbd2fee509a00a7", + "bytes": 1766 + }, { "path": "evidence/artifacts/mac-stage1-client-layer-20260927/README.md", "sha256": "df6c3e4ad05911232fb6d4b027774f9d57730c1a5b59fe973ae382bff9e2a5b8", @@ -36611,6 +36661,11 @@ "sha256": "49cfbdfc06c5626d3e3244cce2efbad58d37cddc059bcb1ea793f195b7350dd5", "bytes": 8718 }, + { + "path": "evidence/hosts/mac-coordinator-64gb-20260925/mac-coordinator-64gb-20260925--llama-cpp--use--20260927.json", + "sha256": "1da651490274b522a8324d389b5c8ddc2fd8ffcc51c873cee6196b125829a9b8", + "bytes": 10830 + }, { "path": "evidence/hosts/mac-coordinator-64gb-20260925/mac-coordinator-64gb-20260925--qdrant--use--20260925.json", "sha256": "500f736cb5e4ea23935fc1e8e9ad7ddb38479129dd59474123962e1aa6e2b98a", From 14b5c52145d9cc42d4c9da36df53ac994009856a Mon Sep 17 00:00:00 2001 From: seathatflowsinourveins <234074349+seathatflowsinourveins@users.noreply.github.com> Date: Sun, 27 Sep 2026 11:28:27 -0400 Subject: [PATCH 3/3] Anti-pattern log: a PR-branch catalog_revision breaks main after the squash #391's receipts cited commits published only on its head branch; the repository deletes merged head branches and CI fetches heads and tags only, so main went red until the revisions were tagged. The 2026-09-25 row now says to tag every such revision (receipt-revision/) before merging and to check it in a full main-plus-tags clone. This PR's own receipt revision is tagged (receipt-revision/b16b84b8). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017d27CMzkbLezdLv7vAStPu --- docs/harness-defaults.md | 2 +- manifests/evidence.json | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/harness-defaults.md b/docs/harness-defaults.md index 053c71e97..7892fcb00 100644 --- a/docs/harness-defaults.md +++ b/docs/harness-defaults.md @@ -90,7 +90,7 @@ When a claim or action proves wrong, correct it where it was relayed and record | 2026-09-25 | Reading a filtered command's output as the raw output | Through rtk 0.50.0's Claude hook, a large `git show :` blob is windowed to about 8 KiB, so a following `tail` reads the window rather than the file end, and `diff` on a missing file exits 1 instead of 2 | Exclude those forms with rtk's four-entry `[hooks] exclude_commands` value and confirm with `rtk hook check`; read an exact blob with native `git show` or `rtk proxy git show` | [Claude hook exclusions at 0.50.0](../recipes/README.md#native-context-mode-and-hooks); the `adoption/bootstrap-linux.sh` reminder; `evidence/receipts/rtk-050-qualification-20260925.json` | | 2026-09-25 | Letting a harness feature rewrite shared repository configuration | Launching an agent with `isolation: "worktree"`, or an agent type that declares it such as the shipped `isolated-builder`, rewrote the shared `core.hooksPath` from the relative `scripts/git-hooks` to the main checkout's absolute path; whenever the main checkout's branch lacked `scripts/git-hooks`, every worktree committed without the gitleaks pre-commit scan | Create the worktree yourself (`git worktree add --no-track -b origin/main`) and pass its path; run `git config --show-origin --get core.hooksPath` before relying on hooks | Coordinator briefs; the `secret-scan` job in `.github/workflows/validate.yml` scans every pull request. Appended 2026-09-27: no shipped agent declares `isolation` any more, and `examples/claude-native/workflows/test-envelope.mjs` fails one that does ([record](decisions/2026-09-27-claude-harness-settings.md)) | | 2026-09-25 | A wait loop whose pattern matches itself | `until ! pgrep -f "unittest discover"; do sleep 3; done` never ended: pgrep never reports its own process (procps-ng `pgrep(1)`), but the waiting shell's command line contains the pattern; one waiter ran for almost ten hours | Bracket one character (`"[u]nittest discover"`), anchor the pattern to the command name, or wait on a PID or exit-marker file, with a timeout | This log; an anchored pattern in `blueprints/gap-wave2-20260923/foundation__quality-evaluation/run_codex_author.sh` | -| 2026-09-25 | Recording evidence against an unpublished commit | Host receipts recorded in a worktree on a local-only commit named it as `catalog_revision`; validation passed locally because worktrees share one object store, while a clone of the branch alone fails | Record receipts only at a commit on GitHub (`git merge-base --is-ancestor origin/main`) and validate in `git clone --no-local --single-branch --branch ` | Partly, in CI: `.github/workflows/validate.yml` runs `scripts/host_receipts.py validate` on a full-history checkout from GitHub, so a never-pushed commit fails there ("is not a commit present in this checkout"). A local worktree accepts a local-only commit, because `git cat-file -e` finds it in the shared object store, and whether the commit is on `main` stays a manual check | +| 2026-09-25 | Recording evidence against an unpublished commit | Host receipts recorded in a worktree on a local-only commit named it as `catalog_revision`; validation passed locally because worktrees share one object store, while a clone of the branch alone fails | Record receipts only at a commit on GitHub (`git merge-base --is-ancestor origin/main`) and validate in `git clone --no-local --single-branch --branch ` | Partly, in CI: `.github/workflows/validate.yml` runs `scripts/host_receipts.py validate` on a full-history checkout from GitHub, so a never-pushed commit fails there ("is not a commit present in this checkout"). A local worktree accepts a local-only commit, because `git cat-file -e` finds it in the shared object store, and whether the commit is on `main` stays a manual check. Appended 2026-09-27: a commit that is published only on a PR branch fails the same way after the squash merge, because the repository deletes merged head branches (`delete_branch_on_merge: true`) and CI fetches only heads and tags. #391's receipts turned `main` red this way. Before merging, tag every such revision with an annotated `receipt-revision/` tag, and check it in a full `git clone --no-local --single-branch --branch main` plus tags; a `--filter=blob:none` clone hides the failure | | 2026-09-25 | Inline code inside a guarded shell command | The PreToolUse secret guard tokenizes inline Python and heredocs, and a segment starting with `set(` after a pipe was blocked as `environment_dump` | Put non-trivial Python in a script file and run that file; never bypass or edit the guard | `scripts/hooks/secret_path_guard.py` (`is_environment_dump`); `evidence/artifacts/token-e2e-ultracode-20260925/README.md` | | 2026-09-25 | Taking a pipeline's exit code as the command's result | A unittest run piped into `tail` returned tail's exit 0 and dropped the summary printed on stderr, so a "suite passed" log rested on tail; a `git push` piped into `tail` hid a refused push | Write stdout and stderr to files and read the command's own exit code and summary; read a push back with `git ls-remote origin refs/heads/` | This log | | 2026-09-26 | Trusting a flag's name instead of the request it sends | Against a local stand-in provider (cases W1 to W3 on Codex 0.155.1 and 0.157.1, local integration evidence), `codex --search exec` sent `external_web_access: false` (Codex's cached web index), the same as no flag, and only `-c web_search="live"` sent `true`; OpenAI parity rests on source reading, and a live `-c web_search="live"` search seen on OpenAI was not retained. That the landscape sweep's GPT-6 lanes, started with `codex --search exec`, searched a cached index is therefore an inference | Pass `-c web_search="live"` when a lane needs current upstream facts, and verify a flag by the request or configuration the tool actually sends | `evidence/artifacts/landscape-sweep-20260926/README.md` (cases W1 to W3); `docs/decisions/2026-09-26-workstation-tool-refresh.md`; `tools/sota-convergence/landscape-sweep/codex_job.py` still passes `--search` at this base, with its fix pending | diff --git a/manifests/evidence.json b/manifests/evidence.json index faa50e40d..0da68aec7 100644 --- a/manifests/evidence.json +++ b/manifests/evidence.json @@ -14003,8 +14003,8 @@ }, { "path": "docs/harness-defaults.md", - "sha256": "bd6677f6fed5170a574206ed366bee91dfbcbf16869b78999e9fc0424de9bb8d", - "bytes": 35935 + "sha256": "a4a17bb12bf5a526b1145a0c844ee2ef8edba297d474a2429acf47218df015de", + "bytes": 36435 }, { "path": "docs/harness-rules-convergence-20260922.md",