Skip to content

Grand-catalog evidence bank on the workstation: 7 WSL winner rows to accepted, local-model eval E2E (promptfoo, Inspect) - #281

Merged
seathatflowsinourveins merged 1 commit into
mainfrom
claude/grand-catalog-evidence-bank-20260925
Sep 25, 2026
Merged

seathatflowsinourveins merged 1 commit into
mainfrom
claude/grand-catalog-evidence-bank-20260925

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Scope

  • What this PR changes: grand-catalog evidence on the workstation nativestack-5975wx-20260925. The receipts are ordered by the WSL rows that the future Linux enforcement would downgrade, and by the north-star action each one supports.
    • New native_proven receipts:
      • simulation data: data-duckdb use, data-pandera use;
      • backtesting: LEAN use;
      • document retrieval for filings: Poppler install plus a superseding use;
      • paper-hosting supervision: systemd use;
      • MCP surfaces: MCP Inspector use;
      • hosting: FastAPI use;
      • local-model evaluation end to end: promptfoo use and Inspect use against this host's llama.cpp server.
  • Base commit: current origin/main, after Token-profile install receipts and a MCPorter use receipt on the workstation (MCPorter host_verified) #277 and Independent reviews of the five WSL data and skfolio receipts (5 agree) #285. The independent reviews of Record 26 independently reviewed host receipts on nativestack-5975wx-20260925 #262's five self-only receipts landed separately in Independent reviews of the five WSL data and skfolio receipts (5 agree) #285, so this PR no longer touches those files. New receipts were recorded at the published commits 7783682f and 5b6014eb, and every catalog_revision is an ancestor of the base. The branch was rebuilt under the hot-file protocol: main's manifests/evidence.json was taken, the files were re-registered, and the matrix and grand list were regenerated.
  • Lane: lane:foundation. Only evidence/hosts/** and the generated reports change; trading-catalog rows appear only as generated matrix/grid output.
  • Owned paths touched: evidence/hosts/nativestack-5975wx-20260925/ (15 new receipt files). Regenerated: catalogs/landscape/component-evidence-matrix.json, docs/component-evidence-matrix.md, catalogs/landscape/new-host-grand-list.json, docs/new-host-grand-list.md. Hot file: manifests/evidence.json, updated in the last commit.

Effect on the grand catalog (generated from the matrix JSON, against current main)

Catalog / layer Winner WSL2 derived status (before → after) Independently reviewed passes (before → after)
foundation / document-retrieval poppler conditional → accepted 1 → 3
foundation / hosting-services fastapi conditional → accepted 0 → 1
foundation / mcp-surfaces mcp-inspector accepted → accepted 0 → 1
foundation / quality-evaluation promptfoo conditional → accepted 0 → 1
foundation / scheduling-supervision systemd conditional → accepted 1 → 2
us-equities / backtesting-engine lean accepted → accepted 0 → 1
us-equities / data-quality-orchestration data-pandera conditional → accepted 0 → 1
us-equities / evaluation-experiments inspect-ai not_established → accepted 0 → 1
us-equities / storage-compute data-duckdb conditional → accepted 1 → 2
  • Winner rows whose WSL2 derived status becomes accepted: 7 (listed above).
  • WSL2 pairs accepted end to end: 48 → 53. host_verified on a host receipt: 33 → 42.
  • WSL2 needs_host: 18 → 13.
  • Rows that would downgrade under Linux enforcement: 6 → 4. Remaining: actions-attest (unpinned), nextjs, postgresql, and data-alpaca-py (its use needs the rotated paper keys).

Receipts (generated from the receipt JSON)

Receipt Stage Class Result Tool version Independent review
nativestack-5975wx-20260925--data-duckdb--use--20260925.json use native_proven pass v1.5.5; source d8cdaa33fda8df955cc76ef58a280f68f4cd43fa independent_session agree
nativestack-5975wx-20260925--data-pandera--use--20260925.json use native_proven pass v0.33.1; source 62f55e2dccf0a199cfe4d6ce3eda0c1d29e2e4e6 independent_session agree
nativestack-5975wx-20260925--fastapi--use--20260925-2.json use native_proven pass 0.141.1 independent_session agree
nativestack-5975wx-20260925--fastapi--use--20260925.json use native_proven pass 0.141.1 independent_session needs_changes
nativestack-5975wx-20260925--inspect-ai--use--20260925-2.json use native_proven pass inspect-ai0.3.266; source ec4dfc6953784dc45b79de3147530c89868c6e26 independent_session agree
nativestack-5975wx-20260925--inspect-ai--use--20260925.json use native_proven pass inspect-ai0.3.266; source ec4dfc6953784dc45b79de3147530c89868c6e26 independent_session needs_changes
nativestack-5975wx-20260925--lean--use--20260925.json use native_proven pass 985ef30ad3ac774218c5ac516b4cb0aa2655730f independent_session agree
nativestack-5975wx-20260925--mcp-inspector--use--20260925-2.json use native_proven pass 2.7.0 independent_session agree
nativestack-5975wx-20260925--mcp-inspector--use--20260925.json use native_proven pass 2.7.0 independent_session needs_changes
nativestack-5975wx-20260925--poppler--install--20260925.json install native_proven pass 26.09.0 independent_session agree
nativestack-5975wx-20260925--poppler--use--20260925-2.json use native_proven pass 26.09.0 independent_session agree
nativestack-5975wx-20260925--promptfoo--use--20260925-2.json use native_proven pass 0.123.1 independent_session agree
nativestack-5975wx-20260925--promptfoo--use--20260925.json use native_proven pass 0.123.1 independent_session needs_changes
nativestack-5975wx-20260925--systemd--use--20260925-2.json use native_proven pass 255.4-1ubuntu8.17 independent_session agree
nativestack-5975wx-20260925--systemd--use--20260925.json use native_proven pass 255.4-1ubuntu8.17 independent_session needs_changes

Recorders and reviewers:

  • Every receipt was recorded by the coordinator session, 106d189e….
  • The reviewers were separate headless claude -p sessions on claude-sonnet-5, which could not edit files and appended reviews only through scripts/host_receipts.py review --kind independent_session:
    • 739ff768… did the first round;
    • a90718ea… did the repair round.

Review and repair (one round)

The first independent review agreed on 5 new receipts and held 5 at needs_changes, all on point 3 (backed claims):

  • FastAPI: the serving environment's versions were not retained.
  • promptfoo and Inspect: the served model was named, but no command read it.
  • MCP Inspector: the claim said "temporary HOME", but isolation came from a project-local index, and the QMD version was unbacked.
  • systemd: runtime, CPU and memory were claimed, but the excerpt was truncated.

Each fix was recorded as a superseding -2 generation from a detached worktree at published origin/main 5b6014eb. They add a serving-environment version readback, a read-only readback of the model server (unit --model basename and --alias, runtime build, /health) and qmd --version, and they reword or drop the unbacked clauses. The repair-round review agreed on all five. The superseded generations stay in the tree as history.

Safety notes

  • MCP Inspector runs only in --cli mode with MCP_AUTO_OPEN_ENABLED=false (no browser), and its version is read with npm ls.
  • The local-model calls are 5 and 4 tiny deterministic requests to the shared llama.cpp server. The API key is read from its file inside the command and never printed.
  • No production store, service or unit was changed. systemd uses only transient units.

Local commands run (on this branch)

scripts/component_matrix.py --write; scripts/new_host_grand_list.py --write
the offline steps of validate.yml's validate job, in order (zizmor and actionlint run in CI only), plus evidence_manifest.py --check -> all passed (0 failures)
verdict_review_gate.py (from a worktree at origin/main) --root . --base origin/main -> passed (0 changed rows, 0 violations)
python -m unittest (uv Python 3.13 with jsonschema/PyYAML/numpy; PROMOTION_GATE_PYTHON set) -> Ran 5387 tests, OK (skipped=612)

Residuals

  • Codex cross-family review: skipped. Codex resets 2026-09-30 23:50 local; the Opus weekly limit resets 2026-09-30 21:00 ET, so the reviewers ran on Sonnet.
  • PostgreSQL 18.6 use: not recorded. No 18.6 binary exists on this host, and every official route needs sudo or a new package source.
  • nextjs: the winner pin 16.3.5 predates the Bump the application-delivery lock to Next.js 16.3.6 for GHSA-vcvr-r3jv-pc5j #252 CVE bump to 16.3.6; that is a wave item.
  • systemd: transient units only. Linger, reboot and power-loss recovery remain the layer's open gaps.
  • Status flips: they take effect in the derived views now. Declared catalog rows change only in the next verdict wave, after 09-30.

🤖 Generated with Claude Code

@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Sep 25, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@seathatflowsinourveins seathatflowsinourveins changed the title Grand-catalog evidence bank on the workstation: 13 WSL winner rows to accepted, local-model eval E2E (promptfoo, Inspect) Grand-catalog evidence bank on the workstation: 12 WSL winner rows to accepted, local-model eval E2E (promptfoo, Inspect) Sep 25, 2026
…ve to accepted

New native_proven receipts for data-duckdb, data-pandera,
LEAN, Poppler (install + superseding use), systemd, MCP Inspector (--cli only),
FastAPI, promptfoo and Inspect (local-model eval end to end on the host's
llama.cpp server). One review/repair round: five receipts held at needs_changes on
backed claims were superseded by -2 generations recorded at published main and
agreed by a fresh reviewer. WSL2 pairs accepted end to end 48 -> 53
(host_verified 27 -> 41), needs_host 18 -> 13. Replayed onto current main under the hot-file protocol; matrix and grand list regenerated;
evidence manifest re-registered under the hot-file protocol.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@seathatflowsinourveins seathatflowsinourveins changed the title Grand-catalog evidence bank on the workstation: 12 WSL winner rows to accepted, local-model eval E2E (promptfoo, Inspect) Grand-catalog evidence bank on the workstation: 7 WSL winner rows to accepted, local-model eval E2E (promptfoo, Inspect) Sep 25, 2026
@seathatflowsinourveins
seathatflowsinourveins force-pushed the claude/grand-catalog-evidence-bank-20260925 branch from 03400f8 to b31b625 Compare September 25, 2026 21:30
@seathatflowsinourveins
seathatflowsinourveins merged commit 8df847b into main Sep 25, 2026
23 of 26 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/grand-catalog-evidence-bank-20260925 branch September 25, 2026 21:48
seathatflowsinourveins added a commit that referenced this pull request Sep 25, 2026
…, token wiring and local-model eval gates (#297)

* Grand-dashboard checkpoint: memory lifecycle v2, retrieval quality v2, token wiring and local-model eval gates

Adds four dated foundation gates for work that merged today (#289, #292, #293, and the #281 local-model receipts), inserted after the existing memory-lifecycle gate; no existing entry changed. States use the renderer's public metadata token alphabet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Re-run checks with the PR body's SOTA sources section

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Scout <scout@local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator Author

Cross-family review (Codex CLI 0.155.1, gpt-6-astra, model_reasoning_effort=max, read-only sandbox), 2026-09-26. Run by the workstation coordinator against a clean checkout of main after merge. Supported findings get a repair PR (one repair round; residuals recorded).

DEFECTS FOUND

  • medium | …--lean--use--20260925.json:29 | The command fails open: a failed primary run or statistics check becomes crc; ! grep then succeeds on the nonexistent control log, and the final success message returns exit 0. Reproduced. | Fix: Exit immediately on primary-check failure; separately execute the control and require its log, expected exit code and empty-order hash.

  • medium | …--fastapi--use--20260925-2.json:36 | Cleanup uses fuser -k -TERM "$PORT"/tcp, which can terminate unrelated processes sharing that numeric port. fuser selects processes by port, without checking ownership by this probe. | Fix: Launch the server in an owned process group and terminate only that group through a cleanup trap.

  • medium | …--data-pandera--use--20260925.json:63, …--data-duckdb--use--20260925.json:62 | Both classify locally authored checks over the repository’s four-row synthetic fixtures as native_proven, causing acceptance promotion. docs/contributing-evidence.md explicitly classifies repository-fixture execution as local_integration. | Fix: Supersede with correctly classified receipts and regenerate the reports; obtain qualifying native evidence before promotion.

  • medium | …--promptfoo--use--20260925-2.json:29, …--inspect-ai--use--20260925-2.json:29 | The native CLIs run, but cleanup deletes their result reports/logs. Only custom Python summaries remain, and the repair reviewers explicitly did not rerun either evaluation. This fails the acceptance policy’s retained-native-result and independent-observation requirements. | Fix: Retain sanitized native reports with hashes and receipt references, then independently verify the E2E claims.

  • medium | …--poppler--install--20260925.json:33, …--systemd--use--20260925-2.json:33 | Retained excerpts omit Poppler’s claimed GOODSIG/VALIDSIG and systemd’s claimed successful workload result/status. Command assertions and reviewer narratives do not retain those native outputs. | Fix: Record concise native status lines before truncation, or attach sanitized output artifacts and supersede the receipts.

  • low | …--promptfoo--use--20260925-2.json:53 | “Existing install under ~/.local/share/codex-ecosystem” contradicts the retained “ephemeral npm install (no stable install found)” output. The same boilerplate misstates FastAPI/Inspect’s ephemeral environments and systemd’s packaged installation. | Fix: Replace the boilerplate with each component’s evidenced installation source.

  • low | …--fastapi--use--20260925-2.json:48 | “Both pins match the stack and winner pins” is false for Uvicorn: it has neither a stack component nor a landscape winner entry. | Fix: State that FastAPI matches the stack/winner pin and Uvicorn matches uv.lock.

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator Author

The GPT-6 cross-family findings on this PR are repaired in #331: superseding receipts, FastAPI, pandera and DuckDB rest on their own upstream test suites at the pinned tags, and the defective receipts carry independent needs_changes reviews.

seathatflowsinourveins pushed a commit that referenced this pull request Sep 26, 2026
…receipts, independent vetoes

Superseding receipts fix fail-open commands, configuration isolation, evidence classes and
unbacked claims; FastAPI, pandera and DuckDB now rest on their own upstream test suites at the
pinned tags. Twelve defective merged receipts carry independent needs_changes reviews.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
seathatflowsinourveins added a commit that referenced this pull request Sep 26, 2026
…receipts, independent vetoes (#331)

Superseding receipts fix fail-open commands, configuration isolation, evidence classes and
unbacked claims; FastAPI, pandera and DuckDB now rest on their own upstream test suites at the
pinned tags. Twelve defective merged receipts carry independent needs_changes reviews.

Co-authored-by: Scout <scout@local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant