Prompt audit 2026-09-27: facts refuter exceptions, hash prose, split-table log check

[seathatflowsinourveins]

Scope

• What this PR changes: it resolves the landscape-sweep template, README and anti-pattern-log items of the 2026-09-27 /claude-api prompt-audit run through cross-family convergence.
  • Two independent lanes judged a frozen packet: the GPT-6 packaged runner and a Claude evidence-reviewer.
  • Split items went to one adjudication round, with each family judging twice (A/B, then B/A). An item was applied only when all four adjudications agreed.
• Base commit: ba1700ad (OSV allowlist: scope each allowed lock to its reviewed advisories, bound to the lock's sha256 and reachability evidence #441). Head: a7470bac, a content commit followed by the registration commit. It was verified on b26add90 (GPT-6 gateway lane on the all-engines OmniRoute instance: static headers and a one-slash model #439). The rebase over OSV allowlist: scope each allowed lock to its reviewed advisories, bound to the lock's sha256 and reachability evidence #441 and Capability-gate host receipts round 2 at main eb5ca78a (supersede the first recording; GPT-6 review agree) + dashboard gate row #442 overlapped only manifests/evidence.json, which was re-registered by the hot-file protocol.
• Lane: lane:foundation
• Paths:
  • tools/sota-convergence/landscape-sweep/templates.json: the facts key only (F3, X7). After "Default to refuted=true when uncertain." one sentence now names the exceptions for unknown fields, maintenance and licenses, in line with common and fit (Top rule as an upstream-verification procedure, merit-only selection, and a dated anti-pattern log (user-approved global instructions) #371, Landscape-sweep templates: OpenSSF Scorecard maintenance rule; licenses never a refutation reason #385). Its unknown-field wording is broader than common's commit-fact rule, which the decision record lists as a limitation. The all-caps scope words stay (F2, kept 4 of 4).
  • tests/test_landscape_sweep_harness.py:
    • a new facts-refuter test;
    • PROMPTS_SHA256_CURRENT recomputed to 11fcd523…0107 with the test's own helpers (PROMPTS_SHA256_20260926 is kept);
    • the license comment made consistent.
  • tools/sota-convergence/landscape-sweep/README.md and evidence/artifacts/landscape-sweep-20260926/README.md: they no longer present the superseded 3adfbed7… as the current template hash (N1).
    • The tools README points at PROMPTS_SHA256_CURRENT and names the ledger entry as the run's registered record.
    • Both date the claim that went stale.
  • docs/harness-defaults.md: on main a blank line ends the anti-pattern table after its eighth row, so GitHub renders the remaining rows as plain text. That line is removed (N2), and four dated rows are added (N3).
  • tests/test_adoption_docs_consistency.py: the log check now rejects a split table (N2).
  • docs/decisions/2026-09-27-prompt-audit-resolution.md and the sanitized evidence under evidence/artifacts/prompt-audit-20260927/: the packet, sources, both rounds' returns, the adjudication inputs and mapping, decisions.json, usage.json, prompt construction, the review round and the controls.
  • manifests/evidence.json: registration only, in the last commit. No generated report changed.
• Not changed:
  • the permission text (X5, X8);
  • the vendored workflow scripts and SHA256SUMS (X6, X10);
  • CLAUDE.md (X9 split, see Residuals).
• Companion PR: Prompt audit rounds 2-3: AGENTS.md F1, F4, S1 and X5b; X9 applied to CLAUDE.md, 4 of 4 blind judgments; dashboard manifest bound #444 (lane:shared) carries the AGENTS.md items (F1, F4). It is held as a draft while X9 gets a second, research-convergence round with an executed comparison, whose result lands in Prompt audit rounds 2-3: AGENTS.md F1, F4, S1 and X5b; X9 applied to CLAUDE.md, 4 of 4 blind judgments; dashboard manifest bound #444.

SOTA sources

• Practice followed:
  • this repository's convergence procedure (docs/convergence-architecture.md) and its split rule (tools/sota-convergence/README.md: a split stays pending);
  • the fail-first shape of docs/decisions/2026-09-26-harness-rules-cleanup.md.
  • The GPT-6 lane is the packaged runner tools/sota-convergence/landscape-sweep/codex_call.sh/codex_job.py: codex-cli 0.157.1, gpt-6-astra at effort max, read-only, live web search, --ignore-user-config.
• F3, X7 (facts exceptions):
  • OpenSSF Scorecard at f92023a3f77879f96e0c9c1305f289d755be4bb6: docs/checks.md, checks/evaluation/maintained.go and checker/check_result.go. They cover the Maintained check and what an inconclusive result means.
  • The repository's own common and fit rules (Top rule as an upstream-verification procedure, merit-only selection, and a dated anti-pattern log (user-approved global instructions) #371, Landscape-sweep templates: OpenSSF Scorecard maintenance rule; licenses never a refutation reason #385), which sweep.js sends in the same prompt as facts.
• N2, N3 (table): the GFM spec 0.29-gfm §4.10: "The table is broken at the first empty line, or beginning of another block-level structure". A pipe inside a cell is escaped.
• F2 (kept): no source shows that the capitals misdirect the current models. The sources checked:
  • Anthropic prompting best practices, which scopes its dial-back advice to Opus 4.5 and 4.6;
  • Claude Code best practices;
  • the OpenAI GPT-5 prompting guide at openai-cookbook 6dc6324f (examples/gpt-5/gpt-5_prompting_guide.ipynb);
  • OpenAI: Using GPT-6.
• N1: the pin itself, PROMPTS_SHA256_CURRENT in tests/test_landscape_sweep_harness.py, is the one source of the current value.
• All sources with quotes: evidence/artifacts/prompt-audit-20260927/sources.json (S1-S14 and seven searches that found nothing) and sources-supplement.json (T1-T3).

Evidence-class table

Claim	Evidence class	Command / receipt
The facts refuter names the exceptions: the new test fails on the base templates and passes on the head	local integration, fail-first	controls/f3x7-1-new-test-on-base-templates.txt (FAILED), f3x7-2-… (OK)
The pin detects the change, and the new pin is the recomputed value	local integration	controls/f3x7-3-base-pin-on-head-templates.txt: the base pin fails with 11fcd523… != f64eec22…
The log check rejects a split table; the base page fails it and the head page passes	synthetic mutant plus local integration	controls/n2-1…n2-4: the base check returns [] on the mutant, the new check reports one error, FAILED on the base page, OK on the head
Both affected modules pass on the head	local integration	controls/pr1-modules-after.txt: Ran 160 tests, OK (skipped=4); the skip reasons are listed
Each decision rests on both families' judgments	retained model judgments plus source review	round1/, adjudication/, decisions.json, usage.json
The prompts' changed wording changes model behavior	not claimed	The overturn check is the next sweep's facts votes (decision record)
CI-equivalent	local	See below. Two full-suite tests fail on this host, and they fail identically on the unmodified base (controls/env-1-two-failures-on-base.txt)

Local commands run

# head e717e3e5; TMPDIR set to a directory outside /tmp; the validate job's local steps in CI order
# (ci_local.sh; skipped here: the pip, actionlint and uv installs and their online audits)
$ cd examples/claude-native/workflows && sha256sum --check --strict SHA256SUMS
exit 0
$ cd examples/claude-native/workflows && node test-envelope.mjs && node test-contract-mutations.mjs
exit 0
$ zizmor --no-config --no-ignores --persona regular --strict-collection --offline .
exit 0
$ python3 scripts/validate.py
exit 0
$ python3 scripts/host_receipts.py validate
exit 0
$ python3 scripts/release_due.py --strict-if-repinned origin/main
exit 0
$ python3 scripts/validate_catalogs.py
exit 0
$ python3 scripts/validate_foundation.py --root . --json
exit 0
$ python3 scripts/landscape.py --root .
exit 0
$ python3 blueprints/blind-catalog-convergence/audit_reports.py --check
exit 0
$ python3 scripts/validate_convergence.py --all-recorded --root . --json
exit 0
$ python3 tools/sota-convergence/build_verdicts.py --check
exit 0
$ python3 scripts/component_matrix.py --check
exit 0
$ python3 scripts/new_host_grand_list.py --check
exit 0
$ python3 scripts/verdict_flip_candidates.py
exit 0
$ python3 tools/sota-convergence/gap_crosswalk.py build --check
exit 0
$ python3 tools/sota-convergence/gap_wave_ledger.py --wave gap-wave2-20260923 --wave gap-wave3-20260923 --owner gap-resolution --check
exit 0
$ python3 scripts/build_ecosystem.py --check
exit 0
$ python3 -m unittest
exit 1: Ran 6803 tests in 727.866s; FAILED (failures=1, errors=1, skipped=765)
  ERROR: tests.test_order_throughput.CapacityRunTests.test_cli_refuses_live_base_url_from_env_file_without_network
  FAIL: tests.test_gpt6_family_tiering_20260926.RunnerTests.test_sigterm_records_the_running_call_as_interrupted_without_counting_it
$ node --test blueprints/native-skill-practice/test-contract.cjs
exit 0

# after the rebase onto ba1700ad (#441, #442), on head a7470bac
$ python3 scripts/validate.py; host_receipts.py validate; component_matrix.py --check; new_host_grand_list.py --check;
  release_due.py --strict-if-repinned origin/main; validate_catalogs.py; build_ecosystem.py --check
exit 0 each
$ python3 -m unittest tests.test_landscape_sweep_harness tests.test_adoption_docs_consistency tests.test_osv_lockfile_coverage tests.test_blind_checkout tests.test_workflow_security_coverage
exit 0: Ran 237 tests; OK (skipped=4)

# the two affected modules, verbose (controls/pr1-modules-after.txt, with the skip reasons)
$ python3 -m unittest -v tests.test_landscape_sweep_harness tests.test_adoption_docs_consistency
exit 0: Ran 160 tests; OK (skipped=4)
# fail-first controls (controls/n2-*, f3x7-*): each file names its command, the files taken from the base,
# the blob of each tested file, and its exit code

The two unittest failures are this host's, not the change's: both fail identically on the unmodified base b26add90 (evidence/artifacts/prompt-audit-20260927/controls/env-1-two-failures-on-base.txt).

Decision record

docs/decisions/2026-09-27-prompt-audit-resolution.md: each item's verdicts, sources, checks with their failing controls, alternatives, overturn conditions and limitations.

Review round

• One evidence-reviewer, covering both diffs, and one read-only GPT-6 codex exec per head reviewed the first local heads before either was pushed. The prompts and answers are in evidence/artifacts/prompt-audit-20260927/review/.
• One repair round fixed the supported findings, and the decision record's Method lists them.
• An independent stack-verifier then re-ran the checks on 70a9a271 and confirmed 11 of 12 claims: the pin recomputed with the test's own helpers, every control, the table rows as full-line sets, registration hashes, token counts, usage tables, the adjudication tallies resolved through mapping.json, and the privacy scan.
  • It refuted one: the run's evidence README still quotes the old hash as reproduced, in its lane-limit list. That copy is now named as an N1 residual, above.
  • It corrected one wording: only one of the two host-environment tests passes dir="/tmp". The control header and the record now describe each mechanism.
  • Both fixes are in the current head.

Residuals

• X9 (CLAUDE.md:3-4) split 2-2. Both families agree the line asks the model to run commands it cannot type, but the adjudications split on the fallback text, with each family choosing its own lane in both orders. Both texts are in the decision record, and no edit was made here. A second round with an executed comparison follows in Prompt audit rounds 2-3: AGENTS.md F1, F4, S1 and X5b; X9 applied to CLAUDE.md, 4 of 4 blind judgments; dashboard manifest bound #444.

• X8: "after the current foundation work" is undefined in eight places. It is permission text, so only the user can define or close it.

• X5: the portable template's top rule has drifted from the host's user-level file. That is the operator's call.

• X6 and X10: decided in round 1, with no change. The scripts are vendored and byte-identity checked, so any change belongs upstream in agent-lab.

• Host: a stray /tmp/.git directory breaks two tests, on the base too. One passes dir="/tmp". The other's fixture starts its runner child without TMPDIR, and the child refuses /tmp. Both checks count any .git entry as a repository (controls/env-1-two-failures-on-base.txt).

• N1, left unchanged on purpose: some dated records still say the packaged templates reproduce 3adfbed7…:

  • the hash-chained ledger note at catalogs/saturation/ledger.json:2846;
  • the run's lane-limit text in lanes.json and catalogs/sota-convergence/manifest-20260926.json;
  • that text's copy as item 1 under "Lane limits" in the run's evidence README, which lists the limits as passed with --limit.

  The corrected, dated statement is at the top of that README.

• Merge order: if this PR and another that adds anti-pattern rows both touch the log, whoever merges second rebases and keeps every row. The new check rejects a table split by a blank line.

Host evidence

Not applicable: no file under evidence/hosts/ changed.

Checklist

• New/changed GitHub Actions are pinned to a full commit SHA with a version comment (none changed).
• New/changed workflows declare top-level permissions: contents: read (none changed).
• No secrets are printed, logged or committed; no new required secret. The evidence was scanned with scripts/validate.py --scan-file, and host paths and the session URL are replaced with placeholders.
• No new paid hosting, subscription or billing surface was introduced.
• Peer-owned untracked files and worktrees were preserved.

🤖 Generated with Claude Code