Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
250 changes: 250 additions & 0 deletions docs/decisions/2026-09-27-prompt-audit-resolution.md

Large diffs are not rendered by default.

5 changes: 4 additions & 1 deletion docs/harness-defaults.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,10 @@ When a claim or action proves wrong, correct it where it was relayed and record
| 2026-09-27 | Silently filtering malformed rows from a measured denominator | `_dicts` in `scripts/component_matrix.py` at `b666c42c` dropped non-object sweep-manifest components and entries | Fail closed with the manifest path, catalog/layer and item index, following the existing convergence input checks | `ConvergenceByLayerTests.test_non_object_manifest_rows_fail_closed` in `tests/test_component_matrix.py` failed with `SystemExit not raised` in all eight fixture cases before the fix |
| 2026-09-27 | Treating a Codex profile as a replacement for config.toml | A draft tiering preregistration claimed the profile loads in place of the base user config. The installed 0.157.1 profile help says it layers on top; [openai/codex rust-v0.157.1 loader L286-334](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/config/src/loader/mod.rs#L286-L334) loads the base and then the profile. [ConfigLayerSource::precedence](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/config/src/config_layer_source.rs) assigns base 20, profile 21, project 25 and session flags 30 | Treat profiles as overrides; a trusted project's config can still win. Pin required worker model/effort/search values on the invocation with `-m` and `-c`. Keep the dated correction and [workstation proof](../evidence/artifacts/codex-worker-lane-host-20260927/README.md) distinct from the separate tiering run | This log |
| 2026-09-27 | Using strict config as a universal worker-profile validator | Observed on codex-cli 0.157.1 on 2026-09-27: `codex --strict-config -p stack-worker exec` fails with `invalid transport` on the profile's partial `[mcp_servers.*]` tables, and `codex --strict-config debug prompt-input probe` refuses the flag as unsupported for `codex debug`. The [pinned loader L594-600](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/config/src/loader/mod.rs#L594-L600) calls [per-file strict validation as ConfigToml, L625-645](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/config/src/loader/mod.rs#L625-L645); profile tables amend transports supplied by the base config | Validate stack-worker with a normal runtime `codex exec -p stack-worker` carrying the lane pins, then inspect its actual events. [`prove_codex_lane.exec_argv`](../tools/adoption/prove_codex_lane.py) defines the invocation; the [retained five-call proof](../evidence/artifacts/codex-worker-lane-host-20260927/prove-live.txt) records the verdicts. A strict failure on a partial layer does not establish a normal-runtime failure | This log; [worker-lane strict-mode observations](decisions/2026-09-26-codex-worker-lane.md#addendum-2026-09-27-start-up-allowances-the-gateway-profile-and-four-base-keys). Recheck on a Codex upgrade |

| 2026-09-27 | Changing a rule in some co-loaded prompt templates but not all | #385 put the unknown-maintenance and license rules into the landscape sweep's `common` and `fit` templates, but the `facts` template, which the refuter receives with `common` and each proposal's `upstream_now`, still said "Default to refuted=true when uncertain" and counted a wrong license as a refuting fact. In the 2026-09-26 run the facts refuter refuted proposals whose `upstream_now` fields were null, which the discover template allows ("null when unknowable"; `evidence/artifacts/landscape-sweep-20260926/returns.json`, anomalyco/opencode and alpacahq/cli) | When a rule changes, update every template that shares the prompt; the facts template states the unknown-field, maintenance and license exceptions | `tests/test_landscape_sweep_harness.py` (the facts-refuter rule test) |
| 2026-09-27 | Leaving a blank line inside a Markdown table | A blank line after the fifth 2026-09-27 row (at `55fc8d17`) ended this log's table, since GFM 0.29-gfm 4.10 says "The table is broken at the first empty line", so every row after it fell outside the table; the shape test read only lines that start with a pipe and passed | Keep a table's rows contiguous from the header to the last row | `tests/test_adoption_docs_consistency.py` (the anti-pattern log check) |
| 2026-09-27 | Restating a linked rule without its scope | `AGENTS.md` required role dispatch for every workflow `agent()` stage, while the Workflow contract of the README it links (`examples/claude-native/workflows/README.md`, its Dispatch by role bullet, from the same commit) exempts the vendored saved scripts | Restate a linked rule with its scope, or link it without restating | This log |
| 2026-09-27 | Quoting a changing value as current in prose | The landscape-sweep README and the 2026-09-26 run's evidence README still said the packaged templates give the 2026-09-26 prompt hash after #385 changed the templates and the test's pin | Point at the pinned source of a changing value instead of quoting it | This log |
| 2026-09-27 | Guessing version-specific CLI syntax or source paths | The custom-agent/profile build tried `codex sandbox linux --help`, which ran `linux` as the command, and guessed an `exec/src/env.rs` URL that returned 404 | Read installed `codex sandbox --help` first; at 0.157.1 the form is `codex sandbox -- <command>`. Inspect the matching release and repository tree before citing source paths; `cli/src/debug_sandbox.rs` calls `protocol/src/shell_environment.rs` at [rust-v0.157.1](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/cli/src/debug_sandbox.rs). A 404 is not an absence check | This log; the [worker addendum](decisions/2026-09-26-codex-worker-lane.md#2026-09-27-addendum-custom-agents-and-context-hub) records returned native results |
| 2026-09-27 | Treating a parsed role field as child authority | The custom-agent README claimed the semantic reviewer's `sandbox_mode = "read-only"` enforced its sandbox; the role loader excludes that field | Trace the applied override list and parent clone in [Codex rust-v0.157.1 role.rs:36–48,119–126,182–189](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/core/src/agent/role.rs#L36-L48); all roles inherit parent sandbox authority | Corrected [custom-agent README](../examples/codex-native/README.md#2026-09-27-custom-agent-instruction-refresh); this log, no spawned-role enforcement test |
| 2026-09-26 | Removing an entire owned list from a preservation check | The writer-identity Prometheus merge replaced two host relabel rules with its bucket rules; stripping the whole key hid the loss | Preserve the original list exactly and append only missing owned rules; verify both the prefix and additions before comparing the remaining config | `host/merge_prometheus.py::verify_merge` in the [writer-identity recipe](../evidence/artifacts/telemetry-writer-identity-20260926/host/); `MergePrometheusTests` in `tests/test_observability_writer_identity_host.py` reproduced the dropped rules before the fix |
Expand Down
5 changes: 3 additions & 2 deletions evidence/artifacts/landscape-sweep-20260926/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,9 @@ coordinator owns and will run from this manifest.
The run used the 2026-09-26 scratchpad prototype of the lane. The same lane is now packaged as
[`tools/sota-convergence/landscape-sweep/`](../../../tools/sota-convergence/landscape-sweep/README.md) (#324). The
run's `prompts_sha256` is `3adfbed7a83e85da3fd7951032e1fa3a579101772a47b211580065c6b42618d4`, the sha256 of the
prototype's staged templates. The packaged templates, filled with this run's values, give the same hash (a local check
in `tests/test_landscape_sweep_harness.py`). The packaged tools produced the returns, lanes, layers, usage records,
prototype's staged templates. The packaged templates, filled with this run's values, gave the same hash until the
2026-09-27 template edits (#385), a local check whose value `tests/test_landscape_sweep_harness.py` keeps as
`PROMPTS_SHA256_20260926`. The packaged tools produced the returns, lanes, layers, usage records,
manifest and source reviews. `tavily-leads.json`, the lead-vetting addendum `tavily-leads-vetting.json` and the
attempts' compact records were written for this record, and each states its method.

Expand Down
Loading
Loading