Skip to content

fix(adoption): report Codex profile drift without replacing values #756

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
seathatflowsinourveins wants to merge 2 commits into from
Closed

fix(adoption): report Codex profile drift without replacing values #756

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4cfab63
fix(adoption): report Codex profile drift by rendered key
seathatflowsinourveins Oct 5, 2026
5cfb02f
chore(evidence): register Codex profile drift check last
seathatflowsinourveins Oct 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions docs/decisions/2026-10-05-codex-profile-drift-check.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Report Codex profile drift without replacing operator values

Date: 2026-10-05. Lane: foundation. North-star action: keep the native research and coding workers' configured model and tool policy observable for US-equities research and historical simulation.

The co-op reported four keys in the installed OmniRoute profile that differed from the NativeStack2604 render, although F9 reported success. This unit reproduces that failure with synthetic profiles; it does not reread the reported host profile or claim host acceptance.

At native-agent-stack@2d849ba1fe1b879144cfa470f0514d3ac3b8e487, `tools/adoption/new_wsl_client_config.py:1475-1521` checks repository wiring, never installed profile files. Its `:2409-2437` profile step creates absent files and preserves existing ones, but labels the step current when it creates nothing, including when an existing file differs. Verification at `:2452-2467` checks login-shell resolution rather than profile contents. The failure is a comparison-scope gap in F9, independently reproduced by a failing check fixture and a failing apply-verification fixture.

## Upstream boundary

Installed Codex 0.160.0's top-level help describes profile-v2 files as an override layer. An expected-render comparator was not found in the installed top-level, doctor or debug help or the version's [release notes](https://github.com/openai/codex/releases/tag/rust-v0.160.0). At openai/codex@a956835d020762cb2b570053af06f643a11c0ecc (rust-v0.160.0), [loader/mod.rs:286-333](https://github.com/openai/codex/blob/a956835d020762cb2b570053af06f643a11c0ecc/codex-rs/config/src/loader/mod.rs#L286-L333) layers the selected profile over the base user config. [merge.rs:96-149](https://github.com/openai/codex/blob/a956835d020762cb2b570053af06f643a11c0ecc/codex-rs/config/src/merge.rs#L96-L149) recursively merges tables and replaces non-table values. Native [config/read parameters:389-407](https://github.com/openai/codex/blob/a956835d020762cb2b570053af06f643a11c0ecc/codex-rs/app-server-protocol/src/protocol/v2/config.rs#L389-L407) expose configuration and its origins, without an expected-render input. Native TOML health checks ([doctor.rs:1188-1201](https://github.com/openai/codex/blob/a956835d020762cb2b570053af06f643a11c0ecc/codex-rs/cli/src/doctor.rs#L1188-L1201)) do not supply the repository's desired values.

F9 already owns that render and a typed additive merge. Extend its check; introduce no installer, client wrapper or dependency. Use stdlib TOML parsing ([tomllib conversion table](https://docs.python.org/3.11/library/tomllib.html#conversion-table)) and native-agent-stack@2d849ba1's existing `strict_equal` (`:1567-1573`) and `plan_merge` (`:1626-1661`). Compare missing and conflicting render-owned paths. Comparing against the merge's expected tree would hide deliberate preservation of operator values.

## Decision

`--check --host NAME` compares the wired stack-worker and OmniRoute profile files under `--home` (defaulting to the user's home) to the selected host render. Plain `--check` remains the repository structural check. Report missing files, missing rendered keys and conflicting typed values using profile basenames and dotted key names. Ignore extra operator-only fields and formatting. `--with-authorization-settings` selects the same comparison scope as the explicit render option; checking it writes nothing and grants no authorization.

Apply retains create-only profile handling. Existing bytes and operator values stay intact. A mismatching profile makes the profile step `drifted`; default verification fails with the key list. Explicitly skipping `codex-files` also omits its profile verification. Dry-run retains its existing no-write planning behavior. No replacement flag is added. Base configuration merging, backups and authorization-write controls keep their existing contract.

Sanitize TOML and OS errors to their exception class, rather than echoing content or host paths. Keep the existence check inside that handler: [python/cpython@v3.11.17, Lib/pathlib.py:1230-1238](https://github.com/python/cpython/blob/v3.11.17/Lib/pathlib.py#L1230-L1238) can propagate permission errors from `exists()`.

This reports file drift against the repository render. It does not prove the effective configuration of a running session, MCP availability, upstream acceptance or new provider execution. The configuration owner decides whether an intentional override should stay and applies any host change through its owned route.

## Evidence and alternatives

The [sanitized receipt](../../evidence/artifacts/codex-profile-drift-check-20261005/receipt.json) retains the red reproduction, repaired module suite and validation separately from pinned source review and native help metadata. Fixtures cover drift, matching values with extra operator fields, merge-kept values, missing keys, malformed TOML, permission errors and explicit authorization comparison. JSON and Markdown modes retain their output formats while returning failure and key-only diagnostics.

A presence-only check repeats the observed false pass. Automatic profile replacement violates the existing operator-value boundary. Native doctor or config/read can establish native syntax or effective configuration, but cannot compare a repository render they do not receive. Read-only static review found the permission-error edge; that finding became a focused regression test. No cross-family readiness approval is claimed; the PR stays draft for the command center's Claude read.

Revisit this glue if upstream ships a supported expected-render comparator, if the owned profile set changes, or if the configuration owner changes the desired-key policy. In particular, adopting profile `skills.config` rule arrays requires a declared comparison policy: the reused merge accepts existing extra rules, and neither current profile renders that array.
168 changes: 168 additions & 0 deletions evidence/artifacts/codex-profile-drift-check-20261005/receipt.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
{
"schema_version": 1,
"date": "2026-10-05",
"lane": "foundation",
"unit": "J-F9DRIFT",
"base_commit": "424b8a778222ae8857aade42c49ae6f7a13b3138",
"reported_defect_pin": "2d849ba1fe1b879144cfa470f0514d3ac3b8e487",
"decision": "docs/decisions/2026-10-05-codex-profile-drift-check.md",
"scope": {
"acceptance": "repository comparison and verification with synthetic profiles and client stubs",
"host_report": "The co-op reported four OmniRoute profile keys differing from its render; this unit did not reread that host profile.",
"host_configuration_reads": false,
"host_configuration_writes": false,
"new_provider_sessions": 0,
"unchanged_upstream_test_suites_run": false,
"scratch": "owned private cache outside every checkout and outside shared tmpfs",
"test_environment": ["TMPDIR points to the owned cache", "PYTHONDONTWRITEBYTECODE=1"]
},
"source_review": [
{
"source": "native-agent-stack@2d849ba1fe1b879144cfa470f0514d3ac3b8e487:tools/adoption/new_wsl_client_config.py:1475-1521",
"finding": "Check validates repository wiring without comparing installed profile files."
},
{
"source": "native-agent-stack@2d849ba1fe1b879144cfa470f0514d3ac3b8e487:tools/adoption/new_wsl_client_config.py:2409-2437",
"finding": "Profiles are create-only; a differing existing file is preserved, but no creation yields a current outcome."
},
{
"source": "native-agent-stack@2d849ba1fe1b879144cfa470f0514d3ac3b8e487:tools/adoption/new_wsl_client_config.py:2452-2467",
"finding": "Apply verification checks login-shell resolution, without profile drift comparison."
},
{
"source": "native-agent-stack@2d849ba1fe1b879144cfa470f0514d3ac3b8e487:tools/adoption/new_wsl_client_config.py:1567-1573,1626-1661",
"finding": "Reuse typed equality and missing/conflicting paths from the existing additive merge; its expected tree deliberately preserves operator values."
},
{
"source": "openai/codex@a956835d020762cb2b570053af06f643a11c0ecc:codex-rs/config/src/loader/mod.rs:286-333",
"finding": "Profile-v2 loads as an override layer above base user configuration."
},
{
"source": "openai/codex@a956835d020762cb2b570053af06f643a11c0ecc:codex-rs/config/src/merge.rs:96-149",
"finding": "Tables merge recursively and non-table values override."
},
{
"source": "openai/codex@a956835d020762cb2b570053af06f643a11c0ecc:codex-rs/app-server-protocol/src/protocol/v2/config.rs:389-407",
"finding": "Native config/read has no expected-render input."
},
{
"source": "https://github.com/openai/codex/releases/tag/rust-v0.160.0",
"finding": "Expected-render comparison not found in these release notes or installed top-level, doctor and debug help."
},
{
"source": "https://docs.python.org/3.11/library/tomllib.html#conversion-table",
"finding": "Use maintained stdlib TOML parsing, preserving typed values."
},
{
"source": "python/cpython@v3.11.17:Lib/pathlib.py:1230-1238",
"finding": "exists() can propagate permission errors; keep it inside the exception-class-only diagnostic handler."
}
],
"executions": [
{
"phase": "installed-client metadata",
"evidence_class": "native_proven",
"commands": ["codex --version", "codex --help"],
"exit_codes": [0, 0],
"observation": "codex-cli 0.160.0; profile-v2 override described in top-level help",
"limit": "Public CLI metadata only; no login, native session, model call or host acceptance."
},
{
"phase": "scratch placement correction",
"evidence_class": "local_integration",
"exit_code": 1,
"observation": "tests/__init__.py rejected checkout-local TMPDIR before any test ran. Moved scratch to owned external cache."
},
{
"phase": "red defect reproduction",
"evidence_class": "synthetic",
"command": "python3 -m unittest tests.test_new_wsl_client_config.CodexProfileDriftTests -v",
"exit_code": 1,
"tests": 3,
"failures": 2,
"observation": "Both drift check and apply verification returned success against a differing profile; matching case passed."
},
{
"phase": "initial focused repair",
"evidence_class": "synthetic",
"command": "python3 -m unittest tests.test_new_wsl_client_config.CodexProfileDriftTests -v",
"exit_code": 0,
"tests": 3
},
{
"phase": "existing expectation correction",
"evidence_class": "local_integration",
"command": "python3 -m unittest tests.test_new_wsl_client_config -q",
"exit_code": 1,
"tests": 190,
"failures": 4,
"duration_seconds": 61.555,
"observation": "Four prior assertions expected success after preserving profiles missing rendered keys. Updated exit expectations while retaining preservation assertions."
},
{
"phase": "focused authorization and preservation integration",
"evidence_class": "local_integration",
"command": "python3 -m unittest tests.test_new_wsl_client_config.CodexProfileDriftTests tests.test_new_wsl_client_config.AuthorizationTests tests.test_new_wsl_client_config.ApplyTests.test_a_differing_profile_file_is_never_overwritten_while_an_existing_config_is_merged -q",
"exit_code": 0,
"tests": 31,
"duration_seconds": 32.769
},
{
"phase": "full module before permission-error regression",
"evidence_class": "local_integration",
"command": "python3 -m unittest tests.test_new_wsl_client_config -q",
"exit_code": 0,
"tests": 192,
"duration_seconds": 66.681
},
{
"phase": "final module with permission-error regression",
"evidence_class": "local_integration",
"fixture_class": "synthetic",
"command": "python3 -m unittest tests.test_new_wsl_client_config -q",
"exit_code": 0,
"tests": 193,
"duration_seconds": 64.098,
"returned_output": "Ran 193 tests in 64.098s\n\nOK"
},
{
"phase": "registry interface correction",
"evidence_class": "local_integration",
"command": "python3 scripts/host_receipts.py register_file --help",
"exit_code": 2,
"observation": "register_file is the existing Python API, not a CLI subcommand; use docs/lanes.md:106-107's documented API invocation."
},
{
"phase": "hot-file derived reports",
"evidence_class": "local_integration",
"commands": ["python3 scripts/component_matrix.py --write", "python3 scripts/new_host_grand_list.py --write"],
"exit_codes": [0, 0],
"observation": "Both reports remain unchanged; only the four files for this unit alter registry entries."
},
{
"phase": "repository integrity and scope",
"evidence_class": "local_integration",
"command": "python3 scripts/validate.py",
"exit_code": 0,
"returned_output": "{\"components\": 69, \"hashed_files\": 10240, \"profiles\": 4, \"receipts\": 208, \"status\": \"passed\"}\nIntegrity and scope checks only; no live provider or GPU execution."
}
],
"comparison_contract": {
"host_check": "--check --host NAME [--home DIR]",
"repository_check": "--check without --host remains structural and reads no installed profile",
"compared_profiles": ["stack-worker.config.toml", "omniroute.config.toml"],
"reported_drift_keys_in_fixture": ["features.standalone_web_search", "model", "model_reasoning_effort", "web_search"],
"operator_only_keys": "ignored",
"authorization_comparison": "included only with the existing --with-authorization-settings flag",
"apply": "create-only profiles retained; codex-files reports drifted and default verify fails on mismatches",
"skip": "explicit codex-files skip omits its profile verification",
"diagnostics": "profile basenames, render-owned dotted key names, exception classes; no values or host paths",
"claims_excluded": ["effective running-session configuration", "MCP availability", "upstream native test acceptance", "provider acceptance", "host repair"]
},
"independent_review": {
"evidence_class": "source_review",
"kind": "read-only static review by a separate bounded GPT agent",
"finding": "Permission errors from exists() needed to remain inside the sanitizing handler; repaired with a focused regression test. Final completeness critic approved the bounded implementation with no additional candidate-class or behavior gap. Its receipt-link and CLI-wording closure notes were completed before commit.",
"cross_family_acceptance": "pending command-center Claude read; PR stays draft"
}
}
18 changes: 14 additions & 4 deletions manifests/evidence.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16914,6 +16914,11 @@
"sha256": "e680ce187ddbb0cde726406710ec3e3272576e276050310b2aca2a7ca47448b7",
"bytes": 22253
},
{
"path": "docs/decisions/2026-10-05-codex-profile-drift-check.md",
"sha256": "7854ddafca2912b5d755a3aaadff9f7154537a633199eb3452214e24694e8218",
"bytes": 5882
},
{
"path": "docs/decisions/2026-10-05-dagu-mise-pin-move.md",
"sha256": "c1eaaffe72e840c7655e05a44e71116174dc1701675d2e08730ba54a8c9a6e87",
Expand Down Expand Up @@ -18514,6 +18519,11 @@
"sha256": "7d634ff5a9195a589923f9be2bf64070fe507bf50b2efc82629150e967ec896e",
"bytes": 14536
},
{
"path": "evidence/artifacts/codex-profile-drift-check-20261005/receipt.json",
"sha256": "1bac7eb84eec5b3be8fc93e5761f588f71bc08ae3ea5b376cd1afca52d3a8f0f",
"bytes": 8686
},
{
"path": "evidence/artifacts/codex-shell-snapshot-cleanup-20260930/listing_diff.py.txt",
"sha256": "7b908845b1dace6922b57582f2ac53ceac51262c17d9973e4e6d5e43afc1f419",
Expand Down Expand Up @@ -54511,8 +54521,8 @@
},
{
"path": "tests/test_new_wsl_client_config.py",
"sha256": "fc53272498f2f2222cc604cfcd485935ad20c4896fff4fc415774fb8bb78ed0f",
"bytes": 282444
"sha256": "90343ceae3671225da08525df23546a95798bfcad76e92ec12277d4c9ebbb015",
"bytes": 288202
},
{
"path": "tests/test_new_wsl_definitive_defaults.py",
Expand Down Expand Up @@ -54986,8 +54996,8 @@
},
{
"path": "tools/adoption/new_wsl_client_config.py",
"sha256": "ae8f67fd2ed4028998bbf7524dc758491ba98c2b9c2b6ed82312c50b9faa4127",
"bytes": 153006
"sha256": "1c4ca8ec7ee0068b4713599f743bb884891fae0db939c47e8b44f9596de082bd",
"bytes": 156613
},
{
"path": "tools/adoption/prove_codex_lane.py",
Expand Down
Loading
Loading