diff --git a/docs/harness-defaults.md b/docs/harness-defaults.md index 60b97e46d..f506d7a31 100644 --- a/docs/harness-defaults.md +++ b/docs/harness-defaults.md @@ -85,6 +85,8 @@ When a claim or action proves wrong, correct it where it was relayed and record | Date | Anti-pattern | What happened | Rule or check that prevents it | Where enforced | | --- | --- | --- | --- | --- | | 2026-09-27 | Accepting a builder's branch on its recipe tests alone | Six pull requests failed CI's `validate` job on repository registry tests their builders never ran: `LockfileInventoryTests.test_every_tracked_lockfile_and_manifest_is_listed` (a new lockfile missing from `.github/osv-scanner-lockfiles.json`) on #425-#428, `RepositoryClassificationTests.test_every_blueprint_value_under_a_label_key_is_classified` on #416, #427 and #429, and `NewWorkflowSecurityCoverageTests.test_all_published_workflows_are_listed_and_covered` on #429 (`validate.yml` runs 36343393516, 36345259037, 36345263362, 36345267040, 36345271562, 36346020915 and 36347738839). That last class skips when zizmor is missing, so a host without zizmor reports a skip, not a failure | A builder's handoff runs the three registry test methods with zizmor on `PATH`, and a skip is not a pass; every push runs them on the tip commit of each pushed ref, checked out in a detached worktree | [`scripts/git-hooks/pre-push`](../scripts/git-hooks/pre-push), which refused a reproduction (an unlisted root-level `requirements-pre-push-probe.txt` committed on HEAD) with "add these to .github/osv-scanner-lockfiles.json", and [`tests/test_pre_push_gate.py`](../tests/test_pre_push_gate.py). The hook is the only mechanical check: the builder row of the [role table](../examples/claude-native/workflows/README.md#dispatch-by-role-2026-09-26) reaches a builder only when its coordinator copies the rule into the brief. `git push --no-verify` bypasses the hook, and CI's `validate` job still runs the whole suite | +| 2026-09-27 | Using a two-slash model slug on a chained gateway | An uncommitted draft of the framework-lane build accepted `sharedgw/cx/gpt-6-astra`. Codex strips one namespace, and only one of letters, digits, `_` and `-`, so any other slug selects fallback metadata with a different prompt template, no Responses Lite or multi-agent, a 272k context window and 10,000-byte tool-output truncation ([openai/codex rust-v0.157.1 manager.rs L763-780](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/models-manager/src/manager.rs#L763-L780), [model_info.rs L99-150](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/models-manager/src/model_info.rs#L99-L150)) | Allow at most one provider segment, of those characters, in both builder and runner; use `sharedgw/gpt-6-astra-max` on 20129. Compare the rendered items' content with `cx/gpt-6-astra-max`, without `id` and `create_time` and with the model names masked. `prompt-input` omits base instructions, so compare those separately using the native bundled catalog and pinned lookup rule ([procedure and source](../tools/sota-convergence/landscape-sweep/README.md#staging-on-the-framework-instance-20129)) | `OmniRouteLaneBuildTests.test_two_slash_model_is_refused_by_builder_and_runner` in [tests/test_landscape_sweep_harness.py](../tests/test_landscape_sweep_harness.py) runs both entry points, `build_args.py` and `codex_call.sh start`: each must exit 2 with the fallback-metadata diagnostic and write nothing. Committed code already refused a second slash; the test failed first on the missing diagnostic and on a dotted namespace (`my.gw/...`) | +| 2026-09-27 | Reading a null call_logs effort column as "effort not sent" | Null `reasoning_effort_requested` or `reasoning_effort_upstream` was treated as evidence of missing effort. The established installed-source check shows those columns are filled only when the response carries encrypted reasoning (OmniRoute 3.8.51, `src/lib/usage/callLogs.ts` L646-653). A first correction then named `requestBody.reasoning` as what was sent, but `requestBody` is the body the gateway received from its client (`open-sse/handlers/chatCore/attemptLogging.ts` L575-582) | Read `pipelinePayloads.providerRequest.reasoning` from `GET /api/usage/call-logs/` at the gateway whose outbound request is in question, with its detailed pipeline logging on (`attemptLogging.ts` L513-516, `callLogs.ts` L1047-1053, `src/lib/db/detailedLogs.ts` L56-64). A non-null `reasoning_effort_upstream` is positive evidence; a null column establishes nothing about the request's effort | The correction/memory rule in [Record a correction the same turn](#record-a-correction-the-same-turn), this log, and the [framework-lane evidence rule](../tools/sota-convergence/landscape-sweep/README.md#staging-on-the-framework-instance-20129); no repository check enforces this interpretation | | 2026-09-27 | Adding an unknown case to a frozen boolean factor | `scripts/component_matrix.py` at `b666c42c` emitted `verdict_winner: unknown` for pending verdicts, although frozen rule 3 gives this factor no unknown case | With no recorded verdict, no component is its winner: emit false while preserving layer states and counts | `ConvergenceByLayerTests.test_no_selection_and_pending_layers` in `tests/test_component_matrix.py` failed on the old value before the fix | | 2026-09-27 | Adding joined row counts as though they count distinct winners | The matrix at `b666c42c` had 62 winner rows and 5 unmatched winners against 66 recorded winners: `data-mlflow` and `mlflow` both match the single `data-mlflow` winner | Explain the row-to-winner cardinality and use `winner_rows[].winner_ids` for the exact per-row mapping | `ConvergenceByLayerTests.test_one_recorded_winner_can_match_multiple_manifest_rows` in `tests/test_component_matrix.py` pins the two-row mapping and failed on the rendered reconciliation claim before the correction | | 2026-09-27 | Silently filtering malformed rows from a measured denominator | `_dicts` in `scripts/component_matrix.py` at `b666c42c` dropped non-object sweep-manifest components and entries | Fail closed with the manifest path, catalog/layer and item index, following the existing convergence input checks | `ConvergenceByLayerTests.test_non_object_manifest_rows_fail_closed` in `tests/test_component_matrix.py` failed with `SystemExit not raised` in all eight fixture cases before the fix | diff --git a/manifests/evidence.json b/manifests/evidence.json index 005c568f2..36bd70bc9 100644 --- a/manifests/evidence.json +++ b/manifests/evidence.json @@ -14008,8 +14008,8 @@ }, { "path": "docs/harness-defaults.md", - "sha256": "392042dea6940417afa0b657035ed5fa388facab574479f90be6965122339a8b", - "bytes": 42680 + "sha256": "1322671e911352efe5556c47f0fe77545bff462e25a046886bc6c3a5aa3aba89", + "bytes": 45559 }, { "path": "docs/harness-rules-convergence-20260922.md", @@ -39458,8 +39458,8 @@ }, { "path": "tests/test_landscape_sweep_harness.py", - "sha256": "e308aa1632a65bb73d043af57a83cfd7f78fab09e4ca717c9eabb2208b59cf39", - "bytes": 173234 + "sha256": "5534e20ae52837c6f9d4a075d13fe71447710e85c66316b76bfcda47bbf7cac5", + "bytes": 186010 }, { "path": "tests/test_lane_packets.py", @@ -40043,13 +40043,13 @@ }, { "path": "tools/sota-convergence/landscape-sweep/README.md", - "sha256": "6d1a061ad803952dafb96d695be740beac3baf2db5a08e17a9dc29b50922ac41", - "bytes": 44743 + "sha256": "3eed54a600a05d572b3ef02e3e4fbf9704f46b8b98bf7aa2611aa3ea0aa16544", + "bytes": 54710 }, { "path": "tools/sota-convergence/landscape-sweep/build_args.py", - "sha256": "6eb1d363a77a43187a5385813cbffe64ed5a6b522e59f90fcc0a6f81a03bd9bd", - "bytes": 37274 + "sha256": "f5c7b4ed0ed274bc6559590dc738c946cc859d6f7d2cb12156df4837103b66f4", + "bytes": 44099 }, { "path": "tools/sota-convergence/landscape-sweep/build_inputs.py", @@ -40063,8 +40063,8 @@ }, { "path": "tools/sota-convergence/landscape-sweep/codex_job.py", - "sha256": "11f80d7ffc4f5aec9468f4855bbdb5169a7a89258218daf5e904fa2fd633e78f", - "bytes": 36712 + "sha256": "8c39915bee5fc1f8aceafff5c3b2df31ca48255c3f6cf8020942345954a3c62e", + "bytes": 41752 }, { "path": "tools/sota-convergence/landscape-sweep/convert.py", diff --git a/tests/test_landscape_sweep_harness.py b/tests/test_landscape_sweep_harness.py index 8aaf7c2db..70a00a1e1 100644 --- a/tests/test_landscape_sweep_harness.py +++ b/tests/test_landscape_sweep_harness.py @@ -687,6 +687,7 @@ def test_lane_home_carries_provider_mcp_servers_and_profile(self): for server in TOKEN_MCP_SERVERS: self.assertIn(f"[mcp_servers.{server}]", config) self.assertNotIn("supports_websockets = true", config) + self.assertNotIn("http_headers", config) # the 20128 default sends no provider headers # The key never reaches a shell snapshot file, and GPT-6 Astra's search is Codex's standalone web.run. for line in ("[features]", "shell_snapshot = false", "standalone_web_search = true", "supports_standalone_web_search = true"): @@ -715,10 +716,118 @@ def test_lane_home_carries_provider_mcp_servers_and_profile(self): self.assertNotIn("quota_stop_percent", codex) self.assertEqual(codex["api_key_placeholder"], "local-loopback") # keyless loopback gateway by default self.assertNotIn("local-loopback", config) # the placeholder is runtime-only, never in the lane config + self.assertEqual(codex["http_headers"], {}) # The staged runner reads the same settings back. lane = codex_job.settings(work) - self.assertEqual((lane["provider"], lane["codex_home"], lane["profile"]), - ("omniroute", work / "codex-home", "stack-worker")) + self.assertEqual((lane["provider"], lane["codex_home"], lane["profile"], lane["http_headers"]), + ("omniroute", work / "codex-home", "stack-worker", {})) + + def test_builder_and_runner_keep_the_name_patterns_equal(self): + # build_args.py checks what it stages and codex_job.py what it runs; the two copies must never drift. + for name in ("MODEL_NAME", "HEADER_VALUE"): + self.assertEqual(getattr(build_args, name).pattern, getattr(codex_job, name).pattern, name) + self.assertEqual(build_args.OMNIROUTE_REQUEST_HEADERS, codex_job.OMNIROUTE_REQUEST_HEADERS) + # Codex strips one namespace for metadata lookup, and only one of [A-Za-z0-9_-]+ (openai/codex rust-v0.157.1, + # codex-rs/models-manager/src/manager.rs L763-780); 20129 therefore needs a one-slash route. + for module in (build_args, codex_job): + for model in ("gpt-6-astra", "cx/gpt-6-astra", "sharedgw/gpt-6-astra-max", "gpt-6-astra-max", + "my_gw/gpt-6.1", "my-gw/gpt-6-astra"): + self.assertTrue(module.MODEL_NAME.fullmatch(model), (module.__name__, model)) + for model in ("sharedgw/cx/gpt-6-astra", "a/sharedgw/cx/gpt-6-astra", "/gpt-6-astra", + "cx//gpt-6-astra", "cx/", "cx/gpt 6", "", "my.gw/gpt-6-astra-max"): + self.assertIsNone(module.MODEL_NAME.fullmatch(model), (module.__name__, model)) + + def test_two_slash_model_is_refused_by_builder_and_runner(self): + # Supplied 2026-09-27 gateway finding, pinned to openai/codex rust-v0.157.1 manager.rs L763-780: Codex strips + # one namespace, and only one of letters, digits, '_' and '-', so a second slash or any other namespace gets + # fallback metadata even if the gateway can route the request. Both entry points refuse before writing. + prompt = temp_dir(self) / "prompt.txt" + prompt.write_text("Reply in JSON.\n", encoding="utf-8") + for model in ("sharedgw/cx/gpt-6-astra", "my.gw/gpt-6-astra-max"): + with self.subTest(entrypoint="build_args.py", model=model): + work, _, done = self.stage_lane("--omniroute-base-url", "http://127.0.0.1:20129/v1", + "--gpt6-model", model) + self.assertEqual(done.returncode, 2, done.stderr) + self.assertIn("fallback metadata", done.stderr) + self.assertIn("manager.rs L763-780", done.stderr) + self.assertFalse((work / "codex-home").exists()) + self.assertFalse((work / "staged.json").exists()) + with self.subTest(entrypoint="codex_call.sh start", model=model): + work, _, done = self.stage_lane("--omniroute-base-url", "http://127.0.0.1:20129/v1", + "--gpt6-model", "sharedgw/gpt-6-astra-max") + self.assertEqual(done.returncode, 0, done.stderr) + staged = json.loads((work / "staged.json").read_text()) + staged["codex"]["model"] = model + write_json(work / "staged.json", staged) + jobs_before = sorted((work / "gpt6").iterdir()) # stage() creates this directory before any job + started = run(["bash", HARNESS / "codex_call.sh", "--work-dir", work, "start", "gpt6-probe", prompt, + HARNESS / "schemas" / "probe.json"]) + self.assertEqual(started.returncode, 2, started.stdout + started.stderr) + self.assertIn("fallback metadata", started.stderr) + self.assertIn("manager.rs L763-780", started.stderr) + self.assertEqual(sorted((work / "gpt6").iterdir()), jobs_before) + + def test_framework_instance_lane_stages_static_provider_headers(self): + # Codex 0.157.1 adds model_providers..http_headers to every request to that provider + # (codex-rs/model-provider-info/src/lib.rs:166-168 and 385-412 at rust-v0.157.1). + work, _, done = self.stage_lane("--omniroute-base-url", "http://127.0.0.1:20129/v1", + "--gpt6-model", "sharedgw/gpt-6-astra-max", + "--omniroute-header", "x-omniroute-compression=allow-lossy", + "--omniroute-header", "X-OmniRoute-No-Memory=1") + self.assertEqual(done.returncode, 0, done.stderr) + headers = {"x-omniroute-compression": "allow-lossy", "x-omniroute-no-memory": "1"} + config = (work / "codex-home" / "config.toml").read_text(encoding="utf-8") + for line in ('model = "sharedgw/gpt-6-astra-max"', 'base_url = "http://127.0.0.1:20129/v1"', + 'http_headers = { "x-omniroute-compression" = "allow-lossy", "x-omniroute-no-memory" = "1" }'): + self.assertIn(line, config) + if sys.version_info >= (3, 11): + import tomllib + provider = tomllib.loads(config)["model_providers"]["omniroute"] + self.assertEqual(provider["http_headers"], headers) + self.assertNotIn("env_http_headers", provider) + codex = json.loads((work / "staged.json").read_text())["codex"] + self.assertEqual((codex["model"], codex["base_url"], codex["http_headers"]), + ("sharedgw/gpt-6-astra-max", "http://127.0.0.1:20129/v1", headers)) + lane = codex_job.settings(work) + self.assertEqual((lane["model"], lane["http_headers"]), ("sharedgw/gpt-6-astra-max", headers)) + + def test_omniroute_headers_take_only_listed_request_switches_and_plain_values(self): + # Only OmniRoute 3.8.51's per-request switches can be staged. Other x-omniroute-* headers carry secrets under + # names without a credential word, such as x-omniroute-self-hop (open-sse/utils/selfHop.ts:1-12) and + # x-omniroute-video-bridge-broker (src/lib/guardrails/videoBridgeBrokerAuth.ts:7-33). + self.assertEqual(build_args.omniroute_headers([]), {}) + self.assertEqual(build_args.omniroute_headers(["X-OmniRoute-Compression=engine:caveman", + "x-omniroute-no-memory=1", "x-omniroute-no-cache=true", + "x-omniroute-strip-reasoning=yes"]), + {"x-omniroute-compression": "engine:caveman", "x-omniroute-no-cache": "true", + "x-omniroute-no-memory": "1", "x-omniroute-strip-reasoning": "yes"}) + refused = [(["x-omniroute-compression"], "NAME=VALUE"), + (["x-omniroute-compression=off", "X-OmniRoute-Compression=default"], "more than once")] + refused += [([f"x-omniroute-compression={value}"], "printable ASCII") + for value in ("", 'a"b', "a\\b", "é", " off", "off ", "a\tb", "a" * 129)] + refused += [([f"{name}=1"], "per-request switches") for name in ( + "x-omniroute-self-hop", "x-omniroute-video-bridge-broker", "x-omniroute-lease-owner", + "x-omniroute-session-id", "x-omniroute-cli-token", "x-omniroute-ws-bridge-secret", + "x-omniroute-feed-signature", "authorization", "x-other", "x-omniroute-", "x-omniroute-a b")] + for specs, needle in refused: + with self.assertRaisesRegex(ValueError, re.escape(needle), msg=specs): + build_args.omniroute_headers(specs) + self.assertEqual(build_args.omniroute_headers(["x-omniroute-compression=" + "a" * 128]), + {"x-omniroute-compression": "a" * 128}) + + def test_omniroute_header_refusals_stage_nothing(self): + work, _, done = self.stage_lane("--omniroute-header", "x-omniroute-self-hop=abc") + self.assertEqual(done.returncode, 2, done.stdout) + self.assertIn("per-request switches", done.stderr) + self.assertNotIn("abc", done.stderr) # a refused value is never echoed + self.assertNotIn("self-hop", done.stderr) # nor a refused name + self.assertFalse((work / "codex-home").exists()) + self.assertFalse((work / "staged.json").exists()) + work = stage_work(self) + done = build(work, "--omniroute-header", "x-omniroute-compression=allow-lossy") + self.assertEqual(done.returncode, 2, done.stdout) + self.assertIn("--gpt6-provider omniroute", done.stderr) + self.assertFalse((work / "staged.json").exists()) def stage_lane_with_skills(self): """Stage the lane with HOME at a fake home: a skill with a reference file and a symlink cycle, a symlinked skill @@ -1046,6 +1155,67 @@ def test_lane_settings_refuse_a_quota_gate_and_a_missing_home(self): with self.assertRaises(codex_job.UsageError): codex_job.settings(self.work) + def test_lane_headers_must_match_the_lane_home_and_are_recorded_per_job(self): + self.lane() + headers = {"x-omniroute-compression": "allow-lossy"} + (self.work / "codex-home" / "config.toml").write_text( + 'model = "sharedgw/gpt-6-astra-max"\nmodel_provider = "omniroute"\n\n[model_providers.omniroute]\n' + 'name = "OmniRoute"\nhttp_headers = { "x-omniroute-compression" = "allow-lossy" }\n', encoding="utf-8") + base = {"provider": "omniroute", "codex_home": "codex-home", "profile": "stack-worker", + "api_key_env": "OMNIROUTE_API_KEY", "api_key_placeholder": "local-loopback", + "model": "sharedgw/gpt-6-astra-max"} + self.settings({**base, "http_headers": headers}) + self.env.pop("OMNIROUTE_API_KEY", None) + result = self.job("gpt6-probe", last=LAST, events=[COMPLETED]) + self.assertEqual((result["exit"], result["model"]), (0, "sharedgw/gpt-6-astra-max")) + argv = self.record()["argv"] + self.assertEqual(argv[argv.index("-m") + 1], "sharedgw/gpt-6-astra-max") + # The job's inputs carry the headers, so a job finished under other headers is rerun, never reused. + self.assertEqual(result["inputs"]["http_headers"], headers) + self.assertEqual(codex_job.job_inputs(b"p", b"s", "m", "omniroute", {}), + codex_job.job_inputs(b"p", b"s", "m", "omniroute")) + # The lane home must carry exactly the staged headers: other values, or headers the record lacks. + if sys.version_info >= (3, 11): + for staged in ({**base, "http_headers": {"x-omniroute-compression": "off"}}, base): + self.settings(staged) + with self.assertRaisesRegex(codex_job.UsageError, "does not carry the staged provider headers"): + codex_job.settings(self.work) + # Headers other than OmniRoute's per-request switches (secret-carrying ones first), malformed values, and any + # header on the native lane are refused. + refused = [({name: "x"}, "per-request switches") for name in ( + "x-omniroute-self-hop", "x-omniroute-video-bridge-broker", "x-omniroute-lease-owner", + "x-omniroute-session-id", "authorization", "x-omniroute-cli-token")] + refused += [({"x-omniroute-compression": 'a"b'}, "printable ASCII"), + ({"x-omniroute-compression": 1}, "printable ASCII"), + (["x-omniroute-compression"], "must be a table")] + for http_headers, needle in refused: + self.settings({**base, "http_headers": http_headers}) + with self.assertRaisesRegex(codex_job.UsageError, re.escape(needle), msg=http_headers): + codex_job.settings(self.work) + self.settings({"model": "gpt-6-astra", "http_headers": headers}) + with self.assertRaisesRegex(codex_job.UsageError, "needs a gateway provider"): + codex_job.settings(self.work) + # Through the real entry point, a refused header starts nothing and is not echoed. + self.settings({**base, "http_headers": {"x-omniroute-self-hop": "x"}}) + started = self.call("start", "gpt6-refused", self.prompt, self.schema) + self.assertEqual(started.returncode, 2, started.stdout + started.stderr) + self.assertIn("per-request switches", started.stderr) + self.assertNotIn("self-hop", started.stderr) + self.assertFalse((self.work / "gpt6" / "gpt6-refused").exists()) + + def test_staged_headers_need_tomllib_and_header_less_lanes_do_not(self): + # Without tomllib (Python 3.9 and 3.10, as macOS's /usr/bin/python3 3.9) the lane home's header table cannot + # be read back, so a lane with staged headers is refused; a header-less lane still runs there. + self.lane() + base = {"provider": "omniroute", "codex_home": "codex-home", "profile": "stack-worker", + "api_key_env": "OMNIROUTE_API_KEY", "model": "cx/gpt-6-astra"} + with mock.patch.dict(sys.modules, {"tomllib": None}): # a None entry makes the import raise ImportError + self.settings({**base, "http_headers": {"x-omniroute-compression": "allow-lossy"}}) + with self.assertRaisesRegex(codex_job.UsageError, r"Python 3\.11"): + codex_job.settings(self.work) + self.settings(base) + self.assertEqual(codex_job.settings(self.work)["http_headers"], {}) + class RunnerTests(RunnerCase): def test_codex_command_line_stdin_and_directory(self): diff --git a/tools/sota-convergence/landscape-sweep/README.md b/tools/sota-convergence/landscape-sweep/README.md index 8560e1344..f8e39f595 100644 --- a/tools/sota-convergence/landscape-sweep/README.md +++ b/tools/sota-convergence/landscape-sweep/README.md @@ -103,7 +103,7 @@ With `build_args.py --gpt6-provider omniroute --codex-host `, the GPT-6 la `build_args.py` writes `/codex-home/` with three files: - `config.toml`: - - the provider block: `model = "cx/gpt-6-astra"`, `model_provider = "omniroute"`, `model_reasoning_effort = "max"`, and `[model_providers.omniroute]` with a loopback `base_url` ending in `/v1`, `env_key = "OMNIROUTE_API_KEY"`, `requires_openai_auth = false` and `wire_api = "responses"`. The fields follow the [Codex config reference](https://developers.openai.com/codex/config-reference). + - the provider block: `model = "cx/gpt-6-astra"`, `model_provider = "omniroute"`, `model_reasoning_effort = "max"`, and `[model_providers.omniroute]` with a loopback `base_url` ending in `/v1`, `env_key = "OMNIROUTE_API_KEY"`, `requires_openai_auth = false` and `wire_api = "responses"`, plus a static `http_headers` table only when `--omniroute-header` is given (see [the framework instance](#staging-on-the-framework-instance-20129)). The fields follow the [Codex config reference](https://developers.openai.com/codex/config-reference). - the `[mcp_servers.*]` tables of `adoption/templates/codex.config.template.toml`. The checkout's own `tools/adoption/render_config.py` renders them for `adoption/hosts/.json`: serena, ai-memory, socraticode, headroom, codebase-memory, qmd and context-mode. - `stack-worker.config.toml`: the Codex worker profile (`--stack-worker-profile`, default `adoption/templates/codex.stack-worker.config.toml`), copied verbatim. - `AGENTS.md`: the host's Codex user instructions, the managed block of `adoption/templates/codex.AGENTS.template.md` read through `tools/adoption/apply_codex_lane.py`'s `agents_block()`: the top rule, rtk-ai/rtk v0.50.0's `hooks/rtk-awareness-full.md` verbatim, and the RTK exactness exceptions. Codex reads `$CODEX_HOME/AGENTS.md` as global instructions, so without it the lane's model got neither the top rule nor RTK's instructions, which the native lane's workers get from `~/.codex/AGENTS.md`. `staged.json` records its `agents_sha256`. @@ -141,7 +141,7 @@ How the runner uses it: - `codex_job.py` sets `CODEX_HOME` to that home, drops `--ignore-user-config` and adds `-p stack-worker`: `codex exec -p stack-worker --skip-git-repo-check -s read-only -m cx/gpt-6-astra -c model_reasoning_effort="max" -c web_search="live" ...`. - The key comes from `$OMNIROUTE_API_KEY` in the harness's environment. For a keyless loopback gateway, upstream's non-interactive setup with no login or API key, the staged placeholder `local-loopback` fills an unset variable; Codex's `env_key` only needs the variable to exist. `--omniroute-require-key` stages no placeholder, so a job without the variable ends with exit 6 before codex starts. - `--quota-stop-percent` is refused with this provider: the quota probe reads the native login, not the gateway's pool. -- A job's `inputs.json` records the provider, so a gateway run never reuses a native job's result. +- A job's `inputs.json` records the provider, so a gateway run never reuses a native job's result. It also records any provider headers, so a job finished under other headers is rerun, not reused. Before a full run through the gateway, run the lane's parity check on the staged home, and do not claim a gateway result as max quality until it passes. The check covers: - max effort reaching the upstream model, shown in the gateway's request log and the rollout's `turn_context`; @@ -151,6 +151,71 @@ Before a full run through the gateway, run the lane's parity check on the staged - reported usage; - whether hosted web search passes through for a custom provider. +#### Staging on the framework instance (20129) + +The default gateway stays 20128: `--omniroute-base-url` defaults to `http://127.0.0.1:20128/v1`, the model to `cx/gpt-6-astra`, and no provider headers are sent. To stage the lane on the framework OmniRoute instance at port 20129, which compresses a request and then passes it to 20128 through its `sharedgw` node: + +```sh +python3 $H/build_args.py --work-dir "$W" --sweep-id "$LANE" --date "$DATE" --smoke mcp-surfaces \ + --gpt6-provider omniroute --codex-host \ + --omniroute-base-url http://127.0.0.1:20129/v1 \ + --gpt6-model sharedgw/gpt-6-astra-max \ + --omniroute-header x-omniroute-compression=allow-lossy +``` + +Use exactly one slash for this lane: `sharedgw/gpt-6-astra-max`. The established 2026-09-27 route check returned HTTP 200 with reasoning: 20129 forwards the bare `gpt-6-astra-max` through `sharedgw` to 20128, whose built-in Codex provider serves it at max. This route observation does not qualify a landscape sweep. + +Every manual framework-lane invocation must pass `-m sharedgw/gpt-6-astra-max`. Under `-p stack-worker`, the profile's `model = "gpt-6-astra"` otherwise overrides the lane home's `model`. The established 2026-09-27 probe omitted `-m`, sent `gpt-6-astra` to 20129 and received six 401 responses. `codex_job.py` already passes `-m` explicitly. Profile precedence follows [openai/codex rust-v0.157.1, config loader L286-334](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/config/src/loader/mod.rs#L286-L334). + +Check prompt parity without a model call using the staged lane home and the runner's working directory, `$W/empty`, for both commands. The control uses `cx/gpt-6-astra-max`; both commands pin the model with `-m` and with the requested prompt-debug config override. `prompt-input` sends no request, so the keyless placeholder fills the key variable: + +```sh +cd "$W/empty" +CODEX_HOME="$W/codex-home" OMNIROUTE_API_KEY=local-loopback codex -p stack-worker -m sharedgw/gpt-6-astra-max \ + debug prompt-input -c model=sharedgw/gpt-6-astra-max > "$W/prompt-sharedgw.json" +CODEX_HOME="$W/codex-home" OMNIROUTE_API_KEY=local-loopback codex -p stack-worker -m cx/gpt-6-astra-max \ + debug prompt-input -c model=cx/gpt-6-astra-max > "$W/prompt-cx.json" +for f in sharedgw cx; do + jq -S 'walk(if type == "object" then del(.id, .create_time) else . end)' "$W/prompt-$f.json" | + sed -e 's#sharedgw/gpt-6-astra-max##g' -e 's#cx/gpt-6-astra-max##g' > "$W/prompt-$f.norm.json" +done +cmp "$W/prompt-sharedgw.norm.json" "$W/prompt-cx.norm.json" +``` + +If no lane home has been staged, use the installed `stack-worker` profile with `-c model_provider=omniroute` for both commands. Compare the rendered items' content, as above: without each item's `id` and `create_time`, with the model names masked, the two files must be identical. Equal item counts alone do not show an identical prompt. At this pin, `prompt-input` returns only `prompt.input`, so it does not expose `base_instructions` ([Codex rust-v0.157.1, prompt_debug.rs](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/core/src/prompt_debug.rs)). Compare the nonempty base instructions separately in `codex debug models --bundled`, applying the pinned namespace/longest-slug-prefix lookup: both names strip to `gpt-6-astra-max` and resolve to the same catalog entry ([manager.rs L745-780](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/models-manager/src/manager.rs#L745-L780)). Label that result as a source/catalog comparison, distinct from the rendered item comparison. Both debug commands run without model inference. Report only the two parity results and keep the prompt private. + +The header option: + +- `--omniroute-header NAME=VALUE` is repeatable and valid only with `--gpt6-provider omniroute`. It renders `http_headers = { "x-omniroute-compression" = "allow-lossy" }` in `[model_providers.omniroute]`. That is Codex's own provider field: Codex 0.157.1 adds these headers to every request to the provider (`codex-rs/model-provider-info/src/lib.rs:166-168`, `build_header_map` at 385-412, rust-v0.157.1; `model_providers..http_headers` in the config reference). +- Codex silently drops a header whose name or value is invalid. Its config loader also ignores a misspelled provider key, because `deny_unknown_fields` there (`lib.rs:135`) is a JSON-schema attribute only. `build_args.py` therefore checks every header before staging and reads `config.toml` back to compare the table. +- Only OmniRoute 3.8.51's per-request switches are accepted, lowercased and each once: `x-omniroute-compression` (`open-sse/handlers/chatCore/headers.ts:35-46`), `x-omniroute-no-cache` (`src/lib/semanticCache.ts:483-486` and `501-504`), `x-omniroute-no-memory` (`headers.ts:18-33`) and `x-omniroute-strip-reasoning` (`headers.ts:48-65`). Values are 1-128 printable ASCII characters without `"` or `\`. +- Other `x-omniroute-*` headers can carry a secret under a name with no credential word, such as `x-omniroute-self-hop` (`open-sse/utils/selfHop.ts:1-12`) and `x-omniroute-video-bridge-broker` (`src/lib/guardrails/videoBridgeBrokerAuth.ts:7-19`). So the names are listed rather than filtered by word. +- The values are recorded in `staged.json` (`codex.http_headers`) and in each job's `inputs.json`. The runner refuses to start when the lane home's `http_headers` differs from `codex.http_headers`. +- Both read-backs parse `config.toml` with `tomllib` (Python 3.11+). Staging already requires it. Without it, the runner refuses a lane with staged headers; a header-less lane still runs on Python 3.9, such as macOS's `/usr/bin/python3`. + +What `allow-lossy` does (read from the installed OmniRoute 3.8.51 source, not measured): + +- OmniRoute reads `x-omniroute-compression` from each request, case-insensitively (`open-sse/handlers/chatCore/headers.ts:41-46`). +- Without an opt-in, OmniRoute replaces lossy steps with the safe pipeline (session-dedup, lite). `allow-lossy` keeps the operator's plan (`open-sse/services/compression/lossyRequestPolicy.ts:29-47`). +- The other values are `off`, `default`, `engine:` and a named combo (`open-sse/services/compression/planResolution.ts:24-58`). No value turns compression on while the instance's master switch is off (`open-sse/services/compression/strategySelector.ts:126-129`). + +Known limits of the chained route: + +- **Headers stop at 20129.** It forwards to 20128 only User-Agent, `x-opencode-*`, `x-session-id` and `x-title`, besides its node's own static custom headers (`open-sse/executors/default.ts:678-685`, `open-sse/utils/opencodeHeaders.ts:102-116`). `x-omniroute-compression` goes no further, and neither do Codex's identity headers (`session-id`, `thread-id`, `x-client-request-id`, `x-codex-*`). Codex's `prompt_cache_key` travels in the request body, and 20128's Codex executor keeps a key it receives and lists it among the body fields it sends upstream (`open-sse/executors/codex.ts:1509-1514` and `1550-1562`). The effect of the lost headers on cache hits is not measured. +- **Model metadata.** Both the builder and runner refuse a second slash, and a provider segment with characters other than letters, digits, `_` and `-`. Codex strips exactly one namespace segment, and only one of those characters ([openai/codex rust-v0.157.1, manager.rs L763-780](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/models-manager/src/manager.rs#L763-L780)); any other slug causes [fallback metadata, model_info.rs L99-150](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/models-manager/src/model_info.rs#L99-L150): a different prompt template, no Responses Lite, no multi-agent, a 272,000-token context window and a 10,000-byte tool-output truncation policy. Use `sharedgw/gpt-6-astra-max` and check prompt parity above. +- **Slashless aliases failed on 20129.** The established 2026-09-27 observation returned 401, `No active credentials for provider: codex`. Its cause was not established from source, so a slashless alias is not a supported alternative for this lane. +- **Null effort columns prove nothing.** OmniRoute populates `call_logs.reasoning_effort_requested` and `reasoning_effort_upstream` only when the response carries encrypted reasoning (installed OmniRoute 3.8.51, `src/lib/usage/callLogs.ts` L646-653). A non-null `reasoning_effort_upstream` is read from the body the gateway sent upstream, so it is positive evidence. +- **What a gateway sent upstream** is `pipelinePayloads.providerRequest` in `GET /api/usage/call-logs/` at that gateway (`open-sse/handlers/chatCore/attemptLogging.ts` L513-516, `src/lib/usage/callLogs.ts` L1047-1053). It is recorded only while the gateway's detailed pipeline logging is on (`call_log_pipeline_enabled`, or `ENABLE_REQUEST_LOGS=true`; `src/lib/db/detailedLogs.ts` L56-64). `requestBody` is the body the gateway received from its client (`attemptLogging.ts` L575-582), not what it sent: 20128's Codex executor rewrites `reasoning.effort` from forced rules, the model suffix and its clamp (`open-sse/executors/codex.ts` L1451-1465). + +Status: the route check above succeeded, but harness acceptance remains open; the manual verification probe without `-m` failed. Before a sweep counts on this route, run one probe job and one discover job through 20129 and record: + +- a valid `last.json`; +- 20129's compression analytics for the request, showing a stacked plan with lossy engines rather than the safe downgrade; +- max effort in 20128's outbound request, evidenced by `pipelinePayloads.providerRequest.reasoning` in 20128's detailed call log, or by a non-null `reasoning_effort_upstream`; +- the cached-input-token ratio against a control job sent straight to 20128. + +Keep failed attempts and their usage. + ## Evidence contract `saturation_ledger.py --check` and `--append` need the following. Each item names the part of the harness that diff --git a/tools/sota-convergence/landscape-sweep/build_args.py b/tools/sota-convergence/landscape-sweep/build_args.py index 88f69b7f7..aa7b1464d 100755 --- a/tools/sota-convergence/landscape-sweep/build_args.py +++ b/tools/sota-convergence/landscape-sweep/build_args.py @@ -5,6 +5,8 @@ [--layers id,id,... | --due-report report.json] [--smoke LAYER_ID] [--stars PATH | --no-stars] [--gpt6-model gpt-6-astra] [--slots 3] [--lock-dir DIR] [--quota-stop-percent PERCENT] [--skills-checked-at YYYY-MM-DD] [--no-embed] [--force] + [--gpt6-provider omniroute --codex-host HOST [--omniroute-base-url URL] + [--omniroute-header NAME=VALUE ...] [--stack-worker-profile PATH] [--omniroute-require-key]] Needs /layers.json and /inputs/ from build_inputs.py. Writes into the work dir: templates.json the repository templates with <>, <> and <> filled @@ -63,8 +65,29 @@ "how many GitHub stars does it have? Answer only in the required JSON.") ARGS_LINE = re.compile(r"^const A = args$", re.MULTILINE) SWEEP_ID = re.compile(r"[a-z0-9][a-z0-9._-]{0,127}") -# Kept equal to codex_job.MODEL_NAME: an optional provider prefix for gateway routes, e.g. "cx/gpt-6-astra". -MODEL_NAME = re.compile(r"(?:[A-Za-z0-9][A-Za-z0-9._-]{0,31}/)?[A-Za-z0-9][A-Za-z0-9._-]{0,63}") +# Kept equal to codex_job.MODEL_NAME: at most one provider segment, e.g. "cx/gpt-6-astra" or the framework +# instance's "sharedgw/gpt-6-astra-max". Codex strips one namespace for metadata lookup, and only one of letters, +# digits, '_' and '-'; another slug gets fallback metadata (openai/codex rust-v0.157.1, +# codex-rs/models-manager/src/manager.rs L763-780). +MODEL_NAME = re.compile(r"(?:[A-Za-z0-9][A-Za-z0-9_-]{0,31}/)?[A-Za-z0-9][A-Za-z0-9._-]{0,63}") +# Static provider headers (--omniroute-header NAME=VALUE), rendered as model_providers.omniroute.http_headers. Codex +# 0.157.1 adds them to every request to the provider and silently skips a name or value that is not a valid HTTP header +# (codex-rs/model-provider-info/src/lib.rs:166-168 and 385-412 at rust-v0.157.1; config reference +# https://developers.openai.com/codex/config-reference), so both are checked here. Kept equal to codex_job.py. +# Names: only OmniRoute 3.8.51's per-request switches, lowercase (field names are case-insensitive, RFC 9110 section +# 5.1). staged.json and every job's inputs.json record the values, and other x-omniroute-* headers carry secrets under +# names without a credential word: x-omniroute-self-hop (open-sse/utils/selfHop.ts:1-12) and +# x-omniroute-video-bridge-broker (src/lib/guardrails/videoBridgeBrokerAuth.ts:7-19), for example. So the switches +# are listed, not filtered by word. +OMNIROUTE_REQUEST_HEADERS = ( + "x-omniroute-compression", # a compression plan or allow-lossy (open-sse/handlers/chatCore/headers.ts:35-46) + "x-omniroute-no-cache", # "true" skips the semantic cache (src/lib/semanticCache.ts:483-486 and 501-504) + "x-omniroute-no-memory", # true, 1 or yes skips memory injection (open-sse/handlers/chatCore/headers.ts:18-33) + "x-omniroute-strip-reasoning", # true, 1 or yes strips reasoning_content (the same file, 48-65) +) +# Values: printable ASCII without '"' or '\' and without a space at either end, at most 128 characters: a valid HTTP +# field value (RFC 9110 section 5.5) that json.dumps writes as a TOML basic string without escapes. +HEADER_VALUE = re.compile(r"[!#-\[\]-~](?:[ !#-\[\]-~]{0,126}[!#-\[\]-~])?") # GPT-6 lane providers. native: Codex's own login (--ignore-user-config). omniroute: a lane-local CODEX_HOME whose # config.toml routes Codex through the local OmniRoute gateway (Codex model_providers with env_key; see # https://developers.openai.com/codex/config-reference) and carries the token-stack MCP servers. @@ -103,6 +126,10 @@ r"""^\s*\[\s*mcp_servers\s*\.\s*(?:"([^"]+)"|'([^']+)'|([A-Za-z0-9_-]+))\s*(?:\.[^\]]*)?\]\s*(?:#.*)?$""") +class UsageError(ValueError): + """A usage refusal before staging any output.""" + + def load_templates() -> dict: return load_json(HERE / "templates.json") @@ -223,12 +250,51 @@ def codex_user_instructions(repo_root: Path) -> str: raise ValueError(str(error)) from None +def omniroute_headers(specs) -> dict[str, str]: + """--omniroute-header NAME=VALUE options as the lane's static provider headers, sorted by name: OmniRoute's + per-request switches (OMNIROUTE_REQUEST_HEADERS), each once, with a HEADER_VALUE value. A refused value is never + echoed, nor a refused name.""" + headers = {} + for number, spec in enumerate(specs or (), 1): + name, separator, value = spec.partition("=") + name = name.lower() + if not separator: + raise ValueError(f"--omniroute-header #{number} must be NAME=VALUE") + if name not in OMNIROUTE_REQUEST_HEADERS: + raise ValueError(f"--omniroute-header #{number}: only OmniRoute's per-request switches can be staged " + f"({', '.join(OMNIROUTE_REQUEST_HEADERS)}); other headers can carry a secret, and " + "staged.json and each job's inputs.json record header values") + if name in headers: + raise ValueError(f"--omniroute-header {name} is given more than once") + if not HEADER_VALUE.fullmatch(value): + raise ValueError(f"--omniroute-header {name}: the value must be 1-128 printable ASCII characters without " + "'\"' or '\\' and without a space at either end") + headers[name] = value + return dict(sorted(headers.items())) + + def stage_lane_home(work: Path, *, model: str, base_url: str, host: str, profile: Path, repo_root: Path, - require_key: bool = False) -> dict: - """Write /codex-home: config.toml (the OmniRoute provider block plus the rendered token MCP servers), - stack-worker.config.toml (the worker profile, verbatim) and AGENTS.md (the host's Codex user instructions: - the top rule and RTK's, as codex_user_instructions reads them), and /empty/.claude/settings.json (context-mode - read access to each pinned skill file by its exact path). Return what staged.json records about them.""" + require_key: bool = False, http_headers: dict | None = None) -> dict: + """Write /codex-home: config.toml (the OmniRoute provider block, with http_headers only when given, plus the + rendered token MCP servers), stack-worker.config.toml (the worker profile, verbatim) and AGENTS.md (the host's + Codex user instructions: the top rule and RTK's, as codex_user_instructions reads them), and + /empty/.claude/settings.json (context-mode read access to each pinned skill file by its exact path). Return + what staged.json records about them. http_headers must come from omniroute_headers.""" + http_headers = dict(sorted((http_headers or {}).items())) + header_lines = [] + if http_headers: + header_lines = [ + "# Static request headers (build_args.py --omniroute-header): Codex 0.157.1 adds http_headers to every", + "# request to this provider (codex-rs/model-provider-info/src/lib.rs:166-168 and 385-412). OmniRoute reads", + "# x-omniroute-compression per request, and allow-lossy keeps its operator compression plan instead of the", + "# safe downgrade (open-sse/handlers/chatCore/headers.ts:41-46, services/compression/lossyRequestPolicy.ts:", + "# 29-47 at 3.8.51). A chained OmniRoute forwards none of these headers to the next gateway: of the", + "# client's headers only User-Agent, x-opencode-*, x-session-id and x-title pass (open-sse/executors/", + "# default.ts:678-685, open-sse/utils/opencodeHeaders.ts:102-116), so Codex's session-id, thread-id and", + "# x-codex-* headers stop at the first gateway too.", + "http_headers = { " + ", ".join(f"{json.dumps(name)} = {json.dumps(value)}" + for name, value in http_headers.items()) + " }", + ] rendered = render_codex_template(repo_root, host) host_label = Path(host).stem if host.endswith(".json") else host # never a private path in the lane record servers_text, servers = mcp_sections(rendered) @@ -277,6 +343,7 @@ def stage_lane_home(work: Path, *, model: str, base_url: str, host: str, profile "requires_openai_auth = false", 'wire_api = "responses"', "supports_standalone_web_search = true", + *header_lines, "", servers_text, ]) @@ -286,16 +353,17 @@ def stage_lane_home(work: Path, *, model: str, base_url: str, host: str, profile (home / "config.toml").write_text(config, encoding="utf-8") (home / f"{LANE_PROFILE}.config.toml").write_bytes(profile_bytes) (home / "AGENTS.md").write_text(agents_text, encoding="utf-8") - try: - import tomllib # Python 3.11+; older interpreters skip this syntax check - except ImportError: - tomllib = None - if tomllib is not None: - for name in ("config.toml", f"{LANE_PROFILE}.config.toml"): - try: - tomllib.loads((home / name).read_text(encoding="utf-8")) - except tomllib.TOMLDecodeError as error: - raise ValueError(f"{LANE_HOME}/{name} is not valid TOML: {error}") from None + import tomllib # mcp_sections has already required it (Python 3.11+) + for name in ("config.toml", f"{LANE_PROFILE}.config.toml"): + try: + tomllib.loads((home / name).read_text(encoding="utf-8")) + except tomllib.TOMLDecodeError as error: + raise ValueError(f"{LANE_HOME}/{name} is not valid TOML: {error}") from None + # Codex's ModelProviderInfo denies unknown fields only in its JSON schema, not when serde loads the config + # (lib.rs:133-136), so a header table under the wrong key would load silently: read it back as Codex will. + provider = tomllib.loads((home / "config.toml").read_text(encoding="utf-8"))["model_providers"]["omniroute"] + if provider.get("http_headers", {}) != http_headers: + raise ValueError(f"{LANE_HOME}/config.toml does not parse back to the requested provider headers") settings = work / LANE_CONTEXT_SETTINGS settings.parent.mkdir(parents=True, exist_ok=True) skill_rules = lane_skill_read_rules(Path.home()) @@ -303,6 +371,7 @@ def stage_lane_home(work: Path, *, model: str, base_url: str, host: str, profile keyless = {} if require_key else {"api_key_placeholder": OMNIROUTE_KEYLESS_PLACEHOLDER} return {"provider": "omniroute", "codex_home": LANE_HOME, "profile": LANE_PROFILE, "api_key_env": OMNIROUTE_KEY_ENV, **keyless, "base_url": base_url, + "http_headers": http_headers, # names and values; omniroute_headers admits only OmniRoute's switches "lane_home": {"host": host_label, "template": str(CODEX_TEMPLATE), "template_sha256": template_sha, "mcp_servers": servers, @@ -444,7 +513,8 @@ def stage(work: Path, *, sweep_id: str, run_date: str, selected: list, test: boo if lane is not None: codex.update(stage_lane_home(work, model=gpt6_model, base_url=lane["base_url"], host=lane["host"], profile=lane["profile"], repo_root=repo_root, - require_key=lane.get("require_key", False))) + require_key=lane.get("require_key", False), + http_headers=lane.get("http_headers"))) harness_files = sorted([*RUNTIME, "sweep.js", "templates.json", *(f"schemas/{name}.json" for name in SCHEMAS)]) staged = {"schema_version": 1, "kind": "landscape_sweep_staging", "sweep_id": sweep_id, "date": run_date, "test": test, "layers": [layer["layer_id"] for layer in selected], "prompts_sha256": digest, @@ -493,6 +563,11 @@ def main(argv=None) -> int: "token MCP servers (required with --gpt6-provider omniroute)") parser.add_argument("--stack-worker-profile", type=Path, help=f"the Codex worker profile overlay (default {STACK_WORKER_PROFILE} in the checkout)") + parser.add_argument("--omniroute-header", action="append", default=[], metavar="NAME=VALUE", + help="a static request header for the gateway (repeatable; --gpt6-provider omniroute only), " + "rendered as model_providers.omniroute.http_headers and recorded in staged.json: one of " + f"OmniRoute's per-request switches ({', '.join(OMNIROUTE_REQUEST_HEADERS)}) with a " + "printable ASCII value, e.g. x-omniroute-compression=allow-lossy") parser.add_argument("--omniroute-require-key", action="store_true", help=f"the gateway requires an API key: stage no {OMNIROUTE_KEYLESS_PLACEHOLDER!r} placeholder, " f"so a job without ${OMNIROUTE_KEY_ENV} ends before codex starts") @@ -517,8 +592,15 @@ def main(argv=None) -> int: raise ValueError("--quota-stop-percent must be above 0 and at most 100") gpt6_model = args.gpt6_model or (OMNIROUTE_DEFAULT_MODEL if args.gpt6_provider == "omniroute" else "gpt-6-astra") + namespace, slash, rest = gpt6_model.partition("/") + if slash and ("/" in rest or not re.fullmatch(r"[A-Za-z0-9_-]+", namespace)): + raise UsageError("--gpt6-model may have one provider segment, of letters, digits, '_' and '-' only: " + "Codex strips only such a namespace, and any other slug gets fallback metadata " + "(openai/codex rust-v0.157.1, codex-rs/models-manager/src/manager.rs L763-780)") if not MODEL_NAME.fullmatch(gpt6_model): raise ValueError("--gpt6-model is not a model name") + if args.omniroute_header and args.gpt6_provider != "omniroute": + raise ValueError("--omniroute-header needs --gpt6-provider omniroute: the native lane has no provider block") lane = None if args.gpt6_provider == "omniroute": if args.quota_stop_percent is not None: @@ -540,7 +622,8 @@ def main(argv=None) -> int: f"{STACK_WORKER_PROFILE} arrives with the Codex worker-lane change; until the " "checkout has it, pass --stack-worker-profile PATH") lane = {"host": args.codex_host, "base_url": args.omniroute_base_url, "profile": Path(profile).resolve(), - "require_key": args.omniroute_require_key} + "require_key": args.omniroute_require_key, + "http_headers": omniroute_headers(args.omniroute_header)} if args.skills_checked_at: date.fromisoformat(args.skills_checked_at) layers = load_json(work / "layers.json") diff --git a/tools/sota-convergence/landscape-sweep/codex_job.py b/tools/sota-convergence/landscape-sweep/codex_job.py index fd6c7c823..0cf85f943 100755 --- a/tools/sota-convergence/landscape-sweep/codex_job.py +++ b/tools/sota-convergence/landscape-sweep/codex_job.py @@ -22,7 +22,13 @@ -m cx/gpt-6-astra ...`. The provider key comes from the variable codex.api_key_env (OMNIROUTE_API_KEY) in the operator's environment. For a keyless loopback gateway, codex.api_key_placeholder ("local-loopback") fills an unset variable, because Codex's env_key only needs it to exist. Without either, a job ends with exit 6 before codex -starts. The quota gate reads the native login only, so it is refused for a gateway lane. +starts. The quota gate reads the native login only, so it is refused for a gateway lane. A lane staged on a chained +OmniRoute instance, such as the framework instance at http://127.0.0.1:20129/v1, names its model with a node alias in +front (`-m sharedgw/gpt-6-astra-max`), and its static provider headers (codex.http_headers, OmniRoute's per-request +switches only, for example x-omniroute-compression = allow-lossy; build_args.py --omniroute-header) sit in the lane +home's [model_providers.omniroute] http_headers. The runner refuses to start when that table differs from +codex.http_headers, and records the headers in each job's inputs.json. The comparison needs tomllib (Python 3.11+): +older interpreters refuse a lane with staged headers and skip it for a header-less lane. --ignore-user-config keeps the host's Codex config out of the lane, the sandbox is read-only, stdin is /dev/null (background `codex exec` otherwise waits on stdin), and the effort is max. Never ultra: ultra lets Codex delegate to @@ -92,8 +98,17 @@ # The prompt is one argv string, as in the 2026-09-26 runner; Linux caps one argument at 131072 bytes. MAX_PROMPT_BYTES = 120_000 JOB_ID = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,127}") -# An optional provider prefix is allowed for gateway routes, e.g. OmniRoute's "cx/gpt-6-astra". -MODEL_NAME = re.compile(r"(?:[A-Za-z0-9][A-Za-z0-9._-]{0,31}/)?[A-Za-z0-9][A-Za-z0-9._-]{0,63}") +# Kept equal to build_args.MODEL_NAME: at most one provider segment, e.g. "cx/gpt-6-astra" or the framework +# instance's "sharedgw/gpt-6-astra-max". Codex strips one namespace for metadata lookup, and only one of letters, +# digits, '_' and '-'; another slug gets fallback metadata (openai/codex rust-v0.157.1, +# codex-rs/models-manager/src/manager.rs L763-780). +MODEL_NAME = re.compile(r"(?:[A-Za-z0-9][A-Za-z0-9_-]{0,31}/)?[A-Za-z0-9][A-Za-z0-9._-]{0,63}") +# Kept equal to build_args.py, which cites their sources: a gateway lane's static provider headers (staged.json +# codex.http_headers, rendered as model_providers..http_headers in the lane home) are OmniRoute's per-request +# switches only, since other x-omniroute-* headers carry secrets, with printable ASCII values. +OMNIROUTE_REQUEST_HEADERS = ("x-omniroute-compression", "x-omniroute-no-cache", "x-omniroute-no-memory", + "x-omniroute-strip-reasoning") +HEADER_VALUE = re.compile(r"[!#-\[\]-~](?:[ !#-\[\]-~]{0,126}[!#-\[\]-~])?") PROVIDERS = ("native", "omniroute") ENV_NAME = re.compile(r"[A-Z_][A-Z0-9_]{0,63}") PROFILE_NAME = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,63}") @@ -161,6 +176,11 @@ def settings(base: Path) -> dict: lock_dir = Path(staged.get("lock_dir") or "locks").expanduser() out["lock_dir"] = lock_dir if lock_dir.is_absolute() else base / lock_dir out["model"] = str(staged.get("model") or DEFAULT_MODEL) + namespace, slash, rest = out["model"].partition("/") + if slash and ("/" in rest or not re.fullmatch(r"[A-Za-z0-9_-]+", namespace)): + raise UsageError("codex.model may have one provider segment, of letters, digits, '_' and '-' only: Codex " + "strips only such a namespace, and any other slug gets fallback metadata " + "(openai/codex rust-v0.157.1, codex-rs/models-manager/src/manager.rs L763-780)") if not MODEL_NAME.fullmatch(out["model"]): raise UsageError(f"codex.model {out['model']!r} is not a model name") stop = staged.get("quota_stop_percent") @@ -180,13 +200,28 @@ def lane_settings(base: Path, staged: dict) -> dict: """The GPT-6 lane's provider. native (the default) runs Codex with --ignore-user-config against the caller's login. A gateway provider (omniroute) runs Codex with CODEX_HOME set to the staged lane-local home /: its config.toml (the provider block and the token MCP servers) and its profile file are - the lane's whole configuration, and the provider's API key comes from the named environment variable.""" + the lane's whole configuration, and the provider's API key comes from the named environment variable. Its static + provider headers (codex.http_headers) must be the ones that config.toml carries.""" provider = str(staged.get("provider") or "native") if provider not in PROVIDERS: raise UsageError(f"codex.provider {provider!r} must be one of {', '.join(PROVIDERS)}") + headers = staged.get("http_headers") + if headers is not None and not isinstance(headers, dict): + raise UsageError("codex.http_headers must be a table of header names and values; restage with build_args.py") + for name, value in (headers or {}).items(): + if name not in OMNIROUTE_REQUEST_HEADERS: # never echoed: a refused name may be anything + raise UsageError(f"codex.http_headers may hold only OmniRoute's per-request switches " + f"({', '.join(OMNIROUTE_REQUEST_HEADERS)}): other headers can carry a secret; restage " + "with build_args.py") + if not (isinstance(value, str) and HEADER_VALUE.fullmatch(value)): + raise UsageError(f"codex.http_headers {name}: the value must be 1-128 printable ASCII characters without " + "'\"' or '\\' and without a space at either end; restage with build_args.py") + headers = dict(sorted((headers or {}).items())) lane = {"provider": provider, "codex_home": None, "profile": None, "api_key_env": None, - "api_key_placeholder": None} + "api_key_placeholder": None, "http_headers": headers} if provider == "native": + if headers: + raise UsageError("codex.http_headers needs a gateway provider: the native lane has no provider block") return lane home = staged.get("codex_home") if not isinstance(home, str) or not home or Path(home).is_absolute() or ".." in Path(home).parts: @@ -204,10 +239,35 @@ def lane_settings(base: Path, staged: dict) -> dict: placeholder = staged.get("api_key_placeholder") if placeholder is not None and not (isinstance(placeholder, str) and PROFILE_NAME.fullmatch(placeholder)): raise UsageError("codex.api_key_placeholder must be a short plain token (a keyless loopback gateway's value)") + carried = lane_home_headers(base / home / "config.toml", provider) + if carried is None and headers: + raise UsageError("comparing codex.http_headers with the lane home config.toml needs Python 3.11+ (tomllib); " + "run the harness with a newer python3, or restage without --omniroute-header") + if carried is not None and carried != headers: + raise UsageError(f"the lane home config.toml does not carry the staged provider headers (codex.http_headers " + f"{sorted(headers)}, model_providers.{provider}.http_headers " + f"{sorted(carried) if isinstance(carried, dict) else carried!r}); restage with build_args.py") lane.update(codex_home=base / home, profile=profile, api_key_env=key, api_key_placeholder=placeholder) return lane +def lane_home_headers(config: Path, provider: str): + """model_providers..http_headers in the lane home's config.toml, {} when absent. None without tomllib + (Python 3.9 and 3.10): lane_settings then refuses staged headers and skips the check for a header-less lane.""" + try: + import tomllib + except ImportError: + return None + try: + parsed = tomllib.loads(config.read_text(encoding="utf-8")) + except (tomllib.TOMLDecodeError, UnicodeDecodeError) as error: + raise UsageError(f"the lane home config.toml is not valid TOML ({error}); restage with build_args.py") from None + providers = parsed.get("model_providers") + table = providers.get(provider) if isinstance(providers, dict) else None + headers = table.get("http_headers") if isinstance(table, dict) else None + return {} if headers is None else headers + + def job_dir(base: Path, job: str) -> Path: if not JOB_ID.fullmatch(job or ""): raise UsageError(f"job id {job!r} must match {JOB_ID.pattern}") @@ -328,11 +388,14 @@ def sha256_hex(data: bytes) -> str: return hashlib.sha256(data).hexdigest() -def job_inputs(prompt: bytes, schema: bytes, model: str, provider: str = "native") -> dict: +def job_inputs(prompt: bytes, schema: bytes, model: str, provider: str = "native", + http_headers: dict | None = None) -> dict: inputs = {"prompt_sha256": sha256_hex(prompt), "schema_sha256": sha256_hex(schema), "model": model, "effort": EFFORT} if provider != "native": # a gateway run never reuses a native job's result, or the other way round inputs["provider"] = provider + if http_headers: # other provider headers (another compression plan, say) make another run + inputs["http_headers"] = dict(sorted(http_headers.items())) return inputs @@ -389,7 +452,7 @@ def start(base: Path, job: str, prompt_file: str, schema_file: str) -> int: prompt_bytes = Path(prompt_file).read_bytes() schema_bytes = Path(schema_file).read_bytes() config = settings(base) # also refuses a broken staged.json - inputs = job_inputs(prompt_bytes, schema_bytes, config["model"], config["provider"]) + inputs = job_inputs(prompt_bytes, schema_bytes, config["model"], config["provider"], config["http_headers"]) changed = False if (directory / "done").exists() and exit_code(directory) == 0: if read_json(directory / "inputs.json") == inputs: