Skip to content

Codex worker profile: serena and codebase-memory required at session start - #436

Closed
seathatflowsinourveins wants to merge 6 commits into
mainfrom
claude/w5-codex-mcp-required-20260927
Closed

seathatflowsinourveins wants to merge 6 commits into
mainfrom
claude/w5-codex-mcp-required-20260927

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Summary

On 2026-09-27 a peer measured serena and codebase-memory missing from GPT-6's first-turn tools on the OmniRoute gateway route. They showed up only after a 75 s sleep in the same turn. At openai/codex rust-v0.157.1, a server that is not required gets one shared 1 s grace, counted from the first tool-list build, and is then left out of that request. A custom provider has no websocket prewarm to hide that. On this host serena answers initialize in 1.53-2.24 s and codebase-memory in 1.20-1.25 s, each timed from its own launch. In the race test's runs without required, the first request left 1.63-1.69 s after codex started, so serena's slower starts miss it. These runs do not show codebase-memory missing it; that rests on the peer's relayed measurement.

This PR marks both servers required in the worker profile. Every -p stack-worker session start now waits for them and fails if one cannot start, so a worker can no longer run without these tools. It also:

  • makes the apply script's dry run work with the stricter profile;
  • makes a failed read-back name each server that did not start;
  • records the decision with sources, alternatives and overturn conditions.

A repair round (bb35c5e1, 4450860b) answers the independent review: see Review fixes. A second round (2ba0e799) answers the GPT-6 cross-family review. It keeps the whole required-server error and corrects the record's timing statements: see GPT-6 review.

Changes

  • adoption/templates/codex.stack-worker.config.toml:
    • adds required = true to the partial [mcp_servers.serena] and [mcp_servers.codebase-memory] tables, with cited source comments;
    • leaves mcp_optional_startup_grace_ms unset.
  • tools/adoption/apply_codex_lane.py:
    • new helpers required_servers(), relaxed_required_flags(), required_failure_text() and required_start_failures() (best effort);
    • the dry run's rehearsal adds -c mcp_servers.<name>.required=false to its one -p stack-worker read, and says so. Its scratch HOME cannot start codebase-memory-mcp 0.11.0 while the account's daemon is running.
    • The apply read-back still starts both servers. On failure the run record keeps the whole required-server error and the names read from it, best effort (required_failure), and the apply prints the rollback command.
  • tools/adoption/prove_codex_lane.py: the docstring now says the profile check starts the required servers.
  • tests/test_codex_worker_lane.py:
    • a native race test: fixture stdio MCP servers, a fake Responses endpoint on loopback, bwrap --unshare-net. Since the repair round, the endpoint answers the first request with a code-mode exec call that lists ALL_TOOLS, so the test checks each server's tool by name;
    • a native apply test whose codebase-memory fixture starts only under the test's HOME;
    • synthetic parser and fake-codex tests (the fake codex now applies -c and enforces required);
    • fixture servers for three older native tests that used /bin/false or a stub;
    • structural profile assertions, including (repair round) the check that the landscape sweep's lane home writes the gateway profile's key filter.
  • docs/decisions/2026-09-26-codex-worker-lane.md: the "2026-09-27 addendum: Required start-up".
  • recipes/README.md: one paragraph on the worker-lane consequence.
  • adoption/templates/codex.omniroute.config.toml (repair round, comment lines only): the gateway lane home's key filter.

Commits, oldest first. The branch was replayed onto 55fc8d17 after #433 merged; the sha before the replay is in brackets.

  • 2da1144d (68c4a8d5): the content above.
  • bb35c5e1 and 4450860b (8ee6d212, 7739be03): the repair round. These are the builder's 890503c3 and 714bfc0c, moved onto the content commit by the coordinator with content unchanged.
  • 97d9b2da (2073901c, coordinator): the decision record's start-up residual. The GPT-6 review found its "only the part past the grace" wrong, and 2ba0e799 replaces it.
  • 2ba0e799: the GPT-6 repair round.
  • b18d9f03: manifests/evidence.json only, registering the branch's seven changed files, last per the docs/lanes.md hot-file protocol. scripts/component_matrix.py --write and scripts/new_host_grand_list.py --write left their outputs unchanged. It replaces the earlier registration commits 6c16591a and a5296a3c.

Review fixes

  1. Line citation. core/src/session/session.rs calls install_initial_mcp_runtime(...) on L1860-1866 and awaits it (.await?) on L1867. The worker template, the two test comments and the decision record said "awaited with ? at ... L1860-1866"; they now cite L1860-1867, and so does this body.

  2. "Both exceed the grace" (decision record, Problem). The handshake probe times each server from its own launch, while the grace ends 1 s after Codex's first tool-list build (tool_catalog.rs L252-258). The record now says only serena's slower starts (up to 2.24 s) are shown past the 1.63 s and 1.69 s at which the failing-first runs sent their first request. It attributes codebase-memory's absence on the gateway route to the relayed peer measurement.

  3. What the race test proves. Before, the profile case asserted the deferred-tool guidance, which shows only that at least one deferred tool was present. The first request names no MCP tool: in a captured profile run, its body held the guidance and neither fixture tool name, and the second request held each name once. Now the fake endpoint answers it with a code-mode exec call whose script lists ALL_TOOLS by name and names no tool itself, as upstream's own code-mode tests do. The test reads the list from the second request's custom_tool_call_output. It asserts that:

    • the profile's first turn lists mcp__serena__fixture_serena and mcp__codebase_memory__fixture_codebase_memory;
    • the relaxed control's first turn lists built-in tools but neither of those;
    • the two lists differ by exactly those two names.

    The comments now say "the first turn lists both tools", not "the first request carries them".

  4. Failing-first attribution. The decision record's evidence table now names the native apply test's before run as the behavioural failing-first evidence: the real codex's rehearsal failed on codebase-memory. The synthetic ApplyFlowTests/RequiredStartTests before run is recorded as what it was: three AttributeErrors for the helpers that did not exist yet.

  5. Stale gateway-lane comment. codex.omniroute.config.toml said the landscape sweep's lane home has "no [shell_environment_policy.filters]" and that an exported OMNIROUTE_API_KEY reaches model-run commands. Since Landscape-sweep OmniRoute lane: exact context-mode read rules for the pinned skills; keep the gateway key out of model shells #393, build_args.py stage_lane_home writes [shell_environment_policy.filters] OMNIROUTE_API_KEY = "exclude" (L259-261 comment, L270-271 table). The comment now says so and keeps the reason: inherit defaults to all and the default excludes are ignored (shell_environment_policy.rs L135-136). The lane test in tests/test_codex_worker_lane.py had the same stale comment. It is corrected, and the test now also asserts that the staged filter equals the profile's. No test pins this template's bytes; the apply test compares the installed copy with the template itself. The evidence manifest does pin them (see Residuals).

GPT-6 review

A cross-family review of a5296a3c (gpt-6-astra at max, read-only, through codex_call.sh) returned repair with three low-severity findings. Each was checked against the pinned source (36650394) and fixed in 2ba0e799.

  1. The failure parser could drop or add a server (apply_codex_lane.py L299).
    • Codex joins the <server>: <error> pairs with "; " unescaped (required.rs L51-58). The review cites rmcp's error formatter as keeping a message's newlines and delimiters.
    • The parser read only the one line holding the marker. So a multiline error lost the later names, and a quoted "; serena: " inside another server's error counted as a failure.
    • Now required_failure_text() keeps the error from its last marker to the end of stderr, newlines included. The run record stores that text, bounded at 2,000 characters, as required_failure.text, beside names_best_effort.
    • Those names are read from the whole text. Every pair Codex joined for a required server is found, but a quoted "; : " can still add one; a test pins that case. The read-back message prints the whole text and marks the names best effort.
    • The new multiline and embedded-delimiter tests failed first on the reviewed head's script.
  2. The added start-up delay (record L827 and this body's residual).
    • The required wait ends before the prewarm worker starts (session.rs L1860-1868), and the optional grace starts at the first tool-list build (tool_catalog.rs L251-277).
    • So an optional server still starting after the required wait can add the shared 1 s grace on top.
    • The record now keeps the timings as observations. It says the added delay depends on the required start-up and on any optional server still starting.
  3. The 60 s is not one deadline, and starting is more than initialize (record L737, L746 and L824).
    • Codex applies startup_timeout_sec separately to the client start, initialize and the first tool listing (rmcp_client.rs L354, L946 and L1015).
    • The required wait awaits the client that start_server_task returns after the first tool listing (required.rs L31, rmcp_client.rs L908-1020). The record now says so.
    • A codebase-memory under another HOME now fails the session start at once if the server exits. If it hangs, the start fails when the stalled step's 60 s timeout ends.
    • The recipe and the template comment now say the servers "answer initialize in 1.2-2.3 s", not "take 1.2-2.3 s to start".

Evidence, with TMPDIR=/var/tmp/claude-codex-startup throughout:

Command (python3 -m unittest -v ...) Class Before: reviewed head's script, new tests After
tests.test_codex_worker_lane.RequiredStartTests tests.test_codex_worker_lane.ApplyFlowTests.test_the_dry_run_relaxes_required_servers_and_the_apply_read_back_starts_them synthetic exit 1, Ran 4 tests, FAILED (failures=2): Lists differ: ['codebase-memory'] != ['codebase-memory', 'serena'], and the read-back message lacked the whole error exit 0, Ran 4 tests, OK

After the repair, before the replay:

  • python3 -m unittest tests.test_codex_worker_lane: exit 0, Ran 63 tests, OK (skipped=9).
  • NAS_CODEX_INTEGRATION=1 python3 -m unittest tests.test_codex_worker_lane.CodexIntegrationTests: exit 0, Ran 9 tests, OK.

At the replayed head b18d9f03 on 55fc8d17:

  • python3 scripts/validate.py: exit 0, {"components": 69, "hashed_files": 7364, "profiles": 4, "receipts": 159, "status": "passed"}.
  • python3 -m unittest tests.test_codex_worker_lane tests.test_adoption_docs_consistency tests.test_claude_repository_evidence tests.test_evidence_manifest tests.test_validate: exit 0, Ran 173 tests, OK (skipped=10).
  • git diff --check 55fc8d17 HEAD: exit 0.

The review's own checks found no other defect:

  • pinned facts: the required selection, the failure text, the .await?, the shared 1 s grace and where it starts, the websocket gate and zero-grace semantics;
  • later releases: rust-v0.157.1 is still the latest stable, and rust-v0.159.0-alpha.10 and rust-v0.158.0-alpha.15.3 keep the grace and the required failure;
  • profile merging, the read-back relaxation and git diff --check;
  • all seven registered files' SHA-256 and sizes.

Its residuals: it started no test suite, Codex process or MCP server, and it did not rerun the host timings or the gateway measurement.

Evidence

codex-cli 0.157.1 on the workstation. Every unittest run set TMPDIR=/var/tmp/claude-codex-startup, outside every repository. The runs keep each command line, its exit code and its output outside the checkout.

Before and after each change:

Command (python3 -m unittest -v ...) Evidence class Before After
tests.test_codex_worker_lane.TemplateTests.test_worker_startup_timeouts_layer_over_user_servers structural template without required: exit 1, Ran 1 test, FAILED (failures=3) TemplateTests: exit 0, Ran 12 tests, OK
NAS_CODEX_INTEGRATION=1 ... CodexIntegrationTests.test_required_servers_start_before_the_first_request_on_a_custom_provider native integration, synthetic inputs base template: exit 1, FAILED (failures=2); first request at 1.63 s, before the fixtures' 5 s start, and a serena that cannot start did not stop the run (exit 0). Strengthened test, on a copy of the tree with the two required = true lines deleted: exit 1, FAILED (failures=3); first request at 1.69 s; the "difference" case reported both tool names missing; the serena that cannot start again exited 0 exit 0, Ran 1 test, OK
NAS_CODEX_INTEGRATION=1 ... CodexIntegrationTests.test_real_app_server_apply_and_byte_exact_rollback native integration, synthetic inputs base script: exit 1, FAILED (failures=1); the rehearsal, under the real codex, failed with "required MCP servers failed to initialize: codebase-memory: handshaking with MCP server failed: connection closed: initialize response". This is the behavioural failing-first evidence for the relaxation. exit 0, Ran 1 test, OK
tests.test_codex_worker_lane.ApplyFlowTests tests.test_codex_worker_lane.RequiredStartTests synthetic base script: exit 1, Ran 21 tests, FAILED (errors=3), all AttributeError for the three helpers that did not exist yet, not a behavioural failure exit 0, Ran 21 tests, OK
NAS_CODEX_INTEGRATION=1 ... tests.test_codex_worker_lane.CodexIntegrationTests native integration, synthetic inputs after the template change, before the fixture fix: exit 1, Ran 8 tests, FAILED (failures=3) exit 0, Ran 9 tests, OK

At the earlier registered head a5296a3c (coordinator), before the GPT-6 round and the replay:

  • python3 scripts/validate.py: exit 0, {"components": 69, "hashed_files": 7348, "profiles": 4, "receipts": 159, "status": "passed"}.
  • env TMPDIR=/var/tmp/claude-codex-startup NAS_CODEX_INTEGRATION=1 python3 -m unittest tests.test_codex_worker_lane tests.test_landscape_sweep_harness tests.test_codex_lane: exit 0, Ran 253 tests, OK (skipped=3), the nine integration tests included.
  • git diff --check e82e6be7 HEAD: exit 0.

At the builder's head 714bfc0c, before registration:

  • env TMPDIR=/var/tmp/claude-codex-startup python3 -m unittest -v tests.test_codex_worker_lane tests.test_landscape_sweep_harness tests.test_codex_lane: exit 0, Ran 253 tests, OK (skipped=12). The skips:

    • 9 CodexIntegrationTests, which need NAS_CODEX_INTEGRATION=1 and run in the next command;
    • shellcheck not installed;
    • CONTEXT_MODE_SECURITY_JS unset;
    • no bash 3.2 binary.

    No skip depends on YAML, so a uv run --with pyyaml==6.0.1 run was not needed.

  • env TMPDIR=/var/tmp/claude-codex-startup NAS_CODEX_INTEGRATION=1 python3 -m unittest -v tests.test_codex_worker_lane.CodexIntegrationTests: exit 0, Ran 9 tests, OK.

  • python3 scripts/validate.py: exit 1, "Publication validation failed:". It reports a SHA-256 mismatch (and, for three, a byte-count mismatch) for exactly the four files git diff --name-only 6c16591a..HEAD lists, the ones this branch has not re-registered. In a throwaway shared clone at 714bfc0c, with those four re-registered through scripts/host_receipts.register_file: exit 0, {"components": 69, "hashed_files": 7348, "profiles": 4, "receipts": 159, "status": "passed"}.

  • The two known full-suite failures, each run alone with env TMPDIR=/var/tmp/claude-codex-startup python3 -m unittest -v <id>, at base e82e6be7 (a shared clone under the same /tmp parent) and at 714bfc0c:

    • tests.test_order_throughput.CapacityRunTests.test_cli_refuses_live_base_url_from_env_file_without_network: exit 1, FAILED (errors=1), SafetyError: credential_file_permissions:worktree, at both;
    • tests.test_gpt6_family_tiering_20260926.RunnerTests.test_sigterm_records_the_running_call_as_interrupted_without_counting_it: exit 1, FAILED (failures=1), "the temporary directory /tmp must lie outside the state directory and every repository", at both.

    With the checkout root and durations replaced, base and head output are identical.

  • python3 scripts/component_matrix.py --write: exit 0, {"flip_rule_violations": 0, "rows": 32, "status": "written"}. python3 scripts/new_host_grand_list.py --write: exit 0, {"status": "written", "layers": 32, "winners": 66}. The tree stayed clean.

  • env TMPDIR=/var/tmp/claude-codex-startup python3 -m unittest -q: exit 1, Ran 6706 tests, FAILED (failures=2, errors=1, skipped=766). The two known failures above, plus tests.test_catalog_freshness_propose.RebuildExplorerSubprocessTests.test_general_publication_validator_passes_after_the_run, whose "Publication validation failed:" lists the same four unregistered hashes. In the re-registered throwaway clone, that test alone: exit 0, Ran 1 test, OK.

Measured probes of the installed servers (not Codex runs):

  • serena answered initialize in 1.53-2.24 s (24 tools), codebase-memory in 1.20-1.25 s (17 tools), three runs each, each timed from that server's own launch.
  • codebase-memory-mcp under a scratch HOME answered nothing in 30 s and printed "CBM could not start because the active account daemon uses a different cache directory ... retry with one consistent CBM_CACHE_DIR". Under the account's HOME it answered in 1.2 s.
  • The apply read-back and the prove script's profile check start both servers from cwd /; serena runs with --project-from-cwd. Probed from / with the account's HOME: serena 1.47-1.53 s (24 tools), codebase-memory 1.20-1.25 s (17 tools), three runs each. That is far inside the 60 s timeout each start-up step has. The host apply itself has not run.

None of these: live provider or model runs, gateway runs, token measurements. Nothing was applied to a host.

Upstream tests

Not run. These are the relevant unchanged tests in openai/codex at rust-v0.157.1 (commit 36650394):

  • codex-rs/core/tests/suite/mcp_optional_startup_grace.rs L40 optional_mcp_startup_grace_controls_initial_turn_tool_catalog, four cases: a custom grace omits a pending server and admits a ready one; a zero grace waits for start-up and respects the start-up timeout.
  • codex-rs/codex-mcp/src/connection_manager_tests.rs:
    • L2871 capture_binding_skips_pending_optional_servers_after_configured_shared_startup_grace;
    • L3027 capture_binding_waits_for_optional_startup_when_shared_grace_is_disabled;
    • L3120 capture_binding_shares_optional_startup_grace_across_connection_sets.
  • codex-rs/core/tests/suite/managed_threads_tests.rs L30 dropping_startup_cleans_up_while_required_mcp_is_stalled.
  • codex-rs/exec/tests/suite/mcp_required_exit.rs L9 exits_non_zero_when_required_mcp_server_fails_to_initialize, found in this round.

The documented invocation is just test -p codex-core, -p codex-mcp or -p codex-exec with the test name as a filter: upstream AGENTS.md L66-67 forbids a direct cargo test, and justfile L87-88 runs cargo nextest run --no-fail-fast. It needs the Rust 1.95.0 that codex-rs/rust-toolchain.toml pins.

This host has no Rust toolchain. cargo --version exits 127, cargo, rustc, rustup, just, cargo-nextest and bazel are not on PATH, and ~/.rustup is empty. No toolchain was installed ad hoc for this.

Two of the tests skip without network (skip_if_no_network! at mcp_optional_startup_grace.rs L43 and managed_threads_tests.rs L31), so a future run must record its skips. The decision record lists the same disposition. The race test is our local integration check, not a substitute.

SOTA sources

All paths are in openai/codex at tag rust-v0.157.1 (commit 36650394c5b38c2990ccf2a3457165ca3e9d9726). They were read with gh api on 2026-09-27, and the repair round read them again from a shallow clone at that commit:

  • required field: codex-rs/config/src/mcp_types.rs L233-235
  • Required-server wait and error text: codex-rs/codex-mcp/src/connection_manager/required.rs L15-59
  • Sorted required list: connection_manager.rs L246-251
  • Start-up steps, each with its own startup_timeout_sec: the client start, initialize and the first tool listing, at codex-mcp/src/rmcp_client.rs L354, L946 and L1015. They run in start_server_task (L908-1020), whose client the required wait awaits at required.rs L31.
  • The prewarm worker starts after the required wait: session.rs L1860-1868
  • Where the wait is awaited: core/src/session/session.rs L1860-1867 (the call on L1860-1866, .await? on L1867), and inside it core/src/session/mcp_runtime.rs L148
  • debug prompt-input starts a thread: core/src/prompt_debug.rs L72-74
  • 1 s optional grace: codex-rs/codex-mcp/src/mcp/mod.rs L195
  • The grace counts from the first tool-list build: connection_manager/tool_catalog.rs L252-258
  • Omitting a pending optional server: connection_manager/tool_catalog.rs L251-277
  • Websocket prewarm gate:
    • core/src/session_startup_prewarm.rs L198-208
    • core/src/client.rs L1020-1028
    • model-provider-info/src/lib.rs L190-192 (custom providers) and L549 (built-in)
  • Grace setting (rejected alternative): config/src/config_toml.rs L318-322
  • mcp:// mention (untested alternative): core/src/session/turn.rs L936-948
  • Deferred-tool guidance, present when at least one deferred tool exists: code-mode-protocol/src/description.rs L15 and L291-293
  • Listing a turn's tools through exec (the repair round's race-test check):
    • core/tests/suite/code_mode.rs run_code_mode_turn_with_builder L264-293
    • code_mode_exports_all_tools_metadata_for_namespaced_mcp_tools L7133-7175
    • custom_tool_output_last_non_empty_text L214-229
    • core/tests/common/responses.rs ev_custom_tool_call L975-985
  • Environment Codex passes to MCP servers (includes HOME): rmcp-client/src/utils.rs L163-164
  • Shell environment policy defaults (inherit all, default excludes ignored) and the no-mixing rule: config/src/shell_environment_policy.rs L135-136 and L106-110
  • The landscape sweep's lane-home key filter, in this repository: tools/sota-convergence/landscape-sweep/build_args.py L259-261 and L270-271 (Landscape-sweep OmniRoute lane: exact context-mode read rules for the pinned skills; keep the gateway key out of model shells #393)
  • Release data via gh api, 2026-09-27:
    • latest stable is rust-v0.157.1;
    • prereleases rust-v0.158.0-alpha.15.3 and rust-v0.159.0-alpha.9 keep the grace, required and the error text;
    • PR #47935 ("Allow cached catalogs to satisfy MCP startup readiness", merged 2026-09-24) adds startup_readiness. It is prerelease only and uses a per-process cache.
  • MCP stdio lifecycle for the fixture server: MCP specification 2025-06-18 (initialize, initialized, tools/list, ping). Upstream's own start-up fixture, codex-rs/rmcp-client/src/bin/test_stdio_server.rs L513-517, likewise holds initialize back until a barrier file appears.
  • Responses stream for the fake endpoint: core/tests/common/responses.rs L717-795 (sse, ev_response_created, ev_assistant_message, ev_completed) and L975-985 (ev_custom_tool_call).

Residuals

  • The upstream tests above were not run on this host (no Rust toolchain).
  • The race test checks the first turn's tools as exec sees them. The first request's body names no MCP tool in code mode, so no byte-level check of that request can name them.
  • The peer's gateway measurement is relayed and not retained on this branch. After the host re-apply, the peer reruns its retained gateway probe. Pass means both tools are present and complete on the first turn.
  • A worker whose HOME is not the account's cannot start codebase-memory while the account daemon runs. It used to run without the tools. Now its session start fails: at once if the server exits, or when the stalled step's 60 s timeout ends if it hangs.
  • Every -p stack-worker launch now waits for both servers, their first tool listing included, before its first request.
    • Observed here: their initialize took 1.2-2.3 s, each timed from its own launch. Without required, the race test's first request left 1.63-1.69 s after codex started. These are observations, not the added delay.
    • The optional grace starts after the required wait, so an optional server still starting then can add the shared 1 s grace on top. The added delay depends on both (GPT-6 finding 2).
  • Before applying, check that the host's user config.toml registers [mcp_servers.codebase-memory] and [mcp_servers.serena] with commands. The profile's partial tables name no command, so without a base registration -p stack-worker fails with "invalid transport". The dry run shows this before any write; register the servers per adoption/templates/codex.config.template.toml first. This hazard came with the earlier Worktree MCP repair, and required raises its stakes.
  • The per-prompt mcp:// mention is untested. startup_readiness exists only in prereleases.
  • Host step for the coordinator, after merge and in a quiet window with no Codex process:
    • The installed stack-worker.config.toml predates this template, so the dry run and the apply refuse it ("exists and differs from the template").
    • Move it aside privately, run the dry run with TMPDIR outside /tmp (the host has a /tmp/.git), then apply.
    • The apply's step-4 read-back now starts serena and codebase-memory from /, strictly. If either does not start (each start-up step has its own 60 s timeout), the apply fails at the read-back. It records the whole error with the names read from it, and prints the rollback command. The stdio probe from / took 1.2-1.5 s, but no host apply has run.
  • The landscape sweep's gateway lane copies the profile verbatim and inherits the wait. Its README is unchanged; the decision addendum covers it.
  • The branch is on origin/main 55fc8d17. It was replayed after Capability gate: promptfoo codex-sdk harness for the Codex jcodemunch, ai-memory and M13 gates, reconciled with native OTel #433 merged: content lines identical, seven files re-registered, FAILS=0. If main moves before merge, the coordinator replays it again.

Recommended lane label: lane:foundation

🤖 Generated with Claude Code

@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Sep 27, 2026
Scout and others added 6 commits September 27, 2026 18:06
…start

A peer measurement on 2026-09-27 found serena and codebase-memory missing from
GPT-6's first-turn tools on the OmniRoute gateway route. At openai/codex
rust-v0.157.1 (commit 36650394) a server that is not required gets one shared
1 s grace from the first tool-list build (codex-mcp/src/mcp/mod.rs L195) and is
then left out of that request (connection_manager/tool_catalog.rs L251-277); a
custom provider has no websocket prewarm to hide it (session_startup_prewarm.rs
L198-208, model-provider-info/src/lib.rs L190-192). Both servers take 1.2-2.3 s
to answer initialize here.

- adoption/templates/codex.stack-worker.config.toml: required = true on both
  partial tables (config/src/mcp_types.rs L233-235; the wait at
  connection_manager/required.rs L15-59, awaited at core/src/session/session.rs
  L1860-1866). mcp_optional_startup_grace_ms stays unset.
- tools/adoption/apply_codex_lane.py: the dry run's rehearsal relaxes the two
  with -c mcp_servers.<name>.required=false for its scratch HOME, since
  codebase-memory-mcp 0.11.0 does not start beside its account daemon under
  another home (measured); the apply read-back and prove_codex_lane.py stay
  strict and name each server that did not start.
- tests/test_codex_worker_lane.py: a native race test (fixture MCP servers,
  fake Responses endpoint, bwrap --unshare-net), a native apply test whose
  codebase-memory fixture starts only under the test's HOME, synthetic parser
  and fake-codex tests, and fixture servers for three older native tests.
- docs/decisions/2026-09-26-codex-worker-lane.md: addendum "Required start-up"
  with sources, alternatives (grace 0 rejected; mcp:// mention untested;
  startup_readiness from PR #47935 is prerelease only) and overturn conditions.
- recipes/README.md: the worker-lane consequence.

Evidence (codex-cli 0.157.1, TMPDIR=/var/tmp/claude-codex-startup):
- structural profile test: before exit 1 FAILED (failures=3); after OK
- native race test, base template: exit 1 FAILED (failures=2), first request at
  1.63 s and a serena that cannot start did not stop the run; after: OK
- native apply test, base script: rehearsal failed on codebase-memory (exit 1);
  after: OK
- ApplyFlowTests + RequiredStartTests: base script FAILED (errors=3); after
  Ran 21 tests OK
- NAS_CODEX_INTEGRATION=1 CodexIntegrationTests: Ran 9 tests OK
No live provider, gateway or model run; not applied to any host.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Resolves the independent review's findings on 68c4a8d and records the
upstream tests that this host cannot run.

- Line citation: at openai/codex rust-v0.157.1, core/src/session/session.rs
  calls install_initial_mcp_runtime(...) on L1860-1866 and awaits it with
  `.await?` on L1867. The template, two test comments and the decision
  record now cite L1860-1867.
- Decision record, Problem: "Both exceed the grace" overstated the probe.
  The stdio probe times each server from its own launch, while the grace
  ends 1 s after Codex's first tool-list build (tool_catalog.rs L252-258).
  In the race test's failing-first runs the first request left 1.63 s and
  1.69 s after codex started. Only serena's slower starts (up to 2.24 s)
  are shown past that point. Codebase-memory's absence on the gateway
  route rests on the relayed peer measurement.
- Race test: the deferred-tool guidance proves only that some deferred
  tool was present. The fake endpoint now answers the first request with a
  code-mode exec call that lists ALL_TOOLS by name, as upstream's
  core/tests/suite/code_mode.rs run_code_mode_turn_with_builder (L264-293)
  and code_mode_exports_all_tools_metadata_for_namespaced_mcp_tools
  (L7133-7175) do. It reads the list from the second request's
  custom_tool_call_output. With the profile the first turn lists
  mcp__serena__fixture_serena and
  mcp__codebase_memory__fixture_codebase_memory. The relaxed control lists
  neither, and the two lists differ by exactly those names.
- Failing-first attribution: the native apply test's before run (real
  codex, rehearsal failed on codebase-memory) is the behavioural evidence.
  The synthetic ApplyFlowTests/RequiredStartTests before run failed with
  AttributeError on the three helpers that did not exist yet.
- Gateway profile template comment: since #393 the landscape sweep's lane
  home writes [shell_environment_policy.filters] OMNIROUTE_API_KEY =
  "exclude" (build_args.py L259-261 and L270-271). The template comment and
  the lane test's comment now say so, and the lane test checks that the
  staged filter equals the profile's.
- Decision record: lists the relevant upstream tests at rust-v0.157.1 and
  records them as not run, because this host has no Rust toolchain.

Evidence (codex-cli 0.157.1, TMPDIR=/var/tmp/claude-codex-startup):
- strengthened race test: exit 0, Ran 1 test, OK. On a copy of the tree
  whose template lacks the two required lines: exit 1, FAILED
  (failures=3), first request at 1.69 s, and the two first turns' tool
  lists were identical.
- tests.test_codex_worker_lane, tests.test_landscape_sweep_harness and
  tests.test_codex_lane together: exit 0, Ran 253 tests, OK (skipped=12).
manifests/evidence.json is not re-registered here; the coordinator does
that on replay.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Wraps the one prose line the repair round left past the file's width, and
records that CodexIntegrationTests passed (Ran 9 tests, OK) both before
and after the race test was strengthened.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The residual stated the servers' start times as the added wait. The record's own timings (L731-735) show each start from its own launch and the first request at 1.63-1.69 s without required, so only the part past the grace is added.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The required-server failure record keeps the whole error text from its
marker to the end of stderr, and names the servers read from it as best
effort (`required_failure` {text, names_best_effort}): Codex joins the
"<name>: <error>" pairs with "; " unescaped (required.rs L51-58 at
rust-v0.157.1), so a multiline error lost later names and a quoted
"; <name>: " can add one. New multiline and embedded-delimiter tests
failed first on the reviewed head's script (FAILED (failures=2)).

The decision record now says start-up includes the first tool listing,
that each start-up step has its own startup_timeout_sec (rmcp_client.rs
L354, L946, L1015), not one overall deadline, and that the added
first-request delay depends on the required start-up plus any optional
server still starting, since the optional grace begins after the
required wait (session.rs L1860-1868, tool_catalog.rs L251-277). The
recipe and template say the servers answer initialize in 1.2-2.3 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@seathatflowsinourveins
seathatflowsinourveins force-pushed the claude/w5-codex-mcp-required-20260927 branch from a5296a3 to b18d9f0 Compare September 27, 2026 22:11
@seathatflowsinourveins

Copy link
Copy Markdown
Owner Author

Owner and order, 2026-09-29 (native-agent-stack-10). This PR is handed to the context-loading gate owner (session cloudflare-security-audit-sota-review, who accepted it). It does not merge before the #381 Gate A window W closes: the Codex stack-worker profile is a frozen instrument, the capability gates qualified the current profile, and the sealed protocol lists codebase-memory as blocked until Q3. After landing it needs its own capability-gate re-run. Triage at main 6d626654: needs_rebase_mechanical (manifests/evidence.json only). Its validate-macos failure on b18d9f03 is in the step "Run the full test suite (gating on macOS)"; the failing test is not identified and needs the run log.

@seathatflowsinourveins

Copy link
Copy Markdown
Owner Author

Custody plan, 2026-10-03: Claude session native-agent-stack-0c takes this PR. Nobody has written about it since the 2026-09-29 handoff, so 0c is taking custody.

Re-read at main dcae68bd0, head b18d9f03:

  • Not on main: the stack-worker template still has only startup_timeout_sec for both servers, and docs/decisions/2026-09-26-codex-worker-lane.md L364-365 holds required = true for a measured trial.
  • The required check validate-macos fails on one test this PR adds (tests/test_codex_worker_lane.py L964): /private/var/... against the printed /var/....
  • On the new distribution, Codex does not wire codebase-memory (client-config-map.json: codex/*/mcp_servers.codebase-memory.* is not_wired). It does wire serena, and openai/codex rust-v0.159.3 and rust-v0.160.0 still default mcp_optional_startup_grace_ms to 1000.

Plan: port and close. A fresh lane:foundation PR on main, after #626:

  • ports the serena scope only and picks the knob by a measured trial at 0.160.0: required = true, mcp_optional_startup_grace_ms = 0 or startup_readiness;
  • repairs the macOS test;
  • carries this PR's codex.omniroute.config.toml comment correction, which is still stale on main;
  • keeps this PR's measurements in a dated addendum.

It merges only after the Gate A owner's exact-head check and the new-WSL configuration owner's ACK. This PR closes once the port is open, citing it.

To object or claim the PR, reply within 2 hours of this comment.

— Claude session native-agent-stack-0c

seathatflowsinourveins pushed a commit that referenced this pull request Oct 3, 2026
… review

Port #436's lane-home key-filter assertion and comment; record the driver's
cleanup deviation, the control-profile hashes, the open model_reasoning_summary
knob and the dated rebase resolution; cite mod.rs:4343-4346; document the
dry run's required-serena read-back; reflow two broken sentences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seathatflowsinourveins added a commit that referenced this pull request Oct 4, 2026
* fix(codex): require Serena before the worker's first turn

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(evidence): register the Serena startup port

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(codex): repair round on the Serena startup port after independent review

Port #436's lane-home key-filter assertion and comment; record the driver's
cleanup deviation, the control-profile hashes, the open model_reasoning_summary
knob and the dated rebase resolution; cite mod.rs:4343-4346; document the
dry run's required-serena read-back; reflow two broken sentences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(evidence): re-register the Serena startup port after the repair round

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Codex lane: read back Serena's effective enabled and required; qualify the first-turn promise (P2 on #674)

Codex waits only for MCP servers whose effective enabled and required are both
true (openai/codex@a956835d, rust-v0.160.0:
codex-rs/codex-mcp/src/connection_manager.rs:270-275), and a permitted cached
start-up satisfies the wait without a live connection
(connection_manager/required.rs:29-35). Both files are byte-identical at
rust-v0.159.3.

- apply_codex_lane.py: the read-back records Serena's effective enabled and
  disabled_reason (codex -p stack-worker mcp get --json, cli/src/mcp_cmd.rs:
  970-983) and its effective required and startup_readiness, derived from
  config/read's layers plus the installed profile at precedence 21, because no
  command that takes --profile prints them (cli/src/main.rs:1861-1885).
  Validation fails on any recorded lookup error, on a missing read-back, and
  unless Serena is effectively enabled and required; an unreadable source
  reports "unknown" and fails. Existing checks are unchanged.
- Template comment and recipe line: promise only what Codex guarantees, and
  name what lifts the wait (inherited enabled = false, project config, -c).
  The template stays TOML-equal.
- Tests: SerenaReadbackTests (8 tests: inherited enabled = false, required
  missing, recorded lookup error, layer precedence, a failed config/read);
  failing first against the unchanged script (Ran 8, FAILED failures=14),
  passing after (Ran 8, OK). The fake codex now waits only for enabled
  required servers, can fail a named lookup, and can report extra layers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Registry: re-register the four files of the P2 repair (last commit, hot-file protocol)

host_receipts.register_file for adoption/templates/codex.stack-worker.config.toml,
recipes/README.md, tests/test_codex_worker_lane.py and
tools/adoption/apply_codex_lane.py. component_matrix.py --write and
new_host_grand_list.py --write changed nothing; validate.py and
evidence_manifest.py --check pass (9,564 files). Only these four files[] rows
changed; receipts[] and convergence_records[] are unchanged, and no
convergence record pins a changed path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Worker-lane tests: emit arrays of tables in the fixture TOML writer

After main's #659 the user template carries [[skills.config]], an array of
tables. This PR's strict-config native test re-emits the rendered template
through emit_toml to register a working fixture Serena, and the writer wrote
the list as JSON objects, which Codex rejects ("Error loading config.toml",
line 17 column 18); main's own copy of the test writes the rendered text
directly, so the failure appeared only on the merge of main into this branch.

The writer now emits a list of tables as [[...]] blocks after the table's
key-value pairs. A structural guard round-trips the rendered user template and
a nested case through emit_toml: failing first against the unchanged writer
(Ran 1, FAILED errors=2), passing after (Ran 1, OK); the native test passes
again against codex-cli 0.159.3 (Ran 1, OK).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Registry: re-register the worker-lane tests after the fixture writer fix (last commit)

host_receipts.register_file for tests/test_codex_worker_lane.py; the two
--write generators changed nothing; validate.py and evidence_manifest.py
--check pass (9,692 files). Only that files[] row changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* New-WSL client config: expectations for Serena's required key (B1, owner Option 2)

The configuration owner approved updating the two expectations, not the code,
and the coordinator ACKed it on 2026-10-04. With this branch's
mcp_servers.serena.required = true merged, the checker counts one more piece,
codex/stack-worker/mcp_servers.serena.required, wired through slot:serena.

- docs/decisions/2026-10-02-new-wsl-client-configuration.md: Decision 2's
  counts become the measured (386, 294, 200, 94, 80, 41, 39, 12), and one dated
  2026-10-04 sentence says why they moved. Decision 14's phrase quotes
  authorization 12 and 80 not wired, which did not move.
- tests/test_new_wsl_client_config.py: the rendered stack-worker serena table
  is {"startup_timeout_sec": 60, "required": True}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Serena startup port: record the 0.160.0 rerun after #626 (receipt block, addendum resolution)

The pinned codex-cli 0.160.0 (adoption/pins-linux-x86_64.json, wrapper and
linux-x64 tarballs matched by SHA-256 and SHA-512, vendored binary
12eb3e81...) ran in a scratch prefix; the host's default codex stayed 0.159.3.

- CodexIntegrationTests: exit 0, Ran 12, OK (skipped=1, the opt-in trial).
- The preregistered startup trial: exit 0; selection A holds (3/3 delayed
  starts expose Serena, 3/3 failed starts fail before any request).
- Scratch checks (a) parse, (b) fail closed on a nonexistent command and
  (c) the real Serena transport the config template renders returned the
  same results under 0.160.0 and 0.159.3. (b) carries negative controls
  without required = true on the same nonexistent command: they get past
  MCP startup (a completed loopback turn, or the model call on the default
  provider with no credentials), while required = true exits 1 with only
  the required-server error, also when the provider requires auth.

evidence/receipts/codex-serena-startup-port-20261003.json gains the dated
rerun_0160_20261004 block (local_integration, no provider execution, no host
apply) and two appended residual notes; residuals[8] (the driver's
process-group kill) and [9] (a real --dry-run) stay open. The addendum gets a
dated Resolution 2026-10-04 paragraph. No earlier text changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Registry: re-register the eight PR-owned files on main 6af8e55 (last commit, hot-file protocol)

main's manifests/evidence.json (taken in the merge) plus register_file for
the seven changed files main already lists and the new receipt. The other
9,871 rows and the receipts and convergence_records sections equal main's,
in main's order. component_matrix.py --check and new_host_grand_list.py
--check pass, so neither report needed --write. The two B1 files are not
listed in main's manifest and are not registered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Registry: re-register the eight PR-owned files on main b629b5b (last commit, hot-file protocol)

main's manifests/evidence.json (taken in the merge) plus register_file for
the seven changed files main already lists and the new receipt. The other
9,874 rows and the receipts and convergence_records sections equal main's,
in main's order. component_matrix.py --check and new_host_grand_list.py
--check pass, so neither report needed --write. The two B1 files are not
listed in main's manifest and are not registered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Scout <scout@local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@seathatflowsinourveins

Copy link
Copy Markdown
Owner Author

Ported and merged as #674 (54eb892). The Codex stack worker now requires Serena before its first turn: required = true on the Serena MCP server in the worker profiles. It was re-accepted against the pinned Codex 0.160.0, including a fail-closed check and its negative control, and passed a cross-family full read (ACCEPT at the merged head). Residuals 8 (the test driver's process-group kill) and 9 (a real --dry-run) stay open and are recorded in the receipt evidence/receipts/codex-serena-startup-port-20261003.json.

@seathatflowsinourveins

Copy link
Copy Markdown
Owner Author

Scope note on the closure above: #674 ports only the Serena half of this PR. It leaves codebase-memory optional, a deliberate scope decision recorded in docs/decisions/2026-09-26-codex-worker-lane.md, "Addendum 2026-10-03: serena at the first turn (port of #436)". That addendum keeps the preregistration, the trial arms, the selection and the codebase-memory retirement with its overturn conditions. Making codebase-memory required again would need a new PR against that addendum.

@seathatflowsinourveins
seathatflowsinourveins deleted the claude/w5-codex-mcp-required-20260927 branch October 8, 2026 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant