feat(evals): add subscription-first Atomic auth

[flora131]

Summary

Adds subscription-first authentication for the Atomic Pier/Harbor eval adapters, with an OpenRouter fallback when subscription credentials are unavailable, and isolates Atomic's mutable agent state from the benchmark git workspace.

Key changes

• Anthropic subscription auth: adapters check for ANTHROPIC_OAUTH_TOKEN and pass it through to the sandbox; both adapters accept the credential without requiring ANTHROPIC_API_KEY.
• OpenAI Codex subscription auth: since Codex has no env-var credential, the adapters read the openai-codex entry out of the host's ~/.atomic/agent/auth.json (falling back to legacy ~/.pi/agent/auth.json), write just that entry to a temp file, upload it into the sandbox, chown/chmod 600 it into $HOME/.atomic/agent/auth.json, and remove the temp copy. No Codex-specific auth env vars are introduced, and credential contents are never printed.
• OpenRouter fallback selection: _select_provider_model swaps to the equivalent openrouter/anthropic/... or openrouter/openai/... model before launch when the subscription credential is missing and OPENROUTER_API_KEY is set, since Atomic's CLI has no subscription→OpenRouter retry at the top level. Includes dash-to-dot slug translation (e.g. claude-opus-4-8 → claude-opus-4.8) for Anthropic's OpenRouter mirrors.
• Sandboxed agent state: routes Atomic's agent dir, session dir, cache, and todo path under the sandbox user's ~/.atomic/agent (via ATOMIC_CODING_AGENT_DIR, --session-dir, and ATOMIC_TODO_PATH) instead of the benchmark repo, preventing the todo tool or session state from polluting the git workspace under evaluation. Directories are created with 700 permissions.
• Session artifact preservation: session transcripts are copied from the sandbox agent dir to /logs after the run for Harbor/Pier artifact parsing, with an EXIT/TERM trap so transcripts are preserved even when a run is killed on timeout.
• openai-codex provider registration: added to both adapters' provider env-key and (Pier) network-allowlist maps (chatgpt.com, auth.openai.com), with no required env keys since auth flows through the uploaded auth.json instead.
• Docs: evals/README.md documents the Anthropic and OpenAI Codex subscription-first flows, their OpenRouter fallback options, and the updated adapter behavior (agent-state isolation, session-copy-on-exit).

Validation

• cd evals && uv run python -c "import atomic_pier, atomic_harbor"
• cd evals && uv run python -m py_compile atomic_pier.py atomic_harbor.py
• ! rg 'CODEX_AUTH|CODEX.*AUTH|AUTH.*CODEX|CODEX_AUTH_JSON|CODEX_TOKEN' evals/atomic_pier.py evals/atomic_harbor.py evals/README.md
• git diff --check -- evals/atomic_pier.py evals/atomic_harbor.py evals/README.md
• git push -u origin feat/evals-subscription-auth ran repository pre-push hooks successfully, including bun run lint, bun run check:file-length, and bun run test:unit.

Goal-run evidence used

The goal ledger and worker receipt reported the eval objective complete: import/py_compile checks passed, forbidden Codex auth-env grep had no matches, Codex auth provisioning reached live openai-codex model execution in a smoke run, and reviewer quorum accepted remaining P3 findings as non-blocking. The PR focuses on the accepted scope and does not include unrelated .atomic/context artifacts.

[claude]

Code Review

Solid PR overall: the subscription-first selection logic is clearly commented, the provider→env-key table refactor in atomic_harbor.py is a real readability win, shlex.quote on --provider/--model closes a latent injection gap, and the README updates match the code. I verified against the CLI source that ATOMIC_CODING_AGENT_DIR is tilde-expanded (packages/coding-agent/src/config.ts:348-350) and ATOMIC_TODO_PATH is a real override, so the literal ~/.atomic/agent env value and the todo redirect both work as intended.

A few findings, ordered by severity:

1. Anthropic API-key users are silently rerouted to OpenRouter (correctness)

_has_subscription_auth("anthropic") only checks ANTHROPIC_OAUTH_TOKEN (evals/atomic_pier.py:222-227, evals/atomic_harbor.py:103-108). A user who runs with a plain ANTHROPIC_API_KEY (no OAuth token) plus an OPENROUTER_API_KEY in the environment will have the run silently redirected to openrouter/anthropic/... — different billing, rate limits, and potentially different model serving — even though a fully valid direct Anthropic credential was available. Suggest:

if provider == "anthropic":
    return bool(
        self._get_env("ANTHROPIC_OAUTH_TOKEN") or self._get_env("ANTHROPIC_API_KEY")
    )

The README section ("fall back … when the subscription token is unavailable") also doesn't mention that an API key alone triggers the reroute, so users could get OpenRouter results while believing they benchmarked the native API.

2. status=$? captures tee's exit code, not Atomic's

In both adapters the run command ends with:

... | grep -v '"type":"message_update"' | stdbuf -oL tee {output}; status=$?; exit $status

(evals/atomic_pier.py:370-372, evals/atomic_harbor.py:273-275.) Without set -o pipefail, $? is the exit status of the last pipeline element (tee), which is ~always 0. So this addition is a no-op relative to the implicit behavior, and an Atomic crash still reports success. If the goal is to propagate the agent's failure past the new traps, use:

...; status=${PIPESTATUS[0]}; exit $status

(or prepend set -o pipefail, though that keeps grep -v's exit-1-on-empty quirk in play). If exit-status propagation wasn't the goal, the status=$?; exit $status suffix can just be dropped — the EXIT trap doesn't clobber $?.

3. Dash→dot slug mapping mangles dated Anthropic model ids

_openrouter_anthropic_model uses re.sub(r"-(\d+)-(\d+)$", r"-\1.\2", model) (evals/atomic_pier.py:230-231). That works for claude-opus-4-8 → claude-opus-4.8, but dated snapshot ids break: claude-haiku-4-5-20251001 → claude-haiku-4-5.20251001 (OpenRouter's slug is anthropic/claude-haiku-4.5). Consider stripping a trailing -\d{8} date segment before the version rewrite, or at least documenting that dated ids aren't supported for fallback.

4. OPENROUTER_API_KEY is injected into the sandbox even when unused (least privilege / security)

Both adapters now pass OPENROUTER_API_KEY into the container whenever the requested provider is anthropic or openai-codex (evals/atomic_pier.py:334-338, evals/atomic_harbor.py:237-238), including runs where subscription auth is present and the primary provider is used. Eval sandboxes execute untrusted task code, so every credential handed in is exfiltratable. If there's no in-run fallback path that consumes it (the _select_provider_model comment says the CLI has no top-level retry), consider only injecting it when provider == "openrouter" was actually selected. If workflow-level fallbackModels is the reason, a one-line comment justifying it would help the next reader.

5. Codex credential briefly sits at a predictable world-visible path (security, low)

_provision_openai_codex_auth uploads to fixed /tmp/atomic-openai-codex-auth.json and only applies 0600 at the install step (evals/atomic_pier.py:254-293). upload_file may not preserve the host-side mode, so there's a window where the credential is readable by other container users, and the file lingers if the follow-up exec_as_agent fails. Low risk in a single-tenant sandbox, but cheap to harden: add chmod 600 to the existing root chown exec, and/or ensure the temp target is removed on failure too.

6. Minor / robustness

• Unquoted $HOME in rm -rf (evals/atomic_pier.py:361, evals/atomic_harbor.py:265): if HOME were ever empty in the exec environment, rm -rf /.atomic/... targets the filesystem root's .atomic. Building the paths as "${HOME:?}"/.atomic/agent/... in the command string keeps expansion and adds a guard.
• TERM trap timing: bash defers trap execution until the foreground pipeline exits, so the TERM trap only fires promptly if the signal also reaches the atomic process (process-group kill). True for typical Docker timeouts, but worth knowing the transcript sync isn't guaranteed on a shell-only SIGTERM.
• Duplicate OPENROUTER_API_KEY in keys when fallback resolves to openrouter (harbor keys.append) — harmless since the env dict dedupes, but easy to skip.

7. Duplication and test coverage

~120 lines (_auth_config_paths through _session_sync_trap_command) are now byte-identical between atomic_pier.py and atomic_harbor.py. The files already shared structure, but this PR doubles the drift surface — a shared evals/atomic_common.py imported by both would keep the auth/fallback logic in one place.

The new pure helpers (_openrouter_anthropic_model, _select_provider_model, _load_openai_codex_auth merge precedence, _has_subscription_auth) are trivially unit-testable, and findings 1 and 3 above are exactly the kind of thing a small pytest file would have caught. The validation steps listed (import + py_compile) only prove the files parse. Even a handful of parametrized cases for the slug mapping and provider selection would be worthwhile.

---

None of these block the core flow (subscription-present paths work as described), but #1 and #2 change observable behavior in the fallback/failure paths and are worth fixing before this lands.

🤖 Review by Claude (Fable 5)