fix(evals): capture stderr separately so diagnostics cannot corrupt the json envelope

[Aaronontheweb]

Problem

The eval harness ran the CLI under test with a merged redirect: > "$STDOUT_FILE" 2>&1. This command sent stderr text into the same file as stdout. The check stdout_json_envelope_valid then reads that file as pure JSON, for --json cases. The CLI writes a correct diagnostic to stderr on some turns, for example [error] Unknown output type from daemon: .... That diagnostic shows correct CLI behavior. It is not a CLI defect.

The extra line broke the JSON parse. The harness then scored a correct run as a failure. A controlled A/B run proved this defect: two of three "failures" for one case came from this exact issue. One of the two failed runs in fact behaved correctly.

Fix

Stdout and stderr now write to two separate files: $STDOUT_FILE and the new $STDERR_FILE. The fix applies at two call sites:

• run_prompt, for single-turn cases
• run_prompt_resume, for multi-turn --resume cases

Stdout stays pure JSON for every --json check. The archive step also copies the stderr file next to the stdout file, in evals/runs/<run-id>/stdout/. A failed run stays diagnosable, because both streams stay present.

Audit

The audit read every assertion helper and every case function in the file. It found that none of them read stderr text from the old merged file.

Assertion helpers checked:

• stdout_contains, stdout_not_contains
• stdout_response_contains, stdout_response_not_contains
• stdout_tool_called
• stdout_json_envelope_valid, stdout_json_tool_called, stdout_json_tool_call_arguments
• stdout_skill_file_read_called

Case functions that read $STDOUT_FILE directly:

• assert_coding_context_worktree_handoff
• assert_approval_set_working_directory_positive
• assert_approval_set_working_directory_retry

Two commits changed evals/run-evals.sh on dev after this branch split off: PR #1886 and PR #1890. The rebase onto dev applied with no conflict. The stream split needed no manual merge.

The audit ran a second time, after the rebase. It checked every new case function those two PRs added:

• assert_skill_cron_tz_timezone and assert_grounding_cron_tz_schedule, from PR feat(reminders): surface reminder scheduling failures loudly (Tier 1) #1886
• assert_approval_session_scratch_disposable, from PR Add typed shell approval store schema 3 #1890

Each new function reads only a stdout_contains, stdout_tool_called, or stdout_json_* helper, or the daemon log. None of them read stderr text. This change does not alter the result of any case, old or new.

Validation

• bash -n evals/run-evals.sh passes.
• dotnet slopwatch analyze finds zero issues.
• No live eval run happened for this change.
• A live eval run needs a real model endpoint. This environment has no such endpoint.
• Verification here stays static: a syntax check and a full code audit, not a live pass/fail comparison.

Companion

PR #1895 carries the product-side half of this same fix. That PR blocks a protocol diagnostic at the source. A protocol diagnostic then never reaches stdout. This PR closes the harness gap that let a stderr diagnostic corrupt the stdout capture.