fix(v1): de-flake the live E2E suite

[mikasenghaas]

Summary

Five targeted fixes for the remaining live-E2E failures on main (post #2262). Each was reproduced and diagnosed locally against the real endpoint before fixing.

1. test_acp_resume_with_tool[openclaw-acp-in-docker] — deterministic upstream 400 on resume

Root cause (captured the exact rejected /responses body locally): with reasoning: true, OpenClaw requests include: ["reasoning.encrypted_content"] and replays reasoning items from its persisted transcript on session resume. Prime's gateway emits encrypted_content in a dotted <fernet>.<base64-metadata> format, which fails OpenClaw's opaque-replay-token allowlist (/^[A-Za-z0-9+/_-]+={0,2}$/ — no dot), so OpenClaw's secret redaction masks the persisted token to gAAAAA…iA==.<metadata> (head6…tail4). The resumed session replays the mangled token → gateway 400 "Invalid request" → every resumed OpenClaw rollout dies. (Live sessions replay from memory, unredacted — which is why only resume broke.) OpenClaw even has a strip-encrypted-content retry, but it only matches invalid_encrypted_content / thinking_signature_invalid error codes; the gateway's generic invalid_request never triggers it.

Fix: OpenClaw reasoning now follows the explicit sampling config (reasoning_effort), never a model-name guess. The old heuristic force-enabled reasoning replay for gpt-5*/o* models, which is exactly the broken path on the default (Prime) endpoint.

Upstream follow-ups worth filing (out of scope here):

• pinference: dot in encrypted_content breaks agents that allowlist OpenAI's dot-free token shape; the opaque 400 also defeats agents' own strip-and-retry fallbacks.
• OpenClaw: allowlist dotted replay tokens (JWT-shaped), or skip redaction for encrypted_content fields.

Validated: failing 5/5 before (3 CI runs + 2 local), passing 4/4 locally after.

2. test_env_id_user_sim_with_tools — 422 mode

The CI model occasionally returns a fully-empty completion; it records as content=None and message_to_wire replays it as "content": null in the next segment's initial messages — strict providers reject the request (content is required unless an assistant message includes tool_calls). Empty completions now replay as "".

3. test_env_id_user_sim_with_tools — score-0 mode

Local transcript: the tool ran and returned hello world [ok-7f3], but the sim-user paraphrased away the "reply with exactly what it returns" requirement, so the assistant dropped the stamp and the reward scored 0. The echo-tool-v1 reward now scores the stamped tool result — per the fixture's own contract ("reward 1.0 only if the model actually called the tool — trivial when the infra works, impossible when it doesn't") — instead of grading whether the model parrots the stamp verbatim.

Validated: failing 4/4 CI runs before; passing 3/3 locally after (2 targeted runs + test_tool placement).

4. test_env_id_user_sim_with_tools — task-leak mode

Local transcript: the sim-user (a strong instruction-follower) reads the scenario's imperative text ("Call the echo_back tool … reply inside <answer></answer> tags") as its own instructions and opens the conversation with <answer>hello world</answer> itself — the assistant is never asked to do anything, no tool ever runs (zero TOOL messages, so 3.'s reward fix can't save it), the sim-user closes with ###DONE###, and the episode scores 0.

Fix: harden the user-sim PERSONA — frame the scenario as what you want the ASSISTANT to do for you, and add an explicit rule that the user seat never performs the task itself (no tool calls, answers, or output formats; it asks the assistant instead).

Validated: failing 2/2 locally before (with fixes 1–3 already in-tree); passing 3/3 after, sibling test_env_id_user_sim still green.

5. test_single_turn/test_agentic[rlm-*], test_tool_state — unretried harness flakes

RLM occasionally ends a turn with zero output (ACP agent produced no visible reply, survives #2262's grace period — the model genuinely emits nothing), and test_tool_state occasionally stalls into its rollout budget (agent timeout). Both record as HarnessError, which the e2e retry include didn't cover (only ProviderError). The conftest now retries HarnessError rollouts too; deterministic harness bugs still fail all three attempts.

Validation

• uv run pytest tests/v1 -m "not e2e" — all passed
• uv run pytest tests/ -m "not e2e" — all passed
• targeted live e2e: openclaw resume ×4, user_sim ×3, tool[subprocess-colocated] ×1 — all passed
• persona fix: test_env_id_user_sim_with_tools ×3 + test_env_id_user_sim ×1 — all passed
• uv run ruff check ., ty check verifiers (3.13), uv run pre-commit run --all-files — passed

---

Note

Medium Risk
Production paths change OpenClaw reasoning defaults, chat serialization, and user-sim behavior—not just tests—though changes are narrow and aimed at known flake/failure modes.

Overview
Targets five live-E2E failure modes with small behavioral fixes in harness, dialect, env, and test defaults.

OpenClaw resume: reasoning on the intercept model is now driven only by sampling.reasoning_effort (not gpt-5/o* name heuristics), avoiding persisted encrypted-reasoning replay that 400s on session resume.

Chat wire: message_to_wire sends content: "" instead of null for assistant turns with no text and no tool calls, so strict providers accept the next request.

User-sim: The sim-user PERSONA frames the scenario as what the assistant should do and forbids the user seat from performing tools, answers, or formatted output itself—fixing cases where the user “completed” the task before the assistant ran tools.

Echo-tool fixture: The echoed reward checks stamped text in tool messages (via content_text), not whether the assistant parroted the stamp.

E2E defaults: Per-seat retries in conftest now include HarnessError alongside ProviderError (still max 2 retries).

^{Reviewed by Cursor Bugbot for commit de73918. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Fix flaky live E2E tests by correcting reward logic, retry config, and assistant message handling

• Adds HarnessError to the retry list alongside ProviderError in _eval_config, so transient harness failures no longer fail seats permanently.
• Fixes the echoed reward in EchoToolTask to check tool messages (not assistant messages) for the expected phrase and token, requiring actual tool execution.
• Sets assistant message content to an empty string instead of null when there are no tool calls in message_to_wire, improving provider compatibility.
• Changes OpenClawHarness to enable reasoning only when sampling.reasoning_effort is explicitly set, removing the model-name heuristic.
• Behavioral Change: echoed reward scores will change for any rollout where the assistant relayed the phrase without invoking the tool.

^{Macroscope summarized de73918.}

[macroscopeapp]

Approvability

Verdict: Approved de73918

This PR contains defensive bugfixes for E2E test flakiness: handling null content for strict API providers, narrowing reasoning detection to explicit opt-in, and updating test retry/validation logic. All changes have clear intent, limited scope, and low risk.

^{You can customize Macroscope's approvability policy. Learn more.}