test(scenarios): eliminate actionCalled-only checks

[lalalune]

Summary

• adds shared CALENDAR scenario assertion helpers that inspect successful result data or post-action payloads
• converts the remaining direct scenario finalChecks arrays away from actionCalled-only checks across calendar, deterministic fixtures, cross-cutting/social/reminder, self-control, lifeops habits/hygiene, relationships, remote, and cloud-apps scenarios
• tightens negative/guidance scenarios with whole-run forbidden-side-effect checks where the intended behavior is to clarify or guide without mutating state
• lowers the testing: pay down the 60 actionCalled-only scenarios to real effect assertions (#9310 subtask C) #11381 action-effect ratchet from post-rebase 25 to 0

Relates to #11381.

Evidence

• Branch rebased onto origin/develop b17e23500c8491e3191abfb0d1cf8f000959de6e; pushed head 88bff0f884e19772ee899a1684c036ff2f0024d2.
• Focused ratchet: bun test packages/scenario-runner/src/action-effect-ratchet.test.ts -> 2 pass, baseline 0.
• Formatting: bunx @biomejs/biome check --config-path biome.json --vcs-enabled=false --files-ignore-unknown=true --no-errors-on-unmatched $(git diff --name-only origin/develop...HEAD | rg '\.ts$') -> checked all 32 changed TypeScript files, no fixes applied.
• Whitespace: git diff --check -> clean.
• Deterministic scenario reports: .github/issue-evidence/11381-actioncalled-paydown-2/deterministic-runs*.
• Evidence note: .github/issue-evidence/11381-actioncalled-paydown-2/README.md.

Deterministic Runs Reviewed

• deterministic-browser-computeruse-progress -> run 52c3d174-9ae6-4208-8f10-7d6816ce6566, passed 1/1; report includes BROWSER screenshot data, COMPUTER_USE_AGENT finish trajectory, and COMPUTER_USE click result.
• deterministic-computeruse-progress-approvals -> run 1b48981e-2407-4b9a-93ba-65c4d9a4beef, passed 1/1; report includes approval prompt result and approval resolution payload.
• deterministic-computeruse-parity-verbs -> run 29392477-7ee9-48ef-b2da-0ee1da4766ff, passed 1/1; report includes set_value, kill_app killed=true, and WINDOW bounds.
• deterministic-gitpathology-actions -> run c5edf5de-23bb-4c75-9ef2-cef395592f6d, passed 1/1; report includes empty-cache reports: [] and finished trajectory JSON artifacts.

Remaining actionCalled-only scenarios

None found by packages/scenario-runner/src/action-effect-ratchet.test.ts at baseline 0.

Verification gaps

• bun run verify fails before typecheck/lint in audit:type-safety-ratchet: as unknown as: 79 / 75 (unrelated repo-wide baseline).
• bun run --cwd packages/scenario-runner typecheck fails before isolating this change on unrelated workspace package/export prerequisites, including @elizaos/plugin-blocker/services/website-blocker/index, @elizaos/plugin-personal-assistant/lifeops/service, @elizaos/plugin-scheduling, @elizaos/shared/steward-session-client, @elizaos/shared/transcripts, @elizaos/shared/local-inference, and @elizaos/capacitor-bun-runtime.
• Live LLM trajectories were not captured for the live-only scenario conversions: no provider keys were present in the environment (OPENAI_API_KEY, ANTHROPIC_API_KEY, GROQ_API_KEY, GOOGLE_GENERATIVE_AI_API_KEY, CEREBRAS_API_KEY, EVAL_MODEL_PROVIDER all missing).

UI

N/A - no UI changes.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 07565124-581b-441f-a295-bcd4bafa0ecc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch chore/11381-actioncalled-calendar-paydown

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[lalalune]

Review pass on #11708 (left draft, not merging).

Validated locally after rebasing the branch onto current origin/develop (e59ddaf):

• bun run --cwd packages/scenario-runner test -- src/action-effect-ratchet.test.ts passed.
• Strict deterministic scenario run passed 4/4 with SCENARIO_USE_LLM_PROXY=1 SCENARIO_LLM_PROXY_STRICT=1 for:
  • deterministic-browser-computeruse-progress
  • deterministic-computeruse-progress-approvals
  • deterministic-computeruse-parity-verbs
  • deterministic-gitpathology-actions
• Biome check passed on the 22 changed TypeScript files.
• Diff whitespace check passed.

I reviewed the new calendar assertion helper and the converted scenarios; the checks are using observed action/result payloads rather than only action-called markers, which matches the paydown goal. No blocking issue found in this pass. CI was still queued/unstable on GitHub, and the PR is draft, so I did not merge or mark ready.

[lalalune]

Fresh review/validation after rebase onto current develop (08cf0678ce), pushed as head ca93160fc7:

• bun run --cwd packages/scenario-runner test -- src/action-effect-ratchet.test.ts -> passed, 2 tests, BASELINE=0.
• Changed TypeScript files: bunx @biomejs/biome check <changed-ts-files> -> checked 32 files, no fixes.
• git diff --check origin/develop...HEAD -> clean.
• Merge-base check confirms current origin/develop is an ancestor of the branch.

I attempted to rerun the deterministic evidence subset (deterministic-browser-computeruse-progress, deterministic-computeruse-progress-approvals, deterministic-computeruse-parity-verbs, deterministic-gitpathology-actions) with strict proxy mode. The first run was polluted by this machine’s main-checkout workspace symlinks: @elizaos/core resolved to /home/shaw/milady/eliza, whose dirty packages/core/src/trajectory-context.ts lacks runWithTrajectoryPurpose. After removing package-local node_modules symlinks in the isolated worktree and preserving symlinks, the runner reached the existing dependency baseline instead (Cannot find package uuid / react-dom), matching the package/export prerequisite gap already documented in the PR.

Keeping draft: ratchet/formats are green, but live trajectory evidence is still absent and deterministic rerun is blocked by local workspace dependency resolution rather than scenario failures.

[lalalune]

Review (revw2) — verified locally, one defect found and fixed on the branch

Verdict: fix pushed, then merge. Adversarial pass on head 88bff0f in a clean worktree against origin/develop.

What checks out

• Conversions are real effect assertions, not larp. All 25 converted scenario modules load and expose custom predicate finalChecks (verified programmatically). Predicates read result.data / payload blobs / turn responseText / forbidden-side-effect sets — not just "the action ran".
• No coverage shrink: 0 scenario files deleted; every conversion strengthens or faithfully re-expresses intent (cross.negative keeps its per-turn forbiddenActions AND adds a whole-run forbidden check).
• Ratchet: action-effect-ratchet.test.ts green at BASELINE = 0; develop still at 25; no develop commits touched these paths since the merge-base.
• pr-deterministic lane: 36/37 passed. The single failure (deterministic-browser-actions, untouched by this PR) is a pre-existing timing race — expected BROWSER_WAIT_FOR_URL to poll at least twice before matching, saw 1 (tab hit the callback URL before the second 50 ms poll). It passed on immediate re-run.
• Mutation check (not vacuous): broke the expected effect text in deterministic-gitpathology-actions' new predicate → lane went red with the real payload in the failure message. Reverted.
• Biome: 32 changed TS files clean.

Defect found → fixed and pushed (798bed9766c)

Converting the gitpathology finalCheck away from actionCalled broke src/__tests__/deterministic-action-coverage.test.ts (2 new red tests): scenarioActionNames() derives coverage from actionName fields on loaded finalChecks, so GIT_PATHOLOGY read as uncovered ("no longer referenced by any scenario" + KNOWN_UNCOVERED drift). Fix: restored the actionCalled entry as a structural coverage marker alongside the custom effect predicate (which remains the real gate). The ratchet doesn't flag mixed check arrays (and doesn't scan this dir at all). Coverage suite now back to develop's profile; scenario still passes in the strict lane.

Pre-existing, NOT this PR (needs its own fix)

deterministic-action-coverage.test.ts > every deterministic message scenario is classified… fails identically on current develop: deterministic-active-view-agent-surface (added in #11610) has message turns + ACTION_PLANNER fixtures but no RESPONSE_HANDLER fixture, so it fits neither STRICT_LLM_ROUTING_SCENARIOS nor PROSE_ONLY. Out of scope here.

Notes / minor

• selfcontrol.integration-with-todos.auto-block inverts the old assertion (was: WEBSITE_BLOCK must fire; now: must NOT fire, expect clarification). Defensible — the user's rule is conditional ("if my workout isn't done by noon") with unspecified sites, so an immediate block was arguably the wrong expectation — but it is an intent change, flagging for awareness.
• expectConfirmedXBlock's /x(?:\.com)?/ first clause is near-vacuous (matches any "x" in the blob); the (hour|60|3600) duration clause carries the check. Cosmetic.
• Live-only scenarios (calendar/lifeops/relationships/remote/selfcontrol/social) were smoke-loaded but not executed — no LLM provider keys in this environment, same gap the PR body documents.

🤖 Generated with Claude Code

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.

[github-actions]

LifeOps Benchmark — eliza

Run ID: lifeops-eliza-28633649056

LifeOps Benchmark

Model: gemma-4-31b
Judge: claude-opus-4-7
Scenarios: 25
pass@1: 0.000
pass@k: 0.000
Total cost: $0.0000

Full artifacts: see the lifeops-run-eliza-28633649056 upload on this run.