feat(cadence): shadow-evaluation mode for direct P1 vs P2 policy comparison (#875) - #881
Conversation
…_p2 (#875) Introduces would_fire_p1 and would_fire_p2 — pure predicates returning (bool, reason) without checking config.policy. The existing should_fire / should_fire_p2 are re-expressed as policy-check + delegate, preserving behavior. This is the substrate for #875 shadow- evaluation mode, which needs to evaluate every policy's would-fire decision per Stop-hook tick regardless of which policy is selected. Tests: 8 new would_fire_p1 cases + 9 new would_fire_p2 cases (disabled / bad-config / divergent / true / determinism / policy-agnostic / should_fire-delegation). Full cadence + hook-cadence suite stays green (188 passed).
Adds [cadence] shadow_mode_enabled TOML key + AELFRICE_CADENCE_ SHADOW_MODE_ENABLED env, default OFF. Resolver follows the env > kwarg > TOML > default precedence used by every other cadence knob. Wired into CadenceConfig as a sixth field; load_cadence_config reads it via _read_bool (same fail-soft pattern as `enabled`). This is the opt-in gate for #875 shadow-evaluation mode. The predicate-evaluation and log-write paths land in subsequent commits; this one just gives them a flag to read. Tests: 7 new cases — default-false, env-wins, kwarg-wins, env-beats- kwarg, unparseable-env-falls-through, TOML-true, wrong-type-ignored (falls back to default with stderr warning).
) Three new public surfaces in src/aelfrice/cadence.py: - shadow_log_path(project_aelfrice_dir, session_id) -> Path. Pure path computation; layout is <aelfrice-dir>/cadence_shadow/<sid>.jsonl, sibling of rebuild_logs/ and cadence_resume_cache.json. - format_shadow_row(session_id, selected_policy, fired, shadow, now) -> str. Pure formatter; ts is supplied by caller (kept outside the formatter to preserve replay-ability in tests and scoring). Schema passes extra shadow keys through so P3 metrics can ride without a schema bump. - append_shadow_row(log_path, row_line) -> None. Mkdir parent, append one line, swallow OSError fail-soft (Stop-hook hot-path). CADENCE_SHADOW_DIRNAME = "cadence_shadow" exposed as a constant for hook-side path derivation in the next commit. Tests: 7 new cases — path layout, no-disk-side-effect on resolve, JSON schema, determinism replay, extra-keys passthrough, append + two-row concatenation, fail-soft on unwritable parent.
Hooks the shadow-log writer into _maybe_fire_cadence_checkpoint. On every Stop tick where cadence is enabled, a new helper _maybe_log_cadence_shadow_tick computes would_fire_p1 + would_fire_p2 against the same inputs the live dispatch reads, derives the selected-policy `fired` boolean, and appends one row to <aelfrice-dir>/cadence_shadow/<sid>.jsonl. Gating: opt-in via [cadence] shadow_mode_enabled (default OFF). When off, the helper returns on the first line at no measurable cost. Cadence enabled=false short-circuits BEFORE shadow eval, so a stray shadow flag on a disabled cadence doesn't write surprise disk content. Fail-soft: any exception in the shadow path traces stderr and returns; live dispatch is unaffected. Tests: 8 new integration cases — disabled-default no-write, P1-no- fire row schema (reason carries the diagnostic), P1-fire row, P2- selected logs both policies, policy=off logs fired=false even when P1 would have fired, two-tick append, cadence-disabled no-write, env override (AELFRICE_CADENCE_SHADOW_MODE_ENABLED=1) flips it on without TOML change. Live-dispatch tests stay green (210 cadence + hook-cadence tests pass total).
…er shadow logs (#875) New module src/aelfrice/cadence_score.py with the scoring surface: - iter_shadow_rows(shadow_dir) — generator over every JSONL line in the cadence_shadow dir; skips malformed lines + unreadable files. - compute_summary(rows, session_filter=None) -> ShadowSummary — aggregates per-policy would_fire counts, selected-policy live fire counts, and a 2x2 P1 vs P2 agreement matrix. - format_report(summary, as_json=False) -> str — human-readable block by default, machine-readable JSON with --json. - resolve_shadow_dir(project_root) — derives the on-disk path. Pure data processing; reads only the shadow log; no live cadence state. Determinism (#605): same rows -> same report, byte-for-byte. Tests: 14 cases covering empty input, totals, session filtering, agreement-matrix counts, selected-policy fire-rate, malformed decision skip, missing-dir tolerance, JSONL parsing edge cases (bad-json line, list-not-dict), and JSON-output roundtrip. The CLI subcommand wrapper lands in the next commit.
Hidden subcommand wrapping the cadence_score module. Flags: --project PATH project root (default: cwd) --session SESSION_ID filter to one session_id --json machine-readable JSON instead of text report Reads <project>/.git/aelfrice/cadence_shadow/*.jsonl, aggregates, prints. Exit codes: 0 (report emitted), 2 (project not a directory). Registered in HIDDEN_SUBCOMMANDS in tests/test_slash_commands.py — no slash file, since this is R&D tooling for the #749 campaign, not a daily workflow verb. May graduate later if the surface stabilises. Tests: 5 new CLI-surface cases — text report, --json roundtrip, --session filter aggregates only matching rows, missing-project returns 2, empty-dir emits zero-row report cleanly.
…mode Documents the new surfaces: would_fire_* policy-agnostic predicates, [cadence] shadow_mode_enabled TOML/env opt-in, the on-disk shadow log layout, the aelf cadence-score CLI subcommand, and the rationale (unblocks #749 campaign comparison without longitudinal flip-and- rebake confound).
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @robotrocketscience, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (10)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
robotrocketscience
added
the
attn:review
PR-size soft capThis PR is over the advisory size threshold:
Bigger PRs collide with more open work, which under the parallel-session workflow tends to produce repeated This is advisory only — nothing is blocked. If the size is intentional (large refactor, module removal, generated code), apply the |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
robotrocketscience
added
the
ready-to-merge
|
merge-train: blocked 1 review thread(s) are unresolved on these files: tests/test_cadence_score.py. Resolve them on the PR (click 'Resolve conversation' on each) and re-add the label. The |
github-actions
Bot
removed
the
ready-to-merge
robotrocketscience
added
the
ready-to-merge
github-actions
Bot
removed
the
ready-to-merge
|
merge-train: merged 310bbb3 → |
2 participants
Summary
Implements #875 cadence shadow-evaluation mode. Lifts the single-policy
mutex that blocked direct P1 vs P2 comparison on identical workload —
the campaign's stated goal. Selected policy still drives live firing;
non-selected policies log decisions to a side-channel for offline
scoring.
Closes #875.
What ships
would_fire_p1/would_fire_p2— policy-agnostic predicatesin
src/aelfrice/cadence.pyreturning(bool, reason). Existingshould_fire/should_fire_p2re-expressed as policy-check +delegate. Behavior identical for live dispatch.
[cadence] shadow_mode_enabledTOML opt-in (default OFF) +AELFRICE_CADENCE_SHADOW_MODE_ENABLEDenv override. Resolverfollows the standard env > kwarg > TOML > default precedence.
.git/aelfrice/cadence_shadow/<session_id>.jsonl. One row perStop-hook tick when enabled. Row schema:
{ "ts": "2026-05-20T23:59:00Z", "session_id": "...", "selected": "p2_ctx_threshold", "fired": true, "shadow": { "p1_every_k_turns": {"would_fire": false, "reason": "fire_idx=8 mod k=15 = 8"}, "p2_ctx_threshold": {"would_fire": true, "reason": "transcript bytes=405000 >= watermark=300000, phase-boundary"} } }aelf cadence-score(hidden subcommand) — reads the on-diskshadow log and emits per-policy fire-rate + 2x2 P1/P2 agreement
matrix. Flags:
--project PATH(default cwd),--session SESSION_IDfilter,--jsonfor machine-readable output.Design properties honored
would_fire_*are pure functions of theirinputs; shadow-log replay reproduces decisions; scoring is
deterministic over the on-disk rows.
ab96e9d3501b1c14): shadow log lives under.git/aelfrice/(gitignored); inspects only inputs the livepolicy module already reads; no
~/.claude/-derived state enters.and returns; live cadence dispatch is unaffected.
[cadence] enabled = trueso a stray flag on disabled cadence doesn't write surprisedisk content.
Sequencing
Unblocks #876 (P3 turn-density-aware design). Per the #876 sequencing
decision: P3 designs become data-driven once shadow data accrues from
this PR.
Test plan
would_fire_*purity cases (disabled / bad-config / divergentinputs / true / determinism replay / policy-agnostic verification
/ should_fire-delegation invariant).
beats-kwarg, unparseable-env-falls-through, TOML-true, wrong-
type-ignored with stderr warning).
JSON schema, determinism, extra-keys passthrough, append + two-
row, fail-soft on unwritable parent).
fire row, P1 fire row, P2 selected logs both policies, policy=
off logs fired=false even when P1 would have fired, two-tick
append, cadence-disabled no-write, env override).
per-policy counts, selected fire-rate, agreement matrix,
session filter, malformed-decision skip, missing-dir tolerance,
JSONL edge cases, JSON roundtrip).
filter, missing-project returns 2, empty-dir zero-row report).
regressions on pre-existing tests.
aelf cadence-score --helpsmoke-tested via uv run aelf.Commits
7 atomic commits per #ab96e9d3501b1c14 (atomic > batched):
refactor(cadence): extract policy-agnostic would_fire_p1 / would_fire_p2feat(cadence): shadow_mode_enabled config + resolverfeat(cadence): shadow-log path resolver + row formatter + appenderfeat(hook): wire Stop-hook shadow-evaluation loggingfeat(cadence): cadence_score module — agreement matrix + fire-ratefeat(cli): aelf cadence-score subcommand for shadow-log reportingdocs(changelog): unreleased entry for #875 cadence shadow-evaluation mode