Skip to content

feat(cadence): p3_substantive dispatch + shadow capture + 4-policy scorer + offline replay (#876) - #920

Merged
github-actions[bot] merged 4 commits into
mainfrom
feat/issue-876-p3-substantive
May 26, 2026
Merged

feat(cadence): p3_substantive dispatch + shadow capture + 4-policy scorer + offline replay (#876)#920
github-actions[bot] merged 4 commits into
mainfrom
feat/issue-876-p3-substantive

Conversation

@yoshi280

@yoshi280 yoshi280 commented May 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

Completes the #876 P3 cadence work and makes the bench gate runnable.
The prior series (#884/#885/#886, PRs 1–3 of 5) shipped the enum
constants, session-ring slots, and the p3_velocity live dispatch, but
stopped there: p3_substantive was never dispatched (selecting it was a
silent no-op), the shadow logger captured only P1/P2, and aelf cadence-score could only report a 2×2 P1-vs-P2 matrix. So the
bench-gated comparison the triage (axis-3 bake) depends on had no P3
data and no way to produce it offline.

This PR closes that gap in four atomic commits.

Commits

  1. feat(cadence): dispatch p3_substantive live in Stop + UPS — adds
    the POLICY_P3_SUBSTANTIVE branch to both dispatchers. Stop owns the
    per-turn classification push (is_substantive_turn
    push_classification onto the session-ring rolling window) and reads
    the window to fire; UPS reads the window read-only (no push) to keep
    it advancing exactly once per turn — a one-turn read lag matching the
    established p3_velocity counter-sharing semantics.

  2. feat(cadence): capture both P3 policies in shadow log
    _maybe_log_cadence_shadow_tick now evaluates would_fire_p3_velocity
    and would_fire_p3_substantive against the same tick inputs and
    records all four policies in the shadow row. This is the bake data
    source.

  3. feat(cadence-score): four-policy pairwise comparison — per-policy
    fire rates already spanned every logged policy; adds a generalised
    pairwise_agreement table (both / a-only / b-only / neither + a
    divergence rate) across every policy pair, so all four compare
    head-to-head. The legacy 2×2 P1-vs-P2 matrix is retained unchanged.

  4. feat(bench): offline cadence-policy replay harness — the
    "replay" half: benchmarks/cadence_replay.py produces the same
    four-policy comparison from a self-contained synthetic fixture, with
    no operator-week of live shadow data. Each tick is fed through every
    would_fire predicate and aggregated by compute_summary.
    Deterministic (v3.0 PHILOSOPHY: natural-language-relatedness gate — deterministic vs embedding #605); sample fixture included.

Determinism / discretion

All predicates are pure functions (#605); the replay is byte-identical
across runs. No ~/.claude/-derived content — the fixture is synthetic
input authored alongside the bench.

Testing

  • New: test_hook_stop_cadence_p3_substantive,
    test_hook_ups_cadence_p3_substantive, the P3-capture test in
    test_hook_stop_cadence_shadow, four-policy/pairwise tests in
    test_cadence_score, and test_cadence_replay (8 tests incl. CLI +
    determinism).
  • Sample replay output (3-tick fixture) shows the head-to-head spread:
    p1 ↔ p3_substantive co-fire on the dense-substantive tick, p2 ↔ p3_velocity co-fire on the high-byte boundary tick.
  • 795 passed / 3 skipped across the cadence/hook/shadow subset.

Closes #876.

Refs #749, #875, #884, #885, #886.

Summary by Sourcery

Wire the P3 substantive cadence policy through live dispatch, shadow logging, scoring, and benchmarking to complete four-policy cadence evaluation and offline replay.

New Features:

  • Enable the P3 substantive cadence policy in both Stop and UPS dispatchers, using rolling substantive-turn classification to decide when to fire.
  • Extend cadence shadow logging to record would_fire decisions for all four policies, including P3 velocity and P3 substantive, per tick.
  • Augment the cadence-score reporting to include a general pairwise policy agreement table across all policies alongside existing per-policy and P1/P2 metrics.
  • Introduce an offline deterministic cadence replay benchmark that replays synthetic fixtures through all four policies and summarizes them via the existing scoring pipeline.

Documentation:

  • Document the new P3 substantive policy wiring, four-policy shadow capture, extended scorer output, and offline replay bench in the v3 changelog.

Tests:

  • Add Stop- and UPS-side integration tests for P3 substantive dispatch, including window maintenance and non-mutation guarantees for UPS.
  • Add tests ensuring the shadow log captures all four policies’ decisions, and that cadence-score correctly computes four-policy per-policy rates and pairwise agreement.
  • Add tests for the cadence replay harness and CLI covering determinism, fixture handling, and JSON/text output.

Summary by CodeRabbit

  • New Features

    • P3 substantive-window cadence policy now integrated into Stop/UPS dispatchers for conversation quality-based trigger decisions
    • Multi-policy pairwise agreement comparison in cadence scoring reports
    • Offline cadence replay tool for deterministic scenario testing
  • Tests

    • Extended integration test coverage for substantive-window behavior with various thresholds and configurations
    • New shadow log evaluation tests verifying all four policies are independently evaluated and logged
    • Comprehensive CLI tests for the cadence replay tool with JSON output support
  • Documentation

    • Updated changelog documenting cadence policy additions and enhanced tooling

Review Change Stack

@yoshi280 yoshi280 added the author-Setr PR coordination mutex label May 26, 2026
@coderabbitai

coderabbitai Bot commented May 26, 2026

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

Extends the cadence policy framework to support P3 substantive-turn density detection. Stop-side implements per-turn classification, stores rolling history in session state, and fires when substantive-window threshold is met. UPS-side reads the same history without mutation and injects rebuilds. Cadence-score now reports pairwise agreement across all four policies, and a new offline replay benchmark enables shadow evaluation of fixture-based scenarios.

Changes

P3 Substantive Cadence Policy Implementation and Analytics

Layer / File(s) Summary
Cadence-score pairwise agreement reporting
src/aelfrice/cadence_score.py, tests/test_cadence_score.py
ShadowSummary gains pairwise_agreement table for all policy-pair combinations alongside the legacy P1/P2 matrix. compute_summary builds per-row fire maps and increments pairwise counts for every observed pair. format_report includes pairwise results in JSON and text. Tests validate four-policy aggregation and pairwise matrix structure.
Offline cadence replay benchmark
benchmarks/cadence_replay.py, benchmarks/fixtures/cadence_replay_sample.json, tests/test_cadence_replay.py
New replay harness evaluates all four policies on synthetic fixtures, emitting shadow-log-shaped rows. CLI accepts fixture JSON, optionally outputs summary as text or JSON file. Sample fixture and comprehensive test coverage validate row shape, per-tick decisions, determinism, and CLI argument handling.
Stop-side P3 substantive dispatch and firing
src/aelfrice/hook.py, tests/test_hook_stop_cadence_p3_substantive.py
Integrated P3 substantive policy into Stop cadence dispatcher. Resolves substantive window/threshold, reads latest user prompt, classifies and pushes per-tick state into session ring, computes rolling substantive ratio, fires when threshold is met, runs rebuild and caches resume state. Tests verify classification semantics, firing thresholds, filler non-substantive classification, disabled-cadence bypass, and window-cap eviction.
UPS-side P3 substantive dispatch and injection
src/aelfrice/hook.py, tests/test_hook_ups_cadence_p3_substantive.py
Integrated P3 substantive policy into UPS cadence dispatcher following read-only contract. Resolves substantive window/threshold, reads existing classification history from session ring without mutation, computes rolling substantive ratio, fires when threshold is met, runs rebuild and caches resume state, and returns rebuild body for injection. Tests verify above/below-threshold behavior and confirm classification immutability.
Extended cadence shadow logging for P3 predicates
src/aelfrice/hook.py, tests/test_hook_stop_cadence_shadow.py
Extended shadow tick logging to evaluate and report P3 velocity and P3 substantive predicates. Populated P3 knobs in shared CadenceConfig. Computed P3 evaluation inputs and both P3 predicates; extended selected-policy to fired mapping and shadow-row payload to include P3 would_fire results. Test validates shadow rows capture all four policies' decisions even when P1 is selected.
Changelog documentation
CHANGELOG/v3.md
Added four Unreleased entries: P3 substantive policy wiring, four-policy shadow logging, four-policy pairwise cadence-score reporting, and offline replay benchmark with sample fixture.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately captures the four main atomic changes: p3_substantive dispatch, shadow capture for all four policies, four-policy scorer, and offline replay. It directly maps to the changeset and is concise and clear.
Description check ✅ Passed The description provides a detailed summary of the PR's purpose, lists four atomic commits with clear scope, documents testing coverage, and closes/references linked issues. All critical template sections are complete with substantive content.
Linked Issues check ✅ Passed The PR successfully delivers all coding objectives from #876: p3_substantive dispatch in both Stop/UPS [hook.py], shadow capture for all four policies [hook.py, test_hook_stop_cadence_shadow.py], four-policy pairwise comparison [cadence_score.py, test_cadence_score.py], and offline deterministic replay harness [cadence_replay.py, test_cadence_replay.py, fixture], with comprehensive tests verifying each requirement.
Out of Scope Changes check ✅ Passed All changes directly support the four-commit P3 completion roadmap: dispatch wiring, shadow logging, scoring generalization, and offline replay. The CHANGELOG documentation and sample fixture are necessary supporting materials. No unrelated or extraneous changes detected.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/issue-876-p3-substantive

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@sourcery-ai

sourcery-ai Bot commented May 26, 2026

Copy link
Copy Markdown

Reviewer's Guide

Implements live dispatch of the p3_substantive cadence policy in both Stop and UPS hooks, extends the shadow cadence logger and scoring CLI to handle all four cadence policies with pairwise comparisons, and adds an offline deterministic replay benchmark plus tests and changelog updates to support P3 cadence evaluation for #876.

Sequence diagram for live p3_substantive cadence dispatch in Stop and UPS hooks

sequenceDiagram
    participant StopHook as Stop_hook_maybe_fire_cadence_checkpoint
    participant UPSHook as UPS_hook_maybe_run_ups_cadence_checkpoint
    participant SessionRing as session_ring
    participant Cadence as cadence_predicates
    participant Rebuilder as context_rebuilder

    StopHook->>Cadence: resolve_cadence_p3_substantive_window
    StopHook->>Cadence: resolve_cadence_p3_substantive_threshold
    StopHook->>Cadence: is_substantive_turn
    StopHook->>SessionRing: push_classification
    StopHook->>SessionRing: read_ring_state
    StopHook->>Cadence: should_fire_p3_substantive
    alt [should_fire_p3_substantive]
        StopHook->>Rebuilder: _run_cadence_rebuild
        Rebuilder-->>StopHook: body
        StopHook->>Rebuilder: _write_cadence_resume_cache
    end

    UPSHook->>Cadence: resolve_cadence_p3_substantive_window
    UPSHook->>Cadence: resolve_cadence_p3_substantive_threshold
    UPSHook->>SessionRing: read_ring_state
    UPSHook->>Cadence: should_fire_p3_substantive
    alt [should_fire_p3_substantive]
        UPSHook->>Rebuilder: _run_cadence_rebuild
        Rebuilder-->>UPSHook: body
    end
Loading

Flow diagram for offline cadence replay benchmark and scoring

flowchart TD
    A[CLI main] --> B[parse_args]
    B --> C[read fixture JSON]
    C --> D[_config_from_fixture]
    D --> E[replay_fixture]
    E --> F[for each tick]
    F --> G[_write_transcript]
    G --> H[estimate_transcript_bytes]
    H --> I[would_fire_p1/p2]
    H --> J[would_fire_p3_velocity]
    F --> K[would_fire_p3_substantive]
    I --> L[build shadow row]
    J --> L
    K --> L
    L --> M[rows list]
    M --> N[compute_summary]
    N --> O[format_report]
    O --> P[stdout]
    O --> Q{args.output?}
    Q -->|yes| R[write JSON summary]
    Q -->|no| P
Loading

File-Level Changes

Change Details Files
Wire p3_substantive cadence policy into Stop and UPS dispatchers with correct session-ring semantics.
  • Add POLICY_P3_SUBSTANTIVE handling in _maybe_fire_cadence_checkpoint with per-turn is_substantive_turn classification pushes into the session ring and thresholded firing based on substantive ratio over a configurable rolling window.
  • Add POLICY_P3_SUBSTANTIVE handling in _maybe_run_ups_cadence_checkpoint that reads the classifications window read-only, mirrors the same window/threshold config, and conditionally triggers context rebuild without mutating ring state.
  • Ensure both paths construct CadenceConfig with the new P3 fields, respect existing fail-soft behaviour, and log informative stderr messages when P3 fires.
src/aelfrice/hook.py
CHANGELOG/v3.md
Extend cadence shadow logging to capture all four policies (p1, p2, p3_velocity, p3_substantive) per tick.
  • Update _maybe_log_cadence_shadow_tick to resolve additional P3 configuration knobs, derive p3_velocity and p3_substantive inputs from session-ring state and transcript bytes, and call would_fire_p3_velocity and would_fire_p3_substantive alongside existing P1/P2 checks.
  • Derive the row-level fired flag from whichever policy is currently selected, now including P3 policies, while still logging all policy decisions into the shadow map.
  • Add tests that seed ring state and transcripts to assert the shadow row includes all four policies and that P3 predicates are evaluated and logged independently of the selected policy.
src/aelfrice/hook.py
tests/test_hook_stop_cadence_shadow.py
Generalise cadence-score to support four-policy statistics and pairwise agreement across all policies.
  • Extend ShadowSummary with a pairwise_agreement field and document the expanded semantics in the class docstring.
  • Enhance compute_summary to track per-row policy fire decisions, maintain the legacy P1/P2 2x2 agreement matrix, and build a lexicographically keyed pairwise_agreement table over all policies present in the shadow data.
  • Update format_report to emit pairwise_agreement in both text and JSON modes, and add tests that validate per-policy counts for P3 policies and the structure and ordering of pairwise entries.
src/aelfrice/cadence_score.py
tests/test_cadence_score.py
Introduce a deterministic offline cadence replay benchmark that feeds synthetic ticks through all policies and reports via cadence_score.
  • Add benchmarks/cadence_replay.py implementing fixture parsing, CadenceConfig construction, synthetic transcript generation, per-tick would_fire evaluation for all four policies, and aggregation via compute_summary with a small CLI wrapper.
  • Ensure the replay harness is deterministic by avoiding external state, using temporary transcripts only, and reusing estimate_transcript_bytes so P2 and p3_velocity see consistent byte counts.
  • Provide a sample JSON fixture and tests that cover replay row shape, per-policy decisions, determinism, CLI behaviour, and JSON output wiring.
benchmarks/cadence_replay.py
benchmarks/fixtures/cadence_replay_sample.json
tests/test_cadence_replay.py
Add focused integration tests for p3_substantive behaviour on both Stop and UPS paths, including ring window semantics.
  • Introduce Stop-side tests that configure p3_substantive via TOML, seed ring classifications, drive hook.stop with substantive and filler prompts, and assert firing behaviour, window threshold logic, and window-cap eviction semantics.
  • Introduce UPS-side tests that configure p3_substantive, seed only the classifications window, invoke _maybe_run_ups_cadence_checkpoint, and assert firing behaviour and that the classifications list is not mutated by UPS.
  • Isolate environment-dependent cadence settings in tests and stub context rebuilder interactions so tests focus on cadence wiring and ring state manipulation.
tests/test_hook_stop_cadence_p3_substantive.py
tests/test_hook_ups_cadence_p3_substantive.py

Assessment against linked issues

Issue Objective Addressed Explanation
#876 Document and decide the P3 density signal source (token-velocity vs substantive-turn classification vs both) in the cadence design/docs, without yet implementing it. The PR implements the substantive-based P3 policy (p3_substantive) in code and wiring (Stop/UPS dispatch, shadow logging, replay), but it does not update or add any design-scoping documentation that records the triage decision among the density options. The issue is explicitly a design-scoping/triage task with no code expected yet.
#876 Decide and document how P3 composes with P1/P2 (standalone, refinement-of-P2, or OR-composition) in the cadence design/docs. The PR treats P3 policies as standalone options in implementation and scoring, but it does not explicitly document or codify a composition decision with P1/P2 as described in the issue. The design axis 2 choice is not resolved or written down; instead, P3 is just added as another independent policy.
#876 Decide and document the sequencing of P3 relative to shadow-evaluation mode (#875) and the broader cadence-policy campaign (i.e., whether P3 waits on, parallels, or precedes #875), updating planning documentation accordingly. The PR adds shadow capture for P3 and a replay/bench harness, but it does not modify any planning or design documents to answer the sequencing question posed in the issue. The triage ask about when to land P3 relative to #875 remains a process/design decision, not addressed by these code changes.

Possibly linked issues

  • #: PR implements and dispatches the P3 substantive cadence policy described in the issue, plus shadow logging and replay tools.

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@github-actions

github-actions Bot commented May 26, 2026

Copy link
Copy Markdown

PR-size soft cap

This PR is over the advisory size threshold:

  • 1329 changed lines (limit: 200)
  • 10 changed files (limit: 3)

Bigger PRs collide with more open work, which under the parallel-session workflow tends to produce repeated attn:merge-conflict cycles (see #602). When practical, split into smaller PRs that each touch a focused surface.

This is advisory only — nothing is blocked. If the size is intentional (large refactor, module removal, generated code), apply the size:override label and this comment will be removed on the next push.

@yoshi280 yoshi280 added the attn:review Needs review (PR open, awaiting reviewer) label May 26, 2026

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 3 issues, and left some high level feedback:

  • The computation of the P3 substantive ratio (reading classifications, slicing by window, summing Trues) is duplicated in the Stop dispatcher, UPS dispatcher, shadow logger, and replay bench; consider factoring this into a small shared helper to keep the window semantics aligned if they ever change.
  • Similarly, the P3 velocity input derivation (bytes-at-last-fire, fire-idx-at-last-fire, transcript-bytes, turns-since) is reimplemented in both the shadow logger and replay bench; centralising that logic behind a helper would reduce the risk of the replay diverging subtly from live behaviour.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The computation of the P3 substantive ratio (reading `classifications`, slicing by window, summing `True`s) is duplicated in the Stop dispatcher, UPS dispatcher, shadow logger, and replay bench; consider factoring this into a small shared helper to keep the window semantics aligned if they ever change.
- Similarly, the P3 velocity input derivation (bytes-at-last-fire, fire-idx-at-last-fire, transcript-bytes, turns-since) is reimplemented in both the shadow logger and replay bench; centralising that logic behind a helper would reduce the risk of the replay diverging subtly from live behaviour.

## Individual Comments

### Comment 1
<location path="src/aelfrice/hook.py" line_range="3284-3293" />
<code_context>
+    if policy == POLICY_P3_SUBSTANTIVE:
</code_context>
<issue_to_address>
**suggestion:** Guard or normalise p3_substantive_window to avoid zero/negative-window edge cases and keep semantics consistent.

Across Stop/UPS/shadow/bench we use `p3_substantive_window` directly in `classifications[-window:]`. If a resolver/fixture ever passes `0` or a negative value, the slice semantics become surprising (`[-0:]` is the full list and negatives expand the slice). Consider normalising the value once (e.g. `max(window, 0)` or early-return on non-positive) and centralising this in a helper to keep P3-substantive behaviour consistent across call sites.

Suggested implementation:

```python
    from aelfrice.session_ring import (  # noqa: PLC0415
        push_classification,
        read_ring_state,
        update_p3_velocity_state,
    )

    def _normalise_p3_substantive_window(window: int) -> int:
        """Clamp the P3 substantive window to zero or greater.

        This keeps downstream slicing semantics (`classifications[-window:]`)
        predictable across call sites.
        """
        return max(window, 0)

        )
        return

    if policy == POLICY_P3_SUBSTANTIVE:
        window = _normalise_p3_substantive_window(
            resolve_cadence_p3_substantive_window(start=cwd)
        )

```

To fully implement the suggestion across the codebase, any place that uses `p3_substantive_window` in `classifications[-window:]` (Stop/UPS/shadow/bench) should normalise the window in the same way before slicing, e.g. by calling a shared helper with the same `max(window, 0)` semantics or inlining that normalisation at each call site. If you prefer early-return semantics for non-positive windows instead, you should replace the `max(window, 0)` behaviour with an explicit guard (e.g. `if window <= 0: return ...`) both here and at those other sites.
</issue_to_address>

### Comment 2
<location path="src/aelfrice/hook.py" line_range="3678" />
<code_context>
+            if isinstance(raw_fire_last, int) and not isinstance(raw_fire_last, bool)
+            else 0
+        )
+        transcript_bytes = estimate_transcript_bytes(tp)
+        turns_since_last_fire = fire_idx - fire_idx_at_last_fire
+
</code_context>
<issue_to_address>
**issue (bug_risk):** Avoid passing a possibly-None transcript path into estimate_transcript_bytes.

In the shadow logger, `tp` can be `None` when the transcript path is missing or malformed, but it’s still passed to `estimate_transcript_bytes`, which likely expects a `Path`/PathLike. Add a small guard here (e.g., treat `None` as zero bytes) so shadow logging doesn’t fail on incomplete payloads, while keeping live cadence behavior unchanged.
</issue_to_address>

### Comment 3
<location path="benchmarks/cadence_replay.py" line_range="73-82" />
<code_context>
+def _config_from_fixture(raw: dict[str, Any]) -> CadenceConfig:
</code_context>
<issue_to_address>
**suggestion:** Fixture-driven ints/floats are cast without validation, which can make the bench brittle on malformed input.

In `_config_from_fixture`, numeric fields are cast via `int(...)` / `float(...)` without validation, so a bad fixture value (e.g. `"k": "ten"`) will raise and stop the replay. Consider catching `ValueError` per field and either falling back to the default or raising a more targeted error that clearly identifies the offending key/value.

Suggested implementation:

```python
    would_fire_p3_velocity,
)
from typing import Any, Callable, Mapping, TypeVar

from aelfrice.cadence_score import compute_summary, format_report

TNum = TypeVar("TNum", int, float)


def _coerce_numeric_field(
    key: str,
    raw_cfg: Mapping[str, Any],
    coerce: Callable[[Any], TNum],
    default: TNum,
) -> TNum:
    """Return a validated numeric config value for the given key.

    The function:
    * Returns ``default`` when the key is absent from ``raw_cfg``.
    * Raises a ValueError with a clear message (including the key and value)
      when the provided value cannot be coerced to the requested numeric type.
    """
    if key not in raw_cfg:
        return default

    value = raw_cfg[key]
    try:
        return coerce(value)
    except (TypeError, ValueError) as exc:
        raise ValueError(
            f"Invalid value for cadence config key {key!r}: {value!r} "
            f"(expected a value coercible to {coerce.__name__})"
        ) from exc

```

To fully implement the comment in `_config_from_fixture`, update each place where numeric fields are currently coerced with `int(...)` / `float(...)` on raw fixture values. For example, change patterns like:

```python
window_days = int(cfg.get("window_days", defaults.window_days))
```

to:

```python
window_days = _coerce_numeric_field(
    "window_days",
    cfg,
    int,
    defaults.window_days,
)
```

and for floats:

```python
some_threshold = _coerce_numeric_field(
    "some_threshold",
    cfg,
    float,
    defaults.some_threshold,
)
```

This ensures malformed numeric values in fixtures raise a targeted `ValueError` that identifies the offending key and value, instead of a generic `ValueError` at the cast site. Apply this replacement to all numeric config fields inside `_config_from_fixture`.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread src/aelfrice/hook.py
Comment thread src/aelfrice/hook.py
Comment thread benchmarks/cadence_replay.py
@yoshi280

Copy link
Copy Markdown
Collaborator Author

[claim:review:garsecg:2026-05-26T18:40:53Z]

The p3_substantive predicate (axis-1 option C) shipped in PR 1/5 but was
never wired into the live dispatchers — only p3_velocity was (PR 3/5).
This adds the POLICY_P3_SUBSTANTIVE branch to both _maybe_fire_cadence_
checkpoint (Stop) and _maybe_run_ups_cadence_checkpoint (UPS).

Stop owns the per-turn classification push: each tick the policy is
active it classifies the last user prompt via is_substantive_turn and
push_classification's it onto the session-ring rolling window, then
reads the last p3_substantive_window entries and fires when the
substantive ratio meets p3_substantive_threshold. UPS reads the window
read-only (no push) to keep the window advancing exactly once per turn
— a one-turn read lag matching the established p3_velocity counter-
sharing semantics. Deterministic (#605), no embeddings.

Without this, selecting policy=p3_substantive was a silent no-op (the
classifications slot was never populated, so the predicate could never
fire).
The shadow-evaluation logger (#875) evaluated only would_fire_p1 and
would_fire_p2, so the cadence_shadow JSONL — the bake data source for
aelf cadence-score — carried no P3 decisions. With p3_velocity and
p3_substantive now live policies, the head-to-head comparison the #876
axis-3 bake needs was impossible to collect.

_maybe_log_cadence_shadow_tick now resolves the P3 knobs, derives the
p3_velocity inputs (byte delta / turns since last fire) and the
p3_substantive input (substantive ratio over the rolling window) from
the same ring state, evaluates both would_fire_p3_* predicates, and
records all four policies in the shadow row. Selected-policy fired
derivation extended to the two P3 branches. All inputs read fail-soft
with the same defaults the live dispatch uses.
per_policy_fire_count/total already spanned every policy present in
the shadow log, so once the logger captures P3 (prior commit) the
per-policy rates cover all four. The only P1/P2-specific surface was
the 2x2 agreement matrix.

Add a generalised pairwise_agreement table to ShadowSummary: for every
unordered policy pair logged on a row, count both / a_only / b_only /
neither (a < b lexicographically). format_report renders a 'pairwise
policy agreement' section with a per-pair divergence rate, and the
--json payload gains a pairwise_agreement object. The legacy 2x2
P1-vs-P2 agreement_matrix is retained unchanged for backward
compatibility, so existing consumers and tests are unaffected.
Completes the #876 bench gate: a deterministic offline replay that
produces the four-policy comparison aelf cadence-score emits from a
live shadow bake, without an operator-week of real data.

benchmarks/cadence_replay.py reads a self-contained synthetic fixture
(per-tick fire_idx / byte counts / prompt / classification window +
the cadence config), feeds each tick through would_fire_p1 / p2 /
p3_velocity / p3_substantive, and emits shadow-log-shaped rows that
compute_summary aggregates exactly as it would a live bake — per-policy
fire rates + pairwise divergence. P2 reads transcript size from a temp
file and p3_velocity takes the int; the harness writes one temp
transcript per tick and uses its actual size for both so the two
predicates see a consistent figure.

Deterministic (#605): same fixture -> byte-identical rows. Discretion:
synthetic fixture only, no live state, no ~/.claude content. Sample
fixture at benchmarks/fixtures/cadence_replay_sample.json.
@robotrocketscience
robotrocketscience force-pushed the feat/issue-876-p3-substantive branch from a0b1581 to 51cea5c Compare May 26, 2026 18:43
@yoshi280 yoshi280 added the ready-to-merge Trigger merge-train: FF main to this PR's head label May 26, 2026
@github-actions

Copy link
Copy Markdown

merge-train: blocked

2 review thread(s) are unresolved on these files: benchmarks/cadence_replay.py, src/aelfrice/hook.py. Resolve them on the PR (click 'Resolve conversation' on each) and re-add the label.

The ready-to-merge label has been removed. Address the issue above and re-add the label when you're ready for another attempt.

@github-actions github-actions Bot removed the ready-to-merge Trigger merge-train: FF main to this PR's head label May 26, 2026
@yoshi280 yoshi280 added the ready-to-merge Trigger merge-train: FF main to this PR's head label May 26, 2026
@github-actions

Copy link
Copy Markdown

merge-train: blocked

2 review thread(s) are unresolved on these files: benchmarks/cadence_replay.py, src/aelfrice/hook.py. Resolve them on the PR (click 'Resolve conversation' on each) and re-add the label.

The ready-to-merge label has been removed. Address the issue above and re-add the label when you're ready for another attempt.

@github-actions github-actions Bot removed the ready-to-merge Trigger merge-train: FF main to this PR's head label May 26, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
src/aelfrice/cadence_score.py (1)

144-162: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Only update legacy P1/P2 agreement when both decisions are present and valid.

Line 161 currently increments the legacy matrix even when P1/P2 entries are missing or malformed, which can overcount (False, False) and skew agreement reporting.

Suggested fix
         p1_fire = False
         p2_fire = False
+        p1_seen = False
+        p2_seen = False
         row_fire: dict[str, bool] = {}
         for policy, decision in shadow.items():
             if not isinstance(decision, dict):
                 continue
@@
             row_fire[policy] = would
             if policy == POLICY_P1_EVERY_K_TURNS:
                 p1_fire = would
+                p1_seen = True
             elif policy == POLICY_P2_CTX_THRESHOLD:
                 p2_fire = would
-        agreement[(p1_fire, p2_fire)] += 1
+                p2_seen = True
+        if p1_seen and p2_seen:
+            agreement[(p1_fire, p2_fire)] += 1
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aelfrice/cadence_score.py` around lines 144 - 162, The legacy P1/P2
agreement matrix is updated unconditionally at the end of the loop, which counts
rows where P1 or P2 entries are missing/malformed; modify the logic so
agreement[(p1_fire, p2_fire)] += 1 only runs when both POLICY_P1_EVERY_K_TURNS
and POLICY_P2_CTX_THRESHOLD were actually present and parsed as booleans: detect
presence while iterating shadow (e.g., track flags or check row_fire keys for
those two policies), and move or guard the increment accordingly so it only
updates agreement when both valid decisions exist.
src/aelfrice/hook.py (1)

3155-3161: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Shadow P3-substantive evaluation is one turn stale.

At Line 3155, shadow evaluation runs before the Stop-side p3_substantive push at Line 3310. But shadow p3_substantive uses pre-push classifications at Lines 3689-3692, so fired/would_fire can diverge from the live decision for the same tick.

This skews shadow rows and downstream pairwise scoring for P3-substantive.

Proposed fix (compute P3-substantive shadow on effective current-turn window, without mutating ring)
@@
     from aelfrice.cadence import (  # noqa: PLC0415
         CadenceConfig,
@@
         estimate_transcript_bytes,
         format_shadow_row,
+        is_substantive_turn,
         read_last_user_prompt,
@@
         last_prompt = read_last_user_prompt(tp)
@@
         raw_classes: Any = (
             state.get("classifications") if isinstance(state, dict) else None
         )
         classifications = raw_classes if isinstance(raw_classes, list) else []
+        current_turn_substantive = is_substantive_turn(last_prompt)
+        effective_classes = classifications + [current_turn_substantive]
         substantive_count = sum(
-            1 for c in classifications[-p3_substantive_window:] if c is True
+            1 for c in effective_classes[-p3_substantive_window:] if c is True
         )

Also applies to: 3310-3327, 3685-3719

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aelfrice/hook.py` around lines 3155 - 3161, Shadow P3-substantive is
computed from stale classifications because _maybe_log_cadence_shadow_tick is
invoked before the Stop-side p3_substantive push; fix by computing the shadow
P3-substantive on the effective current-turn window without mutating the ring or
by invoking the shadow logic after the Stop-side push. Concretely, update
_maybe_log_cadence_shadow_tick (or the caller site around the Stop-side
p3_substantive push) to either (a) accept a snapshot/copy of current
classifications and compute p3_substantive from that snapshot/window so it
matches the live decision, or (b) move the call to
_maybe_log_cadence_shadow_tick to run after the Stop-side p3_substantive action
(the block around your Stop push at the other site), ensuring you reference
p3_substantive, classifications, and ring when locating code to change.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/test_hook_stop_cadence_p3_substantive.py`:
- Line 209: The test unpacks an unused `serr` from the tuple returned by
_run_stop (e.g., "serr, calls, state_dir = _run_stop(...)") which triggers Ruff
RUF059; update the unpacking in both occurrences (around the _run_stop calls at
the two reported sites) to drop or ignore `serr` (use "_, calls, state_dir =
_run_stop(...)" or "calls, state_dir = _run_stop(...)" depending on return
shape) so the unused variable is not created.

---

Outside diff comments:
In `@src/aelfrice/cadence_score.py`:
- Around line 144-162: The legacy P1/P2 agreement matrix is updated
unconditionally at the end of the loop, which counts rows where P1 or P2 entries
are missing/malformed; modify the logic so agreement[(p1_fire, p2_fire)] += 1
only runs when both POLICY_P1_EVERY_K_TURNS and POLICY_P2_CTX_THRESHOLD were
actually present and parsed as booleans: detect presence while iterating shadow
(e.g., track flags or check row_fire keys for those two policies), and move or
guard the increment accordingly so it only updates agreement when both valid
decisions exist.

In `@src/aelfrice/hook.py`:
- Around line 3155-3161: Shadow P3-substantive is computed from stale
classifications because _maybe_log_cadence_shadow_tick is invoked before the
Stop-side p3_substantive push; fix by computing the shadow P3-substantive on the
effective current-turn window without mutating the ring or by invoking the
shadow logic after the Stop-side push. Concretely, update
_maybe_log_cadence_shadow_tick (or the caller site around the Stop-side
p3_substantive push) to either (a) accept a snapshot/copy of current
classifications and compute p3_substantive from that snapshot/window so it
matches the live decision, or (b) move the call to
_maybe_log_cadence_shadow_tick to run after the Stop-side p3_substantive action
(the block around your Stop push at the other site), ensuring you reference
p3_substantive, classifications, and ring when locating code to change.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: bda64743-2f2a-49df-8758-46d33a59d166

📥 Commits

Reviewing files that changed from the base of the PR and between e07ef8d and 51cea5c.

📒 Files selected for processing (10)
  • CHANGELOG/v3.md
  • benchmarks/cadence_replay.py
  • benchmarks/fixtures/cadence_replay_sample.json
  • src/aelfrice/cadence_score.py
  • src/aelfrice/hook.py
  • tests/test_cadence_replay.py
  • tests/test_cadence_score.py
  • tests/test_hook_stop_cadence_p3_substantive.py
  • tests/test_hook_stop_cadence_shadow.py
  • tests/test_hook_ups_cadence_p3_substantive.py

Comment thread tests/test_hook_stop_cadence_p3_substantive.py
@yoshi280 yoshi280 added the ready-to-merge Trigger merge-train: FF main to this PR's head label May 26, 2026
@github-actions
github-actions Bot merged commit 51cea5c into main May 26, 2026
43 checks passed
@github-actions github-actions Bot removed the ready-to-merge Trigger merge-train: FF main to this PR's head label May 26, 2026
@github-actions

Copy link
Copy Markdown

merge-train: merged 51cea5cmain via FF push.

@yoshi280

Copy link
Copy Markdown
Collaborator Author

[release:review:garsecg:2026-05-26T18:51:04Z]

robotrocketscience added a commit that referenced this pull request Jun 10, 2026
Audit #957: feed log is feed.jsonl (not .aelfrice-feed.jsonl); #876
entry described an unshipped p3_turn_density design — rewritten around
the shipped p3_velocity (live) + p3_substantive (scaffolding; dispatch
landed 3.4.0 PR #920); #875 shadow entry matches the per-session log
path, (would_fire, reason) predicates, 2x2 matrix, and real CLI flags;
#870 UPS checkpoint injects the current turn and Stop remains the
cache writer; #887 recent-work block uses regex issue refs with no gh
resolution; #935 slot extractor, #933 index, #937/#941/#932 wording,
#850 test filename.
@robotrocketscience
robotrocketscience deleted the feat/issue-876-p3-substantive branch June 10, 2026 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

attn:review Needs review (PR open, awaiting reviewer) author-Setr PR coordination mutex

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(cadence): P3 turn-density-aware policy — design scoping (#749 P3)

2 participants