fix(bench-gate): the H1 directive gate can be cleared by a one-token lookup table

[robotrocketscience]

Closes #1341.

The H1 bench gate was never re-run after the fix it was waiting on shipped. Re-running it turned up a stale published number, and then a larger problem: the gate cannot certify what it claims to certify.

1. The published number was stale

docs/design/v2_directive_detection.md and docs/design/V2_REENTRY_QUEUE.md have both carried P=0.664 since May. That is the pre-Path-A figure. Path A shipped in #467 and the confirming re-run the memo itself asked for ("needs corpus re-run to confirm") never happened.

Measured against the shipped detector, lab corpus v0.1, 285 rows:

precision 0.706 below 0.8 gate (TP=89, FP=37, FN=6, TN=153, n=285)

P=0.706 / R=0.937. Path A removed 8 false positives, not the ~45 estimated — off by about 5x. And the surviving errors are not the "single dominant pattern" the failure analysis described; they fall into six structurally distinct families (interrogatives the filters miss, attribution, third-person descriptive, use/mention, task verbs outside the prefix bank, and stance/affirmation frames).

2. The gate can be cleared by a one-token lookup table

This is the finding that matters, and it is why this PR ships no detector change.

Partition the corpus deterministically, 163 train / 122 held out. Build the weakest classifier that can be written: read the first word of the prompt, look up the majority label for that word among training rows, answer with it. No grammar, no mood, no notion of durability or task-versus-rule.

held out:  P=0.912  R=0.795   (TP=31, FP=3, FN=8)   -> clears P>=0.80 / R>=0.60

It clears the gate. The real detector scores 0.706 and does not.

The cause is in the corpus. Of 114 distinct opening words, only 7 appear in both classes — 87.9% of rows are decided by their first token. Positives open with deontic/policy vocabulary (always, never, don't, avoid, prefer, use); negatives open with task, question and discourse vocabulary (write, run, check, what, should, ok, please). No row labelled directive opens with a one-shot task verb.

So any rule keyed on head position buys precision for free. It is never charged for the durable directives it would wrongly suppress, because the corpus contains none of them — no "Check every PR for a changelog entry before you approve it", no "Run the full suite before pushing to a shared branch". Those are ordinary standing rules that open with a task verb, and a head-verb filter eats them in production while scoring clean in-corpus.

3. How that was established, and what did not ship

Six family-scoped suppression rules were built against the train split only, each self-measured, then each put through independent adversarial review. Composed, they took the corpus to P=0.953 / R=0.853 — a comfortable pass, and it held up on the held-out split (P=0.944).

Every one of the six was judged both overfit and over-reaching, unanimously, with concrete minimal pairs. Representative: a rule claiming to detect interrogative force by subject-auxiliary inversion actually keyed on a closed pronoun list — should we only run the migration on Postgres is suppressed, should the migration only run on the Postgres shard is not, identical syntax. Another suppressed genuine directives outright: Check each incoming PR for a changelog entry before you approve it.

None of them are in this PR. The one-token baseline explains why they measured so well and reviewed so badly, and a gate that a lookup table passes will bless an overfit detector — it nearly did here.

What this PR contains

• test_directive_corpus_defeats_a_first_token_baseline — a bench-gate guard that trains the baseline on whatever corpus is mounted and fails if it clears the gate. A statement about the corpus, not the detector. Red against v0.1 by design; skips in public CI with the rest of the bench gate.
• The corrected numbers in both design docs, the six-family failure analysis replacing the monolithic-cluster one, a § Gate validity recording the measurement, and corpus v0.2 made the blocking item ahead of further detector iteration.
• Changelog entry.

No change to src/. Detector behaviour is byte-identical.

Verification

# public CI posture (corpus unset)
$ pytest -q tests/bench_gate/test_directive_detection.py
2 skipped

# lab, corpus mounted
$ AELFRICE_CORPUS_ROOT=... pytest -q tests/bench_gate/test_directive_detection.py
FAILED test_directive_detection_gate            - precision 0.706 below 0.8
FAILED test_directive_corpus_defeats_a_first_token_baseline
        - a first-token-only classifier clears the H1 gate on held-out rows
          (P=0.912, R=0.795, TP=31, FP=3, FN=8, train=163, held_out=122)

Both reds are the intended state against v0.1 and both are invisible to public CI.

What this unblocks, and what it does not

#1315 (natural-language capture for time-boxed locks) is blocked on "H1 gate met, or a narrower detector proposed with its own gate". This PR does not clear that blocker — it establishes that clearing it against v0.1 would have meant nothing. The path is corpus v0.2: minimal pairs that break the head-word/class correlation, hard positives (durable rules opening with task verbs, absent today) and hard negatives (one-shot requests opening with policy verbs). Target: class-ambiguous opening words rise from 12.1% to a majority.

Corpus rows stay lab-side throughout. Only aggregate statistics cross the boundary.

Two lab-side corpus-hygiene notes, neither blocking: the label vocabulary is not-directive while tests/corpus/v2_0/README.md documents not_directive, and the provenance field the same README requires to be non-empty is absent from all 285 rows.

Summary by Sourcery

Add a corpus-validity guard for the H1 directive-detection bench gate and correct related documentation and changelog entries.

New Features:

• Introduce a bench-gate test that ensures the directive-detection corpus cannot be solved by a first-token-only baseline classifier before treating gate metrics as evidence.

Bug Fixes:

• Clarify via changelog and design docs that the directive-detection bench gate could be spuriously cleared by a one-token lookup table and record the fix as a corpus-level guard rather than a detector change.

Enhancements:

• Document that the existing directive-detection gate is currently invalid due to corpus issues and outline requirements for corpus v0.2 before further detector iteration.
• Update design and re-entry-queue docs with corrected precision/recall figures and an expanded failure-mode analysis for the directive detector.

Documentation:

• Augment directive-detection design docs with a gate-validity section, corrected performance numbers, and guidance on corpus v0.2, and update the v2 re-entry queue accordingly.

Tests:

• Add a bench-gated test that fails when a first-token-only classifier clears the directive-detection H1 gate, treating this as a corpus problem rather than a detector issue.

Chores:

• Record the updated directive-detection status and gate conditions in the v4 changelog.

[sourcery-ai]

Sorry @robotrocketscience, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Warning

Review limit reached

@robotrocketscience, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 27 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: e88f1e1b-7fd9-4935-9556-d44f8ad97eaf

📥 Commits

Reviewing files that changed from the base of the PR and between 94d0ddb and 4bec0e7.

📒 Files selected for processing (4)

• CHANGELOG/v4.md
• docs/design/V2_REENTRY_QUEUE.md
• docs/design/v2_directive_detection.md
• tests/bench_gate/test_directive_detection.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sourcery-ai]

Reviewer's Guide

Adds a bench-gate validity guard that ensures the directive-detection corpus cannot be cleared by a trivial first-token classifier, and updates design docs, re-entry queue, and changelog to reflect corrected H1 precision/recall numbers and the newly recognized corpus problem without changing detector code.

Flow diagram for the H1 directive corpus validity bench gate

flowchart TD
    A[Run test_directive_corpus_defeats_a_first_token_baseline] --> B[Load lab corpus v0_1 or v0_2]
    B --> C[Partition corpus into train and held_out]
    C --> D[Train first_token_baseline on train]
    D --> E[Evaluate first_token_baseline on held_out]
    E --> F{P >= 0.80 and R >= 0.60?}
    F -->|yes| G[Fail test: corpus can be cleared by one_token baseline]
    F -->|no| H[Pass test: corpus defeats one_token baseline]

    subgraph Gate_reentry_conditions
        I[Detector_precision >= 0.80]
        J[Detector_recall >= 0.60]
        K[Corpus_defeats_one_token_baseline test green]
        I --> L[H1 gate may be treated as valid]
        J --> L
        K --> L
    end

Loading

File-Level Changes

Change	Details	Files
Add a bench-gate test that trains a first-token-only baseline on the directive corpus and fails if it clears the H1 precision/recall gate, making gate passage contingent on corpus quality rather than just detector metrics.	Introduce deterministic train/held-out partitioning based on SHA-1 of row id with a configurable train bucket ceiling. Add a regex-based helper to extract the head word from a prompt, normalising case and skipping bullet/number prefixes. Build a majority-label lookup table keyed by head word over training rows and a simple boolean predictor that returns directive if that word’s directive count exceeds non-directive count. Compute precision and recall for this baseline on held-out rows and assert that it does not meet the H1 P/R thresholds, otherwise failing with a detailed diagnostic message. Skip the validity guard when the corpus row count is below the same minimum as the main directive gate, and mark the test as bench-gated so it only runs in the lab corpus configuration.	tests/bench_gate/test_directive_detection.py
Correct and significantly expand the directive-detection design memo to document the new measurements, multi-family failure analysis, gate invalidity due to a one-token solution, and the requirement for corpus v0.2 before further detector iteration.	Update the status section with the re-run P=0.706 / R=0.937 metrics, TP/FP/FN/TN counts, and explicitly note the gate’s current invalidity. Replace the previous monolithic failure-mode description with a six-family analysis of false positives and a brief treatment of the small false-negative set. Add a Gate validity section that explains the one-token baseline experiment, corpus head-word/class correlation, and its implications for detector design. Add a What corpus v0.2 needs section specifying minimal-pair and hard positive/negative requirements to break the head-word correlation. Revise the Recommendation and Decision asks to prioritise corpus v0.2 creation, require the new validity guard to pass before trusting gate metrics, and clarify Path B’s role after corpus repair.	docs/design/v2_directive_detection.md
Align the H1 re-entry queue documentation with the new directive-detection measurements and explicitly state that gate progress is blocked on corpus quality, not detector changes.	Update the iteration spec entry to quote the corrected P=0.706 / R=0.937 numbers and reference the new measurement date and issue. Document that a first-token-only classifier clears the gate on held-out rows and that, as a result, corpus v0.2 is now the blocking item for H1 re-entry. Clarify that the iteration spec memo’s Gate validity section is the reference for this corpus-blocked status.	docs/design/V2_REENTRY_QUEUE.md
Record the corpus-validity fix and corrected H1 metrics in the changelog, describing the one-token gate issue and the new validity guard as a released fix without altering detector behaviour.	Add a detailed changelog entry explaining that the H1 gate could be cleared by a first-token lookup-table classifier, including its held-out P/R and the underlying head-word/class imbalance in the corpus. Note that six overfit family-scoped suppression rules were rejected and not shipped, and that the new bench-gate validity test now asserts the non-triviality condition. Mention the correction of previously published P=0.664 precision and the true impact of Path A (removing 8 false positives rather than ~45).	CHANGELOG/v4.md

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#1341	Re-run the H1 directive-detection bench gate on the currently shipped detector against lab corpus v0.1, and update design documentation (v2_directive_detection.md and V2_REENTRY_QUEUE.md) to reflect the measured P/R numbers and a corrected failure analysis.	✅
#1341	Revise the directive detector so that it passes the H1 bench gate (P≥0.80, R≥0.60, n≥200) on lab corpus v0.1, using a train/held-out split (both numbers published) and adding public-CI unit tests for any new filter behaviour.	❌	The PR explicitly states that it ships no detector changes (src/ is untouched) and that the H1 gate remains red. Instead of revising the detector to meet P≥0.80 and R≥0.60 on v0.1, it adds a guard test demonstrating that a first-token baseline can clear the gate, and marks the corpus (not the detector) as the blocking item. No new detector rules, no improved P/R, and no unit tests for new filter behaviour are introduced.
#1341	Ensure that only aggregate precision/recall statistics cross the lab/public boundary and that individual corpus rows remain lab-side while addressing the H1 gate and related documentation.	✅

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[github-actions]

PR-size soft cap

This PR is over the advisory size threshold:

• 160 changed lines (limit: 200)
• 4 changed files (limit: 3)

Bigger PRs collide with more open work, which under the parallel-session workflow tends to produce repeated attn:merge-conflict cycles (see #602). When practical, split into smaller PRs that each touch a focused surface.

This is advisory only — nothing is blocked. If the size is intentional (large refactor, module removal, generated code), apply the size:override label and this comment will be removed on the next push.

[robotrocketscience]

[claim:review:Garsecg:2026-08-05T02:10:23Z]

[robotrocketscience]

[claim:review:Kulili:2026-08-05T02:10:40Z]

[robotrocketscience]

[release:review:Kulili:2026-08-05T02:10:48Z]

[robotrocketscience]

Review — the conclusion holds, the guard that encodes it is under-powered

Reviewed the diff, the two design memos and the new bench-gate test. The
finding this PR is built on is right and the docs corrections should land. One
substantive issue with the guard itself, and two things I fixed on the branch.

The conclusion is not at risk

Worth saying plainly before the criticism, so it does not get over-read: the
claim "corpus v0.1 separates its classes by opening vocabulary" rests on a
partition-independent statistic — 7 of 114 distinct opening words appear in
both classes, 87.9% of rows are decided by their first token, and no row
labelled directive opens with a one-shot task verb. None of that depends on
how the rows are split. The decision to ship no detector change, and to make
corpus v0.2 the blocking item, is well supported and I would not touch it.

The guard rests on 34 predictions from one partition

test_directive_corpus_defeats_a_first_token_baseline asserts on a single
sha1(id) % 100 < 60 split. From the counts the PR reports — TP=31, FP=3, FN=8
on 122 held-out rows — precision is computed over 34 positive predictions:

precision 0.9118  n=34   Wilson 95% CI [0.7704, 0.9695]   gate 0.80 -> lower bound BELOW gate
recall    0.7949  n=39   Wilson 95% CI [0.6447, 0.8922]   gate 0.60 -> lower bound above gate

Recall is comfortable. Precision is not: the interval's lower bound is below
the gate the guard is asserting the baseline clears. Concretely, 4 more
false positives on the held-out split takes P to 0.795 and the guard passes —
on an unchanged corpus.

That matters more here than it usually would, because of what this guard is
for. Its whole purpose is to stop a false clearance of the H1 gate. As
written it can produce one itself: corpus v0.2 will add and edit rows, every
edited id moves to a different sha1 bucket, and a re-partition that happens to
land 4 more FPs in the held-out half turns the guard green while the head-word
correlation is untouched. The failure is silent and it reads as "corpus fixed".

Recommended fix: sweep K salted partitions (sha1(salt + id)) and assert on
the aggregate rather than on one draw — either pooled TP/FP/FN across K, which
takes n from 34 to ~34K and tightens the interval directly, or "fails if the
baseline clears on ≥1 of K". Either removes the partition luck and makes the
guard say what the § Gate validity prose already says.

I did not make that change myself. It needs a corpus run to pick K and to
confirm the swept result, the corpus is lab-side, and I am not going to push an
unverifiable edit to a test I cannot execute. Filed as a follow-up rather than
held against this PR — see below.

Not blocking, and why

The guard as written is under-powered, not wrong: it currently fails, correctly,
and having it is strictly better than not having it. Blocking the merge would
also strand the docs corrections, and those are the part with a three-month-old
stale number in them. Landing this and fixing the guard separately is the right
order.

Two things I fixed on the branch

1. Rebased onto github/main — the branch was behind and would not have
   fast-forwarded.
2. docs/design/v2_directive_detection.md § What's being decided still read
   "raising precision from 0.664 to ≥0.80". The head of the same memo now says
   0.706, and § Gate validity says the choice among paths A/B/C is
   uninterpretable — so that section was carrying both the stale number this PR
   exists to correct and a framing the PR supersedes. Marked superseded, with a
   pointer to § Gate validity. (4bec0e74)

One thing left as-is, deliberately

The memo and the CHANGELOG both state P=0.912 / R=0.795 as a flat fact. Given
that this PR's own thesis is that a published number carried forward without its
derivation does real damage, the baseline figure should arguably ship with its
interval and its n. I have not edited the prose for it — that is the author's
call on their own finding, and it is the same question as the guard fix.

[github-actions]

merge-train: merged 4bec0e7 → main via FF push.

[robotrocketscience]

[release:review:Garsecg:2026-08-05T02:18:58Z]

[robotrocketscience]

Both points taken and fixed in PR #1350 (closes #1349).

The guard. Swept to K=200 salted partitions, failing if the baseline clears on any one of them. Your predicted failure was not hypothetical for long — I landed corpus v0.2 shortly after this merged and declared the guard green off a single partition, which is exactly the false clearance you described. The sweep is what actually establishes it: v0.1 clears on 196/200, the union on 0/200 with a maximum precision of 0.754. Chose fail-on-any over pooled counts because pooled predictions across overlapping partitions are not independent, so the Wilson interval on them is too tight in the direction that makes passing easier — and for a guard, a false alarm costing a corpus inspection beats a false clearance blessing an overfit detector.

The point estimate. You were right that it should not have shipped bare, and right to leave it to me. P=0.912 / R=0.795 was one draw from a distribution spanning 0.833–1.000; both memos now carry the swept figure with its n (pooled 0.9381, n=5,978). A memo arguing that numbers carried forward without their derivation do real damage had no business publishing a bare point estimate.

Thanks for the rebase and the § What's being decided catch — that section was carrying both the stale number and a superseded framing, and I had missed it.

One thing your review did not ask for but follows from it: against the union the detector is P=0.665 / R=0.636. Recall was 0.937 with 34 points of headroom on v0.1, which is the premise every iteration path in that memo was written on. It is now barely above the floor, because v0.2's hard positives are durable rules the 29-verb regex never fires on. Recall is the binding constraint from here, not precision.